---
title: "Interact with Databricks foundation models"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Interact with Databricks foundation models}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  markdown: 
    wrap: 72
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Intro

[Databricks](https://docs.databricks.com/en/introduction/index.html)
customers have access to [foundation model
APIs](https://docs.databricks.com/en/machine-learning/foundation-models/index.html)
like DBRX, Meta Llama 3 70B, and Mixtral 8x7B. Databricks also provides
the ability to train and [deploy custom
models](https://docs.databricks.com/en/machine-learning/foundation-models/deploy-prov-throughput-foundation-model-apis.html).

`chattr` supports the following models on Databricks by default:

| Model                     | Databricks Model Name                  | `chattr` Name                |
|---------------------|------------------------------|---------------------|
| DBRX Instruct             | `databricks-dbrx-instruct`             | `databricks-dbrx`            |
| Meta-Llama-3-70B-Instruct | `databricks-meta-llama-3-70b-instruct` | `databricks-meta-llama3-70b` |
| Mixtral-8x7B Instruct     | `databricks-mixtral-8x7b-instruct`     | `databricks-mixtral8x7b`     |

: [Supported Databricks pay-per-token foundation
models](https://docs.databricks.com/en/machine-learning/foundation-models/index.html#pay-per-token-foundation-model-apis)

## Authentication

Databricks requires a
[**host**](https://docs.databricks.com/en/workspace/workspace-details.html#workspace-instance-names-urls-and-ids)
(workspace URL) and
[**token**](https://docs.databricks.com/en/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users)
to authenticate. Both are required for any non-Databricks application,
such as `chattr`, to interact with the models in the Databricks
workspace.

The token can be generated by the user in the workspace under the
developer settings
([docs](https://docs.databricks.com/en/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users))
and the host can be found in the workspaces URL
([docs](https://docs.databricks.com/en/workspace/workspace-details.html#workspace-instance-names-urls-and-ids)).

By default, `chattr` will look for the credentials in environment
variables:

-   `DATABRICKS_HOST`

-   `DATABRICKS_TOKEN`

Use `Sys.setenv()` to set the variable. The downside of using this
method is that the variable will only be available during the current R
session:

``` r
Sys.setenv("DATABRICKS_HOST" = "https://xxxxx.cloud.databricks.com")
Sys.setenv("DATABRICKS_TOKEN" = "####################")
```

A preferred method is to save the secret key to the `.Renviron` file.
This way, there is no need to load the environment variable every time
you start a new R session. The `.Renviron` file is available in your
home directory. Here is an example of the entry:

```         
DATABRICKS_HOST = https://xxxxx.cloud.databricks.com
DATABRICKS_TOKEN = ####################
```

## Change the model

### Supported Models

By default, `chattr` is setup to interact with GPT 4 (`gpt-4`). To
switch to Meta Llama 3 70B you can run:

```{r}
library(chattr)

chattr_use("databricks-meta-llama3-70b")
```

#### Custom Models

If a model doesn't appear in the supported table but is deployed on
[Databricks model
serving](https://docs.databricks.com/en/machine-learning/model-serving/index.html)
as OpenAI-compatible ([configured with `llm/v1/chat` in
mlflow](https://mlflow.org/docs/latest/llms/deployments/index.html#general-configuration-parameters))
then you can specify the model name explicitly with `chattr_use()`

For example if you have deployed a fine-tuned version LLM with an
endpoint name of `"CustomLLM"`:

```{r}
library(chattr)

# use any existing databricks foundation model name (e.g. datarbicks-dbrx)
# then adjust the default model name to 'CustomMixtral'
chattr_use(x = "databricks-dbrx", model = "CustomLLM")
```

## Data files and data frames

Because it is information about your environment and work space, by
default `chattr` avoids sending any data files, and data frame
information to Databricks. Sending this information is convenient
because it creates a shorthand for your requests. If you wish to submit
this information as part of your prompts, use `chattr_defaults()`, for
example:

-   `chattr_defaults(max_data_files = 10)`
-   `chattr_defaults(max_data_frames = 10)`

These two commands will send 10 data frames, and 10 data files as part
of your prompt. You can decide the number to limit this by. The more you
send, the larger your prompt.

If any of these is set to anything but 0, a warning will show up every
time you start the Shiny app:

```         
• Provider: Databricks
• Path/URL: serving-endpoints
• Model: databricks-dbrx-instruct
• Label: DBRX (Databricks)
! A list of the top 10 files will be sent externally to Databricks with every request
To avoid this, set the number of files to be sent to 0 using chattr::chattr_defaults(max_data_files = 0)Î
```