dbsql

Overview ¶

Package dbsql implements the go driver to Databricks SQL

Usage ¶

Clients should use the database/sql package in conjunction with the driver:

import (
	"database/sql"

	_ "github.com/databricks/databricks-sql-go"
)

func main() {
	db, err := sql.Open("databricks", "token:<token>@<hostname>:<port>/<endpoint_path>")

	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()
}

Connection via DSN (Data Source Name) ¶

Use sql.Open() to create a database handle via a data source name string:

db, err := sql.Open("databricks", "<dsn_string>")

The DSN format is:

token:[my_token]@[hostname]:[port]/[endpoint http path]?param=value

Supported optional connection parameters can be specified in param=value and include:

• catalog: Sets the initial catalog name in the session
• schema: Sets the initial schema name in the session
• maxRows: Sets up the max rows fetched per request. Default is 100000
• timeout: Adds timeout (in seconds) for the server query execution. Default is no timeout
• userAgentEntry: Used to identify partners. Set as a string with format <isv-name+product-name>
• useCloudFetch: Used to enable cloud fetch for the query execution. Default is true
• maxDownloadThreads: Sets up the max number of concurrent workers for cloud fetch. Default is 10
• useArrowNativeDecimal: Returns DECIMAL columns as native Arrow decimal128 (via GetArrowBatches); when scanned through database/sql they are returned as lossless strings. Default is false
• authType: Specifies the desired authentication type. Valid values are: Pat, OauthM2M, OauthU2M
• accessToken: Personal access token. Required if authType set to Pat
• clientID: Specifies the client ID to use with OauthM2M
• clientSecret: Specifies the client secret to use with OauthM2M
• useKernel: Routes execution through the SEA-via-kernel backend instead of Thrift. Requires a build with -tags databricks_kernel (CGO_ENABLED=1); the default pure-Go build returns a clear error. Default is false. See the kernel-backend section below
• warehouseId: The bare SQL warehouse id, used by the kernel backend (which addresses a warehouse by id) in preference to the http path. Ignored by the Thrift backend

Supported optional session parameters can be specified in param=value and include:

• ansi_mode: (Boolean string). Session statements will adhere to rules defined by ANSI SQL specification.
• timezone: (e.g. "America/Los_Angeles"). Sets the timezone of the session

Connection via new connector object ¶

Use sql.OpenDB() to create a database handle via a new connector object created with dbsql.NewConnector():

import (
	"database/sql"
	dbsql "github.com/databricks/databricks-sql-go"
)

func main() {
	connector, err := dbsql.NewConnector(
		dbsql.WithServerHostname(<hostname>),
		dbsql.WithPort(<port>),
		dbsql.WithHTTPPath(<http_path>),
		dbsql.WithAccessToken(<my_token>)
	)
	if err != nil {
		log.Fatal(err)
	}

	db := sql.OpenDB(connector)
	defer db.Close()
	...
}

Supported functional options include:

• WithServerHostname(<hostname> string): Sets up the server hostname. The hostname can be prefixed with "http:" or "https:" to specify a protocol to use. Mandatory
• WithPort(<port> int): Sets up the server port. Mandatory
• WithAccessToken(<my_token> string): Sets up the Personal Access Token. Mandatory
• WithHTTPPath(<http_path> string): Sets up the endpoint to the warehouse. Mandatory
• WithInitialNamespace(<catalog> string, <schema> string): Sets up the catalog and schema name in the session. Optional
• WithMaxRows(<max_rows> int): Sets up the max rows fetched per request. Default is 100000. Optional
• WithSessionParams(<params_map> map[string]string): Sets up session parameters including "timezone" and "ansi_mode". Optional
• WithTimeout(<timeout> Duration). Adds timeout (in time.Duration) for the server query execution. Default is no timeout. Optional
• WithUserAgentEntry(<isv-name+product-name> string). Used to identify partners. Optional
• WithCloudFetch (bool). Used to enable cloud fetch for the query execution. Default is true. Optional
• WithMaxDownloadThreads (<num_threads> int). Sets up the max number of concurrent workers for cloud fetch. Default is 10. Optional
• WithAuthenticator (<authenticator> auth.Authenticator). Sets up authentication. Required if neither access token or client credentials are provided.
• WithClientCredentials(<clientID> string, <clientSecret> string). Sets up Oauth M2M authentication.
• WithUseKernel(<useKernel> bool). Routes execution through the SEA-via-kernel backend instead of Thrift. Requires a build with -tags databricks_kernel (CGO_ENABLED=1); the default build returns a clear error. Default is false. See the kernel-backend section below. Optional
• WithWarehouseID(<id> string). The bare SQL warehouse id used by the kernel backend in preference to the http path; ignored by the Thrift backend. Optional

Query cancellation and timeout ¶

Cancelling a query via context cancellation or timeout is supported.

// Set up context timeout
ctx, cancel := context.WithTimeout(context.Background(), 30 * time.Second)
defer cancel()

// Execute query. Query will be cancelled after 30 seconds if still running
res, err := db.ExecContext(ctx, "CREATE TABLE example(id int, message string)")

CorrelationId and ConnId ¶

Use the driverctx package under driverctx/ctx.go to add CorrelationId and ConnId to the context. CorrelationId and ConnId makes it convenient to parse and create metrics in logging.

**Connection Id** Internal id to track what happens under a connection. Connections can be reused so this would track across queries.

**Query Id** Internal id to track what happens under a query. Useful because the same query can be used with multiple connections.

**Correlation Id** External id, such as request ID, to track what happens under a request. Useful to track multiple connections in the same request.

ctx := dbsqlctx.NewContextWithCorrelationId(context.Background(), "workflow-example")

Logging ¶

Use the logger package under logger.go to set up logging (from zerolog). By default, logging level is `warn`. If you want to disable logging, use `disabled`. The user can also utilize Track() and Duration() to custom log the elapsed time of anything tracked.

import (
	dbsqllog "github.com/databricks/databricks-sql-go/logger"
	dbsqlctx "github.com/databricks/databricks-sql-go/driverctx"
)

func main() {
	// Optional. Set the logging level with SetLogLevel()
	if err := dbsqllog.SetLogLevel("debug"); err != nil {
		log.Fatal(err)
	}

	// Optional. Set logging output with SetLogOutput()
	// Default is os.Stderr. If running in terminal, logger will use ConsoleWriter to prettify logs
	dbsqllog.SetLogOutput(os.Stdout)

	// Optional. Set correlation id with NewContextWithCorrelationId
	ctx := dbsqlctx.NewContextWithCorrelationId(context.Background(), "workflow-example")

	// Optional. Track time spent and log elapsed time
	msg, start := logger.Track("Run Main")
	defer log.Duration(msg, start)

	db, err := sql.Open("databricks", "<dsn_string>")
	...
}

The result log may look like this:

{"level":"debug","connId":"01ed6545-5669-1ec7-8c7e-6d8a1ea0ab16","corrId":"workflow-example","queryId":"01ed6545-57cc-188a-bfc5-d9c0eaf8e189","time":1668558402,"message":"Run Main elapsed time: 1.298712292s"}

SEA-via-kernel backend (experimental) ¶

By default the driver uses the Thrift/HiveServer2 backend and is pure Go (CGO_ENABLED=0, go-gettable, cross-compilable). An experimental second backend runs statements over the Statement Execution API via the Rust databricks-sql-kernel, reached through a cgo C ABI. It is opt-in and compiled in only under a build tag, so the default build is unchanged.

To use it, build with the databricks_kernel tag and CGO enabled, and select it per connection with WithUseKernel. The tagged build links the kernel static library, which is not committed — run `make kernel-lib` first to produce it (and the C header) under the cgo link path; `make build-kernel` does both steps:

make kernel-lib      # builds the kernel .a + header at the pinned KERNEL_REV
CGO_ENABLED=1 go build -tags databricks_kernel ./...

connector, _ := dbsql.NewConnector(
	dbsql.WithServerHostname(host),
	dbsql.WithHTTPPath(httpPath),
	dbsql.WithAccessToken(token),
	dbsql.WithUseKernel(true),
)
db := sql.OpenDB(connector)

In a build without the tag, WithUseKernel(true) returns a clear error at connect time rather than silently using Thrift.

Kernel-backend logging uses the same knob as the rest of the driver: the binding-layer trace is emitted through the shared logger at Debug level, so DATABRICKS_LOG_LEVEL=debug or dbsql.SetLogLevel("debug") turns it on and it lands in the driver's normal sink (including a custom logger.SetLogOutput). Where a context is in scope, lines carry the usual connId/corrId/queryId fields. At the default Warn level the lines are suppressed with no cost.

dbsql.SetLogLevel("debug") // or DATABRICKS_LOG_LEVEL=debug

The same level is mapped into the kernel's internal (Rust) log subscriber. Its records are forwarded into the driver's logger, so they use the same output as the Go and Thrift paths, including a local file configured with logger.SetLogOutput. Changing SetLogOutput later safely retargets all three paths. The Rust verbosity is fixed when the first kernel session whose resolved level is not OFF is opened, so set the level before that first non-OFF connect.

For finer control of the Rust verbosity independent of the driver level, set DBSQL_KERNEL_DEBUG to any non-empty value: the callback then defers filtering to RUST_LOG. Filter on the target databricks::sql::kernel (note the colons):

# kernel logs only, at the kernel's own verbosity:
DBSQL_KERNEL_DEBUG=1 RUST_LOG=databricks::sql::kernel=debug ./your_app
# kernel logs plus its HTTP stack:
DBSQL_KERNEL_DEBUG=1 RUST_LOG=debug ./your_app

Supported on the kernel backend: PAT and OAuth (M2M via WithClientCredentials, U2M via the authType=oauthU2M DSN param); reading scalar, nested, and complex-typed results (CloudFetch is transparent); bound query parameters (positional and named); context cancellation during execute; the initial namespace (WithInitialNamespace, applied post-connect via USE CATALOG / USE SCHEMA); metric-view metadata (WithEnableMetricViewMetadata); the retry / backoff policy (WithRetries: RetryWaitMin / RetryWaitMax / RetryMax, including the disable form, forwarded to the kernel's HTTP retry config); and the TLS, proxy, and session-conf (query tags, statement timeout, time zone) options. Federated token providers use one token snapshot during setup. Nothing is silently ignored: WithTimeout, token-provider / external / static authenticators, and custom M2M OAuth scopes (the kernel applies its own) are rejected at connect; staging (PUT/GET/REMOVE on a Unity Catalog volume) is rejected at execute. WithMaxRows is accepted but inert (the kernel manages fetching below the C ABI).

Kernel HTTP requests inherit the driver's ClientTimeout (900s by default). If that value is zero, the kernel uses its own 120s default; zero is neither unlimited nor an immediate timeout.

OAuth U2M is interactive: on a cache miss, connecting launches the system browser and blocks until login completes or the kernel's ~120s callback timeout expires. Because the C ABI can't interrupt session open mid-call, a connection-context deadline is not honored during that window. Use PAT or OAuth M2M for headless / deadline-bound connects. The kernel runs a single, cloud-agnostic in-house U2M flow (OIDC discovery against {host}/oidc, no Azure branching), so on every cloud the kernel path uses the built-in databricks-sql-connector client and requests offline_access + sql. It does NOT forward the cloud-inferred client id / scopes the Thrift path computes: on AWS/GCP those already match, but on Azure the Thrift path uses the Entra-direct app id and <tenant>/user_impersonation scope, which would break the kernel's workspace-federated flow. So the two backends authorize identically on AWS/GCP and deliberately diverge on Azure. Neither backend exposes a U2M-scopes option.

Experimental kernel-only options (rejected by the default backend; the WithKernel* prefix marks them experimental):

• WithKernelTrustedCerts(pem) adds a PEM CA bundle on top of the system roots (for a re-signing proxy or on-prem CA). Required because the kernel's TLS stack does not read SSL_CERT_FILE.
• WithKernelClientCertificate(certPEM, keyPEM) configures a paired mTLS client identity. Both PEM buffers must be non-empty; the certificate may include intermediates and an unencrypted PKCS#8 private key is recommended.
• WithKernelSkipHostnameVerify() skips only the hostname check while keeping chain validation (finer-grained than WithSkipTLSHostVerify).
• WithKernelProxy(KernelProxy{URL, Username, Password, BypassHosts}) sets an explicit HTTP proxy, overriding the HTTP(S)_PROXY / NO_PROXY environment the kernel path otherwise mirrors. The fields are named (not positional) so the four same-typed strings can't be transposed at the call site. Use it for the advanced fields the env-var path can't express: a structured bypass (no-proxy) list, or basic-auth credentials supplied out of band rather than embedded in the URL. An explicit proxy takes precedence over the environment; empty credentials / bypass are passed to the kernel as unset, and a malformed URL is rejected at connect.
• WithKernelRetryOverallTimeout(d) sets the cumulative retry budget across all attempts — the 4th retry knob, alongside the backoff bounds and max attempts carried by the backend-neutral WithRetries (also honored on the kernel path). It is kernel-only because the Thrift WithRetries surface has no overall-budget equivalent; it mirrors the pyo3/napi retry_overall_timeout knob. Zero keeps the kernel's default budget (900s).
• WithKernelMaxChunksInMemory(n) bounds how many decompressed CloudFetch chunks the kernel holds in memory at once — the knob that trades large-result download throughput for peak RSS. Lower it (e.g. 4) to cap memory on wide, row-heavy result sets; raise it for more parallelism at higher memory. It is forwarded as the kernel's client-only cloudfetch_max_chunks_in_memory session conf, stripped before the SEA wire so it never reaches the server. A value <= 0 keeps the kernel's default (16).

Setting any of these without WithUseKernel fails Connect with an error wrapping the sentinel ErrRequiresKernelBackend, detectable with errors.Is.

Features above the backend seam are inherited unchanged: the database/sql connection pool and connection lifecycle. The Go wrapper telemetry interceptor is skipped on the kernel path so it does not duplicate kernel-owned telemetry for the same connection and statements; `enableTelemetry`, `telemetry_batch_size`, and `telemetry_flush_interval` are forwarded to kernel-owned telemetry config. Result types render byte-for-byte identical to the Thrift backend: scalars, DECIMAL (exact string), TIMESTAMP / TIMESTAMP_NTZ (shifted into the session time zone), INTERVAL, nested Array/Map/Struct and VARIANT (as JSON), and GEOMETRY / GEOGRAPHY (WKT). The server query id is surfaced on the success path, so a QueryIdCallback (see below) fires with the real id and EXECUTE_STATEMENT telemetry carries it.

On the read path, context cancellation is honored at result-batch boundaries, not mid-fetch: an in-flight CloudFetch batch runs to completion before the cancel takes effect.

OAuth token caching and HTTP client reuse are inherited from the kernel and need no driver configuration: the kernel caches U2M tokens on disk (~/.config/databricks-sql-kernel/oauth/, with refresh-token lifecycle) and M2M tokens in-memory with background refresh, and reuses a single pooled HTTP client per session across the control-plane, CloudFetch, and auth-refresh calls. There is no driver-side cache or pool knob because these live below the C ABI.

Programmatically Retrieving Connection and Query Id ¶

Use the driverctx package under driverctx/ctx.go to add callbacks to the query context to receive the connection id and query id.

import (
	"github.com/databricks/databricks-sql-go/driverctx"
)

func main() {

	...

	qidCallback := func(id string) {
		fmt.Println("query id: " + id)
	}

	connIdCallback := func(id string) {
		fmt.Println("connection id: " + id)
	}

	ctx := context.Background()
	ctx = driverctx.NewContextWithQueryIdCallback(ctx, qidCallback)
	ctx = driverctx.NewContextWithConnIdCallback(ctx, connIdCallback)

	rows, err1 := db.QueryContext(ctx, `select * from sometable`)

	...

}

Query parameters ¶

Passing parameters to a query is supported when run against servers with version DBR 14.1.

// Named parameters:
p := dbsql.Parameter{Name: "p_bool", Value: true},
rows, err := db.QueryContext(ctx, `select * from sometable where condition=:p_bool`,dbsql.Parameter{Name: "p_bool", Value: true})

// Positional parameters - both `dbsql.Parameter` and plain values can be used:
rows, err := db.Query(`select *, ? from sometable where field=?`,dbsql.Parameter{Value: "123.456"}, "another parameter")

For complex types, you can specify the SQL type using the dbsql.Parameter type field. If this field is set, the value field MUST be set to a string.

Please note that named and positional parameters cannot be used together in the single query.

Staging Ingestion ¶

The Go driver now supports staging operations. In order to use a staging operation, you first must update the context with a list of folders that you are allowing the driver to access.

ctx := driverctx.NewContextWithStagingInfo(context.Background(), []string{"staging/"})

After doing so, you can execute staging operations using this context using the exec context.

_, err1 := db.ExecContext(ctx, `PUT 'staging/file.csv' INTO '/Volumes/main/staging_test/e2etests/file.csv' OVERWRITE`)

Errors ¶

There are three error types exposed via dbsql/errors

DBDriverError - An error in the go driver. Example: unimplemented functionality, invalid driver state, errors processing a server response, etc.

DBRequestError - An error that is caused by an invalid request. Example: permission denied, invalid http path or other connection parameter, resource not available, etc.

DBExecutionError - Any error that occurs after the SQL statement has been accepted such as a SQL syntax error, missing table, etc.

Each type has a corresponding sentinel value which can be used with errors.Is() to determine if one of the types is present in an error chain.

DriverError
RequestError
ExecutionError

The kernel backend (WithUseKernel, see above) additionally wraps every rejection of an option or feature it cannot yet honor with the sentinel ErrNotSupportedByKernel. A caller can detect this case with errors.Is(err, dbsqlerr.ErrNotSupportedByKernel) — for example to fall back to the default (Thrift) backend — rather than matching on the error message text:

if errors.Is(err, dbsqlerr.ErrNotSupportedByKernel) {
	// this option/statement isn't supported on the kernel backend yet;
	// retry with the default backend (omit WithUseKernel).
}

Conversely, a kernel-only option set without WithUseKernel is rejected on the default (Thrift) path with an error wrapping the mirror sentinel ErrRequiresKernelBackend, detectable the same way with errors.Is(err, dbsqlerr.ErrRequiresKernelBackend).

Example usage:

import (
	fmt
	errors
	dbsqlerr "github.com/databricks/databricks-sql-go/errors"
)

func main() {
	...
	_, err := db.ExecContext(ogCtx, `Select id from range(100)`)
	if err != nil {
		if errors.Is(err, dbsqlerr.ExecutionError) {
			var execErr dbsqlerr.DBExecutionError
			if ok := errors.As(err, &execError); ok {
					fmt.Printf("%s, corrId: %s, connId: %s, queryId: %s, sqlState: %s, isRetryable: %t, retryAfter: %f seconds",
					execErr.Error(),
					execErr.CorrelationId(),
					execErr.ConnectionId(),
					execErr.QueryId(),
					execErr.SqlState(),
				    execErr.IsRetryable(),
					execErr.RetryAfter().Seconds(),
				)
			}
		}
		...
	}
	...
}

See the documentation for dbsql/errors for more information.

Retrieving Arrow Batches ¶

The driver supports the ability to retrieve Apache Arrow record batches. To work with record batches it is necessary to use sql.Conn.Raw() to access the underlying driver connection to retrieve a driver.Rows instance. The driver exposes two public interfaces for working with record batches from the rows sub-package:

type Rows interface {
	GetArrowBatches(context.Context) (ArrowBatchIterator, error)
}

type ArrowBatchIterator interface {
	// Retrieve the next arrow.Record.
	// Will return io.EOF if there are no more records
	Next() (arrow.Record, error)

	// Return true if the iterator contains more batches, false otherwise.
	HasNext() bool

	// Release any resources in use by the iterator.
	Close()
}

The driver.Rows instance retrieved using Conn.Raw() can be converted to a Databricks Rows instance via a type assertion, then use GetArrowBatches() to retrieve a batch iterator. If the ArrowBatchIterator is not closed it will leak resources, such as the underlying connection. Calling code must call Release() on records returned by DBSQLArrowBatchIterator.Next().

Example usage:

import (
	...
	dbsqlrows "github.com/databricks/databricks-sql-go/rows"
)

func main() {
	...
	db := sql.OpenDB(connector)
	defer db.Close()

	conn, _ := db.Conn(context.BackGround())
	defer conn.Close()

	query := `select * from main.default.taxi_trip_data`

	var rows driver.Rows
	var err error
	err = conn.Raw(func(d interface{}) error {
		rows, err = d.(driver.QueryerContext).QueryContext(ctx, query, nil)
		return err
	})

	if err != nil {
		log.Fatalf("unable to run the query. err: %v", err)
	}
	defer rows.Close()

	batches, err := rows.(dbsqlrows.Rows).GetArrowBatches(context.BackGround())
	if err != nil {
		log.Fatalf("unable to get arrow batches. err: %v", err)
	}

	var iBatch, nRows int
	for batches.HasNext() {
		b, err := batches.Next()
		if err != nil {
			log.Fatalf("Failure retrieving batch. err: %v", err)
		}

		log.Printf("batch %v: nRecords=%v\n", iBatch, b.NumRows())
		iBatch += 1
		nRows += int(b.NumRows())
		b.Release()
	}
	log.Printf("NRows: %v\n", nRows)
}

Supported Data Types ¶

==================================

Databricks Type --> Golang Type

==================================

BOOLEAN --> bool

TINYINT --> int8

SMALLINT --> int16

INT --> int32

BIGINT --> int64

FLOAT --> float32

DOUBLE --> float64

VOID --> nil

STRING --> string

DATE --> time.Time

TIMESTAMP --> time.Time

DECIMAL(p,s) --> sql.RawBytes

BINARY --> sql.RawBytes

ARRAY<elementType> --> sql.RawBytes

STRUCT --> sql.RawBytes

MAP<keyType, valueType> --> sql.RawBytes

INTERVAL (year-month) --> string

INTERVAL (day-time) --> string

For ARRAY, STRUCT, and MAP types, sql.Scan can cast sql.RawBytes to JSON string, which can be unmarshalled to Golang arrays, maps, and structs. For example:

type structVal struct {
		StringField string `json:"string_field"`
		ArrayField  []int  `json:"array_field"`
}
type row struct {
	arrayVal  []int
	mapVal    map[string]int
	structVal structVal
}
res := []row{}

for rows.Next() {
	r := row{}
	tempArray := []byte{}
	tempStruct := []byte{}
	tempMap := []byte{}
	if err := rows.Scan(&tempArray, &tempMap, &tempStruct); err != nil {
		log.Fatal(err)
	}
	if err := json.Unmarshal(tempArray, &r.arrayVal); err != nil {
		log.Fatal(err)
	}
	if err := json.Unmarshal(tempMap, &r.mapVal); err != nil {
		log.Fatal(err)
	}
	if err := json.Unmarshal(tempStruct, &r.structVal); err != nil {
		log.Fatal(err)
	}
	res = append(res, r)
}

May generate the following row:

{arrayVal:[1,2,3] mapVal:{"key1":1} structVal:{"string_field":"string_val","array_field":[4,5,6]}}