feat(database): add cloudflare d1 support - #36348

ChadRosseau · 2025-11-30T20:59:46Z

User description

SUMMARY

This PR adds support for the D1 database in Superset via a new external engine package:

superset-engine-d1 – Enables Superset to connect to D1 databases.
sqlalchemy-d1 and dbapi-d1 – Underlying packages required by the engine, recursively installed as dependencies of the engine.

The source code for these packages can be found in this organization

After installation, users can connect to D1 databases in Superset using the d1 engine spec. Example connection string:

d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}

This PR also updates docs/docs/configuration/databases.mdx with the appropriate documentation for Cloudflare D1.

TESTING INSTRUCTIONS

All packages have unit tests written. They can be installed by cloning the appropriate repositories and running poetry run pytest

Testing the functionality itself can be done by creating a Cloudflare account and following the connection process.

ADDITIONAL INFORMATION

CodeAnt-AI Description

Add Cloudflare D1 database support and connection instructions

What Changed

Superset documentation now lists Cloudflare D1 in the database reference with an install command and the expected connection string: d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}
The project exposes a "d1" optional dependency group that installs the required D1 client packages to enable D1 connections
A dedicated Cloudflare D1 section was added to the configuration docs with step-by-step install and connection details

Impact

✅ Can connect Superset to Cloudflare D1
✅ Easier installation of D1 dependencies via the "d1" extra
✅ Clearer setup and connection instructions for Cloudflare D1

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

bito-code-review

Code Review Agent Run #dbd316

Actionable Suggestions - 2

docs/docs/configuration/databases.mdx - 2
- Incorrect Documentation · Line 57-57
- Incorrect Documentation · Line 363-375

Review Details

Files reviewed - 1 · Commit Range: f53cd6c..cf3b59c
- docs/docs/configuration/databases.mdx
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.
/pause - Pauses automatic reviews on this pull request.
/resume - Resumes automatic reviews.
/resolve - Marks all Bito-posted review comments as resolved.
/abort - Cancels all in-progress reviews.

Refer to the documentation for additional commands.

Configuration

This repository uses Default Agent You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.

Documentation & Help

AI Code Review powered by

bito-code-review · 2025-11-30T21:05:13Z

 | [Ascend.io](/docs/configuration/databases#ascendio)                       | `pip install impyla`                                                               | `ascend://{username}:{password}@{hostname}:{port}/{database}?auth_mechanism=PLAIN;use_ssl=true`                                                        |
 | [Azure MS SQL](/docs/configuration/databases#sql-server)                | `pip install pymssql`                                                              | `mssql+pymssql://UserName@presetSQL:TestPassword@presetSQL.database.windows.net:1433/TestSchema`                                                       |
 | [ClickHouse](/docs/configuration/databases#clickhouse)                  | `pip install clickhouse-connect`                                                   | `clickhousedb://{username}:{password}@{hostname}:{port}/{database}`                                                                                    |
+| [Cloudflare D1](/docs/configuration/databases#cloudflare-d1)                  | `pip install superset-engine-d1`                                                   | `d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}`                                                                                    |


Incorrect Documentation

This addition references a non-existent package 'superset-engine-d1' and GitHub repository 'sqlalchemy-cf-d1/superset-engine-d1', which could mislead users attempting to configure Cloudflare D1 support. Official Superset documentation and searches confirm no such integration exists yet.

Code Review Run #dbd316

Should Bito avoid suggestions like this for future reviews? (Manage Rules)

Yes, avoid them

Package superset-engine-d1
Repository sqlalchemy-cf-d1/superset-engine-d1

bito-code-review · 2025-11-30T21:05:13Z

+#### Cloudflare D1
+
+To use Cloudflare D1 with superset, install the [superset-engine-d1](https://github.com/sqlalchemy-cf-d1/superset-engine-d1) library.
+
+```
+pip install superset-engine-d1
+```
+
+The expected connection string is formatted as follows:
+
+```
+d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}
+```


Incorrect Documentation

The new Cloudflare D1 section links to a non-existent GitHub repository and package, potentially confusing users. No evidence of official D1 support in Superset exists based on current documentation and searches.

Code Review Run #dbd316

Should Bito avoid suggestions like this for future reviews? (Manage Rules)

Yes, avoid them

Please find the referred packages and its respective code below:
Package superset-engine-d1
Repository sqlalchemy-cf-d1/superset-engine-d1

Side note: You can also find the code for the relevant dependencies with the organization.

rusackas · 2025-12-01T20:23:44Z

Thanks for this! Pinging @betodealmeida for best practices advice here... having the db engine spec on pypi seems OK, but I wonder if it makes more sense (or not) to just add it to the Superset repo directly? ¯\_(ツ)_/¯

ChadRosseau · 2025-12-02T21:43:33Z

Thanks for this! Pinging @betodealmeida for best practices advice here... having the db engine spec on pypi seems OK, but I wonder if it makes more sense (or not) to just add it to the Superset repo directly? ¯_(ツ)_/¯

Thanks for this feedback, we actually had a pretty extensive conversation with @betodealmeida regarding external vs internal engine specs. We agreed that for a project like this where we may need to do more releases in the short term (for example if users run into needed features or bugs), then it's better to have the independent release cycle that comes with an external spec, as opposed to having to align our releases with those of Superset.

The main drawback of an external spec is that if the Superset internal API changes then we will need to adjust our implementation too. However, this is a more hypothetical issue that we think is worth dealing with in return for our own release cycles. Our various libraries are also open-source, so the community is also able to help with such adjustments if needed.

In the future after the implementation has been battle-tested by the community, there's also always the option to officially integrate it as an internal engine spec later on.

That being said, we are always open to other thoughts on the matter. Feel free to let us know if there's anything we haven't considered.

betodealmeida · 2025-12-05T18:59:57Z

Let's also add the package as an optional dependency here:

superset/pyproject.toml

Line 114 in a18b62c

[project.optional-dependencies]

Something like:

d1 = [
    "superset-engine-d1>=0.26.3",
    "sqlalchemy-d1>=0.1.0",
    "dbapi-d1>=0.1.0",
]

This way to use d1 people can do:

pip install "superset[d1]"

And have all dependencies installed.

codeant-ai-for-open-source · 2025-12-05T19:11:54Z

CodeAnt AI is reviewing your PR.

codeant-ai-for-open-source · 2025-12-05T19:14:12Z

Nitpicks 🔍

🔒 No security issues identified
⚡ Recommended areas for review Sensitive Information Exposure The new D1 docs show an example SQLAlchemy URI containing an API token. URIs (and the Extras field) can be logged, stored in the metadata DB, or exposed in UI screenshots — this could leak credentials. The docs should explicitly warn about these risks and recommend secure alternatives (Secure Extra, environment variables, or SQLALCHEMY_CUSTOM_PASSWORD_STORE). URL Encoding / Special Characters Connection tokens and account IDs may contain characters that break URL parsing (e.g., '@', ':', '/'). The D1 section doesn't mention URL-encoding special characters (there are other places in the doc that cover encoding); add a cross-reference or inline guidance to avoid confusing failures.

codeant-ai-for-open-source · 2025-12-05T19:14:41Z

 | [Ascend.io](/docs/configuration/databases#ascendio)                       | `pip install impyla`                                                               | `ascend://{username}:{password}@{hostname}:{port}/{database}?auth_mechanism=PLAIN;use_ssl=true`                                                        |
 | [Azure MS SQL](/docs/configuration/databases#sql-server)                | `pip install pymssql`                                                              | `mssql+pymssql://UserName@presetSQL:TestPassword@presetSQL.database.windows.net:1433/TestSchema`                                                       |
 | [ClickHouse](/docs/configuration/databases#clickhouse)                  | `pip install clickhouse-connect`                                                   | `clickhousedb://{username}:{password}@{hostname}:{port}/{database}`                                                                                    |
+| [Cloudflare D1](/docs/configuration/databases#cloudflare-d1)                  | `pip install superset-engine-d1`                                                   | `d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}`                                                                                    |


Suggestion: Documentation security issue: the new table row shows a full example SQLAlchemy URI with the Cloudflare API token embedded, which encourages pasting secrets into URIs (these often get logged or leaked). Replace the connection-string cell with a link to the detailed section so the table doesn't advertise embedding secrets and reduces chance of accidental token exposure. [security]

Severity Level: Critical 🚨

Suggested change

| [Cloudflare D1](/docs/configuration/databases#cloudflare-d1) | `pip install superset-engine-d1` | `d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}` |

| [Cloudflare D1](/docs/configuration/databases#cloudflare-d1) | `pip install superset-engine-d1` | [See detailed connection instructions below](#cloudflare-d1) |

Why it matters? ⭐

Replacing the inline example URI in the summary table with a link reduces the chance
that readers copy/paste URIs containing secrets into places where they might be logged
or leaked. The existing cell contains only placeholders (no real secret), but the change
is a sensible security-hardening and reduces duplication of the full example that already
exists later in the document.

Prompt for AI Agent 🤖

This is a comment left during a code review. **Path:** docs/docs/configuration/databases.mdx **Line:** 57:57 **Comment:** *Security: Documentation security issue: the new table row shows a full example SQLAlchemy URI with the Cloudflare API token embedded, which encourages pasting secrets into URIs (these often get logged or leaked). Replace the connection-string cell with a link to the detailed section so the table doesn't advertise embedding secrets and reduces chance of accidental token exposure. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.

codeant-ai-for-open-source · 2025-12-05T19:14:41Z

+To use Cloudflare D1 with superset, install the [superset-engine-d1](https://github.com/sqlalchemy-cf-d1/superset-engine-d1) library.
+
+```
+pip install superset-engine-d1
+```
+
+The expected connection string is formatted as follows:
+
+```
+d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}


Suggestion: Missing security and usability guidance in the added Cloudflare D1 section: it shows an example URI embedding the API token without warning about logging/leakage or the need to URL-encode special characters; also it omits the common Docker workflow instruction to persist the driver in images. Update the section to (1) warn not to embed API tokens in URIs and prefer Secure Extra/env vars, (2) mention URL-encoding for tokens with special characters, and (3) show how to add the package to docker/requirements-local.txt so installs persist in container images. [security]

Severity Level: Critical 🚨

Suggested change

To use Cloudflare D1 with superset, install the [superset-engine-d1](https://github.com/sqlalchemy-cf-d1/superset-engine-d1) library.

```

pip install superset-engine-d1

```

The expected connection string is formatted as follows:

```

d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}

To use Cloudflare D1 with Superset, install the [superset-engine-d1](https://github.com/sqlalchemy-cf-d1/superset-engine-d1) library.

pip install superset-engine-d1

If you're running Superset with Docker Compose, persist the dependency in your image by adding the package to `./docker/requirements-local.txt`: ```bash # Run from the repo root: echo "superset-engine-d1" >> ./docker/requirements-local.txt

IMPORTANT SECURITY NOTE: Do not embed raw API tokens directly in SQLAlchemy URIs in production — URIs can be logged, stored in histories, or leaked. Prefer storing credentials in the "Secure Extra" field, environment variables, or using Superset's secure credential mechanisms. If you must include a token in a URI for testing, URL-encode any special characters in the token (for example, replace '+' with '%2B', '@' with '%40', etc.).

The expected connection string (for local/testing examples only) is formatted as follows:

d1://{cloudflare_account_id}:{cloudflare_api_token_urlencoded}@{cloudflare_d1_database_id}

Why it matters? ⭐

The proposed additions fix real omissions: instructing how to persist the dependency in Docker,
warning against embedding tokens in URIs (which can be logged), and advising URL-encoding for special characters
are actionable, improve security posture, and match patterns already used elsewhere in the docs. This is not
mere stylistic noise — it addresses usability and security concerns visible in the new section.

Prompt for AI Agent 🤖

This is a comment left during a code review. **Path:** docs/docs/configuration/databases.mdx **Line:** 365:374 **Comment:** *Security: Missing security and usability guidance in the added Cloudflare D1 section: it shows an example URI embedding the API token without warning about logging/leakage or the need to URL-encode special characters; also it omits the common Docker workflow instruction to persist the driver in images. Update the section to (1) warn not to embed API tokens in URIs and prefer Secure Extra/env vars, (2) mention URL-encoding for tokens with special characters, and (3) show how to add the package to docker/requirements-local.txt so installs persist in container images. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.

codeant-ai-for-open-source · 2025-12-05T19:14:47Z

CodeAnt AI finished reviewing your PR.

murphylee10 · 2025-12-07T20:09:58Z

Let's also add the package as an optional dependency here:

superset/pyproject.toml

Line 114 in a18b62c

[project.optional-dependencies]

Something like:
d1 = [
    "superset-engine-d1>=0.26.3",
    "sqlalchemy-d1>=0.1.0",
    "dbapi-d1>=0.1.0",
]
This way to use d1 people can do:
pip install "superset[d1]"
And have all dependencies installed.

Thanks for the suggestion @betodealmeida! We've added the d1 optional dependency group to pyproject.toml with the three packages included, and also added the alternate pip command to the database documentation.

betodealmeida

This is awesome! Thanks for the contribution!

Co-authored-by: danielalyoshin <daniel.alyoshin@gmail.com> Co-authored-by: Shreyas Rao <raoshreyas2004@gmail.com> Co-authored-by: Murphy Lee <murphylee2004@gmail.com>

ChadRosseau and others added 2 commits November 30, 2025 15:41

feat: added cloudflare d1 connection instructions

f53cd6c

docs: add connection instructions for cloudflare d1

cf3b59c

pull-request-size Bot added the size/S label Nov 30, 2025

github-actions Bot added the doc Namespace | Anything related to documentation label Nov 30, 2025

dosubot Bot added data:connect Namespace | Anything related to db connections / integrations enhancement:db Suggest new DB connections labels Nov 30, 2025

bito-code-review Bot reviewed Nov 30, 2025

View reviewed changes

rusackas requested a review from betodealmeida December 1, 2025 20:21

add d1 packages as optional dependencies in superset

d01186b

codeant-ai-for-open-source Bot added the size:S This PR changes 10-29 lines, ignoring generated files label Dec 5, 2025

codeant-ai-for-open-source Bot reviewed Dec 5, 2025

View reviewed changes

docs: add alternate d1 installation command

fc8c3c9

betodealmeida approved these changes Dec 12, 2025

View reviewed changes

rusackas merged commit e5579ed into apache:master Dec 12, 2025
58 checks passed

	\| [Cloudflare D1](/docs/configuration/databases#cloudflare-d1) \| `pip install superset-engine-d1` \| `d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}` \|
	\| [Cloudflare D1](/docs/configuration/databases#cloudflare-d1) \| `pip install superset-engine-d1` \| [See detailed connection instructions below](#cloudflare-d1) \|

Conversation

ChadRosseau commented Nov 30, 2025 • edited by codeant-ai-for-open-source Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

CodeAnt-AI Description

What Changed

Impact

Checking Your Pull Request

Talking to CodeAnt AI

Example

Preserve Org Learnings with CodeAnt

Example

Retrigger review

Check Your Repository Health

Uh oh!

bito-code-review Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Code Review Agent Run #dbd316

Uh oh!

bito-code-review Bot Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

ChadRosseau Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

bito-code-review Bot Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

ThunderRoar Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

rusackas commented Dec 1, 2025

Uh oh!

ChadRosseau commented Dec 2, 2025

Uh oh!

betodealmeida commented Dec 5, 2025

Uh oh!

codeant-ai-for-open-source Bot commented Dec 5, 2025

Uh oh!

codeant-ai-for-open-source Bot commented Dec 5, 2025

Nitpicks 🔍

Uh oh!

codeant-ai-for-open-source Bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot commented Dec 5, 2025

Uh oh!

murphylee10 commented Dec 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

betodealmeida left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ChadRosseau commented Nov 30, 2025 •

edited by codeant-ai-for-open-source Bot

Loading

bito-code-review Bot left a comment •

edited

Loading

murphylee10 commented Dec 7, 2025 •

edited

Loading