The Genie has spoken And the winners of the Genie-Powered App Challenge are… A few weeks ago, we threw down a challenge: build something incredible with Databricks Apps and Genie Agent. And wow, you delivered. From smart, real-world prob...
- 1287 Views
- 10 replies
- 8 kudos
Thursday
Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, September 17, 2026! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with d...
- 568 Views
- 2 replies
- 3 kudos
a week ago
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 884 Views
- 0 replies
- 3 kudos
2 weeks ago
Databricks Advanced Learning Festival September 16 – October 14, 2026 Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are...
- 81648 Views
- 126 replies
- 48 kudos
4 weeks ago
Virtual Event Time: AMER Sept 23 / 9 AM PTEMEA Sept 24 / 9 AM BST / 10 AM CESTAPJ Sept 24 / 9:30 AM IST / 12 PM SGT / 1 PM JST/KST / 2 PM AEST (Japanese and Korean captions available) Register for the Workshop Note: Marking RSVP on this Community ...
- 730 Views
- 0 replies
- 2 kudos
2 weeks ago
Databricks EMEA Learning Festival Hands On Virtual Training 29–30 September 2026 · 09:00 BST / 10:00 CEST · Fully virtual · Free to join Ready to accelerate your journey on Data & AI? Join us for the EMEA Learning Festival. A two-day, hands-on vir...
- 2988 Views
- 0 replies
- 3 kudos
08-11-2026
Community Champion • Our Community Champion Program celebrates members who consistently contribute their expertise, support fellow practitioners, and help shape a stronger and more collaborative Databricks Community. Every month, we recognize individ...
- 374 Views
- 6 replies
- 6 kudos
Monday
Building a customer 360 requires connecting customer records across disparate data sets and establishing common customer identities. The Customer Entity Resolution Solution Accelerator shows how to build that foundation by translating customer attrib...
- 93 Views
- 0 replies
- 2 kudos
Tuesday
The Weekly Digest • August 31 – September 6 Community Pulse Your weekly pulse: new articles, real answers, and the discussions worth watching. This Week's Community Stars This week's difference-makers – and a shout-out to the new members ear...
- 472 Views
- 1 replies
- 2 kudos
a week ago
Community Activity
- 113 Views
- 5 replies
- 6 kudos
Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...
- 113 Views
- 5 replies
- 6 kudos
Latest Reply
- 6 kudos
1. Pinpointing exactly which operator falls backDon't guess from the query text — go to the Spark UI's SQL/DataFrame tab and look at the query DAG. Photon operators render in orange, standard Spark operators in blue, so you can see visually exactly w...
- 6 kudos
- 11 Views
- 0 replies
- 0 kudos
WordPress to Databricks: Building a Practical Data Pipeline
WordPress is often used as the front end for publishing, ecommerce, forms, memberships, and other online activities. Behind that website, useful information can accumulate quickly. Posts, users, orders, comments, form submissions, and other records c...
- 11 Views
- 0 replies
- 0 kudos
- 84 Views
- 3 replies
- 1 kudos
At what data size do you stop reaching for Spark?
Hi everyone,A question I keep having with my team and I would like to hear how others think about it.A lot of the jobs we run are not big. Plenty of our pipelines process a few gigabytes, some considerably less. We run them on Spark because that is w...
- 84 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
For small workloads (a few MBs to a few GBs), you can prefer single-node or serverless compute within Databricks. This keeps the same governance, monitoring, CI/CD, Unity Catalog, and operational model while avoiding the overhead of a multi-node clus...
- 1 kudos
- 10841 Views
- 12 replies
- 14 kudos
Resolved! Databricks Certified Professional Data Scientist Exam Question Types
Hello,I am not seeing a lot of information regarding the Databricks Certified Professional Data Scientistexam. I took the Associate Developer in Apache Spark Exam last year and the materials for the exam seemed much more focused than what I found for...
- 10841 Views
- 12 replies
- 14 kudos
Latest Reply
- 14 kudos
Could you please share the resources need to clear this certification? can you please help
- 14 kudos
- 2309 Views
- 3 replies
- 0 kudos
Resolved! Publish a Technical Blog
Sir/Madam,I hope you are doing well. I would like to contribute and publish a technical blog on the Databricks Community portal. However, I am not sure about the process and appropriate platform. I would appreciate your guidance on where to start, th...
- 2309 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi Advika,I would like to submit a Partner Technical Blog contribution titled “Controlling Databricks Genie Costs: Unity AI Gateway and Better Genie Agent Design.”The article focuses on practical cost governance for Genie, including Unity AI Gateway ...
- 0 kudos
- 67 Views
- 1 replies
- 1 kudos
Preventing Duplicate Records When Reprocessing Data in Databricks
A pipeline can finish successfully and still produce the wrong result after a retry. Imagine an orders load that writes its data, then fails during a later task. Repeating the load with an append can add the same orders again. Replacing existing rows...
- 67 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Islam_hoti ,The Core Pattern: Identity + OrderingThe article argues that idempotency is not achieved by simple "Appends." Instead, it requires two specific definitions:Identity: A business key (e.g., order_id) that tells you which entity the reco...
- 1 kudos
- 100 Views
- 2 replies
- 1 kudos
Streaming read fails with "Detected a data update" after a restatement job touches the source
Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...
- 100 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
This is a classic "Streaming vs. Batch" impedance mismatch. You are encountering the "Data change detected" exception because standard Spark Structured Streaming (and by extension, the DLT/Lakeflow logic) assumes the source is an append-only event l...
- 1 kudos
- 73 Views
- 1 replies
- 0 kudos
RoadMap DE
The best way to be DE certified??
- 73 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Heya @Elizeu_94!If you are new to the Databricks ecosystem, I would suggest you visiting the Databricks Academy which provides you with amazing free, self-paced courses. You could begin your journey with completing the Databricks Fundamentals badge w...
- 0 kudos
- 45 Views
- 0 replies
- 0 kudos
Databricks Free Edition GitHub push fails with "Resolver error"
Hi,I'm using Databricks Free Edition and connected a GitHub repository through the Databricks GitHub integration. I can clone/connect to the repository and create/edit files, but when I try to Commit & Push, Databricks returns:Error pushing changesRe...
- 45 Views
- 0 replies
- 0 kudos
- 39 Views
- 0 replies
- 0 kudos
help me how to fix
"Triggering new runs for organization 7474657040075149 is currently disabled temporarily." , Why did my job get blocked while running? You clearly said it could run continuously, so why was it blocked?
- 39 Views
- 0 replies
- 0 kudos
- 165 Views
- 4 replies
- 4 kudos
Feature enablement for Foundation Model Unity Catalog permissions
I am trying to restrict workspace users' access to Databricks Foundation Models using the guidance in the Foundation Model Unity Catalog Permissions documentation.I have revoked EXECUTE permission for all users from the system.ai schema. However, wor...
- 165 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
It persists after enablement. The legacy databricks-* endpoints are a separate serving path from both pay-per-token and provisioned throughput, so they aren't gated by the system.ai EXECUTE revoke. It isn't pre-enablement behaviour that clears once t...
- 4 kudos
- 53 Views
- 0 replies
- 0 kudos
Migrate Azure Databricks between cross tenant
How to migrate an Azure Databricks to a different Azure tenant?Does DAB is the best method if a databricks workspace is freshly built on top of just migrating the the workspace resources only like notebook, jobs, cluster, policies, permissions etc? P...
- 53 Views
- 0 replies
- 0 kudos
- 562 Views
- 11 replies
- 1 kudos
Can I run jobs continuously without interruption on a permanent free (Community Edition) account?
Chào mọi người,Tôi hiện đang sử dụng Databricks Community Edition (tài khoản miễn phí) và muốn hỏi về việc chạy các tác vụ theo lịch trình/liên tục trên đó.Cụ thể: 1. Liệu có thể chạy một tác vụ liên tục (ví dụ: tác vụ xử lý dữ liệu trực tuyến hoặc t...
- 562 Views
- 11 replies
- 1 kudos
Latest Reply
- 1 kudos
@FastFoodBro You can use Databricks Free account for personal use only (personal use, learning, experimentation - not for production or commercial use). You can upgrade to a paid plan to access full platform features as you will face interruptions in...
- 1 kudos
- 20051 Views
- 2 replies
- 2 kudos
ADD COLUMN IF NOT EXISTS does not recognize "IF NOT EXIST". How do I add a column to an existing delta table with SQL if the column does not already exist?
How do I add a column to an existing delta table with SQL if the column does not already exist?I am using the following code: <%sqlALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name type; >but it prints the error: <[PARSE_SYNTAX_ERROR] Synta...
- 20051 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Here some SQL devilry that does not require SQL Scripting nor pythons. It adds a my_extra_column FLOAT to the target table with null values. I'm as shocked as anyone that this works, (and if it's undocumented behavior, someone in Databricks Command ...
- 2 kudos
- 55 Views
- 0 replies
- 0 kudos
Deletion Vectors Don’t Delete Data the Way You Think They Do: A Deep Dive into Delta Lake Maintenanc
Hi everyone! In my previous post, I discussed how enabling Deletion Vectors helped reduce our Delta MERGE runtime from 22 minutes down to 6 minutes by eliminating write amplification.However, deferring file rewrites introduces an important architectu...
- 55 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1418 | |
| 1253 | |
| 1187 | |
| 961 |
Latest from our Blog
Beyond the Dashboard: How Transferz Built a Truly Data-Driven Company Culture With AI/BI Genie
Transferz, a fast-growing travel tech platform, eliminated its data team bottleneck by deploying AI/BI Genie — achieving 89,5% adoption across business users achieving 60% operational improvements.Th...
5 Ways to Enhance Databricks AI/BI Dashboard Tables with HTML
Organizations use AI/BI dashboard tables to review accounts, catch problems, and decide what to do next. Yet a plain table can hold all the right data and still be hard to read. One way to improve the...
Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics
Introduction Traditionally, unifying requires building brittle custom integrations for every single source. Lakeflow Connect resolves this with native, fully managed connectors for a full suite of mar...
Meet SDP Rewind: An undo button for your ETL pipelines
A bad deployment slips into your pipeline on Friday, and by Monday, it has been writing incorrect data for three days, with no clean way to roll it back. You can restore one table, but a pipeline is a...
