Grow Your Skills and Earn Rewards! Mark your calendar: January 09 – January 30, 2026 Join us for a three-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. ...
- 126683 Views
- 409 replies
- 143 kudos
12-09-2025
June 15–18, 2026 Are you building the future with data and AI? Then this is your moment. The Call for Proposals for Data + AI Summit 2026 is officially open, and we want to hear from builders, practitioners, and innovators across the data and AI com...
- 7931 Views
- 4 replies
- 6 kudos
12-15-2025
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 4010 Views
- 2 replies
- 9 kudos
10-02-2025
Strengthening cybersecurity isn’t a one-time project — it’s an ongoing process of adapting to new and complex threats. The focus is on modernizing your data infrastructure and analytics capabilities to make security operations smarter, faster, and mo...
- 993 Views
- 1 replies
- 2 kudos
2 weeks ago
2025 wasn’t just another year on the Databricks Community.It was a year of showing up, helping each other, and building trust. Watch the 2025 Community Wrap-Up This video is a small reflection of that journey.It highlights the key contributors of 2...
- 1446 Views
- 13 replies
- 20 kudos
2 weeks ago
Another strong week of engagement, learning, and problem-solving across the community.Let’s take a look at the highlights Standout Contributors These members stepped in with sharp insights, thoughtful follow-ups, and the kind of curiosity that ke...
- 293 Views
- 0 replies
- 0 kudos
Tuesday
Community Activity
- 114 Views
- 2 replies
- 2 kudos
Resolved! Bug Report: SDP (DLT) with autoloader not passing through pipe delimiter/separator
I am noticing a difference between using autoloader in an interactive notebook vs using it in a Spark Declarative Pipeline (DLT Pipeline). This issue seems to be very similar to this other unanswered post from a few years ago. Bug report: the delimit...
- 114 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hey,Okay thanks @nikhilj0421. I have now solved the issue but not with a full refresh of the table. I had tried this previously and even deleted the DLT pipeline hoping that would provide me the clean slate if this lingering schema was an issue but w...
- 2 kudos
- 222 Views
- 3 replies
- 4 kudos
Resolved! Databricks Serverless Pipelines - Incremental Refresh Doubts
Hello everyone,I would like to clarify some doubts regarding how Databricks Pipelines (DLT) behave when using serverless pipelines with incremental updates.In general, incremental processing is enabled and works as expected. However, I have observed ...
- 222 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
Hi @Alf01 , thanks for accepting the solution! To keep you updated, the REFRESH POLICY feature, that I mentioned in my post, is out now! It allows manual control of the refresh strategy (AUTO, INCREMENTAL, INCREMENTAL STRICT, FULL), just as you stat...
- 4 kudos
- 91 Views
- 2 replies
- 0 kudos
Intermittent task execution issues
We're getting intermittent errors:[ISOLATION_STARTUP_FAILURE.SANDBOX_STARTUP] Failed to start isolated execution environment. Sandbox startup failed. Exception class: INTERNAL. Exception message: INTERNAL: LaunchSandboxRequest create failed - Error e...
- 91 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
According to https://learn.microsoft.com/en-us/azure/databricks/release-notes/serverless/, 17.3 is the latest release for serverless and we're on Serverless Environment 4.Here's the trackback:File /databricks/python/lib/python3.12/site-packages/pyspa...
- 0 kudos
- 1092 Views
- 5 replies
- 2 kudos
Resolved! Job fails on clusters only with library dependency
Hello!I have following problem: All my job runs fail when the job uses a library. Even the most basic job (print a string) and the most basic library package (no secondary dependencies, the script does not even import/use the library) fails with `Fai...
- 1092 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi, can you please share detailed soluton, did you include any config in databricks.yml? I'm also having same issue.
- 2 kudos
- 74 Views
- 1 replies
- 0 kudos
Databricks App deployment fails: mysqlclient build error (pkg-config not found)
Hi Community Members,I’m deploying a Python project as a Databricks App, but deployment fails during dependency installation with: ERROR: Failed to build 'MySQL client' pkg-config: not found Exception: Can not find valid pkg-config nameThe dependenc...
- 74 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi All,I’m still looking for guidance on this issue and haven’t been able to find a solution yet. If anyone has encountered something similar or has suggestions on how I can troubleshoot this further, I’d really appreciate your help.Please let me kno...
- 0 kudos
- 66 Views
- 0 replies
- 1 kudos
Invitation to the Databricks Developer Community on Medium
Hi All,We are starting a new Databricks Developer Community publication on Medium, and this is a warm invitation to every Databricks professional, developer, and aspirant who believes in learning by sharing.We are doing this because Databricks learni...
- 66 Views
- 0 replies
- 1 kudos
- 67 Views
- 2 replies
- 2 kudos
Resolved! The Lakeflow connect Gateway setup, do we need to install the agent on-prem?
The Lakeflow connect Gateway setup to connect on-prem SQL serverPlease provide the steps the setup the gateway agent on the on-prem? Where to download this agent? What are the firewall rules for outbound looks like. Kind regards,Asha
- 67 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @Ashash12 ,You need to have proper network connectivity to your on premise SQL Server. As they stated in the docs - connector supports SQL Server on-premises using Azure ExpressRoute and AWS Direct Connect networkinghttps://docs.databricks.com/aws...
- 2 kudos
- 47 Views
- 0 replies
- 1 kudos
Agent Skills
Do you know that it is possible to extend the Assistant with agent skills? It is really straightforward and allows you, in fact, to extend the functionality of databricks. You can create templates for an assistant - I experimented with a template to ...
- 47 Views
- 0 replies
- 1 kudos
- 119 Views
- 3 replies
- 0 kudos
Can't Access Agents
Hi there, I'm very new to Databricks and I wanted to try out Agents. At first, I thought I couldn't access Agents because I was on a Trial account, so I added my CC to the subscription. Unfortunately, I still can't access the feature, not to mention ...
- 119 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi Szymon, thanks for a prompt reply!I followed your suggestions (AWS version) but sadly it seems my account is not on the allowlist for Mosaic Beta.I messaged Databricks support via email, but it seems I also don't have access to the Help Center eit...
- 0 kudos
- 121 Views
- 2 replies
- 1 kudos
Best practices for 3-layer access control in Databricks
Identity and access management model for Databricks and want to implement a clear 3-layer authorization approach:Account level: Account RBAC roles (account admin, metastore admin, etc.)Workspace level: Workspace roles/entitlements + workspace ACLs (c...
- 121 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Thank you for the detailed explanation and for sharing the reference blogs, I may follow up once we complete our initial design draft.
- 1 kudos
- 63 Views
- 0 replies
- 0 kudos
Hive metastore CANNOT access storage
I'm new to Azure Databricks and I'm facing an issue when trying to create a schema or table that points to my Azure Storage account. I keep getting this error:```[EXTERNAL_METASTORE_CLIENT_ERROR.OPERATION_FAILED] Client operation failed: org.apache.h...
- 63 Views
- 0 replies
- 0 kudos
- 67 Views
- 0 replies
- 0 kudos
Great Expectations with Fabric and Databricks-Building Bulletproof Data Pipelines
In this article, I’ll walk you through transforming a basic PySpark notebook into a production-ready data pipeline with comprehensive quality checks using Great Expectations patterns in Microsoft Fabric and Databricks. We’ll start simple and progress...
- 67 Views
- 0 replies
- 0 kudos
- 770 Views
- 4 replies
- 5 kudos
Resolved! switching to Databrick from Ab Initio (an old ETL software)- NEED ADVICE
All courses in market and on youtube as per my knowledge for databrick is outdated as those courses are for community edition. there is no new course for free edition of databrick. i am a working profession and i do not get much time. do you guys kno...
- 770 Views
- 4 replies
- 5 kudos
Latest Reply
- 5 kudos
@pop_smoke What were the arguments that swayed your organization to swtich to Databricks from Ab Initiio?Pure cost basis?
- 5 kudos
- 88 Views
- 0 replies
- 2 kudos
Databricks Assistant
Databricks Assistant can also be used in databricks documentation without login to #databricks. Read and watch databricks news on:https://databrickster.medium.com/databricks-news-2026-week-2-5-january-2026-to-11-january-2026-0bfc6c592051
- 88 Views
- 0 replies
- 2 kudos
- 57 Views
- 1 replies
- 1 kudos
al purpose databricks cluster disappear
Hi, i can see sometime the cluster get disappeared even though it was created some time back using cluster pipeline,what could be the reason to disappear.we can recreate cluster but wanted to know the reason why this cluster get disappeared.thanks!Ve...
- 57 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Ved88 ,30 days after a compute is terminated, it is permanently deleted. To keep an all-purpose compute configuration after a compute has been terminated for more than 30 days, an administrator can pin the compute. Up to 100 compute resources can...
- 1 kudos
Announcements
| User | Count |
|---|---|
| 2247 | |
| 1227 | |
| 1088 | |
| 1039 | |
| 950 |
Latest from our Blog
10 things you no longer need to think about (but if you are, here's how to fix it)
Databricks just released a button to fix all legacy issues in your workspace. Actually, they didn't. But the fact that you felt a momentary surge of hope—or a cynical eye-roll—proves exactly where you...
From Traces to Evaluation: An Autonomous MLflow Agent Harness for AI AgentOps
The Gap in Agent Development Databricks has made it straightforward to deploy AI agents—Model Serving endpoints, automatic MLflow tracing, Unity Catalog integration. But there's a gap between "deploye...
Combining Agent Bricks and ai_parse_document for Smarter Document Workflows
Building a Research Paper Curator for Knowledge Assistants Document-heavy workflows such as research analysis, contract review, and support ticket processing, are a natural fit for AI agents. Building...
Declarative Pipelines meets foreachBatch: Custom Streaming for Advanced Pipelines
If this is the first time you’re hearing about Sinks in Lakeflow Declarative Pipelines (LDP), we highly recommend reading Introducing the SDP Sink API, which explores the recently launched Sinks API, ...
