Databricks Community

why have you posted this here? 

@Marbricks Follow this approach: define the function’s interface first, then connect the real data. Use a temporary function for testing within the same session, and a persistent Unity Catalog function in a development schema for testing with the ext...

Exactly Similar situtation , i have crated my own using Pythopn , VetcorDB and Knowledge graph, you can simulate same or Hire me , i can able to do it for you LOL..See this Video on my Linked In

Hello @TyreseR19, Could you please ensure that both required courses for the Data Analyst learning path were fully completed during the Learning Festival event window? Please verify that every component of each course is marked as completed, includin...

Most modern enterprise data platforms favor managed ingestion (Lakeflow Connect) for standard relational databases like SQL Server. The operational complexity of monitoring 50 managed pipelines is significantly lower than maintaining custom CDC code,...

did you ever figure this out, looking for those specific notebooks myself? I'm not sure if this is helpful, but in the Databricks free edition home page, there is a Databricks fundamentals course, which covers similar things (it is not exactly the sa...

Try this link https://community.databricks.com/t5/data-engineering/how-to-calculate-cost-of-each-table-for-the-specific-databricks/td-p/167086 

@gowri_databrick Parquet's columnar storage organizes data by column rather than by row - all values for order_date are stored together, all order_amount values are stored together and so on. In your commerce scenario with 500 million records, when t...

Hi @SantiNath_Dey, Databricks has solid building blocks for this, and there's even a purpose-built metadata framework you can lean on. At its core, your pipeline needs to do three things...generate surrogate keys at each level, resolve parent referen...

thank you for your response

Yes that's true. There are hundreds (if not 100s) of tasks that can be done without depending on the agents. Databricks indeed has DQX for governance and data quality. Instead of relying on smaller activities (especially when they are repeating in na...

@gowri_databrick Checkpoints in Structured Streaming serve as the ledger and recovery mechanism for streaming queries. It tracks which data has been processed and successfully written, enabling exactly once processing guarantees. A checkpoint contain...

Hi @srikanthp24, Yeah... this is something others have run into as well. When you create a Lakeflow Connect ingestion pipeline via the UI wizard, the "Schedules and notifications" step (step 6) only configures alerts for that pipeline. The gateway pi...

Hi @IM_01,   append_flow is not limited to just reading a source table. You can (and should) include column transformations, filters, and any other DataFrame operations inside the function decorated with @dp.append_flow. The function simply needs to ...