How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 1158 Views
- 0 replies
- 3 kudos
3 weeks ago
Hello all! The next BrickTalks session has been scheduled! In case you missed the last one, BrickTalks are live, expert-led sessions that give the Databricks Community members the opportunity to interact with our Bricksters, ask questions and get to ...
- 168 Views
- 1 replies
- 3 kudos
Monday
BrickCon 2025 | Dec 3–5, 2025 A community-driven conference built by and for Databricks users around the world. Get ready for an unforgettable experience at BrickCon 2025 — the first-ever community-led Databricks conference bringing together da...
- 825 Views
- 3 replies
- 4 kudos
2 weeks ago
Stay on top of customer feedback at scale! The digital world makes it easier than ever for customers to share their opinions, and each review can shape your business decisions. With Large Language Models (LLMs), you can quickly extract and summarize ...
- 84 Views
- 0 replies
- 1 kudos
yesterday
We kept the October momentum going strong — with vibrant technical discussions, creative experiments, and deep dives into AI, streaming, and governance. Here’s a look at what stood out this week in the Databricks Community Fresh Content & Articles...
- 782 Views
- 1 replies
- 5 kudos
Wednesday
As we close out September and step into October, it’s incredible to see how our top contributors continue to lead with consistency, clarity, and collaboration. Week after week, they’ve stayed at the forefront of discussions, setting the tone for know...
- 1441 Views
- 4 replies
- 9 kudos
3 weeks ago
Learn, Level Up, and Get Rewarded! Save the dates: October 10 – October 31, 2025! We're celebrating learning, growth, and career advancement with an exciting opportunity to level up in data engineering, data analysis, machine learning, and genera...
- 164841 Views
- 381 replies
- 114 kudos
08-07-2025
Community Activity
- 1 Views
- 0 replies
- 0 kudos
transformWithStateInPandas. Invalid pickle opcode when updating ValueState with large (float) array
I am getting an error when the entity I need to store in a ValueState is a large array (over 15k-20k items). No error (and works correctly) if I trim the array to under 10k samples. The same error is raised when using it as a value for MapState or as...
- 1 Views
- 0 replies
- 0 kudos
- 19 Views
- 2 replies
- 0 kudos
databricks serverless cluster and poetry private repository
Currently we are evaluating the databricks serverless. It support public repository in poetry as dependency path but it is not supporting private repository as we are not sure whether put the credentials details regarding privare repository.
- 19 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @mahfooz_iiitian ,Databricks supports private repositories only for Notebook-scoped libraries.In serverless you can use do it using pip install (of course store you token in a safe palce):Notebook-scoped Python libraries - Azure Databricks | Micro...
- 0 kudos
- 1757 Views
- 2 replies
- 1 kudos
ModuleNotFound error when using transformWithStateInPandas via a class defined outside the notebook
As per Databricks documentation when I define the class that extends `StatefulProcessor` in a Notebook everything works ok however, execution fails with ModuleNotFound error as soon as the class definition is moved to a file (module) of it's own in a...
- 1757 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
This is no longer an issue; it must be some patch version of DBX Runtime 16.4 fixed it and it works now without doing any changes to original code.Thanks.
- 1 kudos
- 9 Views
- 1 replies
- 0 kudos
Multilingual embedding foundation model request
Are there any plans to offer a foundation model with multilingual support? Llama embed nemotron 8b or qwen3-embedding 8b would help out a lot for users that do not only process data in the English language.
- 9 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @nickv ,There are already some foundation models that supports multiple languages:https://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/supported-models#google-gemma-3 https://learn.microsoft.com/en-us/azure/dat...
- 0 kudos
- 11 Views
- 1 replies
- 0 kudos
Rasterio on shared/standard cluster has no access to proj.db
We try to use rasterio on a Databricks shared/standard cluster with DBR 17.1. Rasterio is directly installed on the cluster as library. Code:import rasterio rasterio.show_versions()Output: rasterio info:rasterio: 1.4.3GDAL: 3.9.3PROJ: 9.4.1GEOS: 3.11...
- 11 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @der,I guess this is related to limitation of standard/shared cluster access mode which you're using. https://docs.databricks.com/aws/en/compute/standard-limitations#network-and-file-system-limitationsYou can try to go with dedicated access mode o...
- 0 kudos
- 31 Views
- 5 replies
- 2 kudos
Resolved! when automatic liquid clustering is enabled, how to know which columns are used for clustering?
Let's say a table is configured to have automatic liquid clustering:ALTER TABLE table1 CLUSTER BY AUTO; How to know which columns were chosen by Databricks?
- 31 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
From the documentation, it seems that in Python, there is such an option, only when creating or replacing a table.# To set clustering columns and auto, which serves as a way to give a hint # for the initial selection. df.writeTo(...).using("delta") ...
- 2 kudos
- 7 Views
- 0 replies
- 0 kudos
Chennai Data Bricks Users?
We have using it any groups to notify upcoming events in Chennai? if yes please add me in that or suggest how to join in that group.If not why not someone to initiate to create a group and add who ever interested in data bricks meetup, so that will h...
- 7 Views
- 0 replies
- 0 kudos
- 8 Views
- 0 replies
- 0 kudos
Metric Views
HiI think I’ve found a reproducible bug / or am misunderstanding some syntax / capabilities of Metric Views when joining a calendar scaffold to an SCD2 table.The same SQL query works perfectly, but the Metric View always returns a constant 1 per mont...
- 8 Views
- 0 replies
- 0 kudos
- 55 Views
- 1 replies
- 0 kudos
Migrating Talend ETL Jobs to Databricks – Best Practices & Challenges
Hi All,I’m currently working on a Proof of Concept (POC) to migrate existing Talend ETL jobs to Databricks. The goal is to leverage Databricks for data processing and orchestration while moving away from Talend.I’d appreciate insights on the followin...
- 55 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Migrating from Talend’s jobs to Databricks requires rebuilding ETL logic using Spark. There is no native one-click converter provided by Databricks, so the typical approach is to audit and refactor each Talend job into code (PySpark or Spark SQL) in ...
- 0 kudos
- 13 Views
- 0 replies
- 0 kudos
Lakebase / Feature Store error: “Failed to get identity details for username” (service principal)
Hello,I’m running into a Lakebase / Feature Store issue related to service principal authentication when trying to log or read from the Databricks Feature Store. Migrating from the legacy online tables. Here’s the exact error:psycopg2.OperationalErr...
- 13 Views
- 0 replies
- 0 kudos
- 40 Views
- 2 replies
- 2 kudos
Unable to use my email with free edition due to being used with another account
I created an account with my email within the Community Edition. Once the free edition became available, I tried to switch over, but I am getting an error message saying the email is already in use.
- 40 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hello @thomastinch! It’s possible that you may have already created a Free Edition account earlier. In that case, try logging in to the Free Edition using the same email ID. It’s also possible that a Free Trial account is already associated with that...
- 2 kudos
- 69 Views
- 5 replies
- 0 kudos
Encountering an error while setting up a single-node cluster on top of aws
Hi Team,I'm trying to create a single-node cluster in Databricks on AWS, but I'm encountering an error. Could you please assist me with this?{ "reason": { "code": "INVALID_ARGUMENT", "type": "CLIENT_ERROR", "parameters": { "databr...
- 69 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
@devpavan Are you using API or terraform to create. Can you please share the json config that you are passing?
- 0 kudos
- 19 Views
- 0 replies
- 0 kudos
Unable to create workspace
Hi Team,we have challenge creating workspace in data bricks account created on top of aws.below are the details:Databricks account name : [email protected]'s LakehouseAWS Account id : 720016114009databricks id: 1ee8765f-b472-4...
- 19 Views
- 0 replies
- 0 kudos
- 29 Views
- 1 replies
- 0 kudos
When Will I get the badge certification?
Hello all,I passed an exam for AWS Databricks Platform Architect AccreditationI would like to know when I can get the certification and badgeThanks
- 29 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Migueldfr , It can take up to 48 hours for badges/ certificates to be issued. Also please check your spam just in case it went there. If you do not receive your badge in 48 hours, please file a ticket with our support team. Please also provide...
- 0 kudos
- 47 Views
- 1 replies
- 0 kudos
Can I reschedule a missed Databricks exam?
Hi everyone,I missed my Databricks certification exam because of a personal emergency and a time zone confusion. The attempt was consumed.Does anyone know if it’s possible to reschedule or get a one-time voucher in situations like this? Any advice wo...
- 47 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Deepak-lv ,Sorry to hear you missed your exam window. Please file a ticket with our support team so they can review the case and determine next steps.Thanks & Regards,@cert-ops
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2112 | |
| 1153 | |
| 947 | |
| 929 | |
| 759 |
Latest from our Blog
Databricks Assistant Data Science Agent: Autonomous Analytics for Petroleum Engineers
Petroleum engineers spend hours wrangling and analyzing data to estimate well production targets. We demonstrate how the Data Science Agent can significantly accelerate this type of analysis and many ...
Azure Databricks — Serverless Private Connectivity to Customer’s Resources (Part-1)
Authors: Kiran Anand, Suraj Karuvel Introduction The serverless model provides customers with a fast and easy option to run their workloads in Azure Databricks. In this model, the compute runs in Dat...
How to Deploy Agent-to-Agent (A2A) Protocol on Databricks Apps?-> End-to-End Guide
Modern AI systems rarely live alone. They collaborate. Google’s Agent-to-Agent (A2A) protocol gives agents a common language (JSON-RPC over HTTP/S, with streaming via SSE) plus a discovery mechanism (...
IoT Ingestion Using Append Flows
IoT (Internet-of-Things) devices are disrupting the manufacturing industry. While estimates vary, most sources agree that there are tens of billions of IoT devices currently operating in manufacturing...
