The Databricks Data Engineer

Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map.

Broadcast joins get passed around as a tip instead of a model: small table equals fast, flip the switch when a join runs long. But the mechanism in almost every write-up is years out of date, and the size Spark checks is not the size that has to fit in memory.

In this episode:
- How to explain broadcast versus shuffle joins at standup, in one sentence, without a whiteboard
- What actually happens on a current Databricks runtime when a join broadcasts, and why the popular explanation stopped being true
- Why the size estimate Spark trusts is not the size that lands, and where to read the real payload
- What adaptive query execution rescues you from, and the two places its hands are tied
- The question to ask before you add a broadcast hint, and the failure signature that tells you a broadcast is what broke

This episode is for Databricks data engineers who write joins every week and treat the broadcast hint as a speed switch. Whether you're mid-level and tired of guessing why one join flies and an identical one falls over, or senior and about to be asked in an interview why Spark chose a sort merge join, you'll walk away able to predict the call before you run the query.

---
Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every week.

LinkedIn: linkedin.com/in/jrlasak
Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake1

Two Databricks engineers sit at adjacent desks. Same title, same pay, same words in the ladder document. Watch a year go by, and they are not doing the same job.

Both get rated strong. Both get told they're ready for more scope. Nobody says the useful thing: the evidence those two are stacking isn't interchangeable, and one of those years makes no sense on the other one's promotion packet.

In this episode:
- Why the data engineering ladder quietly forks, and why that fork is missing from every ladder document
- The five-minute question that tells you which track your last year of Databricks work actually built toward
- How each track reaches staff, and the very different proof each demands
- Where each track stalls a good year, and what to say in your next one on one to get moved off it

This episode is for Databricks data engineers past the junior stage who keep getting strong reviews and vague answers about what's next. Whether you write the cluster policies and ingestion frameworks or ship the tables the business argues from, you'll walk away able to name your track and ask for the work that gets you to the next rung.

---
Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday.

LinkedIn: linkedin.com/in/jrlasak
Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

You inherited Unity Catalog already switched on. You learned the catalogs, the schemas, the grant statements, and never once saw the problem all of it was built to solve. Then someone in a review asks why the company suffered through that migration, and the best you've got is one word: governance.

Here's what that word hides. Before Unity Catalog, one company ran five separate Databricks workspaces, each walled off - its own tables, its own users, its own rules, and nothing connecting them. A new analyst couldn't find where a table lived without three Slack threads. An auditor's one question - who touched customer data last quarter - cost two weeks of digging through old notebooks.

In this episode:
- Why every Unity Catalog design choice is scar tissue over a specific pain from the workspace-per-team era
- How to explain what the migration actually bought your company in sixty seconds, to a product manager or a staff engineer
- The difference between the engineer who says "governance" and the one who gets handed the next platform decision
- Why turning Unity Catalog on does not automatically give you what it promises, and what actually does

This episode is for Databricks data engineers who use Unity Catalog every day but couldn't explain to a product manager why it exists. Whether you joined after the migration and only ever saw the dropdowns, or you're heading into a design conversation where you'll need to justify the work, you'll walk away able to name the world that came before and read every feature as an answer to it.

---
Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday.

LinkedIn: linkedin.com/in/jrlasak
Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Your pipeline hasn't failed in months, and nobody noticed. Then it breaks at 2am, you fix it in twenty minutes, and Slack fills with fire emojis and thank-yous. You're the backbone of the team. You've also been at the same level for three years, and nobody can quite explain why.

It's not bad luck, and it's not a skill gap. The exact thing everyone praises you for, being the one who can always fix it, is the reason they can't afford to move you up.

In this episode:
- Why being the indispensable on-call hero quietly caps the level you can reach
- How one engineer systematized a whole class of Databricks incidents out of existence and got promoted for it
- The difference between measuring your work in saves and measuring it in risk retired
- How to make yourself removable from the hero seat without handing away your job security
- The one self-test that tells you whether your recognition depends on things staying broken

This episode is for Databricks data engineers who carry the pager and keep the platform standing, but keep watching less essential peers get promoted first. Whether you're the war-room legend or the quiet engineer whose prevention work goes uncredited, you'll walk away with three concrete moves to turn indispensability into a promotion case.

---
Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday.

LinkedIn: linkedin.com/in/jrlasak
Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

A nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing to roll back to. The next morning a dashboard is quietly serving half-written garbage, and no one can tell the good files from the wreckage.

That's not a bug in Parquet. Every one of those files is valid, beautifully compressed, doing its job perfectly. The problem lives one level up, in the one thing a folder of Parquet files has never had and a real table cannot live without.

In this episode:
- Why a folder of Parquet files was never actually a table, and what a crashed write silently does to it
- The single idea that turns ACID, time travel, and concurrent writes from three separate Delta features into one
- How Delta really protects you when two pipelines write at once, and why your code should expect a "no" and retry
- A portable question you can point at any storage system to know in one sentence whether you can trust it
- The difference between how junior and senior engineers explain why their team runs on Delta

This episode is for Databricks data engineers who write to Delta tables every day and have never had to explain why it exists. Whether you're prepping for an interview or sitting in a migration meeting, you'll walk away able to explain in sixty seconds what Delta actually bought you, and why the thing that came before it was quietly lying to everyone.

---
Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every Monday, Wednesday, and Friday.

LinkedIn: linkedin.com/in/jrlasak
Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake