Databricks and Spark Interview Questions With Model Answers

Most Spark interview questions are really one question asked in different ways: do you understand what happens on the cluster when your code runs? Candidates who only know the DataFrame API tend to struggle when the interviewer asks why a job is slow, why it ran out of memory, or why a counter printed zero. Candidates who can describe the driver, the executors, lazy evaluation and the shuffle can answer almost anything.

This guide starts with that mental model, then gives model answers for freshers, for experienced candidates and for Databricks-specific questions. Configuration defaults quoted here come from the Apache Spark 4.x documentation as of November 2026. Model answers are illustrative; put them in your own words and connect them to work you have done.

The mental model interviewers are testing

Keep these four ideas in mind and most answers follow from them:

• Driver and executors. Your program runs on the driver, which plans the work. Executors on the worker nodes run tasks on partitions of the data.
• Transformations are lazy, actions are not. The Spark programming guide says transformations create a new dataset from an existing one but do not compute anything straight away; they are computed only when an action needs a result returned to the driver.
• Partitions are the unit of parallelism. One task processes one partition.
• The shuffle is the expensive part. The documentation describes it as Spark’s way of redistributing data so it is grouped differently across partitions, and calls it expensive because it involves disk I/O, serialisation and network I/O.

Spark interview questions for freshers

“What is the difference between a transformation and an action?”

Model answer: “A transformation, such as filter, select or map, describes a new dataset but doesn’t run anything. An action, such as count, collect or writing to storage, triggers execution and returns a result to the driver or writes output. Because transformations are lazy, Spark can look at the whole chain and optimise it before running.”

Why it matters: it explains why an error in a transformation only appears when you call an action, which confuses many beginners.

“What is the difference between narrow and wide transformations?”

Model answer: “In a narrow transformation, such as filter or map, each output partition depends on one input partition, so no data moves between executors. A wide transformation needs data from many partitions, so it causes a shuffle. The documentation lists repartition operations, ‘ByKey’ operations such as groupByKey and reduceByKey, and joins as operations that can cause a shuffle.”

“Why is reduceByKey preferred over groupByKey?”

Model answer: “groupByKey sends every value for each key across the network and then aggregates. reduceByKey combines values within each partition first, so much less data is shuffled. The Spark guide notes that if you are grouping to aggregate per key, reduceByKey or aggregateByKey will perform much better.”

“Why does this counter print zero?”

This is a favourite because it tests whether you understand that code runs in more than one place.

counter = 0
rdd = sc.parallelize(range(100))

def add(x):
    global counter
    counter += x

rdd.foreach(add)
print(counter)   # prints 0 on a cluster, not 4950

Model answer: “The function is serialised and sent to the executors with its own copy of counter. Each executor updates its copy; the driver’s variable never changes. The Spark guide calls the behaviour undefined. For a global sum, use an accumulator, or better still use an aggregation such as sum().”

A strong candidate adds the caveat from the documentation: accumulator updates made inside a lazy transformation such as map() are not guaranteed to run, so update them inside actions.

“What does collect() do, and when is it dangerous?”

Model answer: “It brings the whole dataset back to the driver. That’s fine after a filter that leaves a small result, but on a large dataset the driver can run out of memory. Use take(n) or show() to inspect data, and write large results to storage instead.”

“What is the difference between cache() and persist()?”

Model answer: “Both keep a dataset after it is first computed so later actions can reuse it. persist() lets you choose a storage level; cache() uses the default. For RDDs the documented default is MEMORY_ONLY, where partitions that don’t fit are recomputed when needed. For DataFrames, Spark SQL caches in an in-memory columnar format. I cache only when a dataset is reused several times, and I unpersist it when I’m done.”

Spark interview questions for experienced candidates

“How does Spark choose a join strategy, and when would you broadcast?”

Model answer: “If one side of a join is small, Spark can broadcast it to every executor and avoid shuffling the large side. The threshold is spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MB; setting it to -1 disables automatic broadcasting. I can also use a hint, for example /*+ BROADCAST(r) */ in SQL or .hint("broadcast") on a DataFrame. Hints aren’t guaranteed, because not every strategy supports every join type. For two large tables, Spark typically uses a sort-merge join, which shuffles both sides.”

orders.join(countries.hint("broadcast"), "country_code")

Follow-up to expect: “What goes wrong if you broadcast something too big?” The driver and every executor must hold a copy, so you risk memory errors and long broadcast times.

“What is Adaptive Query Execution?”

Model answer: “AQE re-optimises the query plan while it runs, using statistics collected at shuffle boundaries. It has been enabled by default since Spark 3.2.0. Its main features are: coalescing small post-shuffle partitions, so you don’t have to tune the partition count precisely; converting a sort-merge join to a broadcast join when one side turns out to be small; and splitting skewed partitions in sort-merge joins.”

“How do you deal with data skew?”

Model answer: “First I confirm it: in the Spark UI, one or two tasks in a stage take far longer than the rest or process far more data. Then I find the hot keys, often NULLs or a default value such as an ‘unknown’ country. Options, in roughly the order I’d try them:

1. Filter out or separately handle junk keys such as NULLs if they shouldn’t join anyway.
2. Let AQE’s skew-join handling work. A partition counts as skewed when it is larger than five times the median and larger than 256 MB, by default.
3. Broadcast the smaller side if it fits.
4. Salt the hot key: add a random suffix on the large side, replicate the small side across those suffixes, join, then aggregate.”

“How many shuffle partitions should a job use?”

Model answer: “spark.sql.shuffle.partitions defaults to 200. That’s too many for small data, creating lots of tiny tasks, and too few for very large data, creating huge tasks that spill to disk. With AQE coalescing enabled, I usually set a reasonably high starting number and let Spark merge small partitions at runtime. I check the result in the Spark UI rather than guessing.”

“repartition() or coalesce()?”

Model answer: “repartition always does a full shuffle and can increase or decrease the number of partitions, giving balanced sizes. coalesce reduces partitions by merging existing ones, avoiding a full shuffle, but can leave sizes uneven. Before writing output, I’d use coalesce to reduce the number of small files if the partitions are already fairly even.”

“Why use broadcast variables?”

Model answer: “To send a read-only lookup, such as a mapping table, to each machine once instead of with every task. With DataFrames I’d normally express this as a broadcast join instead, but in RDD code or Python functions a broadcast variable avoids shipping the same object repeatedly.”

Databricks interview questions

Roles that use Databricks add questions about Delta Lake and the platform. Databricks’ documentation describes Delta Lake as the optimised storage layer for tables on the platform and says it is the default format: all tables on Databricks are Delta tables unless specified otherwise.

“What does Delta Lake add on top of Parquet?”

Model answer: “Delta adds a file-based transaction log to Parquet data files. That gives ACID transactions, so readers never see half-written data, and a table history: every write creates a new table version. I can query earlier versions, which helps with audits and with recovering from a bad load.”

DESCRIBE HISTORY sales.orders;

SELECT * FROM sales.orders VERSION AS OF 12;

“How would you upsert a daily batch of changes?”

Model answer: “With MERGE INTO: match on the business key, update when matched and insert when not matched. I’d de-duplicate the source batch first, because the merge fails with an error if several source rows match the same target row: Spark can’t tell which one should win.”

MERGE INTO sales.customers AS t
USING updates AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

“A Delta table has become slow to query. What do you check?”

Model answer: “First, the small-file problem: frequent small writes create many tiny files. OPTIMIZE compacts small files into larger ones. Second, data layout: whether queries can skip files using column statistics, Z-ordering or liquid clustering on the columns people filter by. Third, old files: VACUUM removes data files that the table no longer references. Finally, I’d look at the query plan to see whether the problem is the table or the query.”

Follow-up questions often ask about the trade-off with VACUUM: once old files are removed, you can no longer time-travel to versions older than the retention period, which the Azure Databricks reference gives as seven days by default (the delta.deletedFileRetentionDuration table property).

A debugging scenario to rehearse

Senior interviews often end with an open scenario. Here is an illustrative one and the outline of a strong answer.

“A nightly job that used to take 20 minutes now takes three hours. Nothing in the code has changed. Walk me through it.”

1. Check the input. Has the data volume grown, or has a new value appeared that skews a join key?
2. Open the Spark UI. Which stage got slower? Are a few tasks much slower than the rest (skew), or are all tasks slower (volume or resources)? Is there disk spill?
3. Compare the physical plans. Has a join that used to be broadcast switched to a sort-merge join because a table grew past the threshold?
4. Check the cluster. Did the cluster size, instance type or runtime version change?
5. Check the storage. Has the source table accumulated many small files?
6. Fix one thing, measure, and add monitoring so the next slowdown is caught early.

The order matters: evidence first, changes second. Interviewers mark candidates down for jumping straight to “add more nodes”.

How to prepare in the final week

• Run a few jobs locally or on a free workspace and read the Spark UI for each: stages, tasks, shuffle read and write.
• Call explain() on a join before and after adding a broadcast hint, and describe the difference out loud.
• Prepare two stories from your own work: a performance fix with before-and-after numbers, and a data-quality incident you resolved.
• Know which Spark and Databricks runtime versions you have used. Interviewers notice when candidates describe features that their version did not have.

Spark skills belong on your CV as specific outcomes, such as data volumes, runtime reductions or cost savings. A data analyst resume builder suits analytics-focused roles, while engineering-heavy roles may fit the software engineer resume builder better.

Frequently asked questions

What Spark topics are asked most in interviews?

Lazy evaluation, transformations versus actions, narrow versus wide transformations, shuffles, joins and broadcast joins, caching, partitioning, data skew and reading the Spark UI. Databricks roles add Delta Lake, MERGE, time travel and table maintenance.

Should I learn Spark with Python or Scala for interviews?

Use whichever the employer’s job advert names. PySpark is common in analytics and data engineering roles. The concepts interviewers test, such as shuffles and execution plans, are the same in both.

What Spark questions are asked for freshers?

Expect transformations versus actions, lazy evaluation, groupByKey versus reduceByKey, cache versus persist, what collect() does and why driver-side variables do not update inside foreach.

What Spark questions are asked for experienced candidates?

Expect join strategies and broadcast thresholds, Adaptive Query Execution, skew handling, shuffle partition tuning, small-file problems and an open debugging scenario about a slow or failing job.

Is a Databricks certification required?

Usually not. Employers mainly test practical understanding in the interview. A certification can support your CV, but be ready to explain the concepts in depth either way.

Sources

• RDD Programming Guide (Apache Spark)
• Performance Tuning (Spark SQL) (Apache Spark)
• What is Delta Lake in Databricks? (Databricks)
• Tutorial: Create and manage Delta Lake tables (Databricks)
• MERGE INTO (Databricks)
• VACUUM (Azure Databricks) (Microsoft)