MiniLake — a local Databricks API emulator
Open-source Databricks API emulator, written by data engineers for data engineers. SQL on DuckDB, Delta tables as files, jobs through spark-submit. MIT licensed.
Open-source Databricks API emulator, written by data engineers for data engineers. SQL on DuckDB, Delta tables as files, jobs through spark-submit. MIT licensed.

Open source · MIT · Made by data engineers, for data engineers
MiniLake is a Databricks API emulator.
SQL runs on DuckDB, Delta tables are real files, and jobs run spark-submit in a container.
Point databricks-sdk, the Databricks CLI, Terraform or an MCP agent at
http://localhost:8000; any token is accepted.
Open source today, and it will stay that way.
$
pip install minilake && minilake --port 8000
PyPI
$
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest
GHCR
Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.
What it does
SQL executes on DuckDB, Delta tables are written to disk, and jobs run spark-submit. Everything else in the API is documented below with its status.
Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.
Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.
EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().
spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.
Point the databricks provider at localhost and terraform apply creates real catalogs. Verified on every test run: the real provider is driven through apply, plan and destroy, and a second plan has to come back empty. databricks bundle deploy and bundle run work end to end — jobs execute for real.
One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.
Zero code changes
MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.
STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo.MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.quickstart.py
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ColumnInfo, TableType
# The only line that differs from production
w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
w.tables.create(
name="orders", catalog_name="sales", schema_name="store",
table_type=TableType.MANAGED,
data_source_format="DELTA", storage_location="/data/sales/store/orders",
columns=[
ColumnInfo(name="id", type_text="BIGINT"),
ColumnInfo(name="customer", type_text="STRING"),
ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
],
)
wh = w.warehouses.create(name="dev", cluster_size="Small")
w.statement_execution.execute_statement(
warehouse_id=wh.id,
statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
)
result = w.statement_execution.execute_statement(
warehouse_id=wh.id,
statement="SELECT customer, amount FROM sales.store.orders",
)
print(result.result.data_array) # [['Ana', '459.90']] — really stored
How it works
A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.
For LLM agents
Optional, at /mcp. Several tools collapse multi-call sequences into one, and a resource
describes the DuckDB SQL dialect so agents stop writing Spark SQL. Tools go through MiniLake's own
ASGI stack, so an agent sees the same errors an SDK client would.
register + use
# One line to register it with Claude Code
claude mcp add --transport http minilake http://localhost:8000/mcp
# Then just ask:
# "Use minilake: create a sales catalog with a seeded
# orders table, and tell me the total."
# The agent does this — three calls, no glue code:
# setup_fixture({"catalog": "sales", "schema": "store"})
# seed_table({"full_name": "sales.store.orders", ...})
# run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})
# Enable it on the server side:
MINILAKE_MCP=1 minilake --port 8000
Coverage
Anything out of scope answers 501 NOT_IMPLEMENTED instead of faking a success. The full per-service list is in FEATURES.md.
| Service | Status | What that means |
|---|---|---|
| Unity Catalog | Real | Catalogs, schemas, tables, volumes, functions — each catalog is its own DuckDB database, with native three-part addressing |
| SQL statements | Real | Executed on DuckDB; all three formats and both dispositions, with a typed result manifest |
| EXTERNAL Delta tables | Real | Genuine Delta files; writes go through a generated Spark job, reads through delta_scan() |
| Jobs | Real | spark-submit in sibling containers (or a subprocess fallback), real DAG scheduling, real logs, one-shot runs/submit |
| Workspace, DBFS & Files | Real | File-backed storage with chunked upload; raw-bytes sync is what makes bundle deploy work |
| Clusters | State machine | CRUD plus timed lifecycle transitions — deliberately no Spark compute; that lives in Jobs |
| Permissions | Allow-all CRUD | Real create/read/update, but nothing is ever enforced — this is a single-user tool |
| Identity & SCIM | Unenforced CRUD | Users, groups and service principals, with SCIM filtering and PATCH — records, not credentials; any token is still accepted |
| DLT, Model Registry, Vector Search, Lakeview, Repos | Not implemented | Returns 501 NOT_IMPLEMENTED — loudly, so you never mistake it for working |
Limits
These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.
Any token is accepted and there is exactly one user. Never expose it beyond localhost.
The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.
Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.
The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.
DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.
They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.
Open source
MiniLake is written by data engineers for data engineers. The scope is what pipeline and CI work needs, and the limits above are listed because we would want them listed.
Use it, fork it, ship it.
Source, CI, changelog and issues are public at dmux/minilake.
Report the SDK call that fails in an issue, or add an API group following CONTRIBUTING.md.
Everyone who has committed to the project. See all contributors.
Texto de dmux.github.io tal como lo publicó, leído de su página y mostrado aquí con el formato de El Autómata. El original, en dmux.github.io · Volver a la noticia de El Autómata