MiniLake — The Databricks API, running on your laptop

Open source · MIT · Python 3.11+ · Docker MiniLake is a local Databricks API emulator with real SQL, real Delta Lake and real Spark job execution. Point databricks-sdk, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at http://localhost:8000 — no account, no API key, no cloud bill. $ pip install minilake && minilake --port 8000 PyPI $ docker run -p 8000:8000 ghcr.io/dmux/minilake:latest GHCR Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped. Drop-in fordatabricks-sdk (Python) Databricks CLI Terraform provider Asset Bundles PySpark + Delta MCP agents 30+Databricks API groups emulated, from Unity Catalog to SCIM 67MCP tools for LLM agents, served on the same port 378+tests driven by the real Databricks SDK, not mocks $0cloud spend — everything runs on your machine What you getNot a mock server. A real engine behind a Databricks-shaped API. Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real spark-submit jobs in containers — so a green test locally means something. Unity Catalog, for real Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite. Real execution SQL that actually runs Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types. Real execution Real Delta Lake files EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan(). Real Delta Jobs that really execute spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel. Real Spark Unity Catalog protocol for Spark spark.table("cat.sch.tbl") resolves against MiniLake through the official UCSingleCatalog connector — the same protocol Spark speaks to a real workspace. Wire-compatible A built-in SQL workspace An Athena-style editor at /ui: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab. Batteries included Terraform & Asset Bundles Point the databricks provider at localhost and terraform apply creates real catalogs. Verified on every test run: the real provider is driven through apply, plan and destroy, and a second plan has to come back empty. databricks bundle deploy and bundle run work end to end — jobs execute for real. Wire-compatible An MCP server for agents 67 tools, 8 resources and 4 prompts at /mcp — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL. Real execution Designed for CI One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane. Test friendly Workspace, DBFS, Files, Secrets File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks. Real execution Air-gapped by design DuckDB's delta extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with docker run --network none. Offline ready Databricks-shaped errors Failures come back as {"error_code", "message"}, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud 501 NOT_IMPLEMENTED instead of a convincing lie. Wire-compatible Zero code changesChange the host.

That's the migration.

MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.

Any token is accepted.

No sign-up, no PAT, no OAuth dance to develop against. Types round-trip. STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo.

Physical truth wins.

Declared column types that disagree with the files lose, so mismatches are visible instead of believed. Persistence when you want it. MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start. quickstart.py from databricks.sdk import WorkspaceClient from databricks.sdk.service.catalog import ColumnInfo, TableType # The only line that differs from production w = WorkspaceClient(host="http://localhost:8000", token="dev") w.catalogs.create(name="sales") w.schemas.create(name="store", catalog_name="sales") w.tables.create( name="orders", catalog_name="sales", schema_name="store", table_type=TableType.MANAGED, data_source_format="DELTA", storage_location="/data/sales/store/orders", columns=[ ColumnInfo(name="id", type_text="BIGINT"), ColumnInfo(name="customer", type_text="STRING"), ColumnInfo(name="amount", type_text="DECIMAL(10,2)"), ], ) wh = w.warehouses.create(name="dev", cluster_size="Small") w.statement_execution.execute_statement( warehouse_id=wh.id, statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)", ) result = w.statement_execution.execute_statement( warehouse_id=wh.id, statement="SELECT customer, amount FROM sales.store.orders", ) print(result.result.data_array) # [['Ana', '459.90']] — really stored How it worksOne process.

One port.