Databricks Migration

MicrosoftOfficialPublished by Microsoft

Skill
Workflow
Cloud and DevOps
MIT
Low risk

From the author

Ports existing Databricks notebooks and jobs to Fabric, covering dbutils to notebookutils, secret scopes to Key Vault, DBFS to OneLake, Unity Catalog mapping to schema-enabled Lakehouses, Jobs and Delta Live Tables to Spark Job Definitions and Pipelines, and Photon to the Native Execution Engine. Use whenever Databricks code has to be converted. For Fabric notebook code that is not being migrated, use spark-cli.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The databricks-migration skill provides a structured framework for porting Databricks assets to Microsoft Fabric. It covers the conversion of core components including dbutils to notebookutils, secret scopes to Azure Key Vault, DBFS to OneLake, and Unity Catalog structures to schema-enabled Lakehouses. The skill emphasizes a systematic approach, requiring inventory assessment before prescribing topology, and provides specific mappings for Spark job definitions, MLflow experiments, and cluster configurations to ensure a smooth transition to Fabric's native execution environment.

What it does

Maps dbutils API calls to their notebookutils equivalents.
Provides strategies for migrating Unity Catalog namespaces to Fabric Lakehouse schemas.
Translates Databricks Jobs and Delta Live Tables into Fabric Spark Job Definitions and Pipelines.
Guides the transition from Databricks cluster configurations to Fabric Spark Pools and Environments.
Facilitates the migration of MLflow tracking and model registry workflows.
Defines a multi-notebook migration protocol to manage complex workspace transitions.

When to use it

When porting existing Databricks notebooks, jobs, or data pipelines to Microsoft Fabric.
When needing to resolve namespace collisions during catalog migration.
When replacing proprietary Databricks features like DLT or DBFS with Fabric-native alternatives.
When configuring security and secret management for migrated workloads in Fabric.

Advantages

Ensures architectural alignment with Fabric's native execution engine.
Provides clear, actionable mappings for deprecated Databricks-specific APIs.
Reduces manual effort through structured migration patterns and protocols.
Promotes best practices for library management and environment configuration in Fabric.

Usage examples

“How do I migrate my dbutils.fs.ls calls to Fabric?”: The agent will provide the equivalent notebookutils.fs.ls syntax and explain the necessary path adjustments from DBFS to OneLake.
“I need to move my Databricks Unity Catalog to Fabric. What is the recommended approach?”: The agent will guide you through mapping Unity Catalog catalogs to schema-enabled Lakehouses and explain how to handle namespace preservation.
“How should I handle Databricks Jobs that use Delta Live Tables?”: The agent will explain that DLT has no direct equivalent and will provide a plan to rewrite the logic as parameterized notebooks orchestrated by Fabric Pipelines.
“What is the Fabric equivalent for my Databricks secret scopes?”: The agent will instruct you to use notebookutils.credentials.getSecret, mapping the Databricks scope to an Azure Key Vault URL.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/databricks-migration/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip databricks-migration.zip -d .agents/skills

Documentation by Microsoft

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: databricks-migration (az rest: --headers "x-ms-fabric-skill=databricks-migration"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES
To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
dbutils.widgets has no direct equivalent in Fabric — use notebook parameters (cell tag parameters); notebookutils.runtime.context is execution metadata, not parameter storage. If showing context fields, use documented names such as currentWorkspaceId, currentWorkspaceName, currentNotebookId, currentNotebookName, isForPipeline, and isForInteractive; activityId is the Livy job ID
dbutils.library (runtime library install) has no equivalent — use Fabric Environments for reproducible library management
Map each Unity Catalog catalog to a schema-enabled Lakehouse by default. This preserves the source schema.table hierarchy, with the Lakehouse representing the catalog; collisions arise only if multiple catalogs are intentionally consolidated into one Lakehouse
For an under-specified workspace-wide migration, ask focused questions about inventory, workload topology, security, data locations, and runtime constraints before recommending a Fabric topology
A completed Fabric migration must not retain executable dbutils.* calls in dual-runtime branches or try/except guards — replace the calls and Databricks paths outright

Databricks → Microsoft Fabric Migration

Prerequisite Knowledge

Read these companion documents before executing migration tasks:

COMMON-CORE.md — Fabric REST API patterns, authentication, token audiences, item discovery
COMMON-CLI.md — az rest, az login, token acquisition, Fabric REST via CLI
SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, Spark job execution

For notebook and Lakehouse creation, see spark-cli. For Fabric Warehouse DDL/DML authoring, see sqldw-cli.

Topic	Reference
Migration Orchestrator	migration-orchestrator.md
Migration Workload Map	§ Migration Workload Map
Complete `dbutils` → `notebookutils` Mapping	dbutils-to-notebookutils.md
Unity Catalog → Fabric Lakehouse Schemas	catalog-migration.md
Before/After Code Patterns	code-patterns.md
Cluster Config → Fabric Spark Pools	§ Cluster Config → Fabric Spark Pools
Databricks Jobs → Spark Job Definitions	§ Databricks Jobs → Spark Job Definitions
Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts	§ Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts
MLflow → Fabric ML Experiments	§ MLflow → Fabric ML Experiments
Post-Migration Validation & Testing	validation-testing.md
Migration Gotchas & Troubleshooting	migration-gotchas.md
Multi-Notebook Migration Protocol	§ Multi-Notebook Migration Protocol
Failure Reporting	§ Failure Reporting
Must / Prefer / Avoid	§ Must / Prefer / Avoid
Authentication & Token Acquisition	COMMON-CORE.md § Authentication
Lakehouse Management	SPARK-AUTHORING-CORE.md § Lakehouse Management
Notebook Management	SPARK-AUTHORING-CORE.md § Notebook Management

Context Loading Guide

IMPORTANT — Load only what you need. Do NOT read all resource files upfront. Load the specific file for the phase you are executing:

When	Read This File
User asks to migrate a workspace (full orchestration)	migration-orchestrator.md
Applying code transforms (dbutils, namespaces, paths)	dbutils-to-notebookutils.md + code-patterns.md
Resolving Unity Catalog namespace collisions	catalog-migration.md
Post-migration verification	validation-testing.md
Troubleshooting failures or known issues	migration-gotchas.md

Migration Workload Map

Databricks Component	Fabric Target	Severity	Notes
All-purpose cluster (notebooks, REPL)	Fabric Notebook (Starter Pool or Custom Pool)	Info	No persistent cluster — Fabric provisions compute on session start
Job cluster (automated jobs)	Spark Job Definition (SJD)	Info	SJD maps one-to-one with Databricks Jobs on job clusters
Unity Catalog	Fabric Lakehouse (schema-enabled, one per catalog)	Info	Schema-enabled Lakehouse preserves `schema.table`; default one-Lakehouse-per-catalog has no collision — see catalog-migration.md
Databricks Repos (Git-backed notebooks)	Fabric Git Integration	Info	Connect workspace to Azure DevOps or GitHub; notebooks are synced
Delta Live Tables (DLT)	Fabric Notebooks + Data Pipelines	Blocker	No DLT equivalent — rewrite DLT datasets as parameterized notebook cells with pipeline orchestration
Databricks SQL Warehouses	Fabric Warehouse or Lakehouse SQL Endpoint	Info	SQL warehouse sessions → Warehouse (for write) or SQL Endpoint (for read-only)
MLflow Tracking	Fabric ML Experiments	Info	MLflow SDK is supported in Fabric — see § MLflow
Delta Sharing	OneLake Shortcuts + Fabric external data sharing	Warning	See § Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts
Databricks Feature Store	Feature engineering on Lakehouse/Delta tables + MLflow	Warning	Fabric has no drop-in managed Feature Store; recreate feature tables as Delta tables in a Lakehouse and manage features via notebooks/MLflow. Verify current Fabric feature-store roadmap before committing an approach
dbutils (all sub-modules)	`notebookutils` (most sub-modules)	Info	See dbutils-to-notebookutils.md for full mapping
Scala notebooks	Fabric Notebook (Spark/Scala)	Warning	Scala is supported; swap cell magic `%scala` → `%%spark` and rewrite Databricks-specific APIs/libraries
R notebooks	Fabric Notebook (SparkR)	Warning	SparkR is supported; swap cell magic `%r` → `%%sparkr`, validate package availability, rewrite Databricks-specific APIs

Severity Definitions

Level	Meaning	Action
Blocker	Cannot run in Fabric without redesign or user decision	Stop — surface to user, require resolution
Warning	Migratable but requires validation or architectural decision	Migrate with review flag
Info	Direct substitution-level change	Auto-migrate

`dbutils` → `notebookutils` Quick Reference

The complete side-by-side API table is in dbutils-to-notebookutils.md. The key mappings are:

`dbutils` Call	`notebookutils` Equivalent	Compatibility Note
`dbutils.fs.ls(path)`	`notebookutils.fs.ls(path)`	Direct replacement
`dbutils.fs.cp(src, dest)`	`notebookutils.fs.cp(src, dest)`	Direct replacement
`dbutils.fs.mv(src, dest)`	`notebookutils.fs.mv(src, dest, create_path, overwrite=False)`	⚠️ Signature differs — see dbutils-to-notebookutils.md
`dbutils.fs.rm(path, recurse)`	`notebookutils.fs.rm(path, recurse)`	Direct replacement
`dbutils.fs.mkdirs(path)`	`notebookutils.fs.mkdirs(path)`	Direct replacement
`dbutils.fs.put(path, contents)`	`notebookutils.fs.put(path, contents)`	Direct replacement
`dbutils.fs.head(path, maxBytes)`	`notebookutils.fs.head(path, max_bytes)`	⚠️ Default differs — Python/Scala 100 KB, R 64 KB. See dbutils-to-notebookutils.md
`dbutils.fs.mount(...)`	`notebookutils.fs.mount(source, mountPoint, extraConfigs=None)`	✅ Supported — Microsoft Entra (default), `accountKey`, or `sasToken` auth. For cross-workspace / persistent sharing, prefer OneLake Shortcuts
`dbutils.secrets.get(scope, key)`	`notebookutils.credentials.getSecret(keyVaultUrl, secretName)`	Scope → Key Vault URL; key → secret name
`dbutils.notebook.run(path, timeout, args)`	`notebookutils.notebook.run(name, timeout, args)`	`path` → notebook `name` (relative to workspace)
`dbutils.notebook.exit(value)`	`notebookutils.notebook.exit(value)`	Direct replacement
`dbutils.widgets.get(name)`	See § Widgets Migration	No direct equivalent
`dbutils.library.install(...)`	Not available at runtime — use Fabric Environments	`dbutils.library.restartPython()` → `notebookutils.session.restartPython()`
`dbutils.data.summarize(df)`	`display(df.summary())`	Use `display()` or pandas `describe()`

Widgets Migration

dbutils.widgets has no direct equivalent in Fabric. Use these patterns instead:

Use Case	Fabric Pattern
Pass parameter from parent notebook	Mark a cell in the child notebook as a parameters cell (notebook UI: cell "..." menu → "Mark cell as parameters"). The parent calls `notebookutils.notebook.run("child", arguments={"param": "value"})` — at runtime the engine inserts a new cell beneath the parameters cell that overrides the defaults
Pipeline-driven parameterization	Same parameters-cell mechanism; the Fabric Pipeline notebook activity supplies override values via its Base parameters setting
Centralized cross-notebook config	Use `notebookutils.variableLibrary.getLibrary("<name>")` to read values from a Variable Library item (deployment pipelines activate the right value set per stage)
Interactive selection in notebook	Use `display()` with input cells, IPython widgets (Python only), or Fabric Data Activator

Note: notebookutils.runtime.context does not expose parameter values. It's for execution metadata (workspace/notebook/activity/user IDs, pipeline-vs-interactive flags, etc.). See dbutils-to-notebookutils.md § Runtime Context.

Cluster Config → Fabric Spark Pools

Databricks Cluster Concept	Fabric Spark Equivalent	Notes
All-purpose cluster (interactive)	Starter Pool	Auto-provisioned; no config; ideal for notebooks
Job cluster (single-use for jobs)	Custom Pool (or Starter Pool) attached to SJD	Configure node size, autoscale in Fabric capacity settings
Node type (e.g., `Standard_DS3_v2`)	Fabric node size (Small/Medium/Large/X-Large/XX-Large)	Map by vCore/memory ratio
Autoscale min/max workers	Custom Pool min/max node settings	Available in workspace Spark settings
`spark.conf` in cluster settings	Fabric Environment Spark properties	Move to Environment item; attach to workspace or notebook
`init_scripts` (cluster init)	Fabric Environment install script	Not fully equivalent — only library installs are supported
Databricks Runtime version	Fabric Runtime (1.1 = Spark 3.3, 1.2 = Spark 3.4, 1.3 = Spark 3.5)	Choose matching Spark version; test deprecated APIs
Photon accelerator	Fabric Native Execution Engine (NEE)	Enable in workspace Spark settings; vectorized execution similar to Photon

Databricks Jobs → Spark Job Definitions

Databricks Jobs Concept	Fabric SJD Equivalent	Notes
Job with single notebook task	SJD referencing a notebook	Attach a default Lakehouse; pass parameters via SJD args
Multi-task job (DAG of tasks)	Fabric Data Pipeline orchestrating multiple SJDs/notebooks	Pipeline activities map to job tasks; dependencies = activity dependencies
Job schedule (cron)	Pipeline schedule trigger	Cron expression → recurrence trigger in pipeline
Job parameters	SJD default arguments or notebook cell parameters	Parameters cell in notebook is injected at runtime
Job clusters per task	Pool attached to SJD	Each SJD can specify its Spark pool independently
Databricks Workflows	Fabric Data Pipelines	Full DAG orchestration with conditions, loops, and failure branches

Delegate to spark-cli for SJD creation and notebook deployment.

Databricks Delta Sharing Pattern	Fabric Equivalent
Provider publishes a Delta share	Fabric external data sharing for cross-tenant Fabric data, or a OneLake Shortcut to ADLS Gen2 where the Delta data resides
Recipient reads shared data	Accept the external data share into a Lakehouse (Fabric creates a read-only OneLake Shortcut), or create a direct OneLake Shortcut to accessible ADLS Gen2 data
Cross-workspace table sharing within org	OneLake Shortcuts pointing to another workspace's Lakehouse tables — no data copy
Cross-tenant sharing	Fabric external data sharing — live, read-only, in-place access through a shortcut in the recipient tenant

When producing a migration workload map, include both paths: direct OneLake Shortcuts for accessible ADLS or same-tenant OneLake data, and Fabric external data sharing for native cross-tenant recipient sharing.

MLflow → Fabric ML Experiments

Fabric ML Experiments are built on the MLflow SDK — most code is directly portable:

Databricks MLflow Pattern	Fabric Equivalent	Migration Action
`mlflow.set_tracking_uri("databricks")`	Remove — Fabric tracking is automatic	Delete this line in Fabric notebooks
`mlflow.set_experiment("/path/exp")`	`mlflow.set_experiment("experiment_name")`	Use name only (not path); Fabric creates the Experiment item
`mlflow.log_metric(...)`	`mlflow.log_metric(...)` — identical	No change
`mlflow.log_artifact(...)`	`mlflow.log_artifact(...)` — identical	No change
`mlflow.autolog()`	`mlflow.autolog()` — identical	No change
`mlflow.register_model(...)`	`mlflow.register_model(...)` — identical	Model Registry is available in Fabric ML
Databricks Model Serving	Azure ML Online Endpoints or Fabric Data Activator	No direct Fabric model serving yet — use Azure ML

Must / Prefer / Avoid

MUST DO

Inventory before prescribing a workspace topology — for workspace-wide requests that omit workload inventory, dependencies, security requirements, data locations, or runtime constraints, ask focused clarifying questions and present conditional choices before selecting a Fabric design
Replace all dbutils.* calls using the mapping in dbutils-to-notebookutils.md — dbutils is not available in Fabric notebooks
Migrate dbutils.fs.mount() to notebookutils.fs.mount() (✅ supported — Microsoft Entra default, or accountKey / sasToken from Key Vault). For cross-workspace or persistent sharing, prefer OneLake Shortcuts instead. Always pair mount() with unmount() in try/finally — Fabric mounts are not released automatically on session end
Replace dbutils.secrets.get(scope, key) with notebookutils.credentials.getSecret(keyVaultUrl, secretName) — secret scopes map to Azure Key Vault URLs
Redesign widget-based parameter passing using notebook parameters cells (cell "..." menu → "Mark cell as parameters"); use notebookutils.variableLibrary for centralized cross-notebook config. notebookutils.runtime.context does not expose parameter values
Replace dbutils.library.install*() with Fabric Environments — runtime library installs are not supported in production. dbutils.library.restartPython() maps to notebookutils.session.restartPython() (Python / PySpark only)
Map Unity Catalog namespaces deliberately — default to one schema-enabled Lakehouse per catalog so schema.table is preserved; require a user-approved naming policy only when consolidating multiple catalogs into one Lakehouse. See catalog-migration.md
Map Databricks cluster init scripts to Fabric Environments — cluster-level library installs must move to Environment items

PREFER

Fabric Native Execution Engine (NEE) as the Photon equivalent — enable in workspace Spark settings for vectorized execution on Delta Lake
OneLake Shortcuts over data copy for Delta tables that already exist in ADLS Gen2 — point directly without re-ingesting
Fabric Git Integration as the replacement for Databricks Repos — connect workspace to ADO or GitHub for notebook version control
Fabric ML Experiments for direct MLflow continuity — tracking code requires minimal changes (remove set_tracking_uri)
Medallion architecture when restructuring migrated Databricks catalogs — align bronze, silver, gold Unity Catalog schemas to separate Fabric Lakehouses
Starter Pool for migrating interactive notebook workflows — eliminates cluster startup time that was a common pain point in Databricks job clusters

AVOID

Do not prescribe a one-size-fits-all workspace topology when the source inventory and migration constraints are missing
Do not import dbutils or attempt dbutils = ... assignments in Fabric notebooks — import attempts fail with ModuleNotFoundError, while unresolved dbutils references raise NameError; always use notebookutils
Do not retain dbutils.* calls behind runtime-detection guards (try/except, if IS_DATABRICKS) — replace the calls and Databricks paths outright with notebookutils and Fabric paths
Do not assume Unity Catalog governance policies transfer automatically — RBAC, row-level security, and column masking must be reconfigured in Fabric using workspace roles and Lakehouse permissions
Do not use %pip install in production Fabric notebooks at runtime — use Fabric Environments for stable, versioned library management
Do not attempt to port Delta Live Tables (DLT) pipelines verbatim — DLT has no Fabric equivalent; rewrite as parameterized notebooks orchestrated by Fabric Pipelines
Do not rely on Databricks-specific Spark configurations (e.g., spark.databricks.*) — these are proprietary and will be silently ignored or raise errors in Fabric
Do not use DBFS paths (dbfs:/...) — there is no DBFS in Fabric; all paths must use OneLake abfss:// or Lakehouse-relative paths

Multi-Notebook Migration Protocol

For workspaces with >3 notebooks or individual notebooks >5KB, process one notebook at a time (enumerate → export → transform → summarize → deploy → release) to avoid context overflow. Track each notebook through a status lifecycle (inventory → analyzed → converted → deployed → validated, or failed).

Full protocol, status definitions, and per-notebook summary schema: migration-orchestrator.md § Phase 2.

Failure Reporting

When migration cannot complete (permission failures, unresolvable Blockers such as DLT or OS-level init scripts, namespace collisions with no user-chosen policy, or repeated API failures), emit a structured failure report — do not abandon silently. The report captures phase_reached, blockers[] (item / pattern / reason / recommendation), partial_success counts, and next_steps.

Full report schema and stopping conditions: migration-orchestrator.md § Failure Reporting.

Examples

See dbutils-to-notebookutils.md and code-patterns.md for the full mapping. Key quick references:

dbutils.fs → notebookutils.fs

# Databricks
dbutils.fs.ls("/mnt/bronze/orders/")
dbutils.fs.cp("/mnt/raw/file.csv", "/mnt/archive/file.csv")

# Fabric (replace DBFS/mount paths with OneLake relative paths)
notebookutils.fs.ls("Files/bronze/orders/")
notebookutils.fs.cp("Files/raw/file.csv", "Files/archive/file.csv")

dbutils.secrets → notebookutils.credentials

# Databricks
pwd = dbutils.secrets.get(scope="prod", key="db-password")

# Fabric (scope → Key Vault URL, key → secret name)
pwd = notebookutils.credentials.getSecret("https://myvault.vault.azure.net/", "db-password")

Unity Catalog namespace → Lakehouse schema

# Databricks
df = spark.read.table("prod.silver.customers")

# Fabric (ProdLakehouse represents the source catalog and is attached as context)
df = spark.read.table("silver.customers")

Files

ATTRIBUTION.mdAdded by AutoGineer885 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md22.1 KiB
apm.yml460 B
resources/catalog-migration.md6.5 KiB
resources/code-patterns.md6.7 KiB
resources/dbutils-to-notebookutils.md21.2 KiB
resources/migration-gotchas.md9.3 KiB
resources/migration-orchestrator.md17.4 KiB
resources/validation-testing.md12.4 KiB

resources/migration-orchestrator.md

Migration Orchestrator — Databricks → Fabric

Automated end-to-end migration workflow for porting Databricks workspaces to Microsoft Fabric.

Script pattern: Sections show function signatures and key logic outlines. Generate the full runnable script on demand — this keeps the resource compact and reduces token load.

Required Inputs

Collect these from the user before starting:

Input	Example	How to Obtain
Databricks workspace URL	`https://adb-1234567890123456.12.azuredatabricks.net`	User provides
Databricks auth	Personal access token or OAuth	`databricks auth login` or user provides PAT
Fabric workspace name or ID	`Migration-Target`	User provides
Migration strategy	`lift-and-shift` or `migrate-and-modernize`	Ask user

Derived Values (Auto-Discovered)

Value	Derived From
Databricks workspace ID	Parse from URL (`adb-{workspaceId}`) or `databricks workspace get-status /`
Fabric workspace ID	`GET /v1/workspaces` → filter by `displayName`
Databricks catalog/schema list	`databricks catalogs list` / `databricks schemas list <catalog>`
Notebook inventory	`databricks workspace list --recursive /`
Job inventory	`databricks jobs list --all`

Orchestration Flow

Phase 0: Inventory & Assessment
├── Enumerate notebooks, jobs, DLT pipelines, clusters, secrets, libraries
├── Classify each item by migration complexity (severity matrix)
└── Generate migration manifest with namespace mapping

Phase 1: Environment Setup
├── Create Fabric Lakehouse(s) — one schema-enabled Lakehouse per Unity Catalog catalog
├── Create Fabric Environment(s) — map cluster libraries
└── Create OneLake Shortcuts for external data (ADLS, S3)

Phase 2: Notebook Migration
├── Export notebooks from Databricks workspace
├── Apply code transforms (dbutils→notebookutils, namespace, paths)
├── Deploy to Fabric workspace via REST API
└── Per-notebook validation checkpoint

Phase 3: Job & Orchestration Migration
├── Map single-task Jobs → Spark Job Definitions
├── Map multi-task Jobs → Fabric Data Pipelines
├── Map DLT Pipelines → Notebook + Pipeline combinations
└── Schedule trigger migration

Phase 4: Validation & Cutover
├── Execute migrated notebooks (dry run)
├── Compare outputs where applicable
├── Generate migration report
└── Flag any unresolved blockers

Phase 0: Inventory & Assessment

Step 0.1: Enumerate All Workspace Items

Use Databricks CLI to build a complete inventory:

databricks workspace list / --recursive --output json
databricks jobs list --all --output json
databricks clusters list --output json
databricks pipelines list-pipelines --output json
databricks secrets list-scopes --output json

For Unity Catalog assets:

# Requires Databricks CLI v0.205 or newer. The legacy `databricks unity-catalog ...`
# command group has been removed; use the catalogs/schemas/tables commands below.
databricks catalogs list --output json
databricks schemas list <catalog> --output json
databricks tables list <catalog> <schema> --output json

Step 0.2: Classify Items by Migration Severity

Apply severity to each item based on its migration complexity:

Severity	Criteria	Action
Blocker	Cannot run in Fabric without redesign or user decision	Stop — surface to user
Warning	Migratable but requires validation/performance review	Migrate with review flag
Info	Direct substitution-level change (e.g., `dbutils.fs.ls` → `notebookutils.fs.ls`)	Auto-migrate

Blocker Patterns

Pattern Detected	Why It Blocks	Resolution
Delta Live Tables (DLT) pipeline	No DLT equivalent in Fabric	Rewrite as parameterized notebooks + Pipeline orchestration
`dbutils.widgets.*` (interactive widgets)	No direct equivalent	Redesign as parameters cells + `notebookutils.notebook.run(name, arguments={})`
Unity Catalog namespace collision (only under user-elected multi-catalog consolidation)	Two different `catalog.schema.table` paths would collapse to the same `schema.table` in one Lakehouse	User must supply mapping policy — do not auto-resolve. Default one-Lakehouse-per-catalog mapping has no collision
JDBC/native dependencies installed by `init_scripts` at the OS level	Fabric Environments support Maven and custom JAR libraries, but not arbitrary OS packages, daemon setup, or native system configuration	Move supported JAR/Maven dependencies into the Environment; redesign only the unsupported OS/native setup
Databricks-specific Spark configs (`spark.databricks.*`)	Proprietary — ignored or errors in Fabric	Remove; substitute Fabric equivalents where they exist

Warning Patterns

Pattern Detected	Why It Warns	Resolution
Scala notebooks (`%scala`)	Fabric supports Spark/Scala, but Databricks-specific APIs and libraries may differ, and cell magics change (`%scala` → `%%spark`)	Validate runtime/library compatibility; rewrite Databricks-specific APIs and swap cell magics
R notebooks (`%r`)	Fabric supports SparkR, but package availability may differ, and cell magics change (`%r` → `%%sparkr`)	Validate packages via Fabric Environment; rewrite Databricks-specific APIs and swap cell magics
`%pip install` without version pins	May install incompatible versions on Fabric Runtime	Pin versions in Fabric Environment
`dbutils.fs.mount(...)` usage	Fabric supports `notebookutils.fs.mount()` but prefers Shortcuts for persistence	Migrate to `notebookutils.fs.mount()` + recommend Shortcut
Photon-optimized queries	NEE covers most cases but not all Photon behaviors	Test with NEE enabled; benchmark
MLflow `set_tracking_uri("databricks")`	Must remove for Fabric	Auto-remove (Info if standalone)
Cluster autoscale with 20+ workers	Fabric Custom Pools may have different capacity limits	Verify Fabric capacity SKU supports equivalent scale

Info Patterns (Auto-Migratable)

Pattern Detected	Transformation
`dbutils.fs.ls/cp/rm/mkdirs/put/head`	→ `notebookutils.fs.ls/cp/rm/mkdirs/put/head`
`dbutils.notebook.run(path, timeout)`	→ `notebookutils.notebook.run(name, timeout)`
`dbutils.notebook.exit(value)`	→ `notebookutils.notebook.exit(value)`
`dbutils.secrets.get(scope, key)`	→ `notebookutils.credentials.getSecret(kvUrl, secretName)`
`spark.read.table("catalog.schema.table")`	→ `spark.read.table("schema.table")` (with Lakehouse context)
`SparkSession.builder.getOrCreate()`	Remove — `spark` is pre-instantiated in Fabric

Step 0.3: Namespace Mapping

Unity Catalog uses 3-level naming (catalog.schema.table). A schema-enabled Fabric Lakehouse preserves both lower levels (schema.table), and the Lakehouse itself stands in for the catalog — yielding the full workspace.lakehouse.schema.table path. There is no inherent level collapse.

Default mapping (no collision): create one schema-enabled Lakehouse per Databricks catalog. Schemas and table names are preserved verbatim. This is the recommended structure and requires no renaming.

Optional consolidation (collision risk): collisions only become possible if the user explicitly elects to consolidate multiple catalogs into a single Lakehouse — then two catalog.schema.table paths can map to the same schema.table.

Collision detection (consolidation path only):

For each (catalog, schema, table) triple targeted at the SAME Lakehouse:
  target_key = f"{schema}.{table}"
  If target_key already seen from a DIFFERENT catalog:
    → Mark as COLLISION (Blocker)
    → Record: both source catalog paths map to same Fabric target

Resolution strategies (only when consolidating):

Strategy	When to Use
One schema-enabled Lakehouse per catalog (default)	Clean isolation — schemas and table names preserved, no collision
Prefix schema with catalog name	Consolidating catalogs with overlapping schema names → `prod_silver.customers`, `dev_silver.customers`
Drop non-production catalogs	Only migrate `prod` catalog; skip `dev`/`staging`

⚠️ Never auto-resolve namespace collisions. When the user elects consolidation, surface collisions with the available strategies and let them decide. Silent resolution risks incorrect data bindings.

Step 0.4: Generate Migration Manifest

Output a structured manifest summarizing the assessment:

migration_manifest:
  workspace: <databricks_workspace_url>
  target_fabric_workspace: <fabric_workspace_id>
  strategy: lift-and-shift | migrate-and-modernize
  
  items:
    - path: /Users/alice/ETL/bronze_ingest
      type: notebook
      language: python
      severity: Info
      transforms: [dbutils_fs, namespace_map]
      target: notebooks/bronze_ingest
      
    - path: /Users/alice/ETL/silver_transform  
      type: notebook
      language: python
      severity: Warning
      transforms: [dbutils_fs, dbutils_widgets, namespace_map]
      blockers: []
      warnings: [widgets_require_parameters_cell]
      target: notebooks/silver_transform

  collisions:
    - target_key: silver.customers
      sources: [prod.silver.customers, staging.silver.customers]
      resolution: PENDING_USER_INPUT

  blockers_summary:
    total: 2
    items: [/Shared/DLT/pipeline_config, /Shared/cluster-init/install-drivers.sh]
    
  stats:
    total_items: 47
    auto_migratable: 38
    warnings: 7
    blockers: 2

Phase 1: Environment Setup

Step 1.1: Create Fabric Lakehouses

Create one schema-enabled Lakehouse per Databricks catalog (default), or per the user's consolidation choice. Use the dedicated Lakehouse endpoint with enableSchemas: true — the generic /items endpoint creates a schema-less Lakehouse that cannot preserve Unity Catalog schemas:

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/lakehouses
{
  "displayName": "{lakehouse_name}",
  "description": "Schema-enabled target Lakehouse for Databricks migration",
  "creationPayload": {
    "enableSchemas": true
  }
}

Poll for creation completion via Long-Running Operation (LRO) pattern.

Step 1.2: Create Fabric Environments

Map Databricks cluster libraries to Fabric Environment specifications:

Databricks Source	Fabric Environment Target
`%pip install` lines in notebooks	`Libraries/PublicLibraries/environment.yml` (PyPI / conda dependencies); compute settings live in `Setting/Sparkcompute.yml`
Cluster library list (PyPI)	Fabric Environment PyPI libraries
`init_scripts` with `pip install`	Fabric Environment PyPI libraries
Maven/JAR libraries	✅ Supported — upload custom `.jar` files or add Maven coordinates (POM) to the Fabric Environment; validate runtime compatibility
Custom wheel files	Upload to Environment as custom library

Only cluster init scripts that install OS-level/system packages remain a hard blocker (Fabric Environments support PyPI, conda, custom .jar/wheel, and Maven — not arbitrary system installs).

Step 1.3: Create OneLake Shortcuts

For external data currently accessed via DBFS mounts or external locations:

Databricks Pattern	Fabric Shortcut Target
`dbutils.fs.mount("wasbs://[email protected]/")`	OneLake Shortcut → ADLS Gen2
External location pointing to S3	OneLake Shortcut → Amazon S3
External location pointing to ADLS	OneLake Shortcut → ADLS Gen2

Phase 2: Notebook Migration

Step 2.1: Multi-Notebook Protocol

For workspaces with >3 notebooks or individual notebooks >5KB:

Enumerate — List all notebooks with path, language, and approximate size. Do not read full bodies yet.
Process one-at-a-time — For each notebook: a. Export full source from Databricks (databricks workspace export <path> --format SOURCE --file <local-path> — without --file the source is written to stdout) b. Apply code transforms (see severity matrix above) c. Record structured summary:
- notebook_path (relative)
- detected_patterns[] (pattern IDs)
- severity (max severity across all patterns)
- transforms_applied[]
- unresolved_blockers[] d. Deploy to Fabric workspace
Synthesize — After all notebooks processed, produce unified migration report from summaries.

Step 2.2: Code Transformation Pipeline

Apply transforms in this order (order matters for correctness):

Remove SparkSession.builder.getOrCreate() — spark is pre-instantiated
Replace dbutils.* calls → notebookutils.* (see dbutils-to-notebookutils.md)
Map 3-level namespace → schema-enabled Lakehouse schema.table (default one Lakehouse per catalog; apply prefixing only under user-elected consolidation)
Replace DBFS/mount paths with OneLake-relative paths (Files/... or Tables/...)
Replace dbutils.widgets with parameters cells + notebookutils.notebook.run(name, arguments={})
Remove mlflow.set_tracking_uri("databricks") lines
Remove/adjust proprietary Spark configs (spark.databricks.*)
Flag remaining blockers (DLT, custom JVM/JDBC data sources, OS-level init scripts)

Step 2.3: Deploy Notebooks to Fabric

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items
{
  "displayName": "{notebook_name}",
  "type": "Notebook",
  "definition": {
    "format": "ipynb",
    "parts": [
      {
        "path": "notebook-content.ipynb",
        "payload": "<base64-encoded-notebook-content>",
        "payloadType": "InlineBase64"
      }
    ]
  }
}

Phase 3: Job & Orchestration Migration

Single-Task Jobs → Spark Job Definitions

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items
{
  "displayName": "{job_name}",
  "type": "SparkJobDefinition"
}

Note: The POST /items call above creates an empty SJD shell. It has no executable definition until you follow up with POST .../items/{sjdId}/updateDefinition, supplying a base64-encoded SparkJobDefinitionV1.json part that references the migrated notebook/main file and default Lakehouse. Alternatively, include the definition inline on the initial create (as shown for notebooks in Step 2.3).

Then update the definition to reference the migrated notebook and default Lakehouse.

Multi-Task Jobs → Fabric Data Pipelines

Map the Databricks job DAG to Pipeline activities:

Databricks Task Type	Fabric Pipeline Activity
`notebook_task`	Notebook activity
`spark_python_task`	Notebook activity (wrap .py as notebook)
`sql_task`	Stored Procedure activity or Script activity
`dbt_task`	⚠️ No direct equivalent — use Notebook activity with dbt-fabric
`pipeline_task` (DLT)	Blocker — rewrite as chained Notebook activities

Schedule Migration

Databricks Schedule	Fabric Equivalent
Cron trigger (`quartz_cron_expression`)	Pipeline schedule trigger (recurrence)
File arrival trigger	Pipeline event-based trigger (storage events)
Continuous trigger	Pipeline tumbling window or scheduled recurrence

Phase 4: Validation & Cutover

See validation-testing.md for full validation workflow.

Quick Validation Checklist

Check	Method	Gate
Environments published	`GET /v1/workspaces/{id}/environments/{envId}` → status = "Success"	Phase 1 gate
Notebooks deploy without error	LRO completes with 200	Phase 2 gate
Critical notebooks execute	Run via Fabric REST API → check cell outputs	Phase 3 gate
Data row counts match	Compare Databricks `spark.table(...).count()` vs Fabric equivalent	Cutover gate
No unresolved blockers	All Blocker items either resolved or explicitly excluded	Cutover gate

Failure Reporting

When the migration cannot complete (Blockers with no workaround, permissions failures, or repeated errors), generate a structured failure report:

failure_report:
  workspace: <source_url>
  target: <fabric_workspace>
  phase_reached: <0|1|2|3|4>
  
  blockers:
    - item: /Shared/DLT/pipeline_config
      pattern: delta_live_tables
      severity: Blocker
      reason: "No DLT equivalent in Fabric"
      recommendation: "Rewrite as parameterized notebooks + Pipeline orchestration"
      
    - item: /Shared/cluster-init/install-drivers.sh
      pattern: init_script_system_package
      severity: Blocker
      reason: "Init script installs OS-level packages; Fabric Environments support only PyPI/conda, custom .jar/wheel, and Maven"
      recommendation: "Use a pure-Python alternative, or move the workload to a containerized/Azure ML approach"
  
  partial_success:
    migrated: 38
    skipped: 2
    failed: 0
    
  next_steps:
    - "Resolve 2 blockers manually (DLT rewrite + init-script dependency)"
    - "Exclude the two blocked items from the approved migration batch, then rerun the remaining phases"

Stopping Conditions

Do not attempt workarounds — surface to user and stop:

Permission failures on Databricks workspace or Fabric workspace
Blocker patterns with no known resolution (DLT rewrite required, custom Spark data sources requiring JVM access, OS-level init-script dependencies)
Namespace collisions without user-provided resolution strategy
Repeated API failures (5+ retries with no progress)
Items that exceed Fabric REST API size limits (10MB notebook upload limit)

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python, Needs Node.js

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills-for-fabric · skills/databricks-migration · commit f8fac11

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

Also distributed in

microsoft/skills-for-fabric

AutoGineer is not affiliated with or endorsed by Microsoft or Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download databricks-migration.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Microsoft

Databricks → Microsoft Fabric Migration

Prerequisite Knowledge

Table of Contents

Context Loading Guide

Migration Workload Map

Severity Definitions

dbutils → notebookutils Quick Reference

Widgets Migration

Cluster Config → Fabric Spark Pools

Databricks Jobs → Spark Job Definitions

Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts

MLflow → Fabric ML Experiments

Must / Prefer / Avoid

MUST DO

PREFER

AVOID

Multi-Notebook Migration Protocol

Failure Reporting

Examples

Files

resources/migration-orchestrator.md

Migration Orchestrator — Databricks → Fabric

Required Inputs

Derived Values (Auto-Discovered)

Orchestration Flow

Phase 0: Inventory & Assessment

Step 0.1: Enumerate All Workspace Items

Step 0.2: Classify Items by Migration Severity

Blocker Patterns

Warning Patterns

Info Patterns (Auto-Migratable)

Step 0.3: Namespace Mapping

Step 0.4: Generate Migration Manifest

Phase 1: Environment Setup

Step 1.1: Create Fabric Lakehouses

Step 1.2: Create Fabric Environments

Step 1.3: Create OneLake Shortcuts

Phase 2: Notebook Migration

Step 2.1: Multi-Notebook Protocol

Step 2.2: Code Transformation Pipeline

Step 2.3: Deploy Notebooks to Fabric

Phase 3: Job & Orchestration Migration

Single-Task Jobs → Spark Job Definitions

Multi-Task Jobs → Fabric Data Pipelines

Schedule Migration

Phase 4: Validation & Cutover

Quick Validation Checklist

Failure Reporting

Stopping Conditions

Requirements and compatibility

Security

Attribution and licence

Related skills

Modal

Cloud Infrastructure Network Engineer

Deployment Validation Cloud Architect

Airflow Dag Patterns

Cost Optimization

Hybrid Cloud Networking

Download ZIP

`dbutils` → `notebookutils` Quick Reference