Synapse Migration
MicrosoftOfficialPublished by Microsoft
- Skill
- Workflow
- Software development
- MIT
- Low risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The synapse-migration skill provides a structured, phased workflow for transitioning Synapse Analytics workspaces to Microsoft Fabric. It covers the migration of Spark items, Lake Databases, Linked Services, and Dedicated SQL Pool schema and stored procedures. The skill emphasizes a dependency-ordered approach, requiring gap assessment and explicit approval before conversion. It distinguishes between migrating to a Fabric Lakehouse (schema and code only, no data movement) and a Fabric Warehouse, while providing specific routing for dependent pipeline callers.
What it does
- Discovers and classifies Synapse workspace artifacts.
- Converts Dedicated SQL Pool schema and stored procedures to Spark SQL notebooks or Warehouse objects.
- Migrates Spark items, including notebooks and Spark Job Definitions, to Fabric equivalents.
- Maps Linked Services to Fabric Data Connections or OneLake Shortcuts.
- Manages the migration of approved dependent pipeline callers.
- Validates persisted definitions and artifacts against established contracts.
When to use it
- When migrating a full Synapse workspace to Microsoft Fabric.
- When converting Dedicated SQL Pool schema and procedures to a Lakehouse or Warehouse.
- When porting Spark pools, notebooks, or Spark Job Definitions to Fabric.
- When replacing Synapse Linked Services with Fabric-native connectivity.
Advantages
- Ensures dependency-ordered migration to maintain system integrity.
- Provides clear routing for different workload types (Lakehouse vs. Warehouse).
- Enforces strict validation and approval gates for behavior-changing decisions.
- Supports incremental migration rather than high-risk big-bang cutovers.
- Standardizes the conversion of T-SQL procedures to executable Spark SQL notebooks.
Usage examples
- “Migrate this Dedicated SQL Pool DACPAC to a Fabric Lakehouse and convert the procedures.”
- The agent will assess gaps, obtain approval for procedure-to-notebook mappings, create a migration manifest, convert schema/code artifacts, and validate the results.
- “Migrate this Dedicated SQL Pool to Fabric Warehouse.”
- The agent will assess DDL compatibility, remove unsupported distribution/index declarations, obtain provisioning consent, and prepare for potential data migration.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/synapse-migration/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip synapse-migration.zip -d .agents/skills
Documentation by Microsoft
Telemetry — MANDATORY. Every
api.fabric.microsoft.comcall must carryx-ms-fabric-skill: synapse-migration(az rest:--headers "x-ms-fabric-skill=synapse-migration"), including every LRO poll,fabric_lroand retry. Snippets omit it — add it anyway.
OFFLINE LAKEHOUSE ARTIFACT FAST PATH — DECIDE FIRST. For a no-live-call request with complete supplied inputs and contracts, load only the fast path; do not load
migration-planning-reference.md,dedicated-pool-conversion.md, or implementation scripts. Use one generator for artifacts and final-byte hashes, then one verifier. Do not inspect generated files or run post-pass spot checks. For every optional non-null source default, setmissingValueBehaviortoUseSourceDefaultand restore that default before validation. Do not use this path for a Warehouse target; incomplete and large-procedure requests use their own routes.
CRITICAL NOTES
To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering.
To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering.
Treat a source Dedicated SQL Pool as read-only. Do not run source DDL, DML, stored procedures, or create sample objects.
Dedicated Pool-to-Lakehouse data movement is out of scope. Migrate schema and executable code artifacts only; do not copy source rows or run row-level/business-result equivalence queries. The separate Warehouse route may move data only from a separately approved restored export copy under its own consent gate and restored-source contract.
For Dedicated Pool-to-Lakehouse requests, complete gap assessment, explicit mapping approval, manifest creation, conversion, deployment, and validation in order. Read dedicated-pool-to-lakehouse.md once as the router, then load only the current phase reference.
Stored-procedure
1:1,N:1, andN:Nmappings never apply to views. Views remain schema objects; procedures become traceable Spark SQL notebooks with executable%%sqltransformation cells.Dependent ADF/Synapse callers are conditional scope, not automatic scope. Discover them during assessment, require a separate complete-closure approval, and migrate them only after notebook readiness. Route standalone pipeline requests to
pipeline-migration.Before executing a specialized Dedicated Pool workflow, read the matching contract in dedicated-pool-execution-contracts.md. Its validation, hashing, request-body, resume, and completion-language requirements are mandatory.
Keep Lakehouse and Warehouse paths separate. If the target is unclear, ask before loading path-specific resources. Lakehouse conversion never moves source rows; Warehouse data migration requires its own explicit consent gate and restored-source contract.
For a Fabric Warehouse target, explicitly state that the complete
DISTRIBUTIONandCLUSTERED COLUMNSTORE INDEXdeclarations are removed because Fabric manages distribution, storage, and indexing; a bare mention of either source clause is insufficient.Run setup DDL and CETAS only against the separately approved restored export copy after proving it is not the original production source.
Synapse Analytics to Microsoft Fabric Migration
Core Workflow
Discover the Synapse workspace and classify every source artifact.
Choose targets using workload-routing.md.
Assess gaps before conversion, including runtime, library, connectivity, T-SQL, security, and capacity constraints.
Obtain approval for behavior-changing decisions, target placement, and every stored-procedure-to-notebook mapping.
Create or update
migration-manifest.jsonand persist approval, per-object status, hashes, and checkpoints atomically.Convert and deploy in dependency order, loading only the current phase resource.
Validate persisted definitions and artifacts without executing generated notebooks or pipelines unless a separately approved workflow explicitly requires execution.
Report completed, blocked, skipped, and pending objects without exposing credentials or sensitive values.
For full-workspace orchestration, read migration-orchestrator.md. For a Dedicated Pool-to-Lakehouse migration, use dedicated-pool-to-lakehouse.md as the phase router. For a Warehouse target, load only the matching dw-* resource.
Scope Routing
| Request | Action | Load |
|---|---|---|
| Full Synapse workspace migration | Run phased discovery and migration | migration-orchestrator.md |
| Dedicated SQL Pool schema/code to Lakehouse | Run the strict assessment-to-validation workflow. For guidance-only notebook publication, explicitly state: "The stored procedures were converted from T-SQL to Spark SQL before publication." | dedicated-pool-to-lakehouse.md |
| Guidance-only Dedicated Pool incremental deployment hardening | Answer directly from the focused checklist; do not search scripts or load phase resources. State: "For columns with special characters, enable Delta columnMapping selectively and only with approval." | dedicated-pool-execution-contracts.md |
| Large or complex stored procedure | For a complete offline contract, read only the audit resource once, use one run-local generator and one verifier, do not inspect implementation scripts or generated outputs, and stop after successful verification | dedicated-pool-large-procedure-audit.md |
| Spark pool | Convert configuration and libraries to a Fabric Environment | spark-pool-migration.md |
| Lake Database or HMS | Create or bind the target Lakehouse | lake-database-migration.md or external-hms-migration.md |
| Notebook or Spark Job Definition | Port code, bindings, parameters, and runtime settings | spark-item-migration.md |
| Linked Service | Choose a Fabric Data Connection for external services or a OneLake Shortcut for ADLS Gen2/Blob | connectivity-migration.md |
| Approved dependent procedure callers | Run the internal closure-preserving sub-flow after notebook readback | dedicated-pool-dependent-pipelines.md |
| Standalone pipeline migration | Delegate | pipeline-migration |
| Dedicated SQL Pool to Fabric Warehouse | Run the source extraction, DDL compatibility, approval, optional data, security, and validation route | dw-source-and-extraction.md, then the matching dw-* phase resource |
| Standalone Fabric Warehouse administration | Delegate | sqldw-cli |
| Synapse Link | Use Fabric Mirroring; not covered here | — |
Always report the Lake Database/HMS phase. If none is discovered, mark it NotApplicable (none discovered) while retaining the required target Lakehouse binding.
Load References On Demand
Do not preload all files. Load only the references needed for the current phase:
| Need | Reference |
|---|---|
| Routing, phase order, API audiences, quick examples, troubleshooting, or post-migration handoff | workload-routing.md |
| Dedicated Pool execution variants and response contracts | dedicated-pool-execution-contracts.md |
| Dedicated Pool discovery | dedicated-pool-discovery.md |
| Gap assessment and mapping approval | dedicated-pool-gap-assessment.md |
| Procedure conversion and notebook shape | dedicated-pool-conversion.md |
| Schema/notebook deployment | dedicated-pool-deployment.md |
| Dedicated Pool validation | dedicated-pool-validation.md |
| Runtime/library compatibility | feature-parity.md and library-compatibility.md |
mssparkutils and connector refactoring | utility-api-mapping.md, connector-refactoring.md, and code-patterns.md |
| Capacity planning | capacity-sizing.md |
| Dedicated Pool-to-Warehouse source discovery and extraction | dw-source-and-extraction.md |
| Warehouse DDL compatibility and deployment | dw-ddl-compatibility.md |
| Separately approved Warehouse data migration | dw-data-migration.md |
| Warehouse security and validation | dw-security-validation.md |
| Validation, reporting, and production readiness | validation-testing.md, migration-report.md, and security-governance.md |
Use COMMON-CLI.md only for shared authentication and az rest mechanics not already covered by the phase resource. Use COMMON-CORE.md only for shared Fabric REST patterns.
Must
Preserve every stored-procedure input contract, source default, source decision, and approved target mapping.
Block conversion until gap dispositions, notebook names, dependency grouping, and target workspace placement are explicitly approved.
Use SqlPackage/DMVs for source discovery, Fabric REST for item management, and Fabric Livy statements for Lakehouse schema and Delta execution.
Run the maintained validators required by the selected execution contract; never replace them with ad hoc JSON or cell-shape checks.
Choose the Lakehouse or Warehouse target before loading implementation resources; never apply Warehouse data-movement rules to Lakehouse artifact conversion.
Before persisting any dependent-pipeline source definition, recursively inspect it for secret-bearing or uncertain properties; block the complete selected closure without writing the artifact or mutating a target when any are found.
In every dependent-pipeline response, including guidance-only and pre-approval explanations, state: "Use a
TridentNotebookactivity with exact target identitiestypeProperties.notebookIdandtypeProperties.workspaceId, and pass inputs throughtypeProperties.notebookParameters." Then state: "After deployment, read back each pipeline withPOST /v1/workspaces/{workspaceId}/items/{dataPipelineId}/getDefinitionand validate its canonical hash and structural bindings."For every Linked Service response, first state both replacement classes: Fabric Data Connections for external databases/services and OneLake Shortcuts for ADLS Gen2/Blob storage; then identify which class applies.
Replace
mssparkutilswithnotebookutils, Linked Services with the correct Fabric connection class, andspark.read.synapsesql()with a supported OneLake or JDBC pattern.Externalize workspace and item IDs through parameters or Variable Libraries.
Keep phase status concise, emit bounded heartbeats for long operations, and persist per-object evidence in the manifest.
Prefer
OneLake Shortcuts over copying data.
Fabric Environments for reproducible libraries and Spark settings.
Incremental, dependency-ordered migration over big-bang cutover.
Parameterized notebooks and Variable Libraries for environment promotion.
Lakehouse SQL Endpoint for compatible Serverless SQL Pool read workloads.
Avoid
Do not execute generated notebooks or pipelines as part of artifact migration.
Do not broaden a selected dependent-pipeline closure or route that internal sub-flow to
pipeline-migration.Do not use
sqlcmdagainst the target Fabric Warehouse; use the registered SQL endpoint tooling required by the Warehouse resource.Do not preserve Synapse distribution/storage hints, PolyBase DDL, Linked Service connection strings, or pool-level library installation patterns verbatim.
Do not use target notebooks to orchestrate the migration itself.
Do not hardcode workspace IDs, item IDs, credentials, connection strings, or environment-specific paths.
Examples
Prompt: "Migrate this Dedicated SQL Pool DACPAC to a Fabric Lakehouse and convert the procedures."
Response behavior: Assess gaps first, present and obtain approval for the complete procedure-to-notebook mapping and workspace placement, create the manifest, convert approved schema/code artifacts without moving source rows, validate persisted definitions, and report the resolved Lakehouse and artifact evidence.
Prompt: "Migrate this ADF pipeline to Fabric."
Response behavior: Route to pipeline-migration. Keep it here only when the pipeline is an explicitly approved dependent caller discovered inside an active Dedicated Pool-to-Lakehouse migration.
Prompt: "Migrate this Dedicated SQL Pool to Fabric Warehouse."
Response behavior: Keep this inside synapse-migration, assess DDL compatibility, remove unsupported distribution and columnstore declarations, obtain provisioning consent, and ask separately before any restored-source data migration.
For code transformations and edge cases, load workload-routing.md and the phase-specific resource rather than expanding the main instructions.
Files
- ATTRIBUTION.mdAdded by AutoGineer879 B
- LICENSELicence file added by AutoGineer1.1 KiB
- SKILL.md14.3 KiB
- apm.yml597 B
- references/dedicated-pool-execution-contracts.md14.5 KiB
- references/workload-routing.md6.0 KiB
- resources/capacity-sizing.md4.1 KiB
- resources/code-patterns.md8.4 KiB
- resources/connectivity-migration.md6.4 KiB
- resources/connector-refactoring.md13.8 KiB
- resources/dedicated-pool-conversion.md39.2 KiB
- resources/dedicated-pool-dependent-pipelines.md23.3 KiB
- resources/dedicated-pool-deployment.md20.0 KiB
- resources/dedicated-pool-discovery.md17.8 KiB
- resources/dedicated-pool-gap-assessment.md17.4 KiB
- resources/dedicated-pool-large-procedure-audit.md24.5 KiB
- resources/dedicated-pool-to-lakehouse.md17.9 KiB
- resources/dedicated-pool-validation.md17.2 KiB
- resources/dw-data-migration.md13.7 KiB
- resources/dw-ddl-compatibility.md9.4 KiB
- resources/dw-security-validation.md15.6 KiB
- resources/dw-source-and-extraction.md8.0 KiB
- resources/external-hms-migration.md20.2 KiB
- resources/feature-parity.md5.4 KiB
- resources/lake-database-migration.md34.8 KiB
- resources/library-compatibility.md7.2 KiB
- resources/migration-gotchas.md16.1 KiB
- resources/migration-orchestrator.md63.0 KiB
- resources/migration-planning-reference.md16.3 KiB
- resources/migration-report.md16.9 KiB
- resources/security-governance.md47.4 KiB
- resources/spark-item-migration.md19.2 KiB
- resources/spark-pool-migration.md16.3 KiB
- resources/utility-api-mapping.md5.4 KiB
- resources/validation-testing.md14.5 KiB
- scripts/Invoke-DedicatedPoolTool.ps1Script941 B
- scripts/dedicated-pool-tool/DedicatedPoolMessageSanitizer.csScript2.1 KiB
- scripts/dedicated-pool-tool/DedicatedPoolTool.csproj346 B
- scripts/dedicated-pool-tool/Program.csScript31.4 KiB
- scripts/dedicated_pool_runtime.pyScript116.9 KiB
- scripts/validate_dedicated_pool_spark.pyScript24.8 KiB
resources/lake-database-migration.md
Synapse Lake Database → Fabric Lakehouse Migration
Migrate Synapse Lake Databases and Hive Metastore metadata to Fabric Lakehouses via REST APIs.
Hive Metastore Coverage: Synapse's built-in Hive Metastore (HMS) and Lake Databases share the same underlying catalog. Databases, tables, views, and partitions created via Lake Database designer, Spark SQL (
CREATE DATABASE,CREATE TABLE), or notebook code (df.write.saveAsTable(...)) are all stored in the built-in HMS and are all visible through the Lake Database REST API used in this guide. This means HMS migration for managed/built-in metastores is fully covered here — no separate HMS export/import notebooks are needed.External Hive Metastore (Azure SQL DB / MySQL-backed): If your Synapse workspace uses an external HMS, see external-hms-migration.md for the complete migration guide using JDBC queries. Fabric does not support connecting to an external HMS — all metadata must be migrated into the Fabric Lakehouse catalog. External HMS support in Synapse is deprecated after Spark 3.4.
Prerequisite: Authenticate to both Synapse and Fabric APIs before starting (see COMMON-CLI.md § Authentication Recipes).
Synapse data-plane audience:
https://dev.azuresynapse.netFabric audience:
https://api.fabric.microsoft.com
Phase 1 Overview
Lake Databases must be migrated before Notebooks and Spark Job Definitions so that Fabric Lakehouses exist for notebook lakehouse binding (Phase 2, Step 4).
Phase 1: Lake Database Migration
├── Step 1: Inventory — list databases & tables from Synapse
├── Step 2: Choose mapping mode — schemas vs. separate Lakehouses
├── Step 3: Create Fabric Lakehouse(es)
├── Step 4: Create schemas (if using schema mapping mode)
├── Step 4b: Discover or create ADLS connection (probe-test candidates)
├── Step 5: Create OneLake shortcuts (Delta → Tables/, non-Delta → Files/)
├── Step 6: (Optional) Convert non-Delta tables to Delta
├── Step 7: Validate — verify catalog registration and data accessibility
└── Step 8: (Optional) Validate before proceeding to Phase 2Step 1: Inventory Synapse Lake Databases
List All Databases
Endpoint: GET {endpoint}/databases?api-version=2021-04-01
GET https://{workspaceName}.dev.azuresynapse.net/databases?api-version=2021-04-01Response contains items[] — each item has:
name— database nametype— always"DATABASE"properties.source.location— ADLS Gen2 base path (e.g.,abfss://[email protected]/dbname)properties.source.provider— typically"ADLS"properties.origin.type—"SPARK"(Spark-native) or"SQLOD"(Serverless SQL-originated)
Filter: Only migrate databases where
origin.type == "SPARK"orsource.provider == "ADLS". SkipSQLOD-origin databases — these are Serverless SQL Pool views, not Spark Lake Databases.
Get Database Details
Endpoint: GET {endpoint}/databases/{databaseName}?api-version=2021-04-01
Returns DatabaseEntity with properties.source.location (the warehouse directory path for managed tables).
List Schemas in a Database
Endpoint: GET {endpoint}/databases/{databaseName}/schemas?api-version=2021-04-01
Returns items[] of schema objects. If only default/dbo exists, the database has no custom schemas.
List Tables in a Database
Endpoint: GET {endpoint}/databases/{databaseName}/tables?api-version=2021-04-01
Each table item contains:
name— table nameproperties.tableType—"MANAGED"or absent (external)properties.namespace.databaseName— parent databaseproperties.namespace.schemaName— parent schema (may benullfor default)properties.storageDescriptor.columns[]— column definitions withnameandoriginDataTypeName.typeNameproperties.storageDescriptor.format.formatType—"delta","parquet","csv","orc","avro","textfile", etc.properties.storageDescriptor.source.location— data file path in ADLS Gen2properties.partitioning— partition columns (if Hive-style partitioned)
List Views
Endpoint: GET {endpoint}/databases/{databaseName}/VIEWs?api-version=2021-04-01
Returns view definitions. Extract the SQL for manual recreation in Fabric.
Step 2: Choose Mapping Mode
Ask the user which mapping mode to use. The choice can be made at the workspace level (all databases follow the same pattern) or per database (hybrid mode).
Mode A: Schemas (Default)
All Lake Databases → schemas within one target Lakehouse.
| Synapse | Fabric |
|---|---|
Database sales | Schema sales in target Lakehouse |
Database marketing | Schema marketing in target Lakehouse |
Database default | Schema dbo (Lakehouse default) |
Advantages: Fewer items to manage; cross-schema queries via 2-part names; single SQL endpoint; aligns with Spark Migration Assistant behavior.
Disadvantages: Less isolation; shared OneLake path; harder to assign per-database permissions.
HMS databases (created via Spark SQL
CREATE DATABASE) are ideal candidates for Mode A because they only have adefault/dboschema — no collision risk, simple 1:1 mapping of database name → Fabric schema name.
Mode B: Separate Lakehouses
Each Lake Database → its own Fabric Lakehouse.
| Synapse | Fabric |
|---|---|
Database sales | Lakehouse sales |
Database marketing | Lakehouse marketing |
Advantages: Strong isolation; independent security (OneLake RBAC per Lakehouse); independent SQL endpoints.
Disadvantages: More items to manage; cross-database queries require 3-part names; more shortcuts to create.
Mode C: Hybrid (Per-Database Assignment)
Let the user assign each database individually — some go into a shared Lakehouse as schemas, others get their own dedicated Lakehouse. This is the most flexible option for workspaces with mixed ownership or security requirements.
Example:
| Synapse Database | Target Lakehouse | Maps As | Reason |
|---|---|---|---|
sales | Sales_Lakehouse (dedicated) | Entire Lakehouse | Team-owned, needs own security boundary |
marketing | Marketing_Lakehouse (dedicated) | Entire Lakehouse | Separate team, separate SQL endpoint |
staging_raw | ETL_Lakehouse (shared) | Schema staging_raw | Shared ETL pipeline, same team |
staging_curated | ETL_Lakehouse (shared) | Schema staging_curated | Shared ETL pipeline, same team |
default | ETL_Lakehouse (shared) | Schema dbo | HMS default database, ETL workloads |
When to use Hybrid:
Different databases are owned by different teams and need independent access control
Some databases are tightly related (ETL stages, same domain) and benefit from consolidation
Some databases need dedicated SQL endpoints for separate downstream consumers (Power BI, APIs)
Mix of HMS databases (simple, no schemas) and Lake Database designer databases (may have inner schemas)
User input format — ask the user to provide an assignment map:
{
"databaseAssignments": [
{ "database": "sales", "lakehouse": "Sales_Lakehouse", "mode": "dedicated" },
{ "database": "marketing", "lakehouse": "Marketing_Lakehouse", "mode": "dedicated" },
{ "database": "staging_raw", "lakehouse": "ETL_Lakehouse", "mode": "schema" },
{ "database": "staging_curated", "lakehouse": "ETL_Lakehouse", "mode": "schema" },
{ "database": "default", "lakehouse": "ETL_Lakehouse", "mode": "schema" }
]
}"mode": "dedicated"— create a dedicated Lakehouse for this database; tables go underdboschema (or inner schemas if they exist)"mode": "schema"— map this database as a schema within the shared Lakehouse; the schema name defaults to the database name
Schema collision handling (for databases assigned as
"schema"to the same Lakehouse): Apply the same composite naming rules as Mode A — see below.
Schema Collision Handling (Mode A and Mode C Schema Assignments)
When Synapse databases contain inner schemas beyond default/dbo, a two-level namespace (Database.Schema.Table) must be flattened to one level (Schema.Table).
Naming rules:
| Synapse Source | Databases Have Inner Schemas? | Fabric Schema Name |
|---|---|---|
Database1.dbo.Table1 | No custom schemas in any database | Database1 |
Database1.SchemaA.Table1 | Custom schemas exist | Database1_SchemaA |
Database1.dbo.Table4 | Custom schemas exist | Database1 (drop dbo, use database name only) |
Database2.SchemaA.Table5 | Custom schemas in multiple databases | Database2_SchemaA (no collision) |
Auto-detection logic:
1. List all databases → for each, list schemas
2. If ALL databases have only default/dbo schema:
→ Simple mode: database name = Fabric schema name
3. If ANY database has custom schemas:
→ Composite mode: {database}_{schema} (except dbo → {database})
4. If ONLY ONE database is being migrated with custom schemas:
→ Pass-through mode: inner schema names map 1:1 to Fabric schemas
5. User can always override to Mode B or Mode CNote: In Mode C, collision detection only applies to databases assigned to the same shared Lakehouse. Databases with
"mode": "dedicated"are independent — their inner schemas map directly to Fabric schemas with no collision risk.
Emit a mapping report before creating anything so the user can review name translations:
Mapping Report:
Mode C (Hybrid) — 2 Lakehouses + 1 shared Lakehouse
ETL_Lakehouse (shared):
Synapse staging_raw.dbo.RawOrders → ETL_Lakehouse.staging_raw.RawOrders
Synapse staging_curated.dbo.CleanOrders → ETL_Lakehouse.staging_curated.CleanOrders
Synapse default.dbo.TempData → ETL_Lakehouse.dbo.TempData
Sales_Lakehouse (dedicated):
Synapse sales.dbo.FactSales → Sales_Lakehouse.dbo.FactSales
Synapse sales.dbo.DimCustomer → Sales_Lakehouse.dbo.DimCustomer
Marketing_Lakehouse (dedicated):
Synapse marketing.dbo.Campaigns → Marketing_Lakehouse.dbo.CampaignsStep 3: Create Fabric Lakehouse(es)
Create Lakehouse
Endpoint: POST /v1/workspaces/{workspaceId}/items
{
"displayName": "{lakehouseName}",
"type": "Lakehouse",
"description": "Migrated from Synapse Lake Database",
"creationPayload": {
"enableSchemas": true
}
}Mode A: Create one Lakehouse with
enableSchemas: trueMode B: Create one Lakehouse per database (still use
enableSchemas: trueif database had inner schemas)
Returns HTTP 202 (LRO). Poll
Locationheader untilstatus == "Succeeded". Response includesid(lakehouse ID needed for shortcuts and notebook binding).
Handling name collisions (409):
If a Lakehouse with the same name already exists (HTTP 409), reuse it instead of failing:
resp = requests.post(f"{FABRIC_BASE}/workspaces/{ws_id}/items", headers=fab_headers, json=payload)
if resp.status_code == 409:
# Lakehouse already exists — look it up by name and reuse
items = requests.get(f"{FABRIC_BASE}/workspaces/{ws_id}/items?type=Lakehouse", headers=fab_headers).json()
existing = next((i for i in items.get("value", []) if i["displayName"] == lakehouse_name), None)
if existing:
lakehouse_id = existing["id"]
print(f" Reusing existing Lakehouse: {lakehouse_name} (id={lakehouse_id})")Capture Lakehouse Details
After creation, record these for later phases:
lakehouseId— needed for OneLake shortcuts (Step 5) and notebook binding (Phase 2)workspaceId— needed for notebookmetadata.dependencies.lakehousedisplayName— needed for notebook lakehouse binding
Step 4: Create Schemas (Mode A and Mode C Schema Assignments)
For each Synapse database (and its inner schemas), create the corresponding Fabric schema.
Schemas cannot be created via REST API — they require Spark SQL or the SQL endpoint:
CREATE SCHEMA IF NOT EXISTS Database1;
CREATE SCHEMA IF NOT EXISTS Database1_staging;
CREATE SCHEMA IF NOT EXISTS Database2;
CREATE SCHEMA IF NOT EXISTS Database2_staging;Execute via:
SQL endpoint: Connect to the Lakehouse SQL endpoint and run T-SQL
Fabric notebook: Run in a notebook cell attached to the Lakehouse
Fabric REST API: Execute via Livy session (
POST /v1/workspaces/{workspaceId}/lakehouses/{lakehouseId}/livySessions)
Step 4b: Discover or Create ADLS Connection
Shortcuts require a connectionId — a Fabric Connection object that holds credentials for accessing the ADLS Gen2 storage. The connection's credential (not the caller's Fabric token) is what Fabric uses to read data from storage.
Connection Discovery Strategy
List all connections:
GET /v1/connections(Fabric API)Filter by storage account: Match connections where
connectionDetails.type == "AzureDataLakeStorage"andconnectionDetails.pathcontains the target storage hostnameFilter by container: Parse the container from the connection path. Skip connections locked to a different container than the one containing Synapse data
Score and rank candidates:
| Criteria | Score | Reason |
|---|---|---|
| Root path (no container lock) | +10 | Can access any container |
| Matches target container | +5 | Covers the Synapse data |
| OAuth2 credential | +2 | More likely to have current RBAC |
| WorkspaceIdentity credential | +0 | May lack RBAC, be disabled, or be blocked by policy |
Probe-test each candidate (highest score first): Create a temporary shortcut (
_probe_{tableName}), check the response, then delete the probe. Use the first connection that succeeds.
Probe-Test Logic
# Create a probe shortcut with the candidate connection
probe_payload = {
"name": "_probe_{tableName}",
"path": "Tables",
"target": {"type": "AdlsGen2", "adlsGen2": {
"location": location, "subpath": subpath, "connectionId": candidate_id
}}
}
resp = POST /v1/workspaces/{wsId}/items/{lhId}/shortcuts (probe_payload)
if resp.status_code in (200, 201, 409):
# Connection works — delete probe, use this connection
DELETE /v1/workspaces/{wsId}/items/{lhId}/shortcuts/Tables/_probe_{tableName}
selected_connection_id = candidate_id
elif resp.status_code in (400, 403):
# Credential issue — try next candidate
continueCreating a New Connection
If no existing connection passes the probe, attempt programmatic creation in this order:
Attempt 1: Key-based connection (via ARM listKeys)
Use the ARM token to retrieve the storage account key, then create the connection:
# Step 1: Get storage account key via ARM
arm_url = (
f"https://management.azure.com/subscriptions/{subscription_id}"
f"/resourceGroups/{resource_group}"
f"/providers/Microsoft.Storage/storageAccounts/{storage_account_name}"
f"/listKeys?api-version=2023-05-01"
)
key_resp = requests.post(arm_url, headers=arm_headers)
if key_resp.status_code == 200:
storage_key = key_resp.json()["keys"][0]["value"]
# Step 2: Create connection with Key credential
create_conn = {
"connectivityType": "ShareableCloud",
"displayName": f"{storage_account_name}_{container}_migration",
"connectionDetails": {
"type": "AzureDataLakeStorage",
"creationMethod": "AzureDataLakeStorage",
"parameters": [
{"dataType": "Text", "name": "server", "value": f"https://{storage_account_name}.dfs.core.windows.net"},
{"dataType": "Text", "name": "path", "value": f"/{container}"}
]
},
"privacyLevel": "Organizational",
"credentialDetails": {
"singleSignOnType": "None",
"connectionEncryption": "NotEncrypted",
"skipTestConnection": False,
"credentials": {
"credentialType": "Key",
"key": storage_key
}
}
}
conn_resp = requests.post("https://api.fabric.microsoft.com/v1/connections",
headers=fab_headers, json=create_conn)
if conn_resp.status_code == 201:
selected_connection_id = conn_resp.json()["id"]Requires: The caller must have
Microsoft.Storage/storageAccounts/listKeys/actionon the storage account (typically theStorage Account Key Operator Service RoleorContributorrole).
Attempt 2: WorkspaceIdentity connection
If the Key approach fails (403 on listKeys), fall back to WorkspaceIdentity:
create_conn = {
"connectivityType": "ShareableCloud",
"displayName": f"{storage_account_name}_{container}_migration",
"connectionDetails": {
"type": "AzureDataLakeStorage",
"creationMethod": "AzureDataLakeStorage",
"parameters": [
{"dataType": "Text", "name": "server", "value": f"https://{storage_account_name}.dfs.core.windows.net"},
{"dataType": "Text", "name": "path", "value": f"/{container}"}
]
},
"privacyLevel": "Organizational",
"credentialDetails": {
"singleSignOnType": "None",
"connectionEncryption": "NotEncrypted",
"skipTestConnection": False,
"credentials": {"credentialType": "WorkspaceIdentity"}
}
}Requires: The workspace's managed identity must have
Storage Blob Data ReaderRBAC on the storage account.
Why OAuth2 cannot be created via API
OAuth2 connections require interactive browser consent (authorization code grant flow). The Fabric Connections API explicitly rejects credentialType: "OAuth2" with "CredentialType input is not supported for this API".
Fallback: Manual Portal creation
If both programmatic approaches fail, display the following manual-setup guidance:
No ADLS connection could be created automatically.
Please create one manually:
1. Open Fabric Portal → Settings → Manage connections and gateways
https://app.fabric.microsoft.com/connections
(MSIT: https://msit.powerbi.com/connections)
2. Click '+ New' → Cloud → Azure Data Lake Storage Gen2
3. Server: https://{storageAccount}.dfs.core.windows.net
Path: /{container}
Authentication: OAuth2 (sign in with credentials that have Storage Blob Data Reader)
4. Re-run the migration script — it will discover and probe-test the new connection.
Documentation: https://learn.microsoft.com/fabric/data-engineering/lakehouse-shortcuts#create-a-shortcutCommon Connection Errors
| Error | Cause | Fix |
|---|---|---|
400 "Stored Credential Operation - PowerBIEntityNotFound" | Connection's OAuth token expired or was revoked | Re-authenticate the connection in Fabric Portal, or create a new one |
400 "Stored Credential" (any variant) | Connection credential is invalid, expired, or was rotated | Re-authenticate the connection or create a new one with fresh credentials |
| 403 on shortcut creation | Connection's identity lacks Storage Blob Data Reader on the storage account | Grant RBAC to the connection's identity |
429 Retry-After: {N} | Fabric API rate limit — too many shortcut calls in quick succession | Wait Retry-After seconds, then retry the same request |
Connection reset / ConnectionError | Network-level timeout during shortcut creation (large batch) | Retry with exponential backoff; check network connectivity |
400 "Required property 'connectionId' not found" | Missing connectionId in the shortcut payload | Always include connectionId in the adlsGen2 target |
Step 5: Create OneLake Shortcuts
Shortcut Target Decision
| Table Format | Table Type | Shortcut Location | Result |
|---|---|---|---|
| Delta | Managed | Tables/{schema}/{tableName} | Auto-registers in Lakehouse catalog |
| Delta | External | Tables/{schema}/{tableName} | Auto-registers in Lakehouse catalog |
| Parquet | Managed or External | Files/{schema}/{tableName} | Accessible via Spark; not auto-registered |
| CSV | Managed or External | Files/{schema}/{tableName} | Accessible via Spark; not auto-registered |
| JSON | Managed or External | Files/{schema}/{tableName} | Accessible via Spark; not auto-registered |
| ORC | Managed or External | Files/{schema}/{tableName} | Accessible via Spark; not auto-registered |
| Avro | Managed or External | Files/{schema}/{tableName} | Accessible via Spark; not auto-registered |
Create Shortcut API
Endpoint: POST /v1/workspaces/{workspaceId}/items/{lakehouseId}/shortcuts
Note: The endpoint uses
/items/{lakehouseId}/shortcuts, NOT/lakehouses/{lakehouseId}/shortcuts(which returns 404).
{
"name": "{tableName}",
"path": "Tables/{schemaName}",
"target": {
"type": "AdlsGen2",
"adlsGen2": {
"location": "https://{storageAccount}.dfs.core.windows.net",
"subpath": "/{container}/{path/to/table}",
"connectionId": "{connectionId}"
}
}
}
connectionIdis required. Without it, the API returns 400"Required property 'connectionId' not found". See Step 4b above for how to discover or create the connection.
Shortcut Target Path by Table Type
Managed tables: The data path is under the Synapse workspace warehouse directory:
abfss://{container}@{storage}.dfs.core.windows.net/synapse/workspaces/{workspace}/warehouse/{database}/{tableName}Split this into:
location:https://{storage}.dfs.core.windows.netsubpath:/{container}/synapse/workspaces/{workspace}/warehouse/{database}/{tableName}
External tables: Use the storageDescriptor.source.location path directly:
abfss://{container}@{storage}.dfs.core.windows.net/{custom/path/to/table}Split into:
location:https://{storage}.dfs.core.windows.netsubpath:/{container}/{custom/path/to/table}
Authentication for Shortcuts
Shortcut creation involves two separate authorization checks:
Fabric API authorization (your token): Must have Contributor/Admin role on the Fabric workspace to call the Shortcuts API
Storage data access (connection's credential): The connection's identity must have Storage Blob Data Reader (or higher) on the target ADLS Gen2 storage account/container
The connection credential — not the API caller's token — is what Fabric uses to read data from storage at runtime. See Step 4b for connection discovery.
Shortcut Granularity Strategy
Choose between per-table shortcuts or per-database shortcuts based on the table format:
| Scenario | Strategy | Shortcuts Created | Catalog Registration |
|---|---|---|---|
| All Delta tables | Per-table under Tables/{schema}/ | One per table | Automatic |
| All non-Delta tables | Per-database under Files/{schema}/ | One per database (with tables) | Requires CREATE TABLE USING {format} |
| Mixed formats | Per-table for Delta → Tables/; per-database for non-Delta → Files/ | Hybrid | Automatic for Delta; manual for non-Delta |
Per-database shortcut (non-Delta): Instead of creating 111 individual shortcuts, create one shortcut per database pointing to the warehouse directory. All tables appear as subfolders.
{
"name": "{databaseName}",
"path": "Files/{schemaName}",
"target": {
"type": "AdlsGen2",
"adlsGen2": {
"location": "https://{storageAccount}.dfs.core.windows.net",
"subpath": "/{container}/synapse/workspaces/{workspace}/warehouse/{database}",
"connectionId": "{connectionId}"
}
}
}This creates Files/{schema}/{databaseName}/ containing all table subfolders. Access via:
df = spark.read.parquet("Files/{schema}/{databaseName}/{tableName}")When to use per-database shortcuts: When all tables in the database are non-Delta (parquet, CSV, etc.) and the tables share a common warehouse directory path. This reduces the number of shortcuts from N (tables) to M (databases with tables).
Shortcut Creation Error Cascade
When creating shortcuts in bulk (many tables across many databases), certain errors indicate that all remaining shortcuts will also fail. Abort early to avoid wasting API calls:
| Error | Action | Rationale |
|---|---|---|
| 403 (permission denied) | Abort all remaining shortcuts | The connection lacks Storage Blob Data Reader on the storage account — every subsequent shortcut to the same account will also fail |
| 400 "Stored Credential" | Abort all remaining shortcuts | The connection's credential is invalid — no shortcut using this connection will succeed |
| ConnectionError / reset | Abort all remaining shortcuts | Network-level failure — likely a transient outage affecting all requests |
| 429 (rate limit) | Wait Retry-After seconds, then retry | Transient — the same request will succeed after the cooldown period |
| 409 (conflict) | Continue — treat as success | Shortcut already exists (idempotent re-run) |
| 404 (not found) | Continue — skip this table | The source path doesn't exist; other tables may still be valid |
Implementation pattern:
abort_shortcuts = False
for db_name, tables in db_inventory.items():
if abort_shortcuts:
# Record remaining as skipped
break
for table in tables:
resp = create_shortcut(table, connection_id)
if resp.status_code in (200, 201, 409):
# Success or already exists — continue
pass
elif resp.status_code == 403:
print(f"403 — connection lacks storage access. Aborting remaining shortcuts.")
abort_shortcuts = True
break
elif resp.status_code == 400 and "Stored Credential" in resp.text:
print(f"400 — connection credential invalid. Aborting remaining shortcuts.")
abort_shortcuts = True
break
elif resp.status_code == 429:
retry_after = int(resp.headers.get("Retry-After", 30))
time.sleep(retry_after)
resp = create_shortcut(table, connection_id) # retry once
# else: log and continueWhy abort on 403/400? These are not transient — they indicate a systemic permission or credential issue that affects every shortcut using the same connection. Continuing wastes API quota and produces N identical error messages. Fix the root cause, then re-run.
Step 6: Handle Non-Delta Tables
For non-Delta tables (Parquet, CSV, JSON, ORC, Avro), the shortcut is created under Files/ (Step 5) and data is accessible via Spark. However, these tables are not auto-registered in the Lakehouse catalog. Choose one of the two options below.
Option A: Convert to Delta (Recommended)
Converts the data to Delta format for full catalog registration, SQL endpoint access, and Power BI Direct Lake support.
# Read from the shortcut path
df = spark.read.format("{originalFormat}").load("Files/{schemaName}/{tableName}")
# Write as Delta to the Tables section
df.write.format("delta").mode("overwrite").saveAsTable("{schemaName}.{tableName}")For partitioned tables, preserve partition columns:
df = spark.read.format("parquet").load("Files/{schemaName}/{tableName}")
df.write.format("delta") \
.partitionBy("{partitionCol}") \
.mode("overwrite") \
.saveAsTable("{schemaName}.{tableName}")Note: This creates a physical copy of the data in Delta format. The original shortcut under
Files/remains as a reference.
Advantages: Full Lakehouse catalog registration, SQL endpoint queries, Power BI Direct Lake, V-Order optimization, ACID transactions, time travel.
Option B: Retain Original Format
Keep tables in their legacy format (Parquet, ORC, etc.) under Files/. This avoids data duplication and preserves the original file layout.
Register in the catalog — create an external table definition so Spark SQL and the SQL endpoint can query the data without converting it:
-- Register a non-Delta table pointing to the shortcut path
CREATE TABLE IF NOT EXISTS {schemaName}.{tableName}
USING {format}
LOCATION 'Files/{schemaName}/{tableName}';For example:
-- Parquet table
CREATE TABLE IF NOT EXISTS sales.historical_orders
USING PARQUET
LOCATION 'Files/sales/historical_orders';
-- ORC table
CREATE TABLE IF NOT EXISTS analytics.legacy_events
USING ORC
LOCATION 'Files/analytics/legacy_events';For Hive-style partitioned tables (year=2024/month=01/ directory structure), you must also recover partition metadata after creating the table:
-- Register the partitioned table
CREATE TABLE IF NOT EXISTS {schemaName}.{tableName}
USING PARQUET
PARTITIONED BY ({partitionCols})
LOCATION 'Files/{schemaName}/{tableName}';
-- Recover partitions from directory structure
MSCK REPAIR TABLE {schemaName}.{tableName};For example:
CREATE TABLE IF NOT EXISTS sales.transactions
USING PARQUET
PARTITIONED BY (year INT, month INT)
LOCATION 'Files/sales/transactions';
MSCK REPAIR TABLE sales.transactions;Why
MSCK REPAIR TABLE? Non-Delta tables rely on HMS partition metadata for partition pruning. Unlike Delta (where_delta_logis self-describing), Hive-style partitioned tables need the catalog to know which partitions exist. Without this step, queries read all files instead of pruning to the relevant partitions — causing full scans and poor performance.
After initial registration, if new partitions are added to the source data (e.g., Synapse continues writing year=2025/month=05/), re-run MSCK REPAIR TABLE to pick up the new partitions.
Limitations of retaining original format:
| Capability | Delta (Option A) | Original Format (Option B) |
|---|---|---|
| Lakehouse Explorer UI (Tables section) | Yes | Yes (after CREATE TABLE) |
| SQL endpoint queries | Yes | Yes (after CREATE TABLE) |
| Power BI Direct Lake | Yes | No — requires Delta |
| V-Order optimization | Yes | No |
| ACID transactions / time travel | Yes | No |
| Partition pruning | Automatic | Requires MSCK REPAIR TABLE |
| Data duplication | Yes (new copy) | No (zero-copy via shortcut) |
Decision Guide
Non-Delta table in Synapse:
├── Consumed by Power BI Direct Lake?
│ └── YES → Option A (convert to Delta)
├── Need ACID / time travel / merge operations?
│ └── YES → Option A (convert to Delta)
├── Large table, want to avoid data duplication?
│ └── YES → Option B (retain original format)
├── Read-only / archival data?
│ └── YES → Option B (retain original format)
└── Default recommendation
└── Option A (convert to Delta) — Fabric is Delta-firstStep 7: Validate
After migration, verify:
Lakehouse catalog: Check that Delta tables appear in the Lakehouse Explorer UI under Tables
SQL endpoint: Query migrated tables via the SQL endpoint to confirm schema and data
Row counts: Compare row counts between Synapse and Fabric for each table
Shortcut health: Verify shortcuts are accessible (
notebookutils.fs.ls("Tables/{schema}/"))
Object Type Reference
Full Inventory of Synapse Lake Database Object Types
| Object Type | API Artifact Type | Fabric Support | Migration Action |
|---|---|---|---|
| Delta table (managed) | TABLE (formatType: delta, tableType: MANAGED) | Lakehouse Tables (shortcut) | Shortcut → auto-registers |
| Delta table (external) | TABLE (formatType: delta) | Lakehouse Tables (shortcut) | Shortcut → auto-registers |
| Parquet table (managed) | TABLE (formatType: parquet, tableType: MANAGED) | Lakehouse Files (shortcut) | Shortcut under Files/; Option A (Delta conversion) or Option B (retain + CREATE TABLE + MSCK REPAIR TABLE) |
| Parquet table (external) | TABLE (formatType: parquet) | Lakehouse Files (shortcut) | Shortcut under Files/; Option A or Option B |
| CSV table | TABLE (formatType: csv/textfile) | Lakehouse Files (shortcut) | Shortcut under Files/; Option A (recommended) or Option B |
| JSON table | TABLE (formatType: json) | Lakehouse Files (shortcut) | Shortcut under Files/; Option A (recommended) or Option B |
| ORC table | TABLE (formatType: orc) | Lakehouse Files (shortcut) | Shortcut under Files/; Option A or Option B |
| Avro table | TABLE (formatType: avro) | Lakehouse Files (shortcut) | Shortcut under Files/; Option A or Option B |
| View | VIEW | Not directly migratable | Extract SQL; recreate as Spark SQL CREATE VIEW or SQL endpoint view |
| Schema | SCHEMA | Lakehouse Schema | CREATE SCHEMA IF NOT EXISTS {name} |
| Function (UDF) | FUNCTION | Not migratable via API | Recreate manually via spark.udf.register() or CREATE FUNCTION |
| Partition Info | PARTITIONINFO | Preserved via shortcut | Delta: automatic. Non-Delta: directory structure preserved |
| Relationship | RELATIONSHIP | No Fabric equivalent | Document for reference only |
Decision Tree
For each table in Synapse Lake Database:
├── Is format Delta?
│ ├── YES → Create shortcut under Tables/{schema}/ → auto-registers in catalog ✅
│ └── NO (Parquet/CSV/JSON/ORC/Avro)
│ ├── Option A: Convert to Delta → Shortcut under Files/ → Spark read → write as Delta to Tables/
│ └── Option B: Retain format → Shortcut under Files/ → CREATE TABLE USING {format} → MSCK REPAIR TABLE (if partitioned)
│
├── Is table Managed?
│ ├── YES → Shortcut target = Synapse warehouse directory path
│ │ (ensure Fabric identity has Storage Blob Data Reader on Synapse primary storage)
│ └── NO (External) → Shortcut target = original ADLS Gen2 path
│
└── Is it a View/Function/Relationship?
├── View → Extract SQL, recreate in Fabric
├── Function → Recreate via spark.udf.register()
└── Relationship → Document only (no Fabric equivalent)Step 8: (Optional) Validate Before Proceeding to Phase 2
Run these checks before migrating Notebooks. Notebooks rely on Lakehouses being healthy — missing shortcuts or unregistered tables cause immediate runtime failures.
| Check | How | Pass Criteria |
|---|---|---|
| Shortcut health | notebookutils.fs.ls("Tables/{schema}/") and Files/{schema}/ | All shortcuts resolve; no PathNotFound errors |
| Row counts | Compare SELECT COUNT(*) on Synapse vs. Fabric for each table | Counts match (or are within acceptable tolerance for streaming tables) |
| Schema comparison | Compare column names, types, and order between Synapse and Fabric | Exact match |
| Non-Delta registration | SHOW TABLES IN {schema} | All Option B tables appear in catalog |
| Partition coverage | SHOW PARTITIONS {schema}.{table} for Option B partitioned tables | Partition count matches Synapse |
Do not proceed to Phase 2 until all shortcuts are healthy and row counts match. A notebook that reads from a missing or broken shortcut will fail silently or produce wrong results.
See validation-testing.md → V2: Data Validation for detailed scripts.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs Python
Security
What our scanner noticed· Low risk
- Instructions that use the network
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Microsoft
- Publisher
- Microsoft (Official)
- Source
- microsoft/skills-for-fabric · skills/synapse-migration · commit f8fac11
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- microsoft/skills-for-fabric
AutoGineer is not affiliated with or endorsed by Microsoft or Microsoft.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download synapse-migration.zip