Dataflows CLI

Source Connectors in Dataflow Gen2 Mashup (M)

The source-side companion to connection-management.md (REST connection lifecycle) and output-destinations.md (destination-side M). Documents which Power Query M source connectors are reachable from the Fabric Dataflow Gen2 mashup engine, their argument shapes, navigation patterns, and runtime errors. All claims confirmed by executeQuery probing — the in-band-error contract (HTTP 200 with {"Error":...} in the PQ Arrow Metadata column) is documented at mashup-preview.md § Error handling.

Connector vs binding vs M function

The function and the binding must agree on kind; REST casing differs from M (REST "SQL" ↔ M Sql.Database) — see connection-management.md § Connection Type Examples and § Step 1.

Function inventory

All functions below have their symbol registered in the Fabric mashup engine (Value.Type(<function>) returns [Type]). Runtime behaviour is split into the next three sections.

Full enumeration at runtime via #shared.

Browser functions and gateway scope

Microsoft Learn documents Web.Page and Web.BrowserContents as available in cloud dataflows through an on-premises data gateway. See Using a gateway with the Web connector.

The direct, gateway-free executeQuery runtime probed for this skill returned the following errors, even for inline literals:

For gateway-free authoring and preview, use Web.Contents plus Html.Table. If browser rendering is required, use a gateway-backed Web connection and validate through the supported cloud-dataflow path rather than direct executeQuery.

Lakehouse navigation

Canonical call:

Lakehouse.Contents()

Lakehouse.Contents(null) and Lakehouse.Contents([HierarchicalNavigation = null, CreateNavigationProperties = false, EnableFolding = false]) return the same navigation root. The options-record form is the form used by output-destinations.md § Destination Type: Lakehouse; not required for reads.

Navigation shape

Root — workspaces visible to the bound connection (columns: workspaceId, workspaceName, Data, plus navigation hints ItemKind, ItemName, IsLeaf):

Inside a workspace — one row per Lakehouse / SQL endpoint, columns: lakehouseId, lakehouseName, description, capacityObjectId, extendedProperties, Data, databaseId, ItemKind, ItemName, IsLeaf.

Inside a lakehouse — mixed contents, not a flat list of tables. Columns: Name, Id, Data, Schema, ItemKind, ItemName, IsLeaf. Rows include ItemKind = "Table" (user tables, Schema = "dbo"), ItemKind = "View" (built-in views in schemas sys and queryinsights), and a single ItemKind = "Folder" row with Name = "Files" for OneLake file access.

End-to-end read

section Section1;

shared MyTable = let
    Source = Lakehouse.Contents(),
    Workspace = Source{[workspaceId = "<workspace-guid>"]}[Data],
    Lakehouse = Workspace{[lakehouseId = "<lakehouse-guid>"]}[Data],
    Table = Lakehouse{[Name = "Products"]}[Data]
in
    Table;

Anti-pattern. …{[lakehouseId=...]}[Data]{[Id="Tables"]}[Data] is wrong — there is no Tables folder entry; tables are listed directly at the lakehouse-[Data] level. The engine returns "The key didn't match any rows in the table." ({[Id="Files"]}[Data] is valid — it's the one folder entry that exists.)

PowerPlatform.Dataflows navigation

Read the output of another Dataflow Gen2 with the PowerPlatform.Dataflows connector. The argument is required — call PowerPlatform.Dataflows(null); the no-arg form PowerPlatform.Dataflows() is a compile error. The result is a nav table you drill by index, the same shape pattern as Lakehouse navigation.

Navigation shape

Pass the workspace id through a Variable Library rather than hardcoding a GUID literal — see Variable.Value.

End-to-end read

section Section1;

shared UpstreamDataflow = let
    Source     = PowerPlatform.Dataflows(null),
    Workspaces = Source{[Id = "Workspaces"]}[Data],
    Workspace  = Workspaces{[workspaceId = Variable.Value("currentWorkspaceId")]}[Data],
    Dataflow   = Workspace{[dataflowName = "eval_authoring_basic"]}[Data]
in
    Dataflow;

Note. The first drill is the literal {[Id = "Workspaces"]}[Data] entry, not a workspaceId index — workspaceId selection happens at the next level down. The (null) argument is mandatory.

Credentialed connectors

Without a matching dataflow binding, each function below returns the in-band error "Credentials are required to connect to the <kind> source. (Source at <source>.)" — the credential error itself confirms parsing and arity validation succeeded.

To enable any of these, bind a matching-kind connection per authoring-cli-quickref.md § Connection Binding Quick Patterns. For multi-source documents add [AllowCombine = true].

Pure-data parsers

These transform in-memory values and require no binding — safe primitives for tests and for parsing payloads pulled by a credentialed connector.

Html.Table(html, columnNameSelectorPairs, [options])

Html.Table(Web.Contents("https://example.com/page"),
    {{"Name", ".product-name"}, {"Price", ".product-price"}},
    [RowSelector = ".product-row"])

• Observed engine behavior: in live Dataflow Gen2 refresh, supplying a RowSelector is needed whenever more than one {columnName, cssSelector} pair is given. Microsoft Learn documents the options record — including RowSelector — as optional and does not state this rule; treat it as behavior this skill verified live. Use RowSelector to group the repeating rows.

• Works on tables defined by any repeating CSS pattern, not just <table>.

Csv.Document(binary, [options])

Csv.Document(Web.Contents("https://example.com/data.csv"),
    [Delimiter = ",", Encoding = 65001, QuoteStyle = QuoteStyle.Csv])

Both QuoteStyle.Csv and QuoteStyle.None accepted; optional Columns = <n> to fix arity.

Json.Document(text-or-binary, [encoding])

Json.Document("{""x"": 42}")
Json.Document(Text.ToBinary("{""x"":42}", 65001), 65001)

Lines.FromBinary, #table

Lines.FromBinary(Web.Contents("https://example.com/log.txt"))
#table(type table [Id = Int64.Type, Name = text], {{1, "A"}, {2, "B"}})

Lines.FromBinary streams a binary as a list of lines; #table is the standard inline-table constructor.

#shared

#shared is the global record of every loaded function and identifier. Enumerate connectors at runtime:

let
    Source = #shared,
    AsTable = Record.ToTable(Source),
    SqlFunctions = Table.SelectRows(AsTable, each Text.StartsWith([Name], "Sql."))
in
    SqlFunctions

Returns columns Name and Value (function/type/table/record).

Variable.Value

Variable.Value("<name>") reads a named variable from the dataflow's Variable Library. If the variable does not exist the engine returns "The variable '<name>' could not be found.".

section Section1;
shared CurrentWorkspaceId = Variable.Value("currentWorkspaceId");

Reading a variable value requires a Variable Library configured on the dataflow.

[AllowCombine = true]

For documents that combine more than one credentialed source, prepend the section attribute on the first non-comment line:

[AllowCombine = true]
section Section1;

shared Q = ...;

Without it the privacy firewall blocks evaluation when two or more sources are referenced — see connection-management.md § Operational Pitfalls.

MUST / PREFER / AVOID

MUST

1. Match the M function kind to the binding kind. REST casing differs (REST "SQL" ↔ M Sql.Database) — see connection-management.md § Step 1.

2. Use Html.Table for gateway-free HTML parsing. Web.Page and Web.BrowserContents require an on-premises gateway in cloud dataflows and are disabled in the direct executeQuery runtime tested here.

3. For combined-source documents, add [AllowCombine = true] before section Section1;.

4. Decode the Arrow stream and inspect PQ Arrow Metadata when reading executeQuery results — 200 OK is not success.

PREFER

1. Index by GUID (workspaceId, lakehouseId) over display name for Lakehouse navigation.

2. Pure parsers (Csv.Document, Json.Document, Html.Table) for in-memory data — no binding required.

3. The verified Lakehouse navigation shape (workspaceId → lakehouseId → Name) over folder-style {[Id="Tables"]}.

AVOID

1. Web.BrowserContents in direct executeQuery - use a gateway-backed Web connection when browser rendering is required.

2. Web.Page in direct executeQuery - use a gateway-backed Web connection when the legacy browser function is required.

3. Sql.Database("server") (1-arg form) — engine rejects with arity error. Always supply the database name.

4. {[Id="Tables"]} at the lakehouse-[Data] level — there is no Tables folder; tables are listed directly. ({[Id="Files"]} IS valid — it's the only folder entry.)

See also

• connection-management.md — REST-side connection lifecycle, supportedConnectionTypes, kind casing, binding workflow.

• mashup-preview.md — executeQuery API contract, request body, Arrow stream decoding, embedded errors.

• output-destinations.md — destination-side M (Lakehouse Table, Warehouse, ADX, Azure SQL writes).

• authoring-cli-quickref.md § Connection Binding Quick Patterns — binding a connection to a dataflow definition.

• authoring-script-templates.md — end-to-end bash + PowerShell smoke; includes OData.Feed example.

• dataflows-cli consumption mode § Query Evaluation — Arrow → CSV / pandas decoder template.

• common/DATAFLOWS-AUTHORING-CORE.md — Common Connection Kinds table; 3-part definition structure.