Spark CLI

<!-- Mode reference for the `spark-cli` skill. Loaded on demand from `skills/spark-cli/SKILL.md` when the request matches the `operations` mode. -->

Spark Operations — CLI Skill

This skill provides diagnostics for Microsoft Fabric Spark job failures, Livy session health, and performance bottlenecks using Fabric REST APIs and CLI tools (az rest). All diagnostic operations are read-only; session cleanup (e.g., stopping zombie sessions) requires explicit user confirmation. For Spark development and notebook authoring, use the authoring mode. For interactive PySpark analysis, use the consumption mode.

Table of Contents

The TOC is grouped by purpose. Start at Diagnostic Workflows when triaging an active failure; the earlier sections are foundational references.

1. Fabric Foundations (concepts)

2. CLI Setup & Authentication

3. Spark Sessions, Notebooks & Jobs (background)

4. Spark Monitoring APIs (primary triage surface)

5. Diagnostic Workflows (start here for active triage)

6. Job Failure Diagnostics

7. Livy Session Health

8. Performance Diagnostics

9. Offline / Deep-Dive Tools

Must/Prefer/Avoid

MUST DO

• Always retrieve job/session status before attempting remediation

• Stop at the first specific root cause. A non-empty, actionable job-instance failureReason is sufficient for a high-level diagnosis; use Spark Advisor only when it is absent or generic, and raw logs only when both remain inconclusive.

• For run-history pattern requests, compare job instances, states, durations, and failure reasons. Do not fetch or decode notebook definitions unless the user asks for source-level inspection.

• Request each documented log surface once. Do not probe alternate URL encodings or attempt gzip, deflate, ZIP, or other speculative decoding; if a log is unavailable, report that and continue with the next documented evidence source.

• Use workspace and item discovery from COMMON-CLI.md — never hardcode IDs

• Check Livy session state before submitting diagnostic statements

• Follow the Failure Triage Workflow for systematic diagnosis

• Always check the Spark Advisor API before reading raw logs — it often identifies the root cause immediately

• Use monitoring APIs (no active session required) before attempting Livy-based diagnostics

• Poll job/session status with 10–30 second intervals; timeout diagnostics after 30 minutes

• Always include the Notebook Snapshot URL in diagnostic output — it has the longest retention and enables cell-level inspection in the Fabric UI

PREFER

• Querying job instance history to establish baseline before declaring a regression

• Reusing existing idle sessions for diagnostic queries instead of creating new ones

• Checking capacity utilization when jobs are slow before blaming the Spark code

• Using az rest with JMESPath filtering to extract specific fields from large API responses

• The Spark Advisor API over manual log parsing for skew, task errors, and timeout detection

• Resource Usage API coreEfficiency metric to quantify cluster utilization before recommending scaling

• Job instance history comparison (last 5 runs) to detect regressions before deep-diving

• For MLV refresh scheduling, monitoring, or run-history, use [the mlv mode](../the mlv mode/SKILL.md). For diagnosing the underlying Spark job failure (OOM, skew, shuffle spill), continue with this skill — MLV refreshes execute as Spark jobs and their logs are accessible via the same monitoring APIs.

• MLV failure classification — when diagnosing a failed MLV refresh, classify the error before deep-diving:

  Error Pattern	Category	Diagnosis Path
  MLV_SPARK_SESSION_REQUEST_SUBMISSION_FAILED	Infrastructure	Capacity paused/unavailable, Spark pool misconfigured. Check capacity state first.
  MLV_SELECTED_NOT_FOUND	Configuration	MLV table was deleted/renamed. Verify table exists via SHOW MATERIALIZED LAKE VIEWS IN schema.
  OutOfMemoryError / SparkOutOfMemory	Resource	Source data grew beyond cluster capacity. Check Spark Advisor for memory pressure.
  ShuffleBlockFetchFailed / data skew	Performance	Uneven data distribution. Use Resource Usage API to identify skewed partitions.
  DeltaTableVersionNotFound	Dependency	Source table was vacuumed below retention threshold. Extend delta.logRetentionDuration.
  ConstraintViolationException / ON MISMATCH	Data Quality	DQ constraint dropped/failed rows. Check source data quality upstream.
  Timeout (run > 24 hours)	Scale	Lineage too large for single run. Split into smaller lineage groups across lakehouses.

AVOID

• Killing sessions without checking if they have active statements

• Creating new sessions for every diagnostic query (reuse idle sessions)

• Assuming OOM without checking actual memory metrics from Livy

• Hardcoded workspace or item IDs in diagnostic scripts

• Diagnosing performance without first checking capacity throttling via the Admin API

• Submitting diagnostic statements to sessions in busy state

Examples

Example 1: Diagnose a Failed Notebook

User prompt: "Why did my notebook ETL_Daily fail in workspace Production?"

Agent workflow:

1. Resolves workspace → workspaceId, item → itemId (Notebook)

2. Lists recent Livy sessions, auto-picks the Failed session

3. Queries Spark Advisor → finds TaskError: OutOfMemoryError on executor

4. Queries /stages → confirms data skew (12× max/median ratio in stage 5)

5. Presents report with HIGH findings + fix recommendations

Example 2: Triage Stuck Livy Session

User prompt: "My Livy session abc-1234 is stuck in starting state"

Agent workflow:

1. Uses session ID directly, queries session state

2. Lists all workspace sessions → detects 8 concurrent sessions (capacity pressure)

3. Checks Livy log → no errors, just queued

4. Reports: capacity contention, recommends waiting or cancelling idle sessions

Example 3: Pipeline Failure Root Cause

User prompt: "Diagnose pipeline run 5678 in workspace Analytics"

Agent workflow:

1. Resolves pipeline, calls queryActivityRuns for run 5678

2. Finds 2 Notebook activities: one Succeeded, one Failed

3. Extracts output.result.error.{ename, evalue, traceback} from failed activity

4. Constructs Notebook Snapshot URL for cell-level inspection

5. Presents error details + snapshot link + suggested fix

Quick Start

Environment Setup

Apply environment detection from COMMON-CLI.md to set:

• $FABRIC_API_BASE and $FABRIC_RESOURCE_SCOPE

• $FABRIC_API_URL and $LIVY_API_PATH for Livy operations

Authentication: Use token acquisition from COMMON-CLI.md § Authentication Recipes.

Automated Diagnostic Workflow

When the user provides a simple prompt (e.g., "Diagnose my notebook ETL_Pipeline", "What's wrong with Spark application abc-123", "Check workspace Production for issues"), follow this fast-path summary. For full procedure, edge cases (expired data, pipeline-only sessions), report templates, and retention details, see references/automated-diagnostic-workflow.md.

Entry Points (what the user provides)

Item-Type API Paths

All session API paths follow: $FABRIC_API_URL/workspaces/$workspaceId/<itemTypePath>/$itemId/livySessions/$livyId/applications/$appId/<endpoint> — see SPARK-MONITORING-CORE.md.

Steps at a Glance

Key principle: Always check Spark Advisor first — it's pre-computed and identifies most root causes without log parsing. Pipeline runs have the richest error data via queryActivityRuns (ename, evalue, traceback, cell/line) — see pipeline-diagnosis.md.

Data retention warning: Spark Monitoring API data (logs, stages, advisor) typically expires in minutes to hours after session end. Diagnose failures promptly. If APIs return 404, jump to Step 1b in the reference.

Tier 2 escalation: For truncated data, HTTP 408/504, or DAG/SQL plan visualization, suggest the offline Spark History Server workflow.