executing-spark
Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Executing Spark Code in Fabric (No Notebook) Run arbitrary PySpark or Python code on Fabric Spark compute via the Livy API. No notebook artifact is created or persisted; sessions are ephemeral. Full read/write access to lakehouse Delta tables via Spark SQL. ## Prerequisites - Azure CLI authenticated (`az login`) - A lakehouse in the target workspace (the Livy session runs against it) - Fabric capacity (F or trial) ## Critical: Authentication The Livy API requires a token from `az account get-access-token --resource https://api.fabric.microsoft.com`. Tokens from `fab auth` do **not** work for OneLake storage access inside the Spark session. ```python import subprocess, json result = subprocess.run( ["az", "account", "get-access-token", "--resource", "https://api.fabric.microsoft.com"], capture_output=True, text=True ) token = json.loads(result.stdout)["accessToken"] ``` Do not output or log the token. Pass it directly to the API call. ## Lifecycle ``` 1. Create session POST .../sessions {"kind": "pyspark"} 2. Wait for idle GET .../sessions/{id} poll until state: "idle" (~30-90s) 3. Submit code POST .../sessions/{id}/statements {"code": "...", "kind": "pyspark"} 4. Get result GET .
- Prerequisites
- Critical: Authentication
- Lifecycle
- Getting IDs
- Submitting Code
- What Works
- What Does Not Work
- When to Use This vs Alternatives
- Persisting code as a notebook: poll the definition LRO tightly
- Sessions vs Batch Jobs
- Livy vs Notebook Jobs: reading the outcome
- References
- Related
What does the executing-spark skill do?
Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".
How do I install it?
Run `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From data-goblin/power-bi-agentic-development, a repository with 833 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.