AutoGineer
Generated banner for Firecrawl Build Scrape
Generated logo for Firecrawl Build Scrape

Firecrawl Build Scrape

FirecrawlOfficialCreated by firecrawl

  • Skill
  • Tool integration
  • Automation and integrations
  • ISC
  • No issues found
Download ZIP

From the author

Integrate Firecrawl `/scrape` into product code for single-page extraction. Use when an app already has a URL and needs markdown, HTML, links, screenshots, metadata, or structured page output. Prefer this skill over broader crawl patterns when the feature is page-level.
— the description by firecrawl

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The firecrawl-build-scrape skill provides a standardized approach for integrating the Firecrawl /scrape endpoint into product code. It is designed for scenarios where an application already possesses a specific URL and requires structured content, such as markdown, HTML, links, or metadata, for downstream processing. The skill emphasizes efficient data retrieval by leveraging cached content while providing mechanisms to control freshness via maxAge settings. It serves as the primary primitive for page-level extraction tasks, offering guidance on when to escalate to more complex interaction patterns or search-based discovery.

What it does

  • Extracts content from a single, known URL into formats like markdown or HTML.
  • Supports optional retrieval of links, screenshots, and metadata.
  • Manages content freshness through configurable maxAge parameters.
  • Provides metadata regarding cache state and source status for verification.
  • Integrates with LLM and indexing pipelines for downstream data processing.
  • Offers specific implementation guidance for multiple programming languages.

When to use it

  • When your application already has a specific URL to process.
  • When you need to ingest content for summarization, enrichment, or monitoring.
  • When you require a default extraction primitive before considering interactive browser sessions.
  • When you need to balance performance with data freshness requirements.

Advantages

  • Reduces latency by utilizing cached content for repeat requests.
  • Provides structured output suitable for direct LLM consumption.
  • Offers granular control over cache reuse to meet specific staleness requirements.
  • Simplifies integration with consistent patterns across multiple programming languages.

Usage examples

“How do I scrape a documentation page for my RAG pipeline?”
The agent will guide you to use the /scrape endpoint with the 'onlyMainContent' option to filter out navigation noise, returning clean markdown for your RAG ingestion.
“I need to check if a product page is still live and get its current price.”
The agent will recommend using /scrape with 'maxAge: 0' to ensure a fresh fetch, then advise your code to interpret the returned content for liveness signals.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/firecrawl/firecrawl-build-scrape/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip firecrawl-build-scrape.zip -d .agents/skills

Documentation by firecrawl

Firecrawl Build Scrape

Use this when the application already has the URL and needs content from one page.

Use This When

  • the feature starts from a known URL

  • you need page content for retrieval, summarization, enrichment, or monitoring

  • you want the default extraction primitive before considering /interact

Default Recommendations

  • Return markdown unless the feature truly needs another format.

  • Use onlyMainContent for article-like pages where nav and chrome add noise.

  • Add waits or other rendering options only when the page needs them.

Freshness and Liveness

  • Firecrawl reuses recently indexed content, which is what makes repeat reads of the same URL fast. Set maxAge (milliseconds) to bound how old a reused copy may be, or maxAge: 0 to skip index reuse for a freshness-critical read.

  • Read metadata.cacheState and metadata.cachedAt to see what you actually got.

  • A successful scrape reports what the page returned. Whether the thing the page describes is still active is a source-specific judgment your code makes.

  • See references/freshness-and-liveness.md for the tradeoff, the metadata, and the decision rule.

Common Product Patterns

  • knowledge ingestion from known URLs

  • enrichment from a company, product, or docs page

  • pricing, changelog, and documentation extraction

  • page-level quality checks or monitoring

Escalation Rules

  • If you do not have the URL yet, start with firecrawl-build-search.

  • If content requires clicks, typing, or multi-step navigation, escalate to firecrawl-build-interact.

Implementation Notes

  • Keep the integration narrow: one feature, one URL, one extraction contract.

  • Treat /scrape as the default primitive for downstream LLM or indexing pipelines.

  • Request richer formats only when the consumer needs them, such as links, screenshots, or branding data.

Docs (Source of Truth)

Read the source-of-truth page for your project language before writing integration code:

See Also

  • firecrawl-build

  • firecrawl-build-search

  • firecrawl-build-interact

Files

references/freshness-and-liveness.md

Freshness and Liveness

Two separate questions. Keep them separate in code:

  • Freshness — how old is this content? Controlled by maxAge.

  • Liveness — is the thing the page describes still active? A source-specific judgment your application makes from the content.

maxAge and the Cache Tradeoff

Firecrawl reuses recently indexed content, which is what makes repeat reads of the same URL fast and cheap. maxAge is the maximum age, in milliseconds, of an indexed copy that /scrape may return.

  • Omit maxAge and Firecrawl chooses the window itself, tuning it per domain. This is the right default for most reads.

  • Set maxAge explicitly when the feature has a real staleness bound.

  • maxAge: 0 skips index reuse and takes the live scrape path. It costs latency, and it surfaces live-site failures that a reused copy would have masked, so spend it on reads where staleness would cause a wrong or costly decision.

/parse is always uncached: it ignores a client maxAge and does not store its result, so there is no freshness knob to set there.

Verifying What You Got

From /scrape response metadata:

  • cacheState — "hit" or "miss", present only when index reuse was eligible. With maxAge: 0 the field is absent, which is itself the confirmation that no indexed copy was used.

  • cachedAt — ISO timestamp of the reused copy, present on a "hit".

  • sourceURL — the URL you requested.

  • url — the URL the response came from. Differing values mean the request was redirected. Equal values are not proof that no redirect occurred, because url falls back to the requested URL when the engine reports none.

  • statusCode — the HTTP status of the response.

Deciding Liveness

Firecrawl supplies page evidence; your application interprets it in its own terms. 200 plus non-empty content means the fetch succeeded, not that the item described by the page is still active — plenty of sites serve a full page for a removed record.

  • Read the rendered content for the source's own signals.

  • Prefer a source-specific API or identifier where one exists. Those usually state a status that the rendered page only implies.

  • When the evidence is inconclusive, keep the state unknown and stop before an expensive or irreversible step rather than assuming active.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access, Needs Python, Needs Node.js

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
firecrawl
Publisher
Firecrawl (Official)
Source
firecrawl/skills · skills/build/firecrawl-build-scrape · commit 458d9d0
Licence
ISC License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by firecrawl or Firecrawl.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download firecrawl-build-scrape.zip