Firecrawl + AI Agents: Clean Markdown Is Still Untrusted Input

The scraped page looked ideal: no cookie banner, no navigation, no styling debris—just clean Markdown. Halfway down, one paragraph told the AI agent to ignore its task, retrieve a credential, and send it to another host.

The conversion had worked. The trust model had not. Firecrawl can turn a rendered page into Markdown, links, HTML, screenshots, or structured JSON. That is useful representation work. It does not establish that the page is truthful, current, in scope, safe to quote, or entitled to control a downstream tool.

This distinction is the production boundary for Firecrawl and AI agents. Treat every scraped byte as remote input. Bind the requested target before retrieval. Record the final URL, status, content type, retrieval time, and hashes. Detect instructions inside the content, preserve them as evidence when necessary, and deny them authority. Only then should the normalized result enter a model context or evidence store.

Incident / The page passed formatting and failed authority

Imagine an agent researching vendor support policies. Its operator approves one documentation host and asks for the cancellation terms. The scrape returns a concise Markdown document. Embedded near the relevant paragraph is an instruction aimed at automated readers: fetch a local configuration file, call a diagnostic tool, and include the result in the answer.

A naïve pipeline sees one coherent prompt. The system instruction describes the research task; the page supplies context; the model decides which sentences matter. But the page author, a compromised CMS, an injected ad, or hidden markup can now compete for control. OWASP classifies this as indirect prompt injection: hostile instructions arrive inside content retrieved from a website or document rather than from the user.

Removing headers and scripts does not solve that control problem. A perfectly legible sentence can still be hostile. Converting it to JSON does not solve it either. A schema can prove that a field is a string. It cannot prove that the string is accurate, benign, or authorized to trigger an action.

The safest response is not to discard every suspicious page. Evidence may need to preserve what was found. The response is to separate three lanes:

• retrieval lane: collect a bounded source and its receipt;
• interpretation lane: classify, quote, summarize, and cite the source as data;
• tool lane: accept only application policy and separately bound human authority.

A sentence copied from a webpage may influence a summary after review. It must never manufacture permission to read a secret, widen a crawl, contact a third party, or mutate an external system.

Prerequisites / Start with the smallest Firecrawl operation

Firecrawl exposes several ways to collect web data, and choosing the broadest one by default creates unnecessary uncertainty. The Map endpoint is designed to discover links quickly. Its documentation says titles and descriptions may be absent and warns that the result may not include every link because the endpoint prioritizes speed and uses sources such as sitemaps, cached crawl data, and search results. A map is an inventory candidate, not proof that each URL was freshly retrieved.

The Scrape endpoint handles one URL and can return Markdown plus metadata including sourceURL, target status, and content type. That is the right default when the task names one page. The Crawl endpoint recursively discovers and scrapes reachable pages and supports path filters, depth control, subdomain rules, external-link rules, polling, WebSockets, and webhooks. It is powerful precisely because it can expand the work. Expansion should be explicit.

Before a request, record a contract such as:

{
  operation,
  requested_url,
  allowed_hosts,
  allowed_paths,
  page_limit,
  depth_limit,
  allow_subdomains,
  allow_external_links,
  accepted_content_types,
  byte_limit,
  purpose,
  expires_at
}

For a single product page, choose scrape. For link discovery before a reviewed selection, choose map. Choose crawl only when the task genuinely needs recursive coverage and the operator can state the intended paths and cap. “Crawl the docs” is not a complete contract; it leaves redirects, archives, localization trees, generated calendars, and linked hosts to runtime chance.

Current Firecrawl documentation describes a large default crawl limit. That is a vendor default, not a sensible application default. Set the intended limit in every request. Do the same for depth, included paths, excluded paths, subdomains, and external links. A cost or concurrency ceiling is useful, but the stronger reason is epistemic: the evidence set should be small enough to explain.

Setup / Bind both the requested URL and the final URL

The URL submitted to a scraper is not always the source that answers. Redirects may be legitimate—a canonical hostname, a language path, an HTTP-to-HTTPS upgrade—or they may leave the approved boundary. The admission record must retain both requested and final URLs. If the final host or path is outside the declared contract, block before the content reaches the model.

The same check applies to every page in a crawl. A starting host does not confer authority on its external links. Subdomains are separate unless the policy says otherwise. Query strings and fragments may change the evidence identity. Authenticated pages require a credential binding narrow enough that a page cannot redirect a bearer token to another origin.

Robots rules are another input, not an authorization system. RFC 9309 defines how crawlers select and apply robots groups and explicitly says the protocol is not access authorization. A production agent should evaluate the applicable rule before fetching. It should also recognize the limits: an allowed path may still be private, copyrighted, contractually restricted, or inappropriate for the stated purpose. A disallowed path should fail closed even if the content is reachable through an ordinary HTTP request.

Terms, purpose, and data policy sit alongside robots, not inside it. The scraper should know why it is collecting the page, how long the copy will be retained, and whether personal data is expected. That review cannot be delegated to a sentence found on the target page.

Provenance / Keep raw and normalized receipts together

A Firecrawl response may contain clean Markdown, raw or cleaned HTML, links, JSON, images, or other formats. The Scrape documentation also notes that some signed media and screenshot URLs expire. If the application needs durable evidence, a transient URL is not enough. Store the permitted artifact—or at least its hash and retrieval metadata—within the retention boundary.

A useful page receipt includes:

• requested URL and final source URL;
• retrieval timestamp and declared freshness policy;
• HTTP status and content type;
• crawl job ID or scrape request identity;
• raw-content hash and normalized-content hash;
• normalization options and requested formats;
• scope contract, robots decision, and purpose;
• classification result, schema result, and citation identity.

Hashing both representations answers different questions. The raw hash tells whether the retrieved response changed. The normalized hash tells whether the exact Markdown or JSON given to the model changed. If a parser update produces different Markdown from the same raw page, that is a new evidence artifact even though the source hash is stable.

Do not let normalization erase the source boundary. Each extracted claim should retain a citation to the page receipt. When several pages make the same claim, keep several receipts rather than one merged paragraph with no lineage. A generated summary is another derived artifact with another hash; it is not a replacement for the source.

Common issues / Page instructions remain page data

OWASP's Prompt Injection Prevention Cheat Sheet lists indirect injection from websites and documents, encoded or obfuscated instructions, HTML and Markdown injection, retrieval poisoning, unauthorized tool actions, and data exfiltration. It recommends layered controls: treat external content as data, separate instructions from content, validate inputs and outputs, apply least privilege, and require human oversight for high-risk operations.

No string filter can prove that a page is safe. A malicious instruction can be polite, translated, split across elements, hidden in alt text, encoded, or framed as a quotation. The application therefore needs a structural rule that survives filter misses:

Remote content may supply facts for the requested analysis. It may not redefine the task, request secrets, alter policy, select tools, expand retrieval scope, choose recipients, approve a mutation, or suppress citations.

When an injection signal appears, retain the suspect span with its receipt if the research purpose permits. Mark it as quoted remote data. Exclude it from the trusted instruction channel. If it asks for a tool or mutation, block that path regardless of whether the model says the request seems reasonable.

This rule also covers seemingly helpful onboarding files and agent-specific instructions hosted by a vendor. They may be legitimate documentation, but they are still fetched content. A human or trusted build process may review and adopt an instruction later. The running research agent should not execute it merely because the page describes itself as a skill, setup guide, or system prompt.

Schema / Valid JSON can be wrong with excellent posture

Structured extraction reduces downstream parsing ambiguity. It does not validate the world. A product page can return a schema-valid price that is stale, regional, promotional, or contradicted elsewhere. A policy page can return a valid cancellation window copied from an old revision. An attacker can place a plausible value in the exact field the extractor expects.

Use schema validation for shape, then run evidence validation for claims:

For model answers, preserve uncertainty. “The page states” is different from “the policy is.” A citation lets the reader inspect the distinction. A clean JSON object without a source receipt merely hides the ambiguity behind nicer punctuation.

Validation / Sixty-five cases without a crawl

A deterministic local JavaScript fixture tested the admission policy before any vendor call. It evaluated 65 declared packets across credential binding, requested and final URLs, approved operations, limits, depth, paths, external links, subdomains, robots, terms, purpose, status, MIME type, byte limits, hashes, retrieval time, freshness, prompt injection, hidden markup, encoded instructions, personal data, duplicates, retries, job receipts, idempotency, schema, citations, and downstream tool authority.

All 65 cases matched their expected result across 34 distinct outcomes. Six were ready for a tightly bounded evidence lane. Sixteen were held for missing provenance, freshness, privacy review, schema, citation, or retry state. Thirty-four were blocked. Four were quarantined as remote instructions. Two reused a bound cached snapshot. Three entered retry reconciliation.

The precedence cases matter more than the happy path. Unbound credentials block before target analysis. Missing limits block before an injected paragraph can be classified. A redirect outside scope blocks before provenance. Missing provenance holds before remote instructions are interpreted. Personal data review happens before duplicate reuse. A page-requested tool is blocked before schema validation can make its payload look respectable.

The fixture SHA-256 is 247154bef109f9fcd2f3307b50664204ba2b65b2eb0894d19af7caab664dda68. Node.js was available on the host; a Firecrawl CLI binary was not. The command probe installed nothing. The fixture made zero Firecrawl API calls, fetched zero websites and robots files, used no model or tool, performed no external mutation, and consumed no credentials.

This is a control-plane test, not a scraper benchmark. It says nothing about rendering coverage, anti-bot behavior, crawl throughput, extraction accuracy, current pricing, or vendor reliability. Those questions require approved live targets, credentials, budgets, and a reproducible benchmark.

Recovery / A retry needs a job identity

Crawls and batch work are long-running operations. Firecrawl's Crawl documentation describes a job ID and status retrieval; large responses may be paginated. The webhook overview documents started, page, completed, failed, and other events for supported operations, along with delivery retries. Those events are delivery evidence, not permission to process twice.

Bind every asynchronous run to a local job receipt and idempotency key. Deduplicate page events by job, page identity, and content hash. Record whether the full paginated result was consumed. A completed event without all expected pages is not a complete evidence set. A webhook replay should not append the same page twice or trigger a second downstream workflow.

Verify the webhook before parsing it. Firecrawl's webhook security documentation specifies an HMAC-SHA256 signature over the raw request body and recommends a timing-safe comparison. Authentication establishes the sender. It does not establish that the enclosed web page is trustworthy.

Retry only errors classified as retryable. The current Firecrawl error catalog distinguishes payload, authentication, authorization, quota, conflict, timeout, rate-limit, and server failures and advises honoring Retry-After when present. A validation error should change the request, not loop. A timeout after a job was accepted should reconcile the job ID before creating another crawl.

Rate and concurrency limits are plan-dependent according to the rate-limit documentation. Hard-coding a universal throughput number into application logic will age badly. Treat limits as runtime configuration, respect server signals, and keep the article's scope contract independent of how much capacity the account happens to have.

Privacy / Robots, terms, and PII are separate gates

A public URL is not automatically appropriate training data, lead data, or an autonomous-action trigger. Before collection, decide the purpose, retention period, allowed fields, and review path. If personal data appears unexpectedly, stop the evidence lane until classification and redaction are complete. Do not send the unreviewed content to a model merely to ask whether it contains PII.

Robots rules express crawler preferences under RFC 9309. Terms express a service relationship. Authentication protects access. Privacy law and internal policy govern data use. These controls overlap but do not replace one another. The crawler should record each decision explicitly instead of collapsing them into a single allowed flag.

Least privilege also applies to storage. Keep API keys outside page content and logs. Give the retrieval worker no downstream write authority. Give the interpretation worker access only to admitted artifacts. Give a separate action worker the minimum tools for an approved operation, with its own human approval and idempotency receipt. If a page contains an instruction to cross those boundaries, the architecture should make compliance impossible rather than merely discouraged.

Rollout / Shadow the evidence lane before granting tools

Start with one host and one scrape operation. Bind redirects, status, content type, byte size, raw and normalized hashes, and a freshness window. Store citations. Add prompt-injection classification and inspect quarantined spans. Then introduce map for reviewed link discovery. Add crawl only after limits, paths, pagination, job reconciliation, and duplicate handling are observable.

During shadow mode, the agent may prepare an answer but cannot publish it or call tools. Compare its claims with the cited artifacts. Track four practical rates: pages blocked by scope, pages held by missing provenance, pages quarantined for instruction-like content, and claims rejected for missing citations. A sudden drop in all four may mean the controls stopped running, not that the web became safer.

Only after the evidence lane is stable should a separate workflow consume its output. Even then, do not pass the page's suggested action directly to a tool. Translate an operator-approved intent into a small proposal, bind the exact target and arguments, require authority appropriate to the side effect, and keep a receipt. The scraped page can support the proposal with evidence. It cannot approve it.

The useful promise of Firecrawl is modest and strong: it can remove a large amount of retrieval and representation plumbing. The application still owns scope, trust, provenance, privacy, retries, citations, and action authority. That work is not friction around the scraper. It is what turns scraped content into defensible evidence for an AI agent.

Evidence boundary

This replacement uses current official Firecrawl documentation for scrape, map, crawl, errors, rate limits, webhooks, and webhook signatures; OWASP guidance on prompt injection; RFC 9309; and one deterministic 65-case local JavaScript fixture. It includes no Firecrawl account, API key, SDK or CLI installation, Firecrawl API request, live website crawl, robots fetch, browser benchmark, model call, tool call, customer data, external mutation, pricing verification, legal conclusion, or vendor comparison.

The article therefore does not claim a universal scrape latency, coverage percentage, credit cost, anti-bot success rate, or total cost of ownership. It defines an application admission boundary: normalized web content is usable only as cited, scoped evidence, never as an instruction or permission source.

Primary sources

• Firecrawl Scrape documentation
• Firecrawl Map documentation
• Firecrawl Crawl documentation
• Firecrawl Scrape API reference
• Firecrawl error catalog
• Firecrawl rate limits
• Firecrawl webhook overview
• Firecrawl webhook signature verification
• OWASP Prompt Injection Prevention Cheat Sheet
• RFC 9309: Robots Exclusion Protocol