Single Page Scraping

FireCrawl takes a URL and returns the page's content as clean markdown or structured data, handling the rendering so a Beam agent doesn't need a browser of its own. A Beam agent steps in when a link arrives in an approved source, such as a research request or a monitored page list, reads the URL, and calls FireCrawl to fetch the content under the account's approved crawl settings. The agent writes the returned content into the requesting record or document. Pages that return an error, require a login, or sit behind a paywall get routed to a human to fetch manually.

Whole Site Crawling

FireCrawl can crawl an entire website by following internal links from a starting page, returning content for every page it reaches within set limits. A Beam agent begins a full crawl when an approved research task calls for a complete site rather than one page, reads the starting URL and depth limit, and starts the crawl under the account's approved settings. Once finished, the agent compiles the returned pages into the requested document or dataset. Sites that block crawling, exceed the agreed page limit, or return mostly irrelevant pages get routed to a human to scope down.

Structured Data Extraction

FireCrawl can return specific fields, such as a price or a title, from a page's content using a schema the account defines rather than returning raw text. A Beam agent acts when a page needs a specific value for a downstream record, reads the URL and the approved schema, and calls FireCrawl to return just those fields. The agent writes the extracted values into the target record. Pages that don't fit the schema, return empty fields, or need a new schema get routed to a human to define before the agent proceeds.