Rendered page retrieval

Scrapfly requests a target URL and returns its rendered content through the API. A Beam agent calls the endpoint on a trigger, reads the response body, and applies your approved rule for which elements to keep. The parsed fields are written into your chosen store with a link back to the source page. When the response is empty, truncated, or the page structure no longer matches the rule, the agent skips the write and routes the URL to a person, who confirms whether the site changed or the request needs different settings this time.

Anti-scraping mitigation

Scrapfly handles the machinery that keeps requests from being refused. A Beam agent lets Scrapfly manage the fetch, then reads whether the page came back complete. On a blocked or challenged response, the agent applies your retry rule and records how many attempts each domain needed. Pages that succeed continue into parsing and storage. Domains that fail past the allowed limit are set aside, and the agent notifies a person with the failure details, so someone can decide whether that source is worth a manual look or a settings change.

Parsed JSON delivery

Scrapfly can hand back structured JSON instead of raw HTML. A Beam agent reads those keys, applies your field-mapping rule, and writes the values into your database against the right record. Each run stamps the source URL and collection time so results stay traceable. When a required key is absent, or a value fails the format check your rule defines, the agent holds that record and flags it for a person, keeping partial or malformed rows out of the tables your reports and downstream jobs depend on every day.