On-demand scrape requests

A Beam agent fires a scrape request to Piloterr when a source URL or watchlist entry changes, then reads the returned page payload. It applies your approved parsing rule to pick out the fields you care about and writes them into the target record or dataset. Clean, well matched results file automatically without anyone stepping in. When a page returns partial markup, blocked content, or a layout the rule does not recognize, the agent leaves the raw response attached and routes the case to a person, who confirms the mapping before anything downstream updates from a guess.

Structured field mapping

Each scrape returns raw HTML or JSON that rarely lines up with your schema. On a finished job, the agent reads the response, applies your field map to translate source elements into named columns, and writes the destination row with typed values. Consistent, high confidence rows save without review. Ambiguous fields, missing keys, or values that fail a validation rule are held back: the agent notifies a reviewer with the original and proposed mapping shown side by side, so a human decides how the odd record should be stored rather than letting the mapping guess for them.

Rate limit and retry handling

Large crawls hit rate ceilings and passing failures. When Piloterr returns a throttling or error response, the agent reads the status, applies your retry rule, and reschedules the request inside the allowed window instead of hammering the source. Successful retries continue quietly and the record fills in. Repeated failures, credential problems, or a source that starts refusing requests break the pattern, so the agent pauses that job and escalates to an operator with the error history attached, keeping a stalled crawl from silently dropping data your team expected to arrive on time.