Structured Record Refresh

Diffbot crawls a defined set of web pages and turns each one into structured data, such as a product, article, or organization record, depending on the page type. A Beam agent reads the latest structured output on the schedule the customer has approved and updates the corresponding record in the linked system, whether that's a catalog, a research list, or a lead database. A page that returns clean, complete data updates without any action needed. A page that comes back with missing fields or an unrecognized structure gets flagged for a person to check the source page directly.

Broken Page Flags

A web page Diffbot has been reading can change layout, move, or disappear entirely, which affects whether the crawler can still return usable structured data. A Beam agent reads the crawl result for each page and applies the customer's approved rule for what counts as a failure, such as an empty response, a repeated error, or data missing fields the previous version had. A page returning normally needs no attention. A page failing repeatedly gets flagged to whoever owns that page list, with the URL and the kind of failure noted, rather than being silently dropped from the dataset.

New Entity Discovery Notes

When Diffbot's crawl surfaces a page describing something not yet in the linked system, such as a new organization or product not previously seen, a Beam agent reads the structured data returned and applies the customer's approved rule for what counts as a genuinely new entity versus a duplicate under a different name. A clear new entity gets added to an intake list with its source page attached. Anything that looks like it might already exist under another name gets set aside for a person to confirm before anything is added, since merging incorrectly is harder to undo than waiting.