LLM Call Tracing

Langfuse records each call an application makes to a language model, including the prompt sent, the response received, and how long it took. A Beam agent reads these traces on a schedule, checks them against the approved latency or error rate threshold for that application, and updates a shared dashboard so engineers do not have to query Langfuse directly. When a run of calls shows a much higher failure rate than usual, the agent flags the pattern instead of restarting anything. A human then investigates whether the model, the prompt, or the underlying data changed.

Prompt Version Management

Langfuse keeps a version history of the prompts an application uses, so teams can compare how a change performed against the version before it. A Beam agent reads a newly approved prompt edit from a connected repository and publishes it as the next version in Langfuse, tagging it with the reason for the change. It notes which version is now active against the application record. A prompt edit that has not gone through the team's review step is held and routed to a human rather than published automatically.

Automated Evaluation Scoring

Langfuse can score a model's responses against a rubric or a reference answer, giving a consistent measure of quality across many runs. A Beam agent reads new responses as they come in, applies the approved scoring rule for that application, and records the score against the run in Langfuse. It updates a shared report so the team can watch quality over time without checking each run by hand. Responses that score well below the approved threshold are routed to a human to check whether the issue is the model, the prompt, or the input data.