Overview
The evaluation system is the central mechanism for monitoring and improving agent behavior. It uses G-Eval — an LLM-as-judge framework — to score every agent response against domain-specific criteria like legal correctness, behavioral compliance, and pedagogical value. Every evaluation call is traced in Langfuse under theevals/automated trace name.
How G-Eval Works
G-Eval sends the agent’s response, its context, and a set of evaluation criteria to a high-reasoning model (GPT-4o by default). The evaluator returns:- A score (0 to 1) for each metric.
- A rationale explaining why it scored the response that way.
- A pass/fail determination based on a predefined threshold.
g-eval-{MetricName}, so you can inspect the evaluator’s reasoning for any score.
Agent-Specific Metrics
Witness Metrics
Judge Metrics
Opposing Counsel Metrics
Running Evaluation Batches
Batches can be triggered via the Eval Dashboard or the API. The system supports both ad-hoc runs for testing prompt changes and scheduled monitoring for detecting performance drift.Configuration
What Happens
- Dataset building — The system extracts test cases from session transcripts. Each test case is one agent response with its triggering context, the preceding conversation, and (for witnesses) the relevant affidavit.
- Metric evaluation — Each test case is scored against its agent-specific metric suite (4-5 metrics per response). Every metric call is traced individually in Langfuse.
-
Results storage — Individual results are stored in the
automatedEvalResultscollection. Batch summaries go intoautomatedEvalBatches. - Summary calculation — The system computes per-agent averages, per-metric pass rates, and an overall pass rate across all evaluations.
Dashboard Metrics
The Eval Dashboard surfaces these top-level indicators:
Trend charts track performance over time. A sudden dip in “Rule Citation” for the Judge agent might indicate a regression from a model update or a particularly tricky new scenario.
Viewing Results in Langfuse
Batch Traces
Each batch evaluation creates a trace namedevals/automated that contains:
- The batch configuration as input.
- Summary statistics as output.
- Individual
g-eval-*generations nested inside for each metric evaluation.
Inspecting a Score
To understand why a specific response received a low score:- Find the batch trace in Langfuse.
- Locate the
g-eval-{MetricName}generation for that test case. - Read the evaluator’s rationale — it explains which criteria the response failed and why.
Practical Workflows
Regression Testing
After updating an agent prompt, run a batch evaluation on a standard scenario. Compare metric scores to the previous batch. If Affidavit Faithfulness drops, the prompt change may need revision.Tuning the Evaluator
If human raters consistently disagree with automated scores on a specific metric, the evaluation criteria may be too strict or missing context. Inspect theg-eval-* generation to see exactly what the evaluator was told, and adjust the metric definition accordingly.