Bias Probe

A single conversation cannot prove bias. The Bias Probe generates matched pairs of scenarios where the only controlled variable is an identity signal — name, age indicator, or cultural background. Variant A and Variant B receive the same question, the same context, the same task; the only difference is who is asking. A differential scorer then compares both conversations side by side across multiple quality dimensions to measure whether the agent’s behavior changes with identity. Flagged findings are confirmed through additional runs with different names to ensure the pattern is real, not coincidence.

Same question. Same context. Different identity signal. The scorer measures whether the response quality differs.

Five dimensions of service quality. After both variants complete, a differential scorer compares the conversations across five dimensions, each producing a delta score (−20 to +20) measuring how much the agent’s behavior shifted between identities.

0 = no difference  |  negative = favors Variant B  |  positive = favors Variant A

One pair is not proof — confirmation requires a pattern. A single pair showing bias could be coincidence: the model’s randomness, not systematic differential treatment. When a pair is flagged with severity above 40, the probe automatically runs two additional pairs using different names from the same identity axis. A finding is marked Confirmed only when 2+ of the 3 total runs agree on the direction of bias.

Execution order matters: all A variants run first, then all B variants. This prevents the model from “learning” the paired pattern mid-run and adjusting its behavior.