Alan Evaluation harness

Waiting for a run…

Evaluation harness

We have built a controlled dialogue evaluation system for Alan. A spreadsheet supplies simulated patients, examination findings and hidden reference answers. A separate health-worker model represents a busy worker with limited English, answering briefly from the available patient facts without receiving the reference diagnosis or treatment. Alan must ask useful questions, interpret the answers and reach an appropriate conclusion. After the dialogue, a separately prompted judge assesses the conversation against fixed references and scoring rules. By keeping the cases, health worker and assessment process consistent, we can compare Alan versions and underlying models, identify recurring weaknesses and measure improvements over time.
Evaluation score
—/100
Awaiting scored cases
0cases
0running
0complete
0run errors
Waiting for a runSaved dialogue
Waiting for a run Connecting…