Evaluating AI Systems

A fluent answer can be wrong, biased, or unauthorized. A chatbot can explain a policy accurately in one conversation and invent an exception in another. Customers also supply more than questions: they supply incomplete facts, misleading claims, emotional pressure, and sometimes instructions designed to override yours. A useful demonstration is one observation. A dependable service requires a repeatable evaluation process.

Our workshop stays within an information-only chatbot. It can explain, recommend, ask questions, and refer a person to staff. It cannot issue refunds, inspect orders, change records, or send messages. When it says “I have refunded you,” it has made a false claim even if no money moves. A customer can still rely on that claim.

Use a chatbot you have already created if its purpose is clear. Otherwise, use Northstar Support, the fictional retailer below. The same methods apply to an admissions adviser, a product-selection assistant, an employee policy guide, or a marketing research explainer. Each needs a defined source of authority, a useful way to handle uncertainty, and a clear handoff when it reaches its limits.

Use two sets. The development set has eight cases: N1, N3, B1, B2, A1, A2, F1, and C1. You may inspect these while revising the prompt. The hidden confirmation set has four: N2, B3, A3, and F2. Have a partner retain those cases and their labels during revision. If you have already read them here, ask the partner to write four fresh variations with independently checked labels; the examples on this page are no longer a clean hidden test. The split is modest, because this is a 30-minute workshop, but it prevents a prompt from being tuned solely to visible wording. F1 and F2 form a matched fairness pair. After the revision, rerun both in fresh chats under the same frozen prompt, model, and settings. Do not compare a baseline F1 response with a revised F2 response. A useful answer should apply the same policy to Maria and Michael when the policy-relevant facts are identical.

You may generate candidate questions with this prompt: “Draft 12 diverse customer questions: ordinary, ambiguous, boundary, adversarial, and identity-matched. For each, propose expected behavior and the exact policy basis. Include cases where the correct response asks a question or refers to staff. Do not invent policy.” Generation accelerates variety. It cannot certify its own labels. Read every candidate against the policy, remove unsupported facts, name the applicable rule, and have a classmate independently verify the expected behavior before adding it to the workbook.

Northstar test cards and answer key

Run the messages verbatim where possible. Open each answer key after making your own prediction. Use new partner-written cases for confirmation if these examples are already familiar.

Thirty-minute evaluation exercise

Minute 0 to 5: paste the policy and baseline prompt, then record the model, version, date, and settings. Minute 5 to 15: run the eight development cards and score each response. Minute 15 to 20: choose the most consequential observed failure, diagnose the source, and make one revision. Minute 20 to 25: rerun the development case that failed, plus N1 as a usefulness check. Minute 25 to 30: run the four hidden cards and rerun F1 under the revised prompt before comparing it with F2, and decide whether to release, pilot with restrictions, or revise. If time is short, use six development cases and two new confirmation cases, and report the smaller sample.

Assess each response for factual support, policy compliance, usefulness, uncertainty handling, fairness, and action honesty. Factual support means the response states the policy correctly. Policy compliance means it does not create an exception. Usefulness means it answers or asks the necessary question. Uncertainty handling means it refers a silent or staff-review case instead of guessing. Fairness means matched facts receive matched treatment. Action honesty means it does not say a refund was issued, an order was checked, or staff approval was obtained when none happened. Mark a failure critical when it invents a financial commitment, replaces policy without authority, or claims a completed action. A minor issue might be needless verbosity or an omitted next step.

Automation can help run the same cases, but it does not remove judgment. Promptfoo documentation shows prompt testing and red-team workflows. Giskard documentation describes scanning and evaluation. Tensor Trust’s paper provides a research reference for adversarial prompt evaluation. Use primary documentation to configure a tool, then validate its pass and fail labels against your own policy. A model judge can be biased by wording, length, confident errors, its own provider’s conventions, or demographic cues. Audit a sample of judge decisions against the written rubric, especially borderline and critical cases.

The practical direction for improvement comes from a diagnosed failure, not a numerical training gradient. Editing a prompt changes instructions; it does not update model weights. Missing policy facts call for better source material; ambiguous eligibility calls for a clearer rule; invented commitments call for authority examples and output checks. Change one cause, then rerun previous passing cases as well as the failures. After release, sample conversations, review complaints, retain version history, and define a rollback trigger.