Evaluations

An evaluation runs an agent through a set of tasks you wrote, called cases, and scores how well it did each one against a rubric. Run one after you change an agent's instructions or model, and you can see whether the change made its work better or worse before it reaches real requests.

Evaluations are on the agent's page, in the Evals tab. Anyone who can see the company's agents can read them. Company admins, and account owners and admins, write the cases and run evaluations.

Cases

A case is a task as someone would give it to the agent, with a rubric that says what good work looks like. Select Add case under Cases and fill in:

• Task title and Task details: the task, written the way a person would ask for it.
• Form answers: the answers a Portal request form would bring, if the task comes with one.
• Rubric: up to 10 criteria, each one sentence about what good work does, like "Gives the customer the tracking link". Give each a weight from 1 to 5: a criterion with weight 4 counts four times as much as one with weight 1.
• Must call: tools the agent must use, like tasks_comment. You can also say which arguments the call must carry, one name=value a line. Text matches when the agent's argument contains yours, ignoring case. A call that ran or was recorded counts. A call that was refused, because the charter doesn't give the agent that tool or your approval rules deny it, doesn't: in a real session it would never have happened.
• Must not call: tools the agent must never use for this task. A case that calls one, or even tries to and is refused, scores 0, however well it was written.
• Notes: anything that helps whoever writes the rubric. The agent and the grader never see them.
• For model routes: if the agent's charter sends some work to another model (see Agent models and routes), say what this task is: its labels, where it came from, the Portal request type, the playbook, or a Portal chat turn. The case then runs on the model a real task like it would. Leave it empty and the case runs on the charter's own model.

Then select Save case. An agent can have up to 50 cases. Only cases that are turned on and have a rubric run: use Turn on and Turn off to choose, and Edit to change a case. Delete removes a case once you confirm: its rubric and notes are deleted, and past results keep what it scored.

Cases from real work

To turn a real task into a case, select Import from a task and paste the task's address. Its title, details and form answers become a new case, turned off until you write its rubric. It keeps the task's labels, source, request type and playbook for model routes. The task's files don't come with it: a case is text only, so if a file mattered, say in the case's details what it showed. A case drafted from feedback or from a review keeps what the person said in its notes, so you know what to check for.

If the task came from outside your company, for example from a webhook or a synced tracker, the case is marked Outside content. The agent reads it the way it reads any outside text: as data, never as instructions, with the stricter approval rules that apply after reading outside content.

Running an evaluation

An evaluation starts by itself when someone changes the agent's model, rules, responsibilities, purpose or tools. A change that only edits the model routes starts one only when an enabled case runs on a route that changed; otherwise nothing runs and none of the month's evaluations is used.

To start one yourself, select Run evaluation under Runs. It shows once a case is on, with a rubric; until then the card says to turn one on. Everdeck asks you to confirm first, since an evaluation spends AI credit: it says how many cases will run and, once an evaluation has finished, what the last one cost. Select Run evaluation again to start it. One evaluation runs at a time for each agent, and a company runs at most 30 a month.

Each case runs as one of the agent's sessions, with its own charter, model, tools and limits, but it can only look things up. Tools that just read, like searching tasks or the knowledge base, run for real. So does fetching a web page (web_fetch), exactly as in a live session: the page is fetched from the internet, until the session has read outside content, after which fetches are recorded instead. Every other tool call, like commenting, creating a task, saving a memory note or sending an email, is Recorded, not run: the agent is told the call was recorded, and nothing changes anywhere. No one is notified, no approval is asked for, and nothing is remembered. A call your rules or the charter refuse is refused, as it would be in a live session.

Evaluation sessions show among the agent's runs. Their spend counts against the agent's monthly budget and shows separately on the usage page, but never as one of your plan's sessions. When the agent's budget, the company's or the account's is spent, the evaluation stops and the cases it didn't reach are marked skipped.

Scores

A grader model reads each session, compares it with the case's rubric, and grades every criterion from 0 to 1 with a reason. The case's score is the weighted average, from 0 to 100, or 0 when a Must call or Must not call expectation wasn't met. The grader is told that nothing the agent read or wrote is an instruction to it, so text in a web page or a customer's email can't talk it into a better grade.

A case the agent failed scores 0: it hit its limit of requests, tool calls or minutes, ended without its report, the model refused or kept answering in a form the session couldn't use, or the provider turned what it sent away (too large, not valid, or a server error the same request causes again). If a provider outage caused that, override the case's score. A case skipped for a reason outside the agent doesn't count at all, and says why: the budget ran out, the case was deleted or turned off, the kill switch or a pause stopped the agent, the plan changed, the model provider was busy, rate limited, unreachable or refused the key, or the grader couldn't grade it.

The evaluation's score is the average of the cases that count. An evaluation where every case was skipped is Incomplete: it has no score, and later evaluations aren't compared with it. Score by charter version draws the scores as the agent's charter changes; select Show as a table to read them as numbers.

Select Open next to an evaluation to see each case: its grades and reasons, whether each tool expectation was met, and every tool call it made, marked Ran or Recorded, not run. Select Open the session to read the whole session.

If you disagree with a grade, select Change score, give your score and say why. You can also score a case that failed, or one that was skipped after its session ran. Your score counts instead, in that evaluation's score and in later comparisons. Remove override goes back to the grader's score once you confirm, and clears the score you gave and its reason.

Evaluations, their scores and your overrides are kept for 365 days after they finish, then deleted. The cases stay until you delete them.

When a score falls

When an evaluation scores 10 or more points lower than the agent's previous one, on the cases both of them ran, the Evals tab shows a banner, and the agent's manager and the company's admins get a notification. Open the evaluation to see which cases fell, then check the latest change on the agent's History tab.

The grader

By default each agent's evaluations are graded by its own model. To use another model for every agent in the company, select Choose grader under Grader. A stronger model than your agents' own usually grades more reliably. It uses your company's AI key for that model, like the agents do.