Chat quality audit

The quality audit analyses the exchanges of a chat pipeline to answer one question: did the user get what they needed? When the same cause of dissatisfaction keeps coming back, the audit records it in its report and proposes a fix for the agent: poorly defined instructions, missing knowledge, misread documents, or a model too weak for the task.

Audits are managed in Manager › Tools › Quality audit. An audit never modifies the pipeline: it only observes it.

Creating an audit

1. Click New audit and pick the chat pipeline to audit. A pipeline can only have one active audit at a time.
2. Pick the judge model, the AI that analyses the exchanges. Choose a model at least as capable as the audited agent's: a judge weaker than the agent misses its mistakes. Without a choice, the organisation's default AI is used.
3. Adjust if needed:
   • Pre-filter: an optional decision model that cheaply sets aside clearly successful exchanges before calling the judge.
   • Sample: the share of conversations audited (100% by default). A sampled conversation is audited in full, so that each answer is judged in the light of the reaction that follows it.
   • Inactivity: the delay after which an exchange the user did not reply to is judged anyway (30 minutes by default).
   • Recurrence threshold: the number of occurrences from which an issue is considered recurring and gets a proposed fix (3 by default).
   • What "satisfied" means for this assistant, for example "the user must leave with a case number". The judge takes it into account.

An audit can be paused, resumed, archived or deleted from its Settings tab. Deleting it erases its history, but not the fixes already applied to agents.

How an exchange is judged

Satisfaction mostly shows in the user's reaction, so an exchange is not judged right away:

• when the user writes the next message, the previous exchange is judged with that reaction. Rephrasing the same request or saying "no, that's not it" signal dissatisfaction; thanking or moving on to another question signal satisfaction;
• if the user does not reply, the exchange is judged after the inactivity delay, on the quality of the answer alone. If the user comes back later, the exchange is judged again;
• a 👍 counts as satisfaction without calling the judge. A 👎 forces dissatisfaction, and its optional comment helps the judge find the cause;
• a turn that fails (pipeline error) counts as a technical dissatisfaction, without calling the judge.

Tests run from the Designer are never audited.

What the user sees

While an audit is active, the /app chat window shows a discreet notice ("Exchanges with this assistant are analysed anonymously to improve its quality") and offers 👍 / 👎 under each answer, with an optional comment on 👎. Clicking the same thumb again removes the feedback.

Reading the report

The Report tab covers 7, 30 or 90 days:

• Satisfaction: the share of satisfied exchanges among judged ones, with the number of audited exchanges, the 👍/👎 received and the judge's cost.
• Satisfaction per week: the rate's curve, with a diamond at each applied fix, to see whether the rate went up afterwards.
• Causes of dissatisfaction and By agent: dissatisfaction broken down by category, and the rate of each agent that answered (with its model).
• Recurring issues: the causes that come back, grouped by the judge. Each one shows its observed reasons and the proposed fix. Isolated signals and dismissed issues are folded at the bottom.

:::info Privacy A chat stays private: the report shows no quote from the conversations and no user name. Reasons are rephrased by the judge ("the assistant does not know the cancellation procedure"), and a filter automatically drops any reason that would repeat a passage of the conversation or contain an email address or a phone number. :::

Applying a fix

Depending on the cause, the proposed fix is:

Fix	Effect of "Apply"
Change the instructions	Rewrites the agent's role, objective or constraints. The proposed text can be edited before applying.
Change model	Replaces the agent's AI model with an active model of the organisation.
Complete the knowledge	Advice only (for example, the document to add to a knowledge base).
Review the extraction	Advice only (OCR, chunking, attachments).

Good to know:

• Only the agent is modified, never the pipeline. If the agent is used in several pipelines, the apply window says so: the fix will apply everywhere.
• If the pipeline forces its own model on the agent, changing the agent's model would have no effect: the suggestion becomes advice, to apply in the Designer.
• Undo fix restores the previous value, as long as the agent has not been edited in the meantime.
• A fixed issue that comes back is reopened and flagged Came back.
• Dismiss files away an irrelevant issue; Mark resolved closes it without an automatic fix; Regenerate asks for a new proposal.

Permissions

Action	Permission
Read reports	audit.read
Create, configure, triage issues	audit.write
Apply or undo a fix	audit.write and agent.write

The generic Quality audit profile grants audit read and write, without the right to edit agents. The Pipeline creation profile grants everything.

Cost and retention

Each judged exchange costs one call to the judge model, counted in Finances with the chat turns. It is not charged to the user's quota. The pre-filter and sampling reduce this cost.

Verdicts are deleted with their conversation, according to the organisation's retention period. The weekly trend remains available beyond that: only daily counts are kept.