WorkAI at Scrut
AI at Scrut: from a Q&A assistant to agents
I led design for three stages of AI at Scrut: a Q&A assistant, AI features inside everyday workflows, and a set of agents that check compliance continuously. Compliance customers need to trust what the product tells them, and AI can be wrong. At each stage my job was to work out how far users could rely on the AI, and to design the product so they could always check its work.
The decision
Position it as a junior expert
We built a compliance Q&A assistant and deliberately positioned it as a junior infosec expert. People already know how much to trust a junior colleague: helpful for a first answer, never the final word, always worth checking.
Ask users to learn a new kind of tool
Users work out for themselves how far to trust it.
A junior infosec expert Chosen
A familiar idea. Every answer links to its sources, and with no source the assistant does not answer.
What I did
Trust designed in at each stage
- Designed Teammates to show its work: sources on every answer, and a carefully designed "I can't answer this".
- Designed for review across five workflows: accept, edit and reject, the evidence, and a confidence score.
- Fixed gap analysis after it flagged gaps in our own policy templates, by teaching the system about policy layers.
- Decided when agents speak up, reusing the notification and task system I had built.
The study in one picture
Read the full case study
Password protected. It has the trade-offs in full, the results and how they were measured, and what I would do differently.