Agent Evaluation

In the previous rooms, Agent Discovery, Agent Design, Agent Foundations, and Agent Building, you explored where AI agents can support security workflows, designed the NorthStar Fashion Security Investigation Agent, and progressively built its investigation capabilities.

The agent can retrieve alerts, search related SIEM logs, check IP reputation, consult organisational context, maintain investigation context, and return a structured verdict.

Now it is time to evaluate it.

A convincing final response does not prove that the investigation was performed correctly. The agent could retrieve the wrong alert, search the wrong SIEM field, miss relevant evidence, or reach the expected verdict for the wrong reason.

In this room, you will evaluate the NorthStar agent against reviewed security investigations. You will establish a baseline, diagnose failed behaviour, apply focused improvements, retest the affected cases, and run a complete regression to confirm that previously working behaviour still passes.

The goal is not simply to make the agent produce the expected answer. It is to make its improvements observable, measurable, and supported by evidence.

Learning Objectives

By the end of this room, you will be able to:

• Evaluate agent behaviour using reviewed security investigation cases
• Identify evidence-retrieval and decision-making failures
• Use evaluation results to guide focused agent improvements
• Compare behaviour before and after a change
• Run regression tests to detect previously working behaviour that has been broken

Prerequisites

Before starting this room, you should understand the concepts introduced in:

• Agent Discovery
• Agent Design
• Agent Foundations
• Agent Building

You should also be familiar with AI agents, tools, security investigation workflows, SIEM evidence, and structured agent verdicts.