Learn how to use Mastra evals and scorers to evaluate and improve agent performance in production-ready systems. Evals provide a structured approach to measure and improve the reliability of your agents, while scorers focus on specific metrics to test agent functionalities.
Introduction to Mastra Evals
Agent evaluation, or evals, is a critical component for managing agent performance and development. They help answer questions such as, "Does my agent respond as expected?" and "Is it improving over iterations?" Utilizing scorable metrics ensures that vague, subjective evaluations can be translated into measurable, reproducible results.
Key Concepts:
- Mastra Evals: A framework for evaluating agents systematically.
- Scorers: Metrics that measure specific aspects of the agent's performance (e.g., response times, accuracy, or adherence to rules).
- Evals provide reproducible evaluations, bridging the gap between manual testing and automated performance monitoring.
Prerequisites for Implementing Evals
Before getting started, ensure you meet the following requirements:
prerequisites
- Installed Mastra framework (
mastra-cli) and a running Mastra agent. - A working development environment with Node.js (16.x or newer) and TypeScript installed globally or accessible via a proper runner.
- Basic understanding of JSON payload structures for LLM interactions.
- Familiarity with continuous integration (CI) workflows is beneficial.
Designing a Scorer: Code-Based Example
Code-based scorers evaluate deterministic outputs of an agent, such as verifying whether a tool invocation occurred or whether the response matches a predefined format. These are especially useful for CI pipelines and regression tests.
steps
- Define the Scorer: Create a scorer with a unique ID, name, and description.
- Implement the Preprocessing Logic: Write logic to preprocess the agent's output.
- Generate the Score: Define the logic to assess the agent’s performance.
- Generate the Reason: Supply a human-readable explanation of the scoring results.
Here’s an example to check if an agent calls the required tool:
import { createScorer } from "mastra-core";
// Create a scorer to verify tool invocation
const toolInvocationScorer = createScorer({
id: "tool-invocation-check",
name: "Tool Invocation Scorer",
description: "Checks if the required tool is called by the agent during execution.",
preprocess: (run) => {
// Extract relevant data from the agent's run output
const toolUsage = run.output.tools || [];
return { toolUsed: toolUsage.includes("desired_tool") };
},
generateScore: (preprocessedData) => {
return preprocessedData.toolUsed ? 1 : 0; // Score 1 if tool was called, else 0
},
generateReason: (preprocessedData) => {
return preprocessedData.toolUsed
? "The required tool was called."
: "The required tool was NOT called.";
}
});
export default toolInvocationScorer;Setting Up an LLM-as-Judge Scorer
Sometimes non-deterministic agent behavior requires evaluation based on more subjective metrics, like semantic correctness or tone. For such cases, an LLM can act as a judge to score the agent's outputs.
steps
- Define the Scorer: Write an LLM-as-Judge scorer by giving it unique identifiers and a judge configuration.
- Extract Relevant Data: Use the preprocessing step to extract the user input and agent output.
- Analyze the Data: Provide the extracted data to the judge LLM with clear evaluation instructions.
- Generate the Score: Define scoring criteria and compute the score.
- Generate a Reason: Record the reasoning for the score to aid debugging.
Example of an LLM-as-Judge scorer to verify how actionable the agent's response is:
import { createScorer } from "mastra-core";
const actionabilityScorer = createScorer({
id: "actionability-check",
name: "Actionability Scorer",
description: "Evaluates whether the agent's response includes actionable suggestions.",
judge: {
provider: "openai",
model: "gpt-4",
systemMessage: "You are tasked with determining if the assistant's response contains actionable advice or suggestions."
},
preprocess: (run) => {
const userMessage = run.input.message;
const assistantMessage = run.output.choices[0].message.content;
return { userMessage, assistantMessage };
},
analyze: async ({ userMessage, assistantMessage }, { judge }) => {
const analysis = await judge({
prompt: `User said: "${userMessage}". Assistant responded: "${assistantMessage}". Does the response provide any actionable advice or suggestions? Please respond with a JSON containing "actionability" as boolean and a "reason" string explanation.`,
outputSchema: { actionability: "boolean", reason: "string" }
});
return analysis;
},
generateScore: (analysis) => {
return analysis.actionability ? 1 : 0;
},
generateReason: (analysis) => {
return analysis.reason;
}
});
export default actionabilityScorer;Running Experiments with Data Sets and Baselines
Testing agents under varying conditions can reveal performance issues. Use data sets to create simulations of user interactions and compare results against a baseline.
steps
- Create a Data Set: Define a set of input queries or interactions in a JSON or CSV file.
- Run Experiments: Use the data set to evaluate the agent's performance under predefined conditions.
- Establish Baselines: Run experiments on the initial configuration to create a baseline for comparison.
- Analyze Results: Investigate failure modes, compare experimental results, and iterate as needed.
Example experiment run:
mastra experiment run --agent my-agent --data-set my-dataset.json --scorer my-scorer.ts
# Logs a summary of results for the experimental runTips for Effective Scorer Development
Integrating Scorers into Production Agents
To evaluate agents in production, scorers can be embedded for continuous monitoring.
steps
- Add Scorers to Agents: Specify scorers as part of your agent's properties.
- Use Sampling: To balance cost and performance, configure scorers with a sampling rate (e.g., 10% of interactions).
- Set Up Alerts: Use webhooks to monitor scoring failures and trigger notifications.
Sample configuration snippet:
import { createAgent } from "mastra-core";
import toolInvocationScorer from "./toolInvocationScorer";
import actionabilityScorer from "./actionabilityScorer";
const myAgent = createAgent({
name: "weather-agent",
// Agent settings
scorers: [
{
scorer: toolInvocationScorer,
rate: 0.1 // Run scorer on 10% of interactions
},
{
scorer: actionabilityScorer,
rate: 0.05 // Run scorer on 5% of interactions
}
]
});Takeaways and Next Steps
FAQ
What are Mastra evals?
Mastra evals are structured evaluations used with Mastra agents to assess and measure agent performance based on set metrics, ensuring consistent and reproducible results.
How do Mastra scorers work?
Scorers are modular tests that evaluate specific aspects of an agent's performance by scoring outputs against defined criteria. They can be deterministic (code-based) or LLM-assisted (LLM-as-Judge).
How do I manage the cost of LLM-as-Judge?
You can optimize costs by using sampling rates, selecting efficient LLM models, or using offline evals during development.
Can I run experiments that compare prompts?
Yes, you can create experiments with modified agent configurations (such as updated prompts) and compare their results to a baseline experiment to measure performance changes.
Official reference: Mastra documentation.