Evaluation Metrics

Metrics define how to score the outputs of your models, agents, or pipelines.

What is a metric?

A metric is a scoring definition that evaluates model or agent outputs. In the Evaluator plugin SDK, metrics are inline Python objects passed directly to evaluator.run(...) or evaluator.submit(...).

• Inputs: For custom metrics, inputs define scoring logic composed of dataset fields and model outputs; for judge-based custom metrics, this also includes judge-model inputs (for example, judge prompts/rubrics and configuration).
• Outputs: Row-level scores and aggregate statistics.
• Execution: Metric objects run with dataset, optional runtime configuration, and an optional model or agent target.

• Metric definition: The reusable scoring configuration.
• Metric type: The metric family (for example exact-match, BLEU, LLM-as-a-judge).
• Metric score: The numeric or rubric output produced at evaluation time.

The Evaluation Workflow

[1] Choose and configure a metric object
 |
 v
[2] Select a dataset and execution mode
 |
 v
[3] Create and run an evaluation job
 |
 v
[4] Review row-level and aggregate scores

Quick Start

Minimal sync evaluation with a built-in metric:

1
import os
2
3
from nemo_evaluator.sdk import Evaluator
4
from nemo_platform import NeMoPlatform
5
from nemo_evaluator_sdk import ExactMatchMetric
6
7
client = NeMoPlatform(
8
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
9
    workspace="default",
10
)
11
evaluator: Evaluator = client.evaluator
12
13
metric = ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}")
14
15
result = evaluator.run(
16
    metric=metric,
17
    dataset=[
18
        {"expected": "Paris", "output": "Paris"},
19
        {"expected": "Berlin", "output": "Munich"},
20
    ],
21
)
22
23
print(result.aggregate_scores)

Execution Modes

Metrics can be executed in two modes:

Online Job Targets: Model or Agent

Online evaluation jobs can target either a model (an OpenAI-compatible chat completions endpoint) or an agent (any HTTP endpoint, including agentic systems with tool use and multi-step reasoning). Provide one or the other — the platform routes your request to the correct job type automatically.

See Model Configuration and Agent Configuration for setup details.

Built-in vs. Custom Metrics

• Built-in metrics: Ready-to-use metrics provided by NeMo Platform (for example exact-match, bleu, rouge).
• Custom metrics: Metrics you define for domain-specific evaluation needs.

To configure inline metric objects, see Manage Metrics. For custom metric creation guides, start with Similarity Metrics, LLM-as-a-Judge, or Bring Your Own Metric.

Datasets

Evaluation jobs need dataset input. You can provide data in two ways:

Example of providing a FilesetRef to reference specific files or globs:

1
# Include all files in subdirectory
2
dataset = "my-workspace/my-dataset#subdir/path"
3
4
# Single file
5
dataset = "my-workspace/my-dataset#file.jsonl"
6
7
# Single file in a subdirectory
8
dataset = "my-workspace/my-dataset#subdir/path/file.jsonl"
9
10
# Glob match files
11
dataset = "my-workspace/my-dataset#*.jsonl"
12
13
# Glob match files in subdirectory
14
dataset = "my-workspace/my-dataset#subdir/path/*.jsonl"

Available Metric Types

Use the metric-type pages below to create and configure custom metrics.

Understanding Scores

Scores are the metric outputs produced during evaluation:

Manage Metric Definitions

Create inline metric objects that can be reused from Python helpers or modules. See Manage Metrics for SDK patterns.