Skip to content

Data Model & Ingestion

Every evaluation result is one OpenTelemetry log event named gen_ai.evaluation.result, as defined by the OTel GenAI events conventions. Results travel the OTLP logs path next to your application logs. Data Prepper routes them to their own index, otel-v1-genai-evaluation, where they join the agent spans in otel-v1-apm-span-* on traceId and spanId.

This page is the reference for that record and its storage. To send your first results, start with the Quickstart. The SDK side, including every argument that sets these attributes, is in the SDK guides: Python and TypeScript.

Results are organized the way you test an agent:

Test suite "travel-planner-golden" test.suite.name
└─ Run run_20261105_215438_5df57445 test.suite.run.id (one execution of the suite)
└─ Case "Plan a trip to Lisbon" test.case.id / test.case.name
├─ Input "Plan 3 days in Lisbon under $1,000" → the agent's trace
├─ Expected "Itinerary with flights + hotel ≤ $1,000" → opensearch.test.case.expected
├─ Output the agent's actual answer → the agent's trace
└─ Evaluators run on (input, output, expected, trace):
├─ answer_quality score 4/5, pass → 1 result event
├─ budget_respected pass → 1 result event
└─ safety pass → 1 result event
  • The agent (gen_ai.agent.name, gen_ai.agent.version) is what you evaluate. Every result names the agent and version it judged.
  • A test suite is a named set of cases, stable across runs.
  • A run is one execution of the suite against one version of the agent. It has one result per case per evaluator.
  • A case is one input with an optional expected output. Its input and output live in the agent’s trace, not in the result: the result links to that trace.
  • An evaluator (a code check, an LLM judge, a human) produces one result event per evaluation it scores.
OfflineOnline
What runsA test suite you run on purpose, before a release or in CIReal traffic, sampled and scored as it happens or shortly after
SDKevaluate(), Benchmarkscore()
Suite, run and casetest.suite.name, test.suite.run.id, test.case.id (and usually test.case.name)None
Expected outputopensearch.test.case.expected, when the case has oneNone
Linked toThe root span of the agent run for that caseThe span, trace or session it judged

A result without test.suite.run.id is online. When each offline case runs in its own conversation id (as in the framework examples, so cases don’t share history), the Agent Traces Sessions tab shows every case as a one-turn session next to your real conversations.

opensearch.gen_ai.evaluation.level says what was judged: span, trace, session or tool_call. When it is absent, a result with a spanId is span level and one with only a traceId is trace level. A session-level result carries gen_ai.conversation.id and no trace context.

The verdict is gen_ai.evaluation.score.label: pass or fail (lowercase). Your evaluator, or the SDK from your threshold and direction, decides it. Readers use the stored label and never compute one:

  • error.type is set: evaluator error (no value, no label).
  • gen_ai.evaluation.score.label is pass or fail: Pass or Fail.
  • Anything else: Scored, left out of pass rates. A judge’s own category (for example relevant) is kept in opensearch.gen_ai.evaluation.judge_label when the result also has a verdict.

Pass rate = pass / (pass + fail).

Comparing runs matches cases by id, so keep ids stable:

  • Across runs, a case is test.suite.name + test.case.id.
  • Within a run, a case is test.suite.run.id + test.case.id. All results of a case in one run share these two values.
  • SDK default: when you don’t pass a case id, the SDK uses the first 16 hex characters of the SHA-256 of the input. Editing the input therefore makes a new case, and the old one no longer matches. Pass an explicit case_id (caseId in TypeScript) for an identity that survives edits.
  • test.case.name is a display name only. Renaming a case does not change its identity.
  • Online results have no suite, run or case.

The SDK sends each result as an OTLP LogRecord:

  • event_name = gen_ai.evaluation.result, severity INFO, an empty body, all data in attributes.
  • The trace context of what was judged: traceId + spanId for a span, only traceId for a trace, none for a session or for a case whose task started no span. Offline results are parented to the root span of the agent run for the case.
  • The instrumentation scope is the SDK (opensearch-genai-observability-sdk-py or -ts) with its version.

Data Prepper stores the attributes under attributes with their dotted OTel names, for example attributes.gen_ai.evaluation.score.value. Any OpenTelemetry producer works, as long as it sends the event name and these attributes.

Index type is how the evaluation index template stores the field. Explicit fields are mapped by the template; dynamic fields take the template’s dynamic rules (strings become keyword with ignore_above 256, whole numbers long, other numbers double, booleans boolean). Required means every result should carry it; recommended fields make results easier to group and filter.

The template maps every field that readers filter or group on explicitly, so each new index (a fresh install, or a rollover) can be queried before its first document carries the field. Without the template, a query that references a field no index behind otel-v1-genai-evaluation* has mapped yet fails with Field [...] not found, so on a fresh install create the template before the first result arrives.

FieldIndex typeMappingRequiredMeaning
eventNametext with a keyword (256) subfieldexplicitRequiredgen_ai.evaluation.result. Routing matches this field
timedate_nanosexplicitRequiredWhen the result was recorded
@timestampdate_nanosexplicitRequiredCopy of time made by the pipeline. The Evaluations dataset’s time field
traceIdkeyword (32)explicitOptionalTrace that was judged. Online: the evaluated trace; offline: the agent run of the case
spanIdkeyword (16)explicitOptionalSpan that was judged
instrumentationScope.name / .versionkeyword (128 / 64)explicitOptionalThe producer, for example the SDK and its version
resource.attributes.service.namekeyword (256)dynamicOptionalService that sent the result (your evaluator or CI job)
AttributeIndex typeMappingRequiredMeaning
gen_ai.evaluation.namekeyword (256)explicitRequiredEvaluation (metric) name, for example answer_quality. Results are grouped by it
gen_ai.evaluation.score.valuedoubleexplicitOne of value, label or error.typeNumeric score. Absent on error results
gen_ai.evaluation.score.labelkeyword (256)explicitOne of value, label or error.typepass / fail is the verdict; any other value is a category
gen_ai.evaluation.explanationtextexplicitOptionalThe evaluator’s rationale, up to 8192 characters. Searchable, not aggregatable
error.typekeyword (256)explicitOptionalThe evaluator or the task failed, for example TimeoutError. No value or label
gen_ai.agent.namekeyword (256)explicitRecommendedAgent that was judged. Results without it show as “(no agent name)”
gen_ai.agent.versionkeyword (256)explicitRecommendedIts version. Pinned to keyword so a version like 2 never becomes a number
gen_ai.conversation.idkeyword (256)explicitOptionalSession that was judged (session-level results)
gen_ai.response.idkeyword (256)dynamicOptionalCompletion that was judged
gen_ai.request.modelkeyword (256)dynamicOptionalModel the LLM judge used
gen_ai.prompt.namekeyword (256)dynamicOptionalName of the judge prompt
gen_ai.prompt.versionkeyword (256)explicitOptionalVersion of the judge prompt. Pinned to keyword

OTel does not define these yet. They mirror the OTel names under opensearch..

AttributeIndex typeMappingRequiredMeaning
opensearch.gen_ai.evaluation.levelkeyword (256)explicitOptionalspan, trace, session or tool_call
opensearch.gen_ai.evaluation.evaluator.namekeyword (256)dynamicRecommendedEvaluator that produced the result. One evaluator can produce several evaluations
opensearch.gen_ai.evaluation.evaluator.versionkeyword (256)explicitOptionalEvaluator version. Pinned to keyword
opensearch.gen_ai.evaluation.evaluator.typekeyword (256)dynamicRecommendedllm_judge, code, human or external
opensearch.gen_ai.evaluation.evaluator.descriptionkeyword (256)dynamicOptionalWhat the evaluator checks (criteria or rubric summary)
opensearch.gen_ai.evaluation.score.mindoubleexplicitOptionalLower bound of the scale
opensearch.gen_ai.evaluation.score.maxdoubleexplicitOptionalUpper bound of the scale
opensearch.gen_ai.evaluation.score.thresholddoubleexplicitOptionalPass threshold the verdict was computed from
opensearch.gen_ai.evaluation.score.directionkeyword (256)explicitOptionalmaximize (default) or minimize, for example for latency or cost
opensearch.gen_ai.evaluation.judge_labelkeyword (256)dynamicOptionalThe judge’s own category when the result also has a verdict
opensearch.gen_ai.evaluation.judge.trace_idkeyword (256)dynamicOptionalTrace of the judge’s own run, when the judge is traced

test.suite.name and test.case.name are in the OTel registry; test.suite.run.id and test.case.id are proposed in semantic-conventions#1262. Set only on offline results.

AttributeIndex typeMappingRequiredMeaning
test.suite.namekeyword (256)explicitRequired offlineTest suite, stable across runs
test.suite.run.idkeyword (256)explicitRequired offlineOne run of the suite. Pinned to keyword
test.case.idkeyword (256)explicitRequired offlineOne case. Pinned to keyword. See Case identity
test.case.namekeyword (256)dynamicOptionalReadable case name
opensearch.test.suite.run.namekeyword (256)dynamicOptionalReadable run name, shown in place of the short run id
opensearch.test.suite.run.descriptionkeyword (256)dynamicOptionalWhat changed in this run, up to 1024 characters
opensearch.test.case.expectedkeyword (256)dynamicOptionalExpected output of the case, up to 8192 characters. Strings as is, other values JSON-encoded
opensearch.test.case.expected_truncatedbooleandynamicOptionaltrue when the expected output was cut

OTel cicd.pipeline.* attributes. The SDK sets them inside GitHub Actions, GitLab CI and Jenkins, on results and on agent root spans. See Run evaluations in CI.

AttributeIndex typeMappingRequiredMeaning
cicd.pipeline.namekeyword (256)dynamicOptionalCI workflow or job
cicd.pipeline.run.idkeyword (256)dynamicOptionalCI run id
cicd.pipeline.run.url.fullkeyword (256)dynamicOptionalLink to the CI run

Your own attributes (score(attributes=...), run and case metadata) are added under attributes too and mapped dynamically. The SDK never lets them overwrite the fields above.

A string attribute longer than 256 characters is stored in full in _source and shown in full on the result, but it is not indexed: you cannot filter or aggregate on it, and it is never rejected. That covers long expected outputs, evaluator descriptions, case names and CI URLs. gen_ai.evaluation.explanation is text, so it stays searchable at any length.

Record which prompt a judge used by name and version, not by content:

  • Every result carries gen_ai.prompt.name and gen_ai.prompt.version (for example answer-quality and v3, read together as answer-quality@v3), with the judge model in gen_ai.request.model. Bump the version whenever the prompt changes, so two judge versions are never mixed in one pass rate.
  • Result events do not carry the judge’s system prompt. The same prompt would be copied onto every result, it can hold sensitive data, and it would bloat the index.
  • If you want the full prompt, trace the judge and put the prompt on the judge’s own LLM span as gen_ai.system_instructions (its OTel location), for example with enrich(system_instructions=...) inside the judge call. Pass the judge’s trace id to score() (judge_trace_id=) or EvalScore, so the result links to the judge’s run through opensearch.gen_ai.evaluation.judge.trace_id.

Results are written to otel-v1-genai-evaluation:

  • On backends with ISM and aliases (OpenSearch, Amazon OpenSearch Service), it is a write alias over rolling indices otel-v1-genai-evaluation-000001, -000002, …, managed by an ISM policy.
  • On OpenSearch Serverless, which has no ISM or aliases, it is one index named otel-v1-genai-evaluation, with the same mappings.

Either way, query the results through the index pattern otel-v1-genai-evaluation*.

Where you runWho creates the index
Local stack (Docker Compose)The one-shot genai-evaluation-index-init job, before Data Prepper starts: ISM policy, template and first index with the write alias. It reads docker-compose/data-prepper/otel-v1-genai-evaluation-index-template.json and otel-v1-genai-evaluation-policy.json, and is safe to re-run.
AWS CLI installer, managed domainThe installer, before it creates the OpenSearch Ingestion pipeline: ISM policy, template and first index with the write alias
AWS CDK, managed domainA custom resource, before the pipeline is created: the same three objects
AWS CLI installer and CDK, OpenSearch Serverless (NextGen time series collection)The pipeline sink, on first write: one index otel-v1-genai-evaluation from the template in the pipeline configuration
Your own clusterYou, once, with the three requests below

Every one of these uses the same template JSON. The pipeline sinks also carry the whole template inline (template_content), so an index the sink creates itself still gets the evaluation mappings. See Route results with Data Prepper.

A composable index template named otel-v1-genai-evaluation-index-template, "version": 1, priority 100, for the indices otel-v1-genai-evaluation-*. It starts from Data Prepper’s standard OTel logs template (date_detection off, string attributes keyword with ignore_above 256, numbers dynamic) and adds the evaluation fields listed in Attributes. The local stack, the AWS installer, the CDK and this page use the same name and version, so installs converge: the init job, the installer and Data Prepper replace an installed template only when their own version is higher. A PUT from Dev Tools always replaces it.

otel-v1-genai-evaluation-index-template
PUT _index_template/otel-v1-genai-evaluation-index-template
{
"version": 1,
"index_patterns": [
"otel-v1-genai-evaluation-*"
],
"priority": 100,
"template": {
"settings": {
"plugins.index_state_management.rollover_alias": "otel-v1-genai-evaluation"
},
"mappings": {
"date_detection": false,
"_source": {
"enabled": true
},
"dynamic_templates": [
{
"long_resource_attributes": {
"mapping": {
"type": "long"
},
"path_match": "resource.attributes.*",
"match_mapping_type": "long"
}
},
{
"double_resource_attributes": {
"mapping": {
"type": "double"
},
"path_match": "resource.attributes.*",
"match_mapping_type": "double"
}
},
{
"string_resource_attributes": {
"mapping": {
"type": "keyword",
"ignore_above": 256
},
"path_match": "resource.attributes.*",
"match_mapping_type": "string"
}
},
{
"long_scope_attributes": {
"mapping": {
"type": "long"
},
"path_match": "instrumentationScope.attributes.*",
"match_mapping_type": "long"
}
},
{
"double_scope_attributes": {
"mapping": {
"type": "double"
},
"path_match": "instrumentationScope.attributes.*",
"match_mapping_type": "double"
}
},
{
"string_scope_attributes": {
"mapping": {
"type": "keyword",
"ignore_above": 256
},
"path_match": "instrumentationScope.attributes.*",
"match_mapping_type": "string"
}
},
{
"long_attributes": {
"mapping": {
"type": "long"
},
"path_match": "attributes.*",
"match_mapping_type": "long"
}
},
{
"double_attributes": {
"mapping": {
"type": "double"
},
"path_match": "attributes.*",
"match_mapping_type": "double"
}
},
{
"string_attributes": {
"mapping": {
"type": "keyword",
"ignore_above": 256
},
"path_match": "attributes.*",
"match_mapping_type": "string"
}
}
],
"properties": {
"droppedAttributesCount": {
"type": "integer"
},
"instrumentationScope": {
"properties": {
"droppedAttributesCount": {
"type": "integer"
},
"schemaUrl": {
"type": "keyword",
"ignore_above": 256
},
"name": {
"type": "keyword",
"ignore_above": 128
},
"version": {
"type": "keyword",
"ignore_above": 64
}
}
},
"resource": {
"properties": {
"droppedAttributesCount": {
"type": "integer"
},
"schemaUrl": {
"type": "keyword",
"ignore_above": 256
}
}
},
"severity": {
"properties": {
"number": {
"type": "integer"
},
"text": {
"type": "keyword",
"ignore_above": 32
}
}
},
"body": {
"type": "text"
},
"@timestamp": {
"type": "date_nanos"
},
"time": {
"type": "date_nanos"
},
"observedTime": {
"type": "date_nanos"
},
"traceId": {
"type": "keyword",
"ignore_above": 32
},
"spanId": {
"type": "keyword",
"ignore_above": 16
},
"eventName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"flags": {
"type": "long"
},
"attributes": {
"properties": {
"gen_ai": {
"properties": {
"evaluation": {
"properties": {
"score": {
"properties": {
"value": {
"type": "double"
},
"label": {
"type": "keyword",
"ignore_above": 256
}
}
},
"explanation": {
"type": "text"
},
"name": {
"type": "keyword",
"ignore_above": 256
}
}
},
"agent": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 256
},
"name": {
"type": "keyword",
"ignore_above": 256
}
}
},
"prompt": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 256
}
}
},
"conversation": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"opensearch": {
"properties": {
"gen_ai": {
"properties": {
"evaluation": {
"properties": {
"score": {
"properties": {
"min": {
"type": "double"
},
"max": {
"type": "double"
},
"threshold": {
"type": "double"
},
"direction": {
"type": "keyword",
"ignore_above": 256
}
}
},
"evaluator": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 256
}
}
},
"level": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
},
"test": {
"properties": {
"case": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 256
}
}
},
"suite": {
"properties": {
"run": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"error": {
"properties": {
"type": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}

Roll over at 50 GB or after 24 hours. Its ism_template attaches it to every new otel-v1-genai-evaluation-* index. The policy below only rolls over. The installers add a delete state after a retention period: 7 days by default, set with ISM_RETENTION_DAYS in the local stack, --evaluation-retention-days in the CLI installer and -c evaluationRetentionDays=<days> in the CDK app (0 keeps every index). To keep a longer quality history, raise the retention or set it to 0.

PUT _plugins/_ism/policies/otel-v1-genai-evaluation-policy
{
"policy": {
"description": "Rollover for GenAI evaluation results",
"default_state": "current_write_index",
"states": [
{
"name": "current_write_index",
"actions": [
{
"rollover": {
"min_size": "50gb",
"min_index_age": "24h"
}
}
]
}
],
"ism_template": {
"index_patterns": [
"otel-v1-genai-evaluation-*"
]
}
}
}

Create the policy, then the template, then the first index, so the index picks up both:

PUT otel-v1-genai-evaluation-000001
{
"aliases": {
"otel-v1-genai-evaluation": { "is_write_index": true }
}
}

Check it:

Terminal window
curl -sk -u "admin:$OPENSEARCH_PASSWORD" 'https://localhost:9200/_cat/aliases/otel-v1-genai-evaluation?v'
curl -sk -u "admin:$OPENSEARCH_PASSWORD" 'https://localhost:9200/_plugins/_ism/explain/otel-v1-genai-evaluation-000001?pretty'

Serverless collections have no ISM and no aliases:

  • Write to one index named otel-v1-genai-evaluation. No ISM policy, no alias, no rollover.
  • Use a variant of the template with the same mappings, "index_patterns": ["otel-v1-genai-evaluation"] (the single index, no -*), and no plugins.index_state_management.rollover_alias setting, which is ISM-only. On a NextGen time series collection, set "settings": {"index.timestamp_field": "time"} instead: every document must carry the collection’s timestamp field. The first write creates the index from the template. The AWS CLI installer and CDK pipelines pass exactly this variant as template_content.
  • Retention is set on the collection, with a data lifecycle policy, not with ISM.

The index keeps the default limit of 1000 fields. The built-in attributes use well under 100. If your own attributes and metadata add many keys, raise the limit in two places:

  • In the template: add "index.mapping.total_fields.limit": 2000 under template.settings and PUT the template again. New indices (after the next rollover) get it.

  • On the current write index, now:

    PUT otel-v1-genai-evaluation-000001/_settings
    { "index.mapping.total_fields.limit": 2000 }

When you change the template, raise its version too, so a tool with version 1 does not replace your copy.

Results arrive on the OTLP logs path, mixed with your application logs. Route by the OTLP event name:

  • only events with eventName == "gen_ai.evaluation.result" go to the evaluation index, and
  • the logs route excludes them, so a result is never stored twice.

Routing matches the eventName field of the log record, not an event.name attribute. Matching producers that set only the attribute is a planned follow-up.

otlp-pipeline:
source:
otlp:
port: 21890
ssl: false
route:
- evaluations: 'getEventType() == "LOG" and /eventName == "gen_ai.evaluation.result"'
- logs: 'getEventType() == "LOG" and /eventName != "gen_ai.evaluation.result"'
- traces: 'getEventType() == "TRACE"'
sink:
- pipeline:
name: "otel-logs-pipeline"
routes: ["logs"]
- pipeline:
name: "otel-evaluations-pipeline"
routes: ["evaluations"]
- pipeline:
name: "otel-traces-pipeline"
routes: ["traces"]
otel-evaluations-pipeline:
source:
pipeline:
name: "otlp-pipeline"
processor:
- copy_values:
entries:
- from_key: "time"
to_key: "@timestamp"
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
username: "admin"
password: "<password>"
index: "otel-v1-genai-evaluation" # the write alias

otel-logs-pipeline and otel-traces-pipeline are your existing pipelines. The copy_values step is required: @timestamp is the time field of the evaluation index.

The sink above writes to an alias you created. The local stack and the AWS pipelines also pass the whole template inline, so the sink installs it when it is missing:

sink:
- opensearch:
hosts: ["https://opensearch:9200"]
username: "admin"
password: "<password>"
index_type: custom
index: otel-v1-genai-evaluation # the write alias
template_type: index-template
template_content: |
{ "version": 1, "priority": 100, "template": { ... } } # the whole template, without index_patterns
  • The sink sets the index pattern itself, so the inline copy has no index_patterns. Otherwise it is identical to the template above.
  • Data Prepper does not replace an installed template with the same or a higher version.
  • The inline template is a fallback. If the sink starts before the alias exists, it creates a concrete index otel-v1-genai-evaluation from it, with the evaluation mappings but no rollover. That index then blocks the alias. This is why the local stack and the AWS installer create the alias first.

On Amazon OpenSearch Ingestion, use the same routes and sink. The pipeline cannot create ISM policies, so create the policy, template and first index before the pipeline starts (the AWS installer does). See AWS Managed Services.

opensearch-project/data-prepper#7243 proposes an index_type: genai-evaluation-plain for the OpenSearch sink, which would create the template, policy and alias itself. It is not merged. It is not in a Data Prepper release or in Amazon OpenSearch Ingestion. Do not use it: use index_type: custom with the alias and the inline template, as above. When the index type ships, it uses the same template name and version, so it keeps the template you installed.

The template maps attributes under attributes. with their dotted OTel names, for example attributes.gen_ai.evaluation.score.value, and readers query those names. That is what the otlp source produces by default (output_format: otel). A source that writes another shape (for example log.attributes.*) gets none of the evaluation mappings, and queries on those names find no results. Check a stored result:

Terminal window
curl -sk -u "admin:$OPENSEARCH_PASSWORD" 'https://localhost:9200/otel-v1-genai-evaluation/_search?size=1&filter_path=hits.hits._source.attributes'

Earlier versions of the stack stored evaluation results in the logs index (logs-otel-v1-*) or in daily indices. A pipeline that started before the alias existed may also have created a concrete otel-v1-genai-evaluation index. To move to the evaluation index:

  1. Create the evaluation index (above) and update your pipeline’s routes, so new results go only to otel-v1-genai-evaluation.

  2. Point your queries, saved searches and index patterns at otel-v1-genai-evaluation* instead of the logs index.

  3. Optional: copy the old results across, so their history stays in one place:

    POST _reindex
    {
    "source": {
    "index": "logs-otel-v1-*",
    "query": { "match_phrase": { "eventName": "gen_ai.evaluation.result" } }
    },
    "dest": { "index": "otel-v1-genai-evaluation" }
    }

    Reindexing into the alias writes to its current write index. Results copied this way keep their original time and @timestamp.