Quickstart
This quickstart takes you from an empty stack to evaluation results in OpenSearch. You run a small offline test suite against a real agent (a Strands agent on Amazon Bedrock), score one live trace online, query the results, then change the agent and compare the two runs.
What you build
Section titled “What you build”flowchart LR
A["Your agent<br/>@observe / withObserve"] -->|"OTLP /v1/traces"| C["OTel Collector"]
E["evaluate() / Benchmark / score()"] -->|"OTLP /v1/logs<br/>gen_ai.evaluation.result"| C
C --> D["Data Prepper"]
D -->|"spans"| S[("otel-v1-apm-span-*")]
D -->|"eventName == gen_ai.evaluation.result"| R[("otel-v1-genai-evaluation")]
D -->|"other logs"| L[("logs-otel-v1-*")]
S --- R
Each score is one gen_ai.evaluation.result log event, linked to the agent trace it judged: the result and the agent span share traceId and spanId. The Data Model & Ingestion page describes the event and the index in full.
Prerequisites
Section titled “Prerequisites”- Docker with Docker Compose
- Python 3.10+ or Node.js 20+
- For the Python agent: AWS credentials (the default chain) with access to a Claude model on Amazon Bedrock. Any other Strands model provider works the same way. The TypeScript tab uses a mock agent and needs no model.
-
Start the stack
Terminal window git clone https://github.com/opensearch-project/observability-stack.gitcd observability-stackdocker compose up -dFor evaluations, the stack:
- runs the one-shot
genai-evaluation-index-initjob before Data Prepper starts. It creates the ISM rollover policy, the index template and the first indexotel-v1-genai-evaluation-000001behind theotel-v1-genai-evaluationwrite alias. See The evaluation index. - runs a Data Prepper 2.18.0 snapshot (2.17.0 or later is needed), which routes
gen_ai.evaluation.resultevents, and only those, to the write alias. All other logs still go tologs-otel-v1-*.
The OTel Collector listens on
4318(OTLP/HTTP) and4317(gRPC), OpenSearch on9200, Dashboards on5601. The admin password isOPENSEARCH_PASSWORDin.env.To use your own Data Prepper or an AWS deployment instead, set up the route and the index from Route results with Data Prepper or AWS Managed Services.
- runs the one-shot
-
Install the SDK
Terminal window python -m venv .venv && source .venv/bin/activatepip install "opensearch-genai-observability-sdk-py>=0.3.0" strands-agentsexport OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only,gen_ai_use_latest_invocation_tokensInstall the SDK with the OpenTelemetry context manager:
Terminal window npm install @opensearch-project/genai-observability-sdk-ts@^0.2.0 \@opentelemetry/api @opentelemetry/context-async-hooksUse ES modules (
"type": "module"inpackage.json) and run the examples withnpx tsx. -
Register once at startup
register()sets up two OTLP exporters: traces for the agent spans and logs for the evaluation events. With no endpoint it sends tohttp://localhost:4318, the stack’s collector.agent_nameandagent_versionare stamped on every result, so you can group results by agent version.from opensearch_genai_observability_sdk_py import registerprovider = register(service_name="quickstart-agent",agent_name="quickstart_agent",agent_version="v1",)import { context } from "@opentelemetry/api";import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";import { register } from "@opensearch-project/genai-observability-sdk-ts";// Async code keeps the active span only with a context manager.context.setGlobalContextManager(new AsyncLocalStorageContextManager().enable());const provider = await register({serviceName: "quickstart-agent",agentName: "quickstart_agent",agentVersion: "v1",});For another collector, Data Prepper or Amazon OpenSearch Ingestion, pass
endpoint=(an OTLP base URL, no/v1/traces), ortraces_endpoint=andlogs_endpoint=together, plussigv4=for OpenSearch Ingestion. See the Python and TypeScript SDK pages. -
Run an offline test suite
A test suite is data, scorers are functions, and one
evaluate()call is one run. Save this asquickstart_evals.py(orquickstart_evals.ts). The Python agent answers four questions with thev1prompt “Answer in as few words as possible.”, so a one-word answer can failanswer_length. The TypeScript mock agent answers from a table, and its last case fails on purpose."""Agent evaluations quickstart: a Strands agent on Amazon Bedrock, two scorers, one offline run.pip install "opensearch-genai-observability-sdk-py>=0.3.0" strands-agentsexport OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only,gen_ai_use_latest_invocation_tokens"""import osfrom strands import Agentfrom strands.models.bedrock import BedrockModelfrom opensearch_genai_observability_sdk_py import Op, enrich, evaluate, mark_agent_root, observe, register, scorerAGENT_NAME = "quickstart_agent"AGENT_VERSION = os.environ.get("AGENT_VERSION", "v1")PROMPTS = {"v1": "Answer in as few words as possible.", "v2": "Answer in one complete sentence."}provider = register(service_name="quickstart-agent", agent_name=AGENT_NAME, agent_version=AGENT_VERSION)def text(role: str, content: str) -> list[dict]:return [{"role": role, "parts": [{"type": "text", "content": content}]}]@observe(name=AGENT_NAME, op=Op.INVOKE_AGENT)def agent(question: str) -> str:mark_agent_root() # this span is the agent run in Agent Tracesenrich(input_messages=text("user", question), **{"gen_ai.agent.version": AGENT_VERSION})model = BedrockModel(model_id="us.anthropic.claude-haiku-4-5-20251001-v1:0", region_name="us-west-2")answer = str(Agent(name=AGENT_NAME, model=model, system_prompt=PROMPTS[AGENT_VERSION],callback_handler=None)(question)).strip()enrich(output_messages=text("assistant", answer))return answer@scorer(name="contains_expected", evaluator_type="code", description="The answer contains the expected text")def contains_expected(input, output, expected) -> bool:return expected.lower() in output.lower() # a bool is a pass / fail verdict@scorer(name="answer_length", threshold=0.5, score_min=0, score_max=1, evaluator_type="code",description="The answer is a sentence, not a single word")def answer_length(input, output, expected) -> float:return min(len(output) / 40.0, 1.0) # passes at 20 characters or moreresult = evaluate(name="quickstart-suite", # test.suite.name, stable across runstask=agent,data=[{"case_id": "opensearch", "case_name": "What is OpenSearch", "input": "What is OpenSearch?", "expected": "search"},{"case_id": "otlp", "case_name": "OTLP acronym", "input": "What does OTLP stand for?", "expected": "OpenTelemetry"},{"case_id": "france", "case_name": "Capital of France", "input": "Capital of France?", "expected": "Paris"},{"case_id": "hamlet", "case_name": "Author of Hamlet", "input": "Who wrote Hamlet?", "expected": "Shakespeare"},],scores=[contains_expected, answer_length],run_name=f"{AGENT_VERSION} {PROMPTS[AGENT_VERSION].lower().rstrip('.')}",)print("run id:", result.summary.run_id)provider.force_flush() # export spans and evaluation events before exitimport { context } from "@opentelemetry/api";import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";import { Op, defineScorer, evaluate, markAgentRoot, observe, register } from "@opensearch-project/genai-observability-sdk-ts";context.setGlobalContextManager(new AsyncLocalStorageContextManager().enable());const provider = await register({ serviceName: "quickstart-agent", agentName: "quickstart_agent", agentVersion: "v1" });const ANSWERS: Record<string, string> = {"What is OpenSearch?": "OpenSearch is an open source search and analytics suite.","What does OTLP stand for?": "OpenTelemetry Protocol.","Capital of France?": "Paris is the capital of France.","Who wrote Hamlet?": "I am not sure.",};// A plain function here; evaluate() also awaits async tasks and scorers.const agent = observe({ name: "quickstart_agent", op: Op.INVOKE_AGENT }, (question: string) => {markAgentRoot();return ANSWERS[question] ?? "I don't know.";});const containsExpected = defineScorer({ name: "contains_expected", evaluatorType: "code", evaluatorDescription: "The answer contains the expected text" },(_input, output, expected) => String(output).toLowerCase().includes(String(expected).toLowerCase()),);const answerLength = defineScorer({ name: "answer_length", threshold: 0.5, scoreMin: 0, scoreMax: 1, evaluatorType: "code" },(_input, output) => Math.min(String(output).length / 40, 1),);const result = await evaluate({name: "quickstart-suite",task: (input) => agent(String(input)),data: [{ caseId: "opensearch", caseName: "What is OpenSearch", input: "What is OpenSearch?", expected: "search" },{ caseId: "otlp", caseName: "OTLP acronym", input: "What does OTLP stand for?", expected: "OpenTelemetry" },{ caseId: "france", caseName: "Capital of France", input: "Capital of France?", expected: "Paris" },{ caseId: "hamlet", caseName: "Author of Hamlet", input: "Who wrote Hamlet?", expected: "Shakespeare" },],scores: [containsExpected, answerLength],runName: "v1 baseline",});console.log("run id:", result.summary.runId);await provider.forceFlush();Run it (
python quickstart_evals.py, ornpx tsx quickstart_evals.ts).evaluate()prints a summary with the run id. The run sends:- 4 agent traces (
invoke_agent quickstart_agent). Each root span carriestest.suite.name,test.suite.run.idandtest.case.id. - 8
gen_ai.evaluation.resultevents, one per case per scorer, each parented to its case’s agent root span, with the expected output inopensearch.test.case.expected. With the Python agent,franceusually answers “Paris.” and failsanswer_length; with the TypeScript mock,hamletfails both scorers.
Keep
case_idstable: runs are compared case by case on it. Without one, the SDK derives the id from a hash of the input, so editing a question makes a new case. - 4 agent traces (
-
Score live traffic online
Online results have no suite or run: they judge real traces. The simplest form is
score()while the agent span is active; the result joins that span. Later, an evaluator job can score stored traces by id (trace_id=,span_id=), as shown in Evaluation & Scoring."""Score live traffic online: the result joins the active agent span."""from strands import Agentfrom strands.models.bedrock import BedrockModelfrom opensearch_genai_observability_sdk_py import Op, enrich, mark_agent_root, observe, register, scoreAGENT_NAME, AGENT_VERSION = "quickstart_agent", "v2"provider = register(service_name="quickstart-agent", agent_name=AGENT_NAME, agent_version=AGENT_VERSION)def text(role: str, content: str) -> list[dict]:return [{"role": role, "parts": [{"type": "text", "content": content}]}]@observe(name=AGENT_NAME, op=Op.INVOKE_AGENT)def agent(question: str) -> str:mark_agent_root()enrich(input_messages=text("user", question), **{"gen_ai.agent.version": AGENT_VERSION})model = BedrockModel(model_id="us.anthropic.claude-haiku-4-5-20251001-v1:0", region_name="us-west-2")answer = str(Agent(name=AGENT_NAME, model=model, system_prompt="Answer in one complete sentence.",callback_handler=None)(question)).strip()enrich(output_messages=text("assistant", answer))score( # inline: the active span (this agent run) is judged"answer_mentions_city",1.0 if "Paris" in answer else 0.0,threshold=1,evaluator="city_check",evaluator_type="code",)return answerprint(agent("What is the capital of France?"))provider.force_flush()import { context } from "@opentelemetry/api";import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";import { Op, markAgentRoot, register, score, withObserve } from "@opensearch-project/genai-observability-sdk-ts";// Without a context manager, score() cannot see the active span and the result is not linked to the trace.context.setGlobalContextManager(new AsyncLocalStorageContextManager().enable());const provider = await register({ serviceName: "quickstart-agent", agentName: "quickstart_agent", agentVersion: "v1" });await withObserve("quickstart_agent", { op: Op.INVOKE_AGENT }, async () => {markAgentRoot();const answer = "Paris is the capital of France.";score({name: "answer_mentions_city",value: answer.includes("Paris") ? 1 : 0,threshold: 1,evaluator: "city_check",evaluatorType: "code",});return answer;});await provider.forceFlush(); -
Check that the results arrived (optional)
Data Prepper batches writes, so results usually show up within 60 to 90 seconds:
Terminal window curl -sk -u "admin:$OPENSEARCH_PASSWORD" \'https://localhost:9200/otel-v1-genai-evaluation/_search?size=0' \-H 'Content-Type: application/json' \-d '{"query":{"term":{"attributes.gen_ai.agent.name":"quickstart_agent"}},"aggs":{"runs":{"terms":{"field":"attributes.test.suite.run.id","missing":"(online)"}}}}'You see one bucket per run with 8 results each (one run per language you ran) and an
(online)bucket with 1 per online run. If the results are inlogs-otel-v1-*instead, the route is missing: see Route results with Data Prepper. -
Query the results
The results are documents in
otel-v1-genai-evaluation*. Query them with PPL, in Discover or Query Workbench in OpenSearch Dashboards, or with the PPL API. Pass rate per evaluation and run:Terminal window curl -sk -u "admin:$OPENSEARCH_PASSWORD" https://localhost:9200/_plugins/_ppl \-H 'Content-Type: application/json' \-d '{"query": "source = otel-v1-genai-evaluation* | where `attributes.test.suite.name` = '"'"'quickstart-suite'"'"' | stats count() by `attributes.test.suite.run.id`, `attributes.gen_ai.evaluation.name`, `attributes.gen_ai.evaluation.score.label`"}'The cases of one run, with their expected output and the trace each result judged:
source = otel-v1-genai-evaluation*| where `attributes.test.suite.run.id` = '<run id>'| fields `attributes.test.case.id`, `attributes.gen_ai.evaluation.name`, `attributes.gen_ai.evaluation.score.value`, `attributes.gen_ai.evaluation.score.label`, `attributes.opensearch.test.case.expected`, traceId, spanIdThe agent’s input and output are not on the results. Open a result’s
traceIdin Agent Traces to see the case’s agent run. -
Run again and compare
Python: run the suite again with the
v2prompt (“Answer in one complete sentence.”):Terminal window AGENT_VERSION=v2 python quickstart_evals.pyTypeScript: change the mock answer to
"Who wrote Hamlet?"to"William Shakespeare wrote Hamlet.", setagentVersion: "v2"andrunName: "v2 fixed hamlet", and run the suite again.Then put the two runs side by side, case by case. Cases match by
test.case.id:source = otel-v1-genai-evaluation*| where `attributes.test.suite.name` = 'quickstart-suite' and `attributes.gen_ai.evaluation.name` = 'answer_length'| stats count() by `attributes.test.case.id`, `attributes.gen_ai.agent.version`, `attributes.gen_ai.evaluation.score.label`In the tested Python run,
francefailedanswer_lengthwithv1(“Paris.”) and passed withv2. A real model’s answers vary, so your cases can differ.
Prompt for your coding agent
Section titled “Prompt for your coding agent”Paste this into your coding agent, in your agent’s repository, to instrument it and add evaluations the same way:
# Instrument this agent for OpenSearch Agent Traces and agent evaluations
Instrument the agent app in this repository so every run shows up in OpenSearch **Agent Traces** and its quality scores land in the evaluation index (`otel-v1-genai-evaluation`). Use the OpenSearch GenAI Observability SDK and the OpenTelemetry GenAI semantic conventions. Read the SDK's README and `docs/agent-evaluations.md` first, and use only APIs that exist in the installed version.
Values: OTLP endpoint `<OTLP_ENDPOINT>` (base URL, for example `http://localhost:4318`), service `<SERVICE_NAME>`, agent `<AGENT_NAME>`, version `<AGENT_VERSION>`, test suite `<SUITE_NAME>`.
## 1. Install and register- Python needs SDK 0.3.0 and TypeScript SDK 0.2.0 or later. Python: `pip install "opensearch-genai-observability-sdk-py>=0.3.0"`. TypeScript: `npm install @opensearch-project/genai-observability-sdk-ts@^0.2.0 @opentelemetry/context-async-hooks`. If npm fails with `ERESOLVE` against the framework's OpenTelemetry 2.x peers, install with `--legacy-peer-deps`, keep that in `.npmrc`, keep `@opentelemetry/api` on 1.9.x, run the unchanged app once and add back any peer it now misses (Strands: `@modelcontextprotocol/client`).- Call `register()` once at startup, before the agent is built: Python `register(service_name=, agent_name=, agent_version=)`; TypeScript `await register({ serviceName, agentName, agentVersion })`, preceded by `context.setGlobalContextManager(new AsyncLocalStorageContextManager().enable())` so async spans nest.- Endpoint from the environment, never hardcoded: `OTEL_EXPORTER_OTLP_ENDPOINT=<OTLP_ENDPOINT>` (the SDK appends `/v1/traces` and `/v1/logs`). Separate pipelines: set both traces and logs URLs (`traces_endpoint` + `logs_endpoint`, or both `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`; Data Prepper defaults 21890 and 21892). Amazon OpenSearch Ingestion (OSIS): HTTPS only, `sigv4={"service": "osis", "region": ...}` behind a config switch, `compression` equal to the pipeline's. Evaluation results travel as OTLP **logs**, so the logs path must work too.- Do not call the framework's own exporter setup (`StrandsTelemetry().setup_otlp_exporter()`, Strands `setupTracer()`): `register()` owns the global tracer provider. Flush before a script exits: `provider.force_flush()` / `await provider.forceFlush()`.
## 2. Traces, spans and sessions- One trace per agent turn (one user request). Its **root span is the agent span**: `gen_ai.operation.name=invoke_agent`, `gen_ai.agent.name`, `gen_ai.agent.version`, `gen_ai.conversation.id` (stable for the whole conversation; it builds Sessions), `gen_ai.input.messages` (this turn's user message) / `gen_ai.output.messages` in the GenAI parts format (pass objects; the SDK serializes them) `[{"role": "user", "parts": [{"type": "text", "content": "..."}]}]`, and the token usage of **this turn only**.- Wrap each turn: Python `@observe(name="<AGENT_NAME>", op=Op.INVOKE_AGENT)` on a function or method that takes the user input, TypeScript `withObserve("<AGENT_NAME>", { op: Op.INVOKE_AGENT }, async (span) => ...)`. Inside it call `mark_agent_root()` / `markAgentRoot()` and `enrich(session_id=<conversation id>, input_messages=..., output_messages=..., **{"gen_ai.agent.version": "<AGENT_VERSION>"})` (TypeScript `enrich({ sessionId, inputMessages, outputMessages, "gen_ai.agent.version": ... })`); In Python, `enrich` messages replace the decorator's automatic argument capture. Return the agent span's hex `trace_id` / `span_id` with the answer, for evaluations.- Let framework auto-instrumentation produce the inner spans: Strands, LangChain, OpenAI and others emit `invoke_agent`, `chat` (with `gen_ai.usage.input_tokens` / `output_tokens`, `gen_ai.request.model`) and `execute_tool` spans through the global provider. Give the framework agent the same name `<AGENT_NAME>` and pass it the conversation id (Strands `trace_attributes` / `traceAttributes` with `gen_ai.conversation.id`). For Strands set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only` (messages as span attributes) and, in Python, also `gen_ai_use_latest_invocation_tokens`, before the first agent is built (set it in code at import so every entry point, including the CI runner, gets it), and also set this turn's usage on your agent span (`enrich(input_tokens=, output_tokens=)` from `result.metrics.latest_agent_invocation.usage`; only Data Prepper back-fills a root that has none); Strands TypeScript reports lifetime token totals, so set this turn's usage on your agent span (`enrich({ inputTokens, outputTokens })` from `result.metrics.agentInvocations.at(-1).usage`). Wrap with `observe(op=Op.EXECUTE_TOOL)` / `observe(op=Op.CHAT)` + `enrich(model=, input_tokens=, output_tokens=)` only what is not traced.- Root rules: if the agent runs inside an HTTP handler or job span, mark the agent span with `mark_agent_root()` (span attribute `derived.gen_ai.is_root=true`). Never continue an incoming `traceparent` whose parent span is not exported to the same OpenSearch: start a new root instead (TypeScript: run the turn inside `context.with(ROOT_CONTEXT, ...)`; Python: `token = context.attach(Context())` before the turn, `context.detach(token)` after), or the trace has no root.- Never put secrets or credentials in attributes.
## 3. Evaluations (each score is one `gen_ai.evaluation.result` log event)- **Online** (live traffic: after each turn or in a sampled job): `score(name, value, trace_id=, span_id=` of the agent span`, conversation_id=, evaluator=, evaluator_type="code" | "llm_judge" | "human", evaluator_description=, threshold=, direction=` (or `label="pass"/"fail"`)`, score_min=, score_max=)`; TypeScript `score({ name, value, traceId, spanId, conversationId, evaluator, evaluatorType, ... })`, always with `traceId` / `spanId`.- **LLM judges**: add `judge_model`, `judge_prompt_name`, `judge_prompt_version` (camelCase in TypeScript); never put the judge's system prompt on results (no `judge_system_instructions`). Tracing the judge is optional; if you trace it, never as your agent (no `invoke_agent`, not `<AGENT_NAME>`), and pass its trace id as `judge_trace_id`.- **Offline** suite: a dataset file of cases with a stable case id, case name, input and expected output. Python and TypeScript: `evaluate(name="<SUITE_NAME>", task=, data=[{"case_id", "case_name", "input", "expected"}], scores=[...], run_name=, run_description=, agent_name=, agent_version=)` with scorers declared by `@scorer(name=, evaluator_type=, description=, threshold=, score_min=, score_max=, judge_model=, judge_prompt_name=, judge_prompt_version=)` (TypeScript `await evaluate({...})` with `defineScorer()`; it accepts async tasks; Python async agents: `asyncio.run` inside the task). A declaration applies only to results whose `evaluator` is unset or the scorer's own; a wrapper of a library metric declares that metric as its `evaluator`. Cases you run yourself or import: `Benchmark`; Python `Benchmark(name=, run_name=, agent_name=, agent_version=)` and `log(case_id=, case_name=, input=, output=, expected=, trace_id=, span_id=, scores=[EvalScore(...)])`; TypeScript `new Benchmark("<SUITE_NAME>", { runName, agentName, agentVersion })` and `bench.log({ caseId, caseName, input, output, expected, traceId, spanId, scores })`; await each case yourself, then `close()`. Results get `test.suite.name`, `test.suite.run.id`, `test.case.id` / `test.case.name`, `opensearch.test.case.expected`.- **CI**: expose the suite as one command that exits non-zero when an evaluation's mean is below its bar or a case errors. For GitHub Actions, adapt the Python SDK's `examples/github-action` (`action.yml`, `example-workflow.yml`; its `run_suite.py` calls `register()` itself, so the suite module must not register on import; its default SDK install spec is `opensearch-genai-observability-sdk-py>=0.3.0`, and pin the action to a full commit SHA); in TypeScript write the equivalent workflow step. The SDK adds `cicd.pipeline.*` in CI itself.
## 4. Verify, then reportRun a two-turn conversation (score each turn online) and one offline suite run; flush; poll for up to 2 minutes (spans and evaluation events are indexed separately and either can lag); query OpenSearch (`_plugins/_ppl` or `_search`):- `otel-v1-apm-span-*`: each trace has one root (`parentSpanId = ""`): your `invoke_agent` span with `attributes.gen_ai.agent.name`, `.agent.version`, `.conversation.id`, `.input.messages`, `.output.messages`, `.usage.*` and `derived.gen_ai.is_root`; below it `chat` spans with `gen_ai.usage.*` and `execute_tool` spans in the same trace; both turns share one conversation id, and turn 2's root tokens are not turn 1 + turn 2.- `otel-v1-genai-evaluation*` (a write alias over `-000001`, or one index on OpenSearch Serverless): `eventName = gen_ai.evaluation.result` documents for the online scores (no `test.suite.run.id`) and the offline run (`test.suite.name`, `test.suite.run.id`, `test.case.id`), whose `traceId` / `spanId` match agent root spans, and no `gen_ai.system_instructions`. If they land in `logs-otel-v1-*`, the pipeline lacks the `eventName` route.- If OpenSearch Dashboards is reachable: Agent Traces (Traces, Spans, Sessions tabs) lists each turn by its agent span.
Checklist (confirm each with evidence):- [ ] SDK installed; `register()` called once before the agent is built; endpoint from env; flush on exit.- [ ] Each turn = one trace whose root is the `invoke_agent` agent span, marked as agent root, with agent name, version, conversation id, input/output messages and per-turn tokens.- [ ] Framework `chat` / `execute_tool` spans nest under it, with token usage and model.- [ ] Online `score()` results attached to the agent span, with evaluator name, type, description and rule.- [ ] Offline run with suite name, run id, stable case ids/names and expected output; judge model and prompt name/version, no judge system prompt.- [ ] A CI command or workflow that runs the suite and fails on regression.- [ ] Spans and evaluation events confirmed in OpenSearch; app behavior unchanged.Next steps
Section titled “Next steps”- Evaluation & Scoring: every option of
score(),evaluate()andBenchmark, LLM judges and verdict rules. - Run evaluations in CI: run the suite on every pull request and fail on regressions.
- Data Model & Ingestion: every attribute, the index template, and Data Prepper routing.
- Evaluation Integrations: keep DeepEval, RAGAS, MLflow or pytest and send their results.