Skip to content

Quickstart

This quickstart takes you from an empty stack to evaluation results in OpenSearch. You run a small offline test suite against a real agent (a Strands agent on Amazon Bedrock), score one live trace online, query the results, then change the agent and compare the two runs.

flowchart LR
    A["Your agent<br/>@observe / withObserve"] -->|"OTLP /v1/traces"| C["OTel Collector"]
    E["evaluate() / Benchmark / score()"] -->|"OTLP /v1/logs<br/>gen_ai.evaluation.result"| C
    C --> D["Data Prepper"]
    D -->|"spans"| S[("otel-v1-apm-span-*")]
    D -->|"eventName == gen_ai.evaluation.result"| R[("otel-v1-genai-evaluation")]
    D -->|"other logs"| L[("logs-otel-v1-*")]
    S --- R

Each score is one gen_ai.evaluation.result log event, linked to the agent trace it judged: the result and the agent span share traceId and spanId. The Data Model & Ingestion page describes the event and the index in full.

  • Docker with Docker Compose
  • Python 3.10+ or Node.js 20+
  • For the Python agent: AWS credentials (the default chain) with access to a Claude model on Amazon Bedrock. Any other Strands model provider works the same way. The TypeScript tab uses a mock agent and needs no model.
  1. Start the stack

    Terminal window
    git clone https://github.com/opensearch-project/observability-stack.git
    cd observability-stack
    docker compose up -d

    For evaluations, the stack:

    • runs the one-shot genai-evaluation-index-init job before Data Prepper starts. It creates the ISM rollover policy, the index template and the first index otel-v1-genai-evaluation-000001 behind the otel-v1-genai-evaluation write alias. See The evaluation index.
    • runs a Data Prepper 2.18.0 snapshot (2.17.0 or later is needed), which routes gen_ai.evaluation.result events, and only those, to the write alias. All other logs still go to logs-otel-v1-*.

    The OTel Collector listens on 4318 (OTLP/HTTP) and 4317 (gRPC), OpenSearch on 9200, Dashboards on 5601. The admin password is OPENSEARCH_PASSWORD in .env.

    To use your own Data Prepper or an AWS deployment instead, set up the route and the index from Route results with Data Prepper or AWS Managed Services.

  2. Install the SDK

    Terminal window
    python -m venv .venv && source .venv/bin/activate
    pip install "opensearch-genai-observability-sdk-py>=0.3.0" strands-agents
    export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only,gen_ai_use_latest_invocation_tokens
  3. Register once at startup

    register() sets up two OTLP exporters: traces for the agent spans and logs for the evaluation events. With no endpoint it sends to http://localhost:4318, the stack’s collector. agent_name and agent_version are stamped on every result, so you can group results by agent version.

    from opensearch_genai_observability_sdk_py import register
    provider = register(
    service_name="quickstart-agent",
    agent_name="quickstart_agent",
    agent_version="v1",
    )

    For another collector, Data Prepper or Amazon OpenSearch Ingestion, pass endpoint= (an OTLP base URL, no /v1/traces), or traces_endpoint= and logs_endpoint= together, plus sigv4= for OpenSearch Ingestion. See the Python and TypeScript SDK pages.

  4. Run an offline test suite

    A test suite is data, scorers are functions, and one evaluate() call is one run. Save this as quickstart_evals.py (or quickstart_evals.ts). The Python agent answers four questions with the v1 prompt “Answer in as few words as possible.”, so a one-word answer can fail answer_length. The TypeScript mock agent answers from a table, and its last case fails on purpose.

    """Agent evaluations quickstart: a Strands agent on Amazon Bedrock, two scorers, one offline run.
    pip install "opensearch-genai-observability-sdk-py>=0.3.0" strands-agents
    export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only,gen_ai_use_latest_invocation_tokens
    """
    import os
    from strands import Agent
    from strands.models.bedrock import BedrockModel
    from opensearch_genai_observability_sdk_py import Op, enrich, evaluate, mark_agent_root, observe, register, scorer
    AGENT_NAME = "quickstart_agent"
    AGENT_VERSION = os.environ.get("AGENT_VERSION", "v1")
    PROMPTS = {"v1": "Answer in as few words as possible.", "v2": "Answer in one complete sentence."}
    provider = register(service_name="quickstart-agent", agent_name=AGENT_NAME, agent_version=AGENT_VERSION)
    def text(role: str, content: str) -> list[dict]:
    return [{"role": role, "parts": [{"type": "text", "content": content}]}]
    @observe(name=AGENT_NAME, op=Op.INVOKE_AGENT)
    def agent(question: str) -> str:
    mark_agent_root() # this span is the agent run in Agent Traces
    enrich(input_messages=text("user", question), **{"gen_ai.agent.version": AGENT_VERSION})
    model = BedrockModel(model_id="us.anthropic.claude-haiku-4-5-20251001-v1:0", region_name="us-west-2")
    answer = str(Agent(name=AGENT_NAME, model=model, system_prompt=PROMPTS[AGENT_VERSION],
    callback_handler=None)(question)).strip()
    enrich(output_messages=text("assistant", answer))
    return answer
    @scorer(name="contains_expected", evaluator_type="code", description="The answer contains the expected text")
    def contains_expected(input, output, expected) -> bool:
    return expected.lower() in output.lower() # a bool is a pass / fail verdict
    @scorer(name="answer_length", threshold=0.5, score_min=0, score_max=1, evaluator_type="code",
    description="The answer is a sentence, not a single word")
    def answer_length(input, output, expected) -> float:
    return min(len(output) / 40.0, 1.0) # passes at 20 characters or more
    result = evaluate(
    name="quickstart-suite", # test.suite.name, stable across runs
    task=agent,
    data=[
    {"case_id": "opensearch", "case_name": "What is OpenSearch", "input": "What is OpenSearch?", "expected": "search"},
    {"case_id": "otlp", "case_name": "OTLP acronym", "input": "What does OTLP stand for?", "expected": "OpenTelemetry"},
    {"case_id": "france", "case_name": "Capital of France", "input": "Capital of France?", "expected": "Paris"},
    {"case_id": "hamlet", "case_name": "Author of Hamlet", "input": "Who wrote Hamlet?", "expected": "Shakespeare"},
    ],
    scores=[contains_expected, answer_length],
    run_name=f"{AGENT_VERSION} {PROMPTS[AGENT_VERSION].lower().rstrip('.')}",
    )
    print("run id:", result.summary.run_id)
    provider.force_flush() # export spans and evaluation events before exit

    Run it (python quickstart_evals.py, or npx tsx quickstart_evals.ts). evaluate() prints a summary with the run id. The run sends:

    • 4 agent traces (invoke_agent quickstart_agent). Each root span carries test.suite.name, test.suite.run.id and test.case.id.
    • 8 gen_ai.evaluation.result events, one per case per scorer, each parented to its case’s agent root span, with the expected output in opensearch.test.case.expected. With the Python agent, france usually answers “Paris.” and fails answer_length; with the TypeScript mock, hamlet fails both scorers.

    Keep case_id stable: runs are compared case by case on it. Without one, the SDK derives the id from a hash of the input, so editing a question makes a new case.

  5. Score live traffic online

    Online results have no suite or run: they judge real traces. The simplest form is score() while the agent span is active; the result joins that span. Later, an evaluator job can score stored traces by id (trace_id=, span_id=), as shown in Evaluation & Scoring.

    """Score live traffic online: the result joins the active agent span."""
    from strands import Agent
    from strands.models.bedrock import BedrockModel
    from opensearch_genai_observability_sdk_py import Op, enrich, mark_agent_root, observe, register, score
    AGENT_NAME, AGENT_VERSION = "quickstart_agent", "v2"
    provider = register(service_name="quickstart-agent", agent_name=AGENT_NAME, agent_version=AGENT_VERSION)
    def text(role: str, content: str) -> list[dict]:
    return [{"role": role, "parts": [{"type": "text", "content": content}]}]
    @observe(name=AGENT_NAME, op=Op.INVOKE_AGENT)
    def agent(question: str) -> str:
    mark_agent_root()
    enrich(input_messages=text("user", question), **{"gen_ai.agent.version": AGENT_VERSION})
    model = BedrockModel(model_id="us.anthropic.claude-haiku-4-5-20251001-v1:0", region_name="us-west-2")
    answer = str(Agent(name=AGENT_NAME, model=model, system_prompt="Answer in one complete sentence.",
    callback_handler=None)(question)).strip()
    enrich(output_messages=text("assistant", answer))
    score( # inline: the active span (this agent run) is judged
    "answer_mentions_city",
    1.0 if "Paris" in answer else 0.0,
    threshold=1,
    evaluator="city_check",
    evaluator_type="code",
    )
    return answer
    print(agent("What is the capital of France?"))
    provider.force_flush()
  6. Check that the results arrived (optional)

    Data Prepper batches writes, so results usually show up within 60 to 90 seconds:

    Terminal window
    curl -sk -u "admin:$OPENSEARCH_PASSWORD" \
    'https://localhost:9200/otel-v1-genai-evaluation/_search?size=0' \
    -H 'Content-Type: application/json' \
    -d '{"query":{"term":{"attributes.gen_ai.agent.name":"quickstart_agent"}},"aggs":{"runs":{"terms":{"field":"attributes.test.suite.run.id","missing":"(online)"}}}}'

    You see one bucket per run with 8 results each (one run per language you ran) and an (online) bucket with 1 per online run. If the results are in logs-otel-v1-* instead, the route is missing: see Route results with Data Prepper.

  7. Query the results

    The results are documents in otel-v1-genai-evaluation*. Query them with PPL, in Discover or Query Workbench in OpenSearch Dashboards, or with the PPL API. Pass rate per evaluation and run:

    Terminal window
    curl -sk -u "admin:$OPENSEARCH_PASSWORD" https://localhost:9200/_plugins/_ppl \
    -H 'Content-Type: application/json' \
    -d '{"query": "source = otel-v1-genai-evaluation* | where `attributes.test.suite.name` = '"'"'quickstart-suite'"'"' | stats count() by `attributes.test.suite.run.id`, `attributes.gen_ai.evaluation.name`, `attributes.gen_ai.evaluation.score.label`"}'

    The cases of one run, with their expected output and the trace each result judged:

    source = otel-v1-genai-evaluation*
    | where `attributes.test.suite.run.id` = '<run id>'
    | fields `attributes.test.case.id`, `attributes.gen_ai.evaluation.name`, `attributes.gen_ai.evaluation.score.value`, `attributes.gen_ai.evaluation.score.label`, `attributes.opensearch.test.case.expected`, traceId, spanId

    The agent’s input and output are not on the results. Open a result’s traceId in Agent Traces to see the case’s agent run.

  8. Run again and compare

    Python: run the suite again with the v2 prompt (“Answer in one complete sentence.”):

    Terminal window
    AGENT_VERSION=v2 python quickstart_evals.py

    TypeScript: change the mock answer to "Who wrote Hamlet?" to "William Shakespeare wrote Hamlet.", set agentVersion: "v2" and runName: "v2 fixed hamlet", and run the suite again.

    Then put the two runs side by side, case by case. Cases match by test.case.id:

    source = otel-v1-genai-evaluation*
    | where `attributes.test.suite.name` = 'quickstart-suite' and `attributes.gen_ai.evaluation.name` = 'answer_length'
    | stats count() by `attributes.test.case.id`, `attributes.gen_ai.agent.version`, `attributes.gen_ai.evaluation.score.label`

    In the tested Python run, france failed answer_length with v1 (“Paris.”) and passed with v2. A real model’s answers vary, so your cases can differ.

Paste this into your coding agent, in your agent’s repository, to instrument it and add evaluations the same way:

Prompt for your coding agent
# Instrument this agent for OpenSearch Agent Traces and agent evaluations
Instrument the agent app in this repository so every run shows up in OpenSearch **Agent Traces** and its quality scores land in the evaluation index (`otel-v1-genai-evaluation`). Use the OpenSearch GenAI Observability SDK and the OpenTelemetry GenAI semantic conventions. Read the SDK's README and `docs/agent-evaluations.md` first, and use only APIs that exist in the installed version.
Values: OTLP endpoint `<OTLP_ENDPOINT>` (base URL, for example `http://localhost:4318`), service `<SERVICE_NAME>`, agent `<AGENT_NAME>`, version `<AGENT_VERSION>`, test suite `<SUITE_NAME>`.
## 1. Install and register
- Python needs SDK 0.3.0 and TypeScript SDK 0.2.0 or later. Python: `pip install "opensearch-genai-observability-sdk-py>=0.3.0"`. TypeScript: `npm install @opensearch-project/genai-observability-sdk-ts@^0.2.0 @opentelemetry/context-async-hooks`. If npm fails with `ERESOLVE` against the framework's OpenTelemetry 2.x peers, install with `--legacy-peer-deps`, keep that in `.npmrc`, keep `@opentelemetry/api` on 1.9.x, run the unchanged app once and add back any peer it now misses (Strands: `@modelcontextprotocol/client`).
- Call `register()` once at startup, before the agent is built: Python `register(service_name=, agent_name=, agent_version=)`; TypeScript `await register({ serviceName, agentName, agentVersion })`, preceded by `context.setGlobalContextManager(new AsyncLocalStorageContextManager().enable())` so async spans nest.
- Endpoint from the environment, never hardcoded: `OTEL_EXPORTER_OTLP_ENDPOINT=<OTLP_ENDPOINT>` (the SDK appends `/v1/traces` and `/v1/logs`). Separate pipelines: set both traces and logs URLs (`traces_endpoint` + `logs_endpoint`, or both `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`; Data Prepper defaults 21890 and 21892). Amazon OpenSearch Ingestion (OSIS): HTTPS only, `sigv4={"service": "osis", "region": ...}` behind a config switch, `compression` equal to the pipeline's. Evaluation results travel as OTLP **logs**, so the logs path must work too.
- Do not call the framework's own exporter setup (`StrandsTelemetry().setup_otlp_exporter()`, Strands `setupTracer()`): `register()` owns the global tracer provider. Flush before a script exits: `provider.force_flush()` / `await provider.forceFlush()`.
## 2. Traces, spans and sessions
- One trace per agent turn (one user request). Its **root span is the agent span**: `gen_ai.operation.name=invoke_agent`, `gen_ai.agent.name`, `gen_ai.agent.version`, `gen_ai.conversation.id` (stable for the whole conversation; it builds Sessions), `gen_ai.input.messages` (this turn's user message) / `gen_ai.output.messages` in the GenAI parts format (pass objects; the SDK serializes them) `[{"role": "user", "parts": [{"type": "text", "content": "..."}]}]`, and the token usage of **this turn only**.
- Wrap each turn: Python `@observe(name="<AGENT_NAME>", op=Op.INVOKE_AGENT)` on a function or method that takes the user input, TypeScript `withObserve("<AGENT_NAME>", { op: Op.INVOKE_AGENT }, async (span) => ...)`. Inside it call `mark_agent_root()` / `markAgentRoot()` and `enrich(session_id=<conversation id>, input_messages=..., output_messages=..., **{"gen_ai.agent.version": "<AGENT_VERSION>"})` (TypeScript `enrich({ sessionId, inputMessages, outputMessages, "gen_ai.agent.version": ... })`); In Python, `enrich` messages replace the decorator's automatic argument capture. Return the agent span's hex `trace_id` / `span_id` with the answer, for evaluations.
- Let framework auto-instrumentation produce the inner spans: Strands, LangChain, OpenAI and others emit `invoke_agent`, `chat` (with `gen_ai.usage.input_tokens` / `output_tokens`, `gen_ai.request.model`) and `execute_tool` spans through the global provider. Give the framework agent the same name `<AGENT_NAME>` and pass it the conversation id (Strands `trace_attributes` / `traceAttributes` with `gen_ai.conversation.id`). For Strands set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental,gen_ai_span_attributes_only` (messages as span attributes) and, in Python, also `gen_ai_use_latest_invocation_tokens`, before the first agent is built (set it in code at import so every entry point, including the CI runner, gets it), and also set this turn's usage on your agent span (`enrich(input_tokens=, output_tokens=)` from `result.metrics.latest_agent_invocation.usage`; only Data Prepper back-fills a root that has none); Strands TypeScript reports lifetime token totals, so set this turn's usage on your agent span (`enrich({ inputTokens, outputTokens })` from `result.metrics.agentInvocations.at(-1).usage`). Wrap with `observe(op=Op.EXECUTE_TOOL)` / `observe(op=Op.CHAT)` + `enrich(model=, input_tokens=, output_tokens=)` only what is not traced.
- Root rules: if the agent runs inside an HTTP handler or job span, mark the agent span with `mark_agent_root()` (span attribute `derived.gen_ai.is_root=true`). Never continue an incoming `traceparent` whose parent span is not exported to the same OpenSearch: start a new root instead (TypeScript: run the turn inside `context.with(ROOT_CONTEXT, ...)`; Python: `token = context.attach(Context())` before the turn, `context.detach(token)` after), or the trace has no root.
- Never put secrets or credentials in attributes.
## 3. Evaluations (each score is one `gen_ai.evaluation.result` log event)
- **Online** (live traffic: after each turn or in a sampled job): `score(name, value, trace_id=, span_id=` of the agent span`, conversation_id=, evaluator=, evaluator_type="code" | "llm_judge" | "human", evaluator_description=, threshold=, direction=` (or `label="pass"/"fail"`)`, score_min=, score_max=)`; TypeScript `score({ name, value, traceId, spanId, conversationId, evaluator, evaluatorType, ... })`, always with `traceId` / `spanId`.
- **LLM judges**: add `judge_model`, `judge_prompt_name`, `judge_prompt_version` (camelCase in TypeScript); never put the judge's system prompt on results (no `judge_system_instructions`). Tracing the judge is optional; if you trace it, never as your agent (no `invoke_agent`, not `<AGENT_NAME>`), and pass its trace id as `judge_trace_id`.
- **Offline** suite: a dataset file of cases with a stable case id, case name, input and expected output. Python and TypeScript: `evaluate(name="<SUITE_NAME>", task=, data=[{"case_id", "case_name", "input", "expected"}], scores=[...], run_name=, run_description=, agent_name=, agent_version=)` with scorers declared by `@scorer(name=, evaluator_type=, description=, threshold=, score_min=, score_max=, judge_model=, judge_prompt_name=, judge_prompt_version=)` (TypeScript `await evaluate({...})` with `defineScorer()`; it accepts async tasks; Python async agents: `asyncio.run` inside the task). A declaration applies only to results whose `evaluator` is unset or the scorer's own; a wrapper of a library metric declares that metric as its `evaluator`. Cases you run yourself or import: `Benchmark`; Python `Benchmark(name=, run_name=, agent_name=, agent_version=)` and `log(case_id=, case_name=, input=, output=, expected=, trace_id=, span_id=, scores=[EvalScore(...)])`; TypeScript `new Benchmark("<SUITE_NAME>", { runName, agentName, agentVersion })` and `bench.log({ caseId, caseName, input, output, expected, traceId, spanId, scores })`; await each case yourself, then `close()`. Results get `test.suite.name`, `test.suite.run.id`, `test.case.id` / `test.case.name`, `opensearch.test.case.expected`.
- **CI**: expose the suite as one command that exits non-zero when an evaluation's mean is below its bar or a case errors. For GitHub Actions, adapt the Python SDK's `examples/github-action` (`action.yml`, `example-workflow.yml`; its `run_suite.py` calls `register()` itself, so the suite module must not register on import; its default SDK install spec is `opensearch-genai-observability-sdk-py>=0.3.0`, and pin the action to a full commit SHA); in TypeScript write the equivalent workflow step. The SDK adds `cicd.pipeline.*` in CI itself.
## 4. Verify, then report
Run a two-turn conversation (score each turn online) and one offline suite run; flush; poll for up to 2 minutes (spans and evaluation events are indexed separately and either can lag); query OpenSearch (`_plugins/_ppl` or `_search`):
- `otel-v1-apm-span-*`: each trace has one root (`parentSpanId = ""`): your `invoke_agent` span with `attributes.gen_ai.agent.name`, `.agent.version`, `.conversation.id`, `.input.messages`, `.output.messages`, `.usage.*` and `derived.gen_ai.is_root`; below it `chat` spans with `gen_ai.usage.*` and `execute_tool` spans in the same trace; both turns share one conversation id, and turn 2's root tokens are not turn 1 + turn 2.
- `otel-v1-genai-evaluation*` (a write alias over `-000001`, or one index on OpenSearch Serverless): `eventName = gen_ai.evaluation.result` documents for the online scores (no `test.suite.run.id`) and the offline run (`test.suite.name`, `test.suite.run.id`, `test.case.id`), whose `traceId` / `spanId` match agent root spans, and no `gen_ai.system_instructions`. If they land in `logs-otel-v1-*`, the pipeline lacks the `eventName` route.
- If OpenSearch Dashboards is reachable: Agent Traces (Traces, Spans, Sessions tabs) lists each turn by its agent span.
Checklist (confirm each with evidence):
- [ ] SDK installed; `register()` called once before the agent is built; endpoint from env; flush on exit.
- [ ] Each turn = one trace whose root is the `invoke_agent` agent span, marked as agent root, with agent name, version, conversation id, input/output messages and per-turn tokens.
- [ ] Framework `chat` / `execute_tool` spans nest under it, with token usage and model.
- [ ] Online `score()` results attached to the agent span, with evaluator name, type, description and rule.
- [ ] Offline run with suite name, run id, stable case ids/names and expected output; judge model and prompt name/version, no judge system prompt.
- [ ] A CI command or workflow that runs the suite and fails on regression.
- [ ] Spans and evaluation events confirmed in OpenSearch; app behavior unchanged.