
How to see what your AI agents actually did, with AgentCore, OpenTelemetry and CloudWatch
In this article
- Why Logs Alone Fail for LLM Apps
- What a Good Agent Trace Contains
- Setting Up LLM Observability on AWS With AgentCore
- Agents Running Outside AgentCore
- Sampling, Privacy and Cost Trade-Offs
- From Traces to Answers: Dashboards, Alarms and Evaluations
- How Signiance Helps
A customer asks your support agent about refunds. The agent answers confidently, quotes last year’s policy, and the request finishes in 2.4 seconds with an HTTP 200. Your logs show nothing wrong. This is the gap LLM observability closes: logs tell you that a request happened, while traces tell you what the agent actually did, which model it called, which tool it picked, what came back and how many tokens it burned along the way.
For startups moving agents from pilot to production, that difference decides whether a bad answer takes ten minutes or two days to explain. This guide covers what to capture, how to set it up on AWS with Amazon Bedrock AgentCore and OpenTelemetry, and the sampling and privacy trade-offs to settle before you turn everything on.
Why Logs Alone Fail for LLM Apps
A traditional web request is mostly deterministic. If the status code is fine and latency is normal, the user probably got what they wanted. An agent breaks that assumption in three ways.
One request is many steps. A single user turn can trigger a planning call, two tool calls, a retrieval step and a final model call. Flat log lines from each step are hard to stitch back together, especially under concurrent traffic.
Failures are quiet. The worst agent bugs never throw an exception. The agent picks the wrong tool, retrieves a stale document or loops one extra time. Everything returns 200, and the only symptom is a wrong answer or a bigger bill.
Cost lives inside the request. Token usage varies by prompt, context size and number of reasoning cycles. Without per-step token counts, you cannot tell which feature or customer is driving spend.
A trace solves this by giving every step a span with a parent, a start and end time, attributes and a status. Put together, the spans form a timeline of one conversation turn that you can read like a call stack.
What a Good Agent Trace Contains
The industry is converging on the OpenTelemetry GenAI semantic conventions as the shared vocabulary for LLM tracing. They are still at Development status, which means names can change between releases, so pin your instrumentation versions. The shape is already useful:
- Agent spans such as
invoke_agent support-agent, carryinggen_ai.agent.nameand the conversation ID. - Inference spans named like
chat <model>, withgen_ai.provider.name,gen_ai.request.model,gen_ai.usage.input_tokensandgen_ai.usage.output_tokens. - Tool spans named
execute_tool <tool>, withgen_ai.tool.nameandgen_ai.tool.call.id. - Retrieval and memory spans for RAG lookups and memory reads or writes.
On top of these standard attributes, add a few of your own. The ones that pay off fastest are tenant or customer ID, feature name, prompt version and deployment environment. They let you answer questions like “did prompt v14 raise token usage for enterprise tenants?” without guessing.
Frameworks do much of this for you. Strands Agents, LangChain, LangGraph and CrewAI emit OpenTelemetry spans, and libraries such as OpenInference, OpenLLMetry and OpenLIT cover others. Strands, for example, can opt into the newest conventions with OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental.
Setting Up LLM Observability on AWS With AgentCore
If your agent runs on Amazon Bedrock AgentCore Runtime, most of the plumbing exists already. The setup is four steps.
1. Enable CloudWatch Transaction Search once per account. Spans are stored as structured logs, and AgentCore trace views depend on this. In the console, open Application Signals, choose Transaction Search, enable it and tick the option to ingest spans as structured logs. From the CLI, after adding a CloudWatch Logs resource policy that lets X-Ray write to the aws/spans log group:
aws xray update-trace-segment-destination --destination CloudWatchLogs
# Optional: index a percentage of spans as trace summaries
aws xray update-indexing-rule --name "Default" \
--rule '{"Probabilistic": {"DesiredSamplingPercentage": 10}}'
2. Add the AWS Distro for OpenTelemetry (ADOT) to your agent.
# requirements.txt
aws-opentelemetry-distro>=0.18.0
boto3
Then start the agent through the instrumentation wrapper, for example CMD ["opentelemetry-instrument", "python", "main.py"] in your Dockerfile.
3. Pass a session ID on every invocation. Send the X-Amzn-Bedrock-AgentCore-Runtime-Session-Id header so all turns of a conversation group together in the Sessions view. If an upstream service already started a trace, pass its traceparent or X-Amzn-Trace-Id header so the agent’s spans join it.
4. Add business context to your spans. Since November 16, 2026, AgentCore merges your OTEL_RESOURCE_ATTRIBUTES with its own defaults instead of replacing them, so you can safely set values like deployment.environment.name=prod,team=support and OTEL_SERVICE_NAME=support-agent. For per-request context, add attributes in code:
from opentelemetry import trace
tracer = trace.get_tracer("support-agent")
def lookup_policy(tenant_id: str, policy_version: str):
with tracer.start_as_current_span("lookup_refund_policy") as span:
span.set_attribute("app.tenant_id", tenant_id)
span.set_attribute("app.policy_version", policy_version)
# ... fetch and return the policy
Agents created since July 2026 in supported Regions send traces, prompts and structured logs to a single per-agent log group, /aws/bedrock-agentcore/runtimes/<agent_id>-<endpoint_name>. That makes it easier to scope IAM access and encryption per agent. Older agents can opt in with UNIFIED_TRACES_DESTINATION_ENABLED=true. AgentCore memory resources emit spans by default; gateway, built-in tool and identity tracing are switched on per resource in the console.
Agents Running Outside AgentCore
Many startups run agents on Lambda, ECS or EKS. You can still send the same telemetry to CloudWatch with ADOT and a handful of environment variables:
AGENT_OBSERVABILITY_ENABLED=true
OTEL_PYTHON_DISTRO=aws_distro
OTEL_PYTHON_CONFIGURATOR=aws_configurator
OTEL_RESOURCE_ATTRIBUTES=service.name=support-agent,aws.log.group.names=/aws/bedrock-agentcore/runtimes/support-agent
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=/aws/bedrock-agentcore/runtimes/support-agent,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp
Create the log group first, grant the role permission to write traces (the AWSXrayWriteOnlyPolicy managed policy covers this) and logs, and run the process with opentelemetry-instrument. On Lambda, add the AWS Lambda Layer for OpenTelemetry and set AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument.
Sampling, Privacy and Cost Trade-Offs
Tracing everything feels safe until the CloudWatch bill arrives. AWS’s own documentation warns that the ADOT default samples 100 percent of traces, which can mean up to 20 times the ingestion volume of the recommended 5 percent rate. A sensible starting point for most services:
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.05
Agents are a special case. Conversations are expensive and rare compared with web requests, and a single bad session is often what you need to see. Many teams keep a higher rate for agent services in early production, then lower it once evaluations and alarms catch most regressions. Decide per service, not globally.
Privacy needs the same deliberate choice. Prompts and completions routinely contain personal data. The OpenTelemetry conventions keep message content out of span attributes by default and make capture opt-in, while AgentCore lets you keep model payloads on spans for debugging. A practical rule: capture full content in staging, and in production either capture metadata only or redact before export. CloudWatch Logs data protection policies can mask sensitive fields in the log groups that hold your spans.
From Traces to Answers: Dashboards, Alarms and Evaluations
Once spans flow, the payoff is in what you build on them.
- Curated views. CloudWatch generative AI observability, generally available since October 2025 at no extra charge beyond standard telemetry pricing, shows latency, token usage and errors across AgentCore runtime, gateways, memory, tools and identity.
- Alarms on what matters. Alert on token usage per session, tool error rate and p95 latency per agent, not just on 5xx errors.
- Quality, not just health. Pair traces with evaluations. Amazon CloudWatch Omni, announced in September 2026, shows prompts, model calls, tool calls and sub-agent interactions in one timeline and runs evaluators against live traffic. If you are still on the console experience, AgentCore Evaluations gives you similar scoring.
Teams that get here stop debating why an agent misbehaved. They open the trace, find the span and fix the cause.
How Signiance Helps
Cloud Managed Service team runs production AWS workloads for startups, including AI agents. We can enable Transaction Search and ADOT across your agents, define the custom attributes and sampling rules that fit your traffic, set up redaction and data protection, and build the dashboards and alarms that tell you when quality or cost drifts, then keep it all running as you grow.
If your agents are live but you are still debugging from log lines, book a free consultation or a free AI Native readiness assessment at signiance.com, and we will map out your tracing setup together.
