Agent Telemetry - AG2
Agent
AG2 ships a TelemetryMiddleware that emits OpenTelemetry spans for a single agent's work — its turns, LLM calls, tool executions, and human-in-the-loop interactions.
The middleware follows the OpenTelemetry GenAI Semantic Conventions, so traces can be exported to any compatible backend — Jaeger, Grafana Tempo, Datadog, Honeycomb, Langfuse, and others.
Tip
To trace the interactions between agents in a network — the hub's dispatch, channels, agent lifecycles, and tasks — see Network Telemetry. The two compose: an agent's invoke_agent span links back to the network envelope that triggered it.
Installation
pip install "ag2[openai,tracing]"
Quick Start
<br> 1<br> 2<br> 3<br> 4<br> 5<br> 6<br> 7<br> 8<br> 9<br>10<br>11<br>12<br>13<br>14<br>15<br>16<br>17<br>18<br>19<br>20<br>21<br>22<br>23<br>24<br>25<br>26<br>27<br>28<br>29<br>30<br>31<br> |
<br>from opentelemetry import trace<br>from opentelemetry.sdk.resources import Resource<br>from opentelemetry.sdk.trace import TracerProvider<br>from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter<br>from ag2 import Agent<br>from ag2.config import OpenAIConfig<br>from ag2.middleware.builtin import TelemetryMiddleware<br># 1. Configure OpenTelemetry<br>resource = Resource.create(attributes={"service.name": "ag2-quickstart"})<br>tracer_provider = TracerProvider(resource=resource)<br>tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))<br>trace.set_tracer_provider(tracer_provider)<br># 2. Create agent with telemetry middleware<br>agent = Agent(<br> "assistant",<br> prompt="You are a helpful assistant.",<br> config=OpenAIConfig(model="gpt-4o-mini"),<br> middleware=[<br> TelemetryMiddleware(<br> tracer_provider=tracer_provider,<br> agent_name="assistant",<br> ),<br> ],<br>)<br># 3. Run -- spans are emitted automatically<br>import asyncio<br>reply = asyncio.run(agent.ask("What is the capital of France?"))<br> |
Trace Hierarchy
Each ask() call produces a root span with child spans for LLM calls, tool executions, and human input:
invoke_agent assistant
|-- chat gpt-4o-mini # LLM API call
|-- execute_tool get_weather # tool execution
|-- chat gpt-4o-mini # LLM call after tool result
+-- await_human_input assistant # human-in-the-loop
Span Types
Every span includes an ag2.span.type attribute:
ag2.span.type |
Operation name | Triggered by |
|---|---|---|
agent |
invoke_agent |
on_turn -- wraps the full agent turn |
llm |
chat |
on_llm_call -- each LLM API call |
tool |
execute_tool |
on_tool_execution -- each tool invocation |
human_input |
await_human_input |
on_human_input -- human-in-the-loop |
Semantic Attributes
Spans carry standard OpenTelemetry GenAI attributes:
| Attribute | Span types | Description |
|---|---|---|
gen_ai.operation.name |
All | Operation: invoke_agent, chat, execute_tool, await_human_input |
gen_ai.agent.name |
agent, human_input | Agent name |
gen_ai.provider.name |
agent, llm | LLM provider (e.g. openai, anthropic) -- auto-detected |
gen_ai.request.model |
agent, llm | Model name (e.g. gpt-4o-mini) -- auto-detected |
gen_ai.response.model |
llm | Resolved model name from response |
gen_ai.response.finish_reasons |
llm | Finish reasons (e.g. ["stop"], ["tool_calls"]) |
gen_ai.usage.input_tokens |
llm | Prompt token count |
gen_ai.usage.output_tokens |
llm | Completion token count |
gen_ai.usage.cache_creation_input_tokens |
llm | Tokens used to create prompt cache (Anthropic) |
gen_ai.usage.cache_read_input_tokens |
llm | Tokens read from prompt cache (Anthropic, OpenAI, Gemini) |
gen_ai.tool.name |
tool | Tool function name |
gen_ai.tool.call.id |
tool | Tool call ID |
gen_ai.tool.type |
tool | Tool type (always function) |
Content Capture
By default, message content, tool arguments, and results are included in spans. To disable content capture for privacy-sensitive environments:
<br>1<br>2<br>3<br>4<br>5<br> |
<br>TelemetryMiddleware(<br> tracer_provider=tracer_provider,<br> agent_name="assistant",<br> capture_content=False, # omits messages, tool args, and results<br>)<br> |
When content capture is enabled (the default), spans include these additional attributes:
| Attribute | Span type | Content |
|---|---|---|
gen_ai.input.messages |
llm | JSON request messages |
gen_ai.output.messages |
llm | JSON response messages |
gen_ai.tool.call.arguments |
tool | Tool call arguments (JSON) |
gen_ai.tool.call.result |
tool | Tool execution result |
ag2.human_input.prompt |
human_input | Prompt shown to human |
ag2.human_input.response |
human_input | Human's response |
Warning
With capture_content=True, message content, tool arguments, and human input will appear in your tracing backend. Ensure your backend has appropriate access controls.
Custom Span Attributes
Use span_attributes to stamp custom key-value pairs onto spans the middleware emits. This is useful for routing or filtering traces by tenant, environment, deployment, or any other label your backend supports. Every span will carry these attributes.
<br>1<br>2<br>3<br>4<br>5<br>6<br>7<br>8<br> |
<br>TelemetryMiddleware(<br> tracer_provider=tracer_provider,<br> agent_name="assistant",<br> span_attributes={<br> "deployment": "production",<br> "ag2.org.id": "org-abc123",<br> },<br>)<br> |
Tip
This is the right place to add tenant or organization identifiers when your tracing backend filters traces by span-level labels (for example, Google Cloud Trace label filters or Datadog tags).
Note
If a key in span_attributes collides with an intrinsic attribute set by the middleware (such as ag2.span.type or gen_ai.usage.input_tokens), the middleware's value always wins.
Configuration
TelemetryMiddleware accepts:
| Parameter | Type | Default | Description |
|---|---|---|---|
tracer_provider |
`TracerProvider | None` | Global provider |
capture_content |
bool |
True |
Include message/tool content in spans |
agent_name |
`str | None` | "unknown" |
provider_name |
`str | None` | None |
model_name |
`str | None` | None |
span_attributes |
`dict[str, str] | None` | None |
Tool Execution Example
<br> 1<br> 2<br> 3<br> 4<br> 5<br> 6<br> 7<br> 8<br> 9<br>10<br>11<br>12<br>13<br>14<br>15<br>16<br>17<br>18<br>19<br>20<br>21<br>22<br> |
<br>from ag2 import Agent<br>from ag2.config import OpenAIConfig<br>from ag2.middleware.builtin import TelemetryMiddleware<br>from ag2.tools import tool<br>@tool<br>def get_weather(city: str) -> str:<br> """Get weather information for a city."""<br> return f"Sunny, 72F in {city}"<br>agent = Agent(<br> "weather_agent",<br> prompt="Use the get_weather tool to answer weather questions.",<br> config=OpenAIConfig(model="gpt-4o-mini"),<br> tools=[get_weather],<br> middleware=[<br> TelemetryMiddleware(<br> tracer_provider=tracer_provider,<br> agent_name="weather_agent",<br> ),<br> ],<br>)<br> |
Backend Integration
Since TelemetryMiddleware emits standard OpenTelemetry spans, any OTLP-compatible backend works — Grafana Tempo, Jaeger, Datadog, Honeycomb, Langfuse, and others. Swap the ConsoleSpanExporter from the Quick Start above for an OTLPSpanExporter pointed at your backend (or a local OpenTelemetry Collector); nothing else about the instrumentation changes.