Telemetry and metrics

Virtual MCP Server (vMCP) provides comprehensive observability through OpenTelemetry instrumentation. You can export traces and metrics to monitor backend operations and workflow executions.

Telemetry types

vMCP supports two types of telemetry:

• Traces: Track requests across vMCP and its backends, showing the full path of tool calls, resource reads, and workflow executions
• Metrics: Counters and histograms for backend request rates, error rates, and latency distributions

For general ToolHive observability concepts including trace structure and metrics, see the observability overview.

Enable telemetry

Configure telemetry by creating a shared MCPTelemetryConfig resource, then referencing it from your VirtualMCPServer. This enables reuse across multiple resources, Kubernetes-native secret references for OTLP auth headers, CA bundle ConfigMap references, and per-server serviceName overrides.

telemetry-config.yaml

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPTelemetryConfig
metadata:
  name: shared-otel
  namespace: toolhive-system
spec:
  openTelemetry:
    enabled: true
    endpoint: otel-collector.monitoring:4318
    insecure: true
    tracing:
      enabled: true
      samplingRate: '0.05'
    metrics:
      enabled: true
  prometheus:
    enabled: true

virtualmcpserver.yaml

apiVersion: toolhive.stacklok.dev/v1beta1
kind: VirtualMCPServer
metadata:
  name: my-vmcp
  namespace: toolhive-system
spec:
  groupRef:
    name: my-group
  telemetryConfigRef:
    name: shared-otel
    serviceName: my-vmcp
  incomingAuth:
    type: anonymous

The serviceName field on telemetryConfigRef overrides the telemetry service name for this specific VirtualMCPServer, giving it a distinct identity in your observability backend. If omitted, it defaults to the server name with a thv- prefix.

When you update the MCPTelemetryConfig resource, the operator detects the change and triggers a rolling update of affected deployments.

MCPTelemetryConfig fields

Field	Description	Default
spec.openTelemetry.enabled	Enable OpenTelemetry export	false
spec.openTelemetry.endpoint	OTLP collector endpoint (hostname:port)	-
spec.openTelemetry.insecure	Use HTTP instead of HTTPS	false
spec.openTelemetry.tracing.enabled	Enable tracing	false
spec.openTelemetry.tracing.samplingRate	Trace sampling rate (0.0-1.0)	"0.05"
spec.openTelemetry.metrics.enabled	Enable OTLP metrics export	false
spec.prometheus.enabled	Expose /metrics endpoint	false
spec.prometheus.metricsOnTransportPort	Also serve /metrics on the vMCP transport port	true

Prometheus metrics are served on the vMCP pod's dedicated diagnostics port 9464. Set spec.prometheus.metricsOnTransportPort: false to stop duplicating the endpoint on the vMCP transport port; the transport-port default is expected to flip in a future release.

For inline VirtualMCPServer telemetry (spec.config.telemetry), the same metricsOnTransportPort field applies, and an optional spec.config.telemetry.prometheusPort overrides the diagnostics port (9464 by default). The MCPTelemetryConfig resource does not expose a port override.

Export to observability backends

Export to Jaeger via OpenTelemetry Collector

Deploy an OpenTelemetry Collector configured to export to Jaeger:

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

Then create an MCPTelemetryConfig resource that points to the collector:

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPTelemetryConfig
metadata:
  name: jaeger-otel
  namespace: toolhive-system
spec:
  openTelemetry:
    enabled: true
    endpoint: otel-collector:4318
    insecure: true
    tracing:
      enabled: true
    metrics:
      enabled: true

Metrics collection

vMCP supports two methods for collecting metrics:

• Push via OpenTelemetry: Set spec.openTelemetry.metrics.enabled: true to push metrics to your OTel Collector via OTLP.
• Pull via Prometheus: Set spec.prometheus.enabled: true to expose a /metrics endpoint on the dedicated diagnostics port (9464 by default, overridable via spec.config.telemetry.prometheusPort on the VirtualMCPServer). By default, the endpoint is also served on the vMCP transport port for compatibility; set metricsOnTransportPort: false to disable the duplicate.

Backend metrics

These metrics track requests to individual MCP server backends:

Metric	Type	Description
toolhive_vmcp_backends_discovered	Gauge	Number of backends discovered
toolhive_vmcp_backend_requests	Counter	Total requests per backend
toolhive_vmcp_backend_errors	Counter	Total errors per backend
toolhive_vmcp_backend_requests_duration	Histogram	Duration of backend requests
mcp.client.operation.duration	Histogram	MCP client operation duration (mcp_client_operation_duration on /metrics)

Workflow metrics

These metrics track workflow execution across backends:

Metric	Type	Description
toolhive_vmcp_workflow_executions	Counter	Total workflow executions
toolhive_vmcp_workflow_errors	Counter	Total workflow execution errors
toolhive_vmcp_workflow_duration	Histogram	Duration of workflow executions

Rate limit metrics

When rate limiting is enabled on the VirtualMCPServer, these metrics track token bucket decisions and the health of the Redis backend that stores the buckets. Each metric carries namespace and server attributes identifying the resource:

Metric	Type	Description
toolhive_rate_limit_decisions	Counter	Bucket decisions. Additional attributes: decision (allowed or rejected), scope (shared or per_user), operation_type (server or tool)
toolhive_rate_limit_redis_errors	Counter	Redis errors during rate limit checks. Additional attribute: error_type (timeout, connection, auth, or other)
toolhive_rate_limit_fail_open	Counter	Requests allowed because the rate limit check failed open after a Redis error. Additional attribute: error_type (matches redis_errors)
toolhive_rate_limit_check_latency	Histogram	Duration of the atomic Redis Lua rate limit check, in seconds (each check, including failures)

For the counting semantics (allowed increments per bucket, rejected only for the first rejected bucket) and example PromQL queries, see Observe rate limit activity in the rate limiting guide.

Optimizer metrics

When the vMCP optimizer is enabled, these metrics track tool-finding and tool-calling performance:

Metric	Type	Description
toolhive_vmcp_optimizer_find_tool_requests	Counter	Total FindTool calls
toolhive_vmcp_optimizer_find_tool_errors	Counter	Total FindTool errors
toolhive_vmcp_optimizer_find_tool_duration	Histogram	Duration of FindTool calls
toolhive_vmcp_optimizer_find_tool_results	Histogram	Number of tools returned per call
toolhive_vmcp_optimizer_token_savings_percent	Histogram	Token savings percentage per call
toolhive_vmcp_optimizer_call_tool_requests	Counter	Total CallTool calls
toolhive_vmcp_optimizer_call_tool_errors	Counter	Total CallTool errors
toolhive_vmcp_optimizer_call_tool_not_found	Counter	CallTool calls where tool was not found
toolhive_vmcp_optimizer_call_tool_duration	Histogram	Duration of CallTool calls

Distributed tracing

vMCP creates client-side spans for backend operations with the following span names:

• tools/call <tool_name> - Tool calls to backends
• resources/read - Resource reads from backends
• prompts/get <prompt_name> - Prompt retrieval from backends
• list_capabilities - Backend capability discovery

Each span includes attributes for the target backend (target.workload_id, target.workload_name, target.base_url) and the relevant MCP attributes (mcp.method.name, gen_ai.tool.name, mcp.resource.uri).

Next steps

• Set up audit logging for structured request and authorization event tracking

Related information

• Observability concepts - overview of ToolHive's observability architecture
• Kubernetes telemetry guide - telemetry for MCPServer resources
• OpenTelemetry tutorial - set up a local observability stack