Telemetry

OpenTelemetry utilities for the Substrate project.

Overview

The shared-telemetry package provides a set of utilities for instrumenting your application with OpenTelemetry tracing. It wraps the OpenTelemetry API to provide a simpler interface for common tracing operations.

The Substrate web application is fully instrumented with:

• Automatic HTTP instrumentation via OpenTelemetry auto-instrumentations
• GraphQL operation tracing via Apollo Server plugin
• Database query tracing via Prisma extensions
• Manual span creation for custom operations

Configuration

Environment Variables

Configure OpenTelemetry using the following environment variables:

Variable	Description	Default	Required?
OTEL_SERVICE_NAME	Service name for traces	substrate-web	No
OTEL_EXPORTER_OTLP_ENDPOINT	OTLP exporter endpoint URL	-	Yes (for export)
OTEL_EXPORTER_OTLP_HEADERS	JSON object of headers for OTLP exporter	{}	No
OTEL_LOG_LEVEL	OpenTelemetry log level	INFO	No

Example .env configuration:

# Enable OpenTelemetry tracing
OTEL_SERVICE_NAME="substrate-web"
OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318/v1/traces"
OTEL_LOG_LEVEL="INFO"

Instrumentation Setup

The web application uses Next.js instrumentation hooks to initialize OpenTelemetry. The setup in apps/web/src/instrumentation.ts includes:

1. NodeSDK - Configures the OpenTelemetry SDK with resource attributes and exporters
2. Auto-instrumentations - Automatically traces HTTP requests, database calls, and other operations
3. Custom plugins - Apollo GraphQL and Prisma instrumentation

What's automatically traced:

• HTTP requests (incoming and outgoing)
• GraphQL queries and mutations
• Database queries via Prisma
• Next.js API routes

Local Development with Jaeger

To view traces locally, run Jaeger using Docker:

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Then configure your .env:

OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318/v1/traces"

Access the Jaeger UI at http://localhost:16686

Installation

This package is internal to the Substrate monorepo and is available as a workspace dependency:

{
  "dependencies": {
    "shared-telemetry": "^0.0.1"
  }
}

API Reference

getTracer(name?)

Get a tracer instance for creating spans.

Parameters:

• name (optional): The tracer name. Defaults to 'substrate'.

Returns: An OpenTelemetry Tracer instance.

import { getTracer } from "shared-telemetry";

const tracer = getTracer("my-service");

createSpan(name, tracerName?)

Create a new span for tracing an operation.

warning

This function returns a span that must be manually ended with span.end(). Consider using withSpan or withSpanSync instead for automatic lifecycle management.

Parameters:

• name: The name of the span.
• tracerName (optional): The tracer name. Defaults to 'substrate'.

Returns: An OpenTelemetry Span instance.

import { createSpan } from "shared-telemetry";
import { SpanStatusCode } from "@opentelemetry/api";

const span = createSpan("operation");
try {
  // ... your code
  span.setStatus({ code: SpanStatusCode.OK });
} catch (error) {
  span.recordException(error);
  span.setStatus({ code: SpanStatusCode.ERROR });
  throw error;
} finally {
  span.end(); // Always end the span!
}

withSpan(name, fn, tracerName?)

Wrapper to trace an async function with a span. The span is automatically created, set as active, and ended when the function completes.

Parameters:

• name: The name of the span.
• fn: An async function that receives the span as its argument.
• tracerName (optional): The tracer name. Defaults to 'substrate'.

Returns: A promise that resolves to the result of the function.

import { withSpan, addSpanAttributes } from "shared-telemetry";

export async function GET(request: Request) {
  return withSpan("api.hello.get", async (span) => {
    addSpanAttributes({
      "http.route": "/api/hello",
      "http.method": "GET",
    });
    return new Response("Hello, from API!");
  });
}

withSpanSync(name, fn, tracerName?)

Wrapper to trace a synchronous function with a span. The span is automatically created, set as active, and ended when the function completes.

Parameters:

• name: The name of the span.
• fn: A synchronous function that receives the span as its argument.
• tracerName (optional): The tracer name. Defaults to 'substrate'.

Returns: The result of the function.

import { withSpanSync } from "shared-telemetry";

const result = withSpanSync("sync-operation", (span) => {
  span.setAttribute("key", "value");
  return performSyncOperation();
});

getActiveSpan()

Get the current active span from the context.

Returns: The active Span or undefined if no span is active.

import { getActiveSpan } from "shared-telemetry";

const span = getActiveSpan();
if (span) {
  span.setAttribute("custom.attribute", "value");
}

addSpanAttributes(attributes)

Add attributes to the current active span. This is a convenience function that safely adds attributes only if there is an active span.

Parameters:

• attributes: A record of attribute key-value pairs. Values can be strings, numbers, or booleans.

import { addSpanAttributes } from "shared-telemetry";

addSpanAttributes({
  "http.method": "GET",
  "http.status_code": 200,
  "cache.hit": true,
});

recordException(error)

Record an exception on the current active span. This will also set the span status to ERROR.

Parameters:

• error: The error to record.

import { recordException } from "shared-telemetry";

try {
  await riskyOperation();
} catch (error) {
  recordException(error);
  throw error;
}

extractTraceHeaders()

Extract trace context headers for propagation to downstream services.

Returns: A record of header key-value pairs suitable for HTTP request headers.

import { extractTraceHeaders } from "shared-telemetry";

const headers = extractTraceHeaders();
await fetch("https://api.example.com", {
  headers: {
    ...headers,
    "Content-Type": "application/json",
  },
});

Best Practices

1. Use withSpan for async operations: It handles span lifecycle automatically, including error recording.

2. Use withSpanSync for sync operations: Same benefits as withSpan but for synchronous code.

3. Add meaningful attributes: Use addSpanAttributes to add context that will help with debugging and analysis.

4. Propagate context: Use extractTraceHeaders when making HTTP calls to downstream services to maintain trace continuity.

5. Record exceptions: Use recordException to capture error details in your traces.

See Also

• OpenTelemetry Documentation
• OpenTelemetry JavaScript API

GraphQL Plugin

The package also exports an Apollo Server telemetry plugin via the shared-telemetry/graphql subpath:

import { createTelemetryPlugin } from "shared-telemetry/graphql";

createTelemetryPlugin()

Creates an Apollo Server plugin that automatically traces GraphQL operations.

Features:

• Creates spans for each GraphQL operation (queries, mutations, subscriptions)
• Tracks individual field resolver executions
• Sanitizes sensitive variables (passwords, tokens) from trace attributes
• Captures operation type, name, and document
• Preserves error status across the request lifecycle

Usage with Apollo Server:

import { ApolloServer } from "@apollo/server";
import { createTelemetryPlugin } from "shared-telemetry/graphql";

const server = new ApolloServer({
  typeDefs,
  resolvers,
  plugins: [createTelemetryPlugin()],
});

Span Attributes:

Attribute	Description
graphql.operation.name	The operation name
graphql.operation.type	query, mutation, or subscription
graphql.document	The GraphQL query document
graphql.variable.<name>	Variable values (sensitive fields excluded)
graphql.field.name	Resolver field name
graphql.field.type	Resolver return type
graphql.field.path	Resolver field path

Peer Dependencies:

The GraphQL plugin requires @apollo/server and graphql as peer dependencies. These are marked as optional in the package to avoid requiring them for consumers that only use the core telemetry utilities.