Spans

A span captures contextual information about requests and responses as they're processed through the router's request lifecycle (pipeline). The information from spans can be used when displaying traces in your application performance monitors (APM).

Spans configuration

Router request lifecycle services

A router's request lifecycle has three major services:

• Router service - Handles an incoming request before it is parsed. Works within a context of opaque bytes.

• Supergraph service - Handles a request after it has been parsed and before it is sent to the subgraph. Works within a GraphQL context.

• Subgraph service - Handles a request after it has been sent to the subgraph. Works within a GraphQL context.

The router, supergraph and subgraph sections are used to define custom span configuration for each service:

1telemetry:
2  instrumentation:
3    spans:
4      router:
5        attributes: {}
6          # ...     
7      supergraph:
8        attributes: {}
9          # ...
10      subgraph:
11        attributes: {}
12          # ...      

attributes

Spans may have attributes attached to them from the router pipeline. These attributes are used to filter and group spans in your APM.

Attributes may be drawn from standard attributes or selectors.

The attributes that are available depend on the service of the pipeline.

1telemetry:
2  instrumentation:
3    spans:
4      router:
5        attributes:
6          # Standard attributes
7          http.response.status_code: true
8          # Custom attributes
9          "my_attribute":
10            response_header: "x-my-header"

You can also have conditions on custom attributes using selectors. You can only have conditions on a selector at the same execution level. Example you can't have a condition on response_header if you want to set an attribute from request_header.

1telemetry:
2  instrumentation:
3    spans: 
4      router: 
5        attributes:    
6          on_error: 
7            response_status: reason
8            condition:
9              not:
10                eq:
11                - response_status: code
12                - 200

default_attribute_requirement_level

The default_attribute_requirement_level option sets the default attributes to attach to spans, as defined by OpenTelemetry semantic conventions.

Valid values:

• required (default)

• recommended

1telemetry:
2  instrumentation:
3    spans:
4      # Set the default requirement level
5      default_attribute_requirement_level: required

Attributes can be configured individually, so that required attributes can be overridden or disabled. For example, http.response.status_code is set individually to override the standard value:

1telemetry:
2  instrumentation:
3    spans:
4      # Set the default requirement level
5      default_attribute_requirement_level: required
6      router:
7        attributes:
8          # Disable standard attribute
9          http.response.status_code: false

mode

The mode option enables the router spans to either use legacy attributes in the router, or those defined in the OpenTelemetry specification.

Valid values:

• spec_compliant

• deprecated (default)

spec_compliant

This mode follows the OpenTelemetry spec. Attributes that were previously added to spans that did not follow conventions are now removed.

You will likely gain a significant performance improvement by using this mode as it reduces the number of attributes that are added to spans.

1telemetry:
2  instrumentation:
3    spans:
4      mode: spec_compliant

For now this is not the default, however it will be in a future release.

deprecated

This mode is the default and follows the previous behavior if you have not configured the mode option to spec_compliant.

1telemetry:
2  instrumentation:
3    spans:
4      mode: deprecated

Attributes are added to spans that do not follow OpenTelemetry conventions.

Span status

By default spans are marked in error only if the http status code is different than 200. If you want to mark a span in error for other reason you can override the otel.status_code attribute which is responsible to mark a span in error or not. If it's in error then otel.status_code = error, if not it will be ok.

Naming

By default, we will use a span naming convention that aligns with the current semantinc conventions for GraphQL server in OpenTelemetry which means the root span name must be of format <graphql.operation.type> <graphql.operation.name> provided that graphql.operation.type and graphql.operation.name are available.

If you want to change the name of spans we're creating for each services you can override this value by setting the otel.name attribute using any selectors you want.

Here is an example if you want to mark the router and supergraph span in error if you have a graphql error in the payload and you want to enforce the router span name to be graphql_router.

1telemetry:
2  instrumentation:
3    spans:
4      router:
5        attributes:
6          otel.name:
7            static: graphql_router # Override the span name to graphql_router 
8          otel.status_code:
9            static: error
10            condition:
11              eq:
12              - true
13              - on_graphql_error: true
14      supergraph:
15        attributes:
16          otel.status_code:
17            static: error
18            condition:
19              exists:
20                response_errors: $[0].extensions.code # Here is an example to get the first error code, `on_graphql_error` is also available for supergraph

Span configuration example

An example configuration of telemetry.spans in router.yaml sets both standard and custom attributes for the router service:

1telemetry:
2  instrumentation:
3    spans:
4      default_attribute_requirement_level: required
5      mode: spec_compliant
6      router:
7        attributes:
8          # Standard attributes (http)
9          dd.trace_id: false
10          http.request.body.size: false
11          http.response.body.size: false
12          http.request.method: false
13          # ...
14  
15          # Conditional custom attribute
16          otel.status_description: # You can conditionally put a status description attribute on your span if it respect the condition
17            static: "there was an error"
18            condition: # http response status code != 200 or it contains a graphql error in the payload
19              any:
20              - not:
21                  eq:
22                  - response_status: code
23                  - 200
24              - eq:
25                - on_graphql_error
26                - true
27          # Custom attributes
28          "acme.custom_1":
29            trace_id: datadog
30          "acme.custom_2":
31            response_header: "X-CUSTOM2"
32            default: "unknown"
33          "acme.custom_3":
34            env: "ENV_VAR"
35          "static_attribute":
36            static: "my_static_value"
37            # ...
38          
39      supergraph:
40        attributes: {}
41          # ...
42      subgraph:
43        attributes: {}
44          # ...      

Spans configuration reference