GraphOS reporting

The GraphOS Router and Apollo Router Core can report operation usage metrics to GraphOS that you can then visualize in GraphOS Studio. These metrics also enable powerful GraphOS features like schema checks.

Enabling usage reporting

You enable usage reporting in the router by setting the following environment variables:

1export APOLLO_KEY=<YOUR_GRAPH_API_KEY>
2export APOLLO_GRAPH_REF=<YOUR_GRAPH_ID>@<VARIANT_NAME>

Usage reporting via OpenTelemetry Protocol (OTLP)Since 1.49.0

Prior to router v1.49.0, all GraphOS reporting was performed using a private tracing format called Apollo Usage Reporting protocol.

As the ecosystem around OpenTelemetry (OTel) has rapidly expanded, Apollo evaluated migrating its internal tracing system to use an OTel-based protocol.

Starting in v1.49.0, the router can use OpenTelemetry Protocol (OTLP) to report operation usage metrics to GraphOS. The benefits of reporting via OTLP include:

• A comprehensive way to visualize the router execution path in GraphOS Studio.

• Additional spans that were previously not included in Studio traces, such as query parsing, planning, execution, and more.

• Additional metadata such as subgraph fetch details, router idle / busy timing, and more.

Configuring usage reporting via OTLP

You can enable usage reporting via OTLP by an option that can also configure the ratio of traces sent via OTLP and Apollo Usage Reporting protocol:

• In router v1.x, this is controlled using the experimental_otlp_tracing_sampler option and is disabled by default.

• In router v2.x and later, this option is renamed to otlp_tracing_sampler and is enabled by default.

The supported values of the OTLP sampler option are the following:

• always_off: send all traces via Apollo Usage Reporting protocol. Default for v1.x.

• always_on: send all traces via OTLP. Default for v2.x and later.

• 0.0 - 1.0: the ratio of traces to send via OTLP (for example, 0.6 = 60% OTLP, 40% Apollo Usage Reporting protocol).

The OTLP sampler is applied after the common tracing sampler. In the following example, the common sampler samples traces at 1% of all traffic. The OTLP sampler sets its ratio to 0.7. This results in 0.7% of all traffic having traces sent via OTLP, and the remaining 0.3% of all traffic having traces sent via Apollo Usage Reporting protocol:

1telemetry:
2  apollo:
3    # Send 0.7 OTLP / 0.3 Apollo
4    otlp_tracing_sampler: 0.7
5
6  exporters:
7    tracing:
8      common:
9        # Sample traces at 1% of all traffic
10        sampler: 0.01

Reporting field-level traces

In their responses to your router, your subgraphs can include field-level traces that indicate how long the subgraph took to resolve each field in an operation. By analyzing this data in GraphOS Studio, you can identify and optimize your slower fields:

Your subgraph libraries must support federated tracing (also known as FTV1 tracing) to provide this data.

• To confirm support, check the FEDERATED TRACING entry for your library on this page.

• Consult your library's documentation to learn how to enable federated tracing.

  • If you use Apollo Server with @apollo/subgraph, federated tracing support is enabled automatically.

Subgraph trace sampling

By default, the router requests subgraph trace data from operations with a 1% sampling probability per operation. In most cases, this provides a sufficient sample size while minimizing latency for most operations (traces can affect latency because they increase the size of subgraph response payloads).

You can customize your router's trace sampling probability by setting the following options in your YAML config file:

1telemetry:
2  apollo:
3    # In this example, the trace sampler is configured  
4    # with a 50% probability of sampling a request.
5    # This value can't exceed the value of tracing.common.sampler.
6    field_level_instrumentation_sampler: 0.5
7
8  exporters:
9    tracing:
10      common:
11        # FTV1 uses the same trace sampling as other tracing options,
12        # so this value is also required.
13        sampler: 0.5

Disabling field-level traces

To completely disable requesting and reporting subgraph trace data, set field_level_instrumentation_sampler to always_off:

1telemetry:
2  apollo:
3    field_level_instrumentation_sampler: always_off

Experimental local field metrics

Apollo Router can send field-level metrics to GraphOS without using FTV1 tracing. This feature is experimental and is not yet displayable in GraphOS Studio. To enable this feature, set the experimental_local_field_metrics option to true in your router configuration:

1telemetry:
2  apollo:
3    experimental_local_field_metrics: true

Advanced configuration

send_headers

Provide this field to configure which request header names and values are included in trace data that's sent to GraphOS. By default, no header information is sent to GraphOS as a security measure.

1telemetry:
2  apollo:
3    field_level_instrumentation_sampler: 0.01 # (default)
4    send_headers:
5      only: # Include only headers with these names
6        - referer

Supported values:

1send_headers: none

1send_headers: all

send_variable_values

Provide this field to configure which GraphQL variable values are included in trace data that's sent to GraphOS. By default, no variable information is sent to GraphOS as a security measure.

1telemetry:
2  apollo:
3    field_level_instrumentation_sampler: 0.01 # (default)
4    send_variable_values:
5      except: # Send all variables EXCEPT ones with these names
6        - first

Supported values:

1send_variable_values: none

1send_variable_values: all

1telemetry:
2  apollo:
3    # The percentage of requests will include HTTP request and response headers in traces sent to GraphOS Studio.
4    # This is expensive and should be left at a low value.
5    # This cannot be higher than tracing->common->sampler
6    field_level_instrumentation_sampler: 0.01 # (default)
7
8    # Include HTTP request and response headers in traces sent to GraphOS Studio
9    send_headers: # other possible values are all, only (with an array), except (with an array), none (by default)
10      except: # Send all headers except referer
11        - referer
12
13    # Include variable values in Apollo in traces sent to GraphOS Studio
14    send_variable_values: # other possible values are all, only (with an array), except (with an array), none (by default)
15      except: # Send all variable values except for variable named first
16        - first
17  exporters:
18    tracing:
19      common:
20        sampler: 0.5 # The percentage of requests that will generate traces (a rate or `always_on` or `always_off`)

errors

You can configure whether the router reports GraphQL error information to GraphOS, and whether the details of those errors are redacted. You can customize this behavior globally and override that global behavior on a per-subgraph basis.

By default, your router does report error information, and it does redact the details of those errors.

• To prevent your router from reporting error information at all, you can set the send option to false.

• To include all error details in your router's reports to GraphOS, you can set the redact option to false.

Your subgraph libraries must support federated tracing (also known as FTV1 tracing) to provide errors to GraphOS. If you use Apollo Server with @apollo/subgraph, federated tracing support is enabled automatically.

To confirm support:

• Check the FEDERATED TRACING entry for your library on the supported subgraphs page.

• If federated tracing isn't enabled automatically for your library, consult its documentation to learn how to enable it.

• Note that federated tracing can also be sampled (see above) so error messages might not be available for all your operations if you have sampled to a lower level.

See the example below:

1telemetry:
2  apollo:
3    errors:
4      subgraph:
5        all:
6          # By default, subgraphs should report errors to GraphOS
7          send: true # (default: true)
8          redact: false # (default: true)
9        subgraphs:
10          account: # Override the default behavior for the "account" subgraph
11            send: false