GraphOS reporting
Send router operation metrics to GraphOS
The GraphOS Router and Apollo Router Core can report operation usage metrics to GraphOS that you can then visualize in GraphOS Studio. These metrics also enable powerful GraphOS features like schema checks.
Enabling usage reporting
You enable usage reporting in the router by setting the following environment variables:
1export APOLLO_KEY=<YOUR_GRAPH_API_KEY>
2export APOLLO_GRAPH_REF=<YOUR_GRAPH_ID>@<VARIANT_NAME>
Usage reporting via OpenTelemetry Protocol (OTLP)Since 1.49.0
Prior to router v1.49.0, all GraphOS reporting was performed using a private tracing format called Apollo Usage Reporting protocol.
As the ecosystem around OpenTelemetry (OTel) has rapidly expanded, Apollo evaluated migrating its internal tracing system to use an OTel-based protocol.
Starting in v1.49.0, the router can use OpenTelemetry Protocol (OTLP) to report operation usage metrics to GraphOS. The benefits of reporting via OTLP include:
A comprehensive way to visualize the router execution path in GraphOS Studio.
Additional spans that were previously not included in Studio traces, such as query parsing, planning, execution, and more.
Additional metadata such as subgraph fetch details, router idle / busy timing, and more.
Configuring usage reporting via OTLP
You can enable usage reporting via OTLP by an option that can also configure the ratio of traces sent via OTLP and Apollo Usage Reporting protocol:
In router v1.x, this is controlled using the
experimental_otlp_tracing_sampler
option and is disabled by default.In router v2.x and later, this option is renamed to
otlp_tracing_sampler
and is enabled by default.
The supported values of the OTLP sampler option are the following:
always_off
: send all traces via Apollo Usage Reporting protocol. Default for v1.x.always_on
: send all traces via OTLP. Default for v2.x and later.0.0 - 1.0
: the ratio of traces to send via OTLP (for example, 0.6 = 60% OTLP, 40% Apollo Usage Reporting protocol).
The OTLP sampler is applied after the common tracing sampler. In the following example, the common sampler samples traces at 1% of all traffic. The OTLP sampler sets its ratio to 0.7. This results in 0.7% of all traffic having traces sent via OTLP, and the remaining 0.3% of all traffic having traces sent via Apollo Usage Reporting protocol:
1telemetry:
2 apollo:
3 # Send 0.7 OTLP / 0.3 Apollo
4 otlp_tracing_sampler: 0.7
5
6 exporters:
7 tracing:
8 common:
9 # Sample traces at 1% of all traffic
10 sampler: 0.01
Reporting field-level traces
In their responses to your router, your subgraphs can include field-level traces that indicate how long the subgraph took to resolve each field in an operation. By analyzing this data in GraphOS Studio, you can identify and optimize your slower fields:
Your subgraph libraries must support federated tracing (also known as FTV1 tracing) to provide this data.
To confirm support, check the
FEDERATED TRACING
entry for your library on this page.Consult your library's documentation to learn how to enable federated tracing.
If you use Apollo Server with
@apollo/subgraph
, federated tracing support is enabled automatically.
Subgraph trace sampling
By default, the router requests subgraph trace data from operations with a 1% sampling probability per operation. In most cases, this provides a sufficient sample size while minimizing latency for most operations (traces can affect latency because they increase the size of subgraph response payloads).
You can customize your router's trace sampling probability by setting the following options in your YAML config file:
1telemetry:
2 apollo:
3 # In this example, the trace sampler is configured
4 # with a 50% probability of sampling a request.
5 # This value can't exceed the value of tracing.common.sampler.
6 field_level_instrumentation_sampler: 0.5
7
8 exporters:
9 tracing:
10 common:
11 # FTV1 uses the same trace sampling as other tracing options,
12 # so this value is also required.
13 sampler: 0.5
telemetry.apollo.field_level_instrumentation_sampler
cannot exceed the value of telemetry.exporters.tracing.common.sampler
.Disabling field-level traces
To completely disable requesting and reporting subgraph trace data, set field_level_instrumentation_sampler
to always_off
:
1telemetry:
2 apollo:
3 field_level_instrumentation_sampler: always_off
Experimental local field metrics
Apollo Router can send field-level metrics to GraphOS without using FTV1 tracing. This feature is experimental and is not yet displayable in GraphOS Studio.
To enable this feature, set the experimental_local_field_metrics
option to true
in your router configuration:
1telemetry:
2 apollo:
3 experimental_local_field_metrics: true
Advanced configuration
send_headers
Provide this field to configure which request header names and values are included in trace data that's sent to GraphOS. By default, no header information is sent to GraphOS as a security measure.
1telemetry:
2 apollo:
3 field_level_instrumentation_sampler: 0.01 # (default)
4 send_headers:
5 only: # Include only headers with these names
6 - referer
Supported values:
Value / Type | Description |
---|---|
| Set send_headers to the string value none to include no header information in reported traces.YAML
|
| Set send_headers to the string value all to include all header information in reported traces.YAML
|
| An array of names for the headers that the router will report to GraphOS. All other headers are not reported. See the example above. |
| An array of names for the headers that the router will not report to GraphOS. All other headers are. Uses the same format as the only example above. |
send_variable_values
Provide this field to configure which GraphQL variable values are included in trace data that's sent to GraphOS. By default, no variable information is sent to GraphOS as a security measure.
1telemetry:
2 apollo:
3 field_level_instrumentation_sampler: 0.01 # (default)
4 send_variable_values:
5 except: # Send all variables EXCEPT ones with these names
6 - first
Supported values:
Value / Type | Description |
---|---|
| Set send_variable_values to the string value none to include no variable information in reported traces.YAML
|
| Set send_variable_values to the string value all to include all variable information in reported traces.YAML
|
| An array of names for the variables that the router will report to GraphOS. All other variables are not reported. Uses the same format as the except example above. |
| An array of names for the variables that the router will not report to GraphOS. All other variables are reported. See the example above. |
1telemetry:
2 apollo:
3 # The percentage of requests will include HTTP request and response headers in traces sent to GraphOS Studio.
4 # This is expensive and should be left at a low value.
5 # This cannot be higher than tracing->common->sampler
6 field_level_instrumentation_sampler: 0.01 # (default)
7
8 # Include HTTP request and response headers in traces sent to GraphOS Studio
9 send_headers: # other possible values are all, only (with an array), except (with an array), none (by default)
10 except: # Send all headers except referer
11 - referer
12
13 # Include variable values in Apollo in traces sent to GraphOS Studio
14 send_variable_values: # other possible values are all, only (with an array), except (with an array), none (by default)
15 except: # Send all variable values except for variable named first
16 - first
17 exporters:
18 tracing:
19 common:
20 sampler: 0.5 # The percentage of requests that will generate traces (a rate or `always_on` or `always_off`)
errors
You can configure whether the router reports GraphQL error information to GraphOS, and whether the details of those errors are redacted. You can customize this behavior globally and override that global behavior on a per-subgraph basis.
By default, your router does report error information, and it does redact the details of those errors.
To prevent your router from reporting error information at all, you can set the
send
option tofalse
.To include all error details in your router's reports to GraphOS, you can set the
redact
option tofalse
.
Your subgraph libraries must support federated tracing (also known as FTV1 tracing) to provide errors to GraphOS. If you use Apollo Server with @apollo/subgraph
, federated tracing support is enabled automatically.
To confirm support:
Check the
FEDERATED TRACING
entry for your library on the supported subgraphs page.If federated tracing isn't enabled automatically for your library, consult its documentation to learn how to enable it.
Note that federated tracing can also be sampled (see above) so error messages might not be available for all your operations if you have sampled to a lower level.
See the example below:
1telemetry:
2 apollo:
3 errors:
4 subgraph:
5 all:
6 # By default, subgraphs should report errors to GraphOS
7 send: true # (default: true)
8 redact: false # (default: true)
9 subgraphs:
10 account: # Override the default behavior for the "account" subgraph
11 send: false
apollo_operation_id
from the context.