Router Logging to stdout

Configure logging to stdout

You can configure GraphOS Router or Apollo Router Core logging to be directed to stdout, and its output format can be set to text or JSON.

For general logging configuration, refer to Router Logging Configuration.

stdout configuration

enabled

The stdout logging output is disabled by default.

To enable stdout logging, set the enabled option to true:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true

format

You can configure the logging output format. The default format depends on how the router is run:

  • In an interactive shell, text is the default.

  • In a non-interactive shell, json is the default.

You can explicitly set the format in router.yaml with telemetry.exporters.logging.stdout.format:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: text

tty_format

You can configure the log format when you're running on an interactive shell. This is useful during development.

If both format and tty_format are configured then the output depends on the environment where the router is run:

  • In an interactive shell, tty_format will take precedence.

  • In a non-interactive shell, format will take precedence.

You can explicitly set the format in router.yaml with telemetry.exporters.logging.stdout.tty_format:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: json
7         tty_format: text

rate_limit

The rate at which log messages are produced can become too high, especially for request processing errors. To prevent the router from filling its logs with redundant messages, you can use the rate_limit option to set the logging rate limit.

You can set the logging rate limit for each log location, where different log messages are rate limited independently, and log lines with the same message but different sets of attributes are limited under the same rate. This rate limiting only applies to logging to stdout and doesn't affect events sent to trace exporters like OTLP, which have their own sampling configuration.

To enable rate limiting, set the rate_limit option:

YAML
router.yaml
1telemetry:
2  exporters:
3    logging:
4      stdout:
5        format: json
6        rate_limit:
7          capacity: 1  # number of allowed messages during the rate limiting interval
8          interval: 3s

For configuration options specific to each output format, see the text and json format references.

Configuration reference

OptionValuesDefaultDescription
enabledtrue|falsefalseEnable or disable stdout logging.
formattext|jsonSee the format documentation for details.
tty_formattext|jsonSee the format documentation for details.

Logging output format

You can configure logging to be output in different formats:

Each format has its own specific settings.

text

The text format is human-readable and ideal for development and debugging. It is the default logging output format.

To use the text format, in router.yaml enable telemetry.exporters.logging.stdout and set the format as text:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: text # The default text format will be used

The text format can also be used as a key in YAML, telemetry.exporters.logging.stdout.format.text, to specify advanced configuration options:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format:
7           text:
8              ansi_escape_codes: true
9              display_filename: true
10              display_level: true
11              display_line_number: true
12              display_target: true
13              display_thread_id: true
14              display_thread_name: true
15              display_timestamp: true
16              display_resource: true
17              display_span_list: true
18              display_current_span: true
19              display_service_name: true
20              display_service_namespace: true
21              display_trace_id: true # true|false|open_telemetry|hexadecimal|decimal|datadog|uuid
22              display_span_id: true

Example text output:

JSON
12023-10-30T15:49:34.174435Z  INFO trace_id: bbafc3f048b6137375dd78c10df18f50 span_id: 40ede28c5df1b5cc main ThreadId(01) span_name{span_attr_1="span_attr_1" span_attr_2="span_attr_2"}: event_target: event_file.rs:32: event_attr_1="event_attr_1" event_attr_2="event_attr_2"

text configuration reference

OptionValuesDefaultDescription
ansi_escape_codestrue|falsetrueUse ansi terminal escape codes.
display_filenametrue|falsefalseThe filename where the log event was raised.
display_leveltrue|falsetrueThe level of the log event, e.g. INFO, WARN, ERROR, TRACE.
display_line_numbertrue|falsefalseThe line number where the event was raised.
display_targettrue|falsefalseThe module name where the event was raised.
display_thread_idtrue|falsefalseThe id of the thread where the event was raised.
display_thread_nametrue|falsefalseThe name of the thread where the event was raised.
display_timestamptrue|falsetrueThe timestamp of when the event was raised.
display_service_nametrue|falsefalseThe service name as configured in metrics common.
display_service_namespacetrue|falsefalseThe service namespace as configured in metrics common.
display_trace_idtrue|false|open_telemetry|hexadecimal|decimal|datadog|uuidfalseThe trace id of the span in which the event was raised.
display_span_idtrue|falsefalseThe span ID of the span in which the event was raised.
display_span_listtrue|falsetrueA list of all spans to root in which the event was raised and all of their attributes.
display_current_spantrue|falsetrueThe span in which the event was raised and all of its' attributes.

json

The json format is a machine-readable format ideal for consumption by application performance monitors (APMs).

The router supports structured JSON output provided by tracing-subscriber.

To use the json format, in router.yaml enable telemetry.exporters.logging.stdout and set the format as json:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: json

Each log entry will be a single well-formed JSON document that is ideal for processing in your APM tool of choice.

Example default json output:

JSON
stdout
1{
2  "timestamp": "2023-10-30T14:09:34.771388Z",
3  "level": "INFO",
4  "trace_id": "54ac7e5f0e8ab90ae67b822e95ffcbb8",
5  "span_id": "d52e3478c718b8a9",
6  "fields": {
7    "event_attr_1": "event_attr_1",
8    "event_attr_2": "event_attr_2"
9  },
10  "target": "event_target"
11}

You can configure which attributes are included in the JSON output by specifying telemetry.exporters.logging.stdout.format.json as a key in router.yaml:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: 
7           json:
8             display_filename: false
9             display_level: true
10             display_line_number: false
11             display_target: false
12             display_thread_id: false
13             display_thread_name: false
14             display_timestamp: true
15             display_current_span: true
16             display_span_list: true
17             display_resource: true
18             display_trace_id: true # true|false|open_telemetry|hexadecimal|decimal|datadog|uuid
19             display_span_id: true

Example json output:

JSON
1{
2  "timestamp": "2023-10-30T15:47:52.570482Z",
3  "level": "INFO",
4  "trace_id": "54ac7e5f0e8ab90ae67b822e95ffcbb8",
5  "span_id": "d52e3478c718b8a9",
6  "fields": {
7    "event_attr_1": "event_attr_1",
8    "event_attr_2": "event_attr_2"
9  },
10  "target": "event_target",
11  "filename": "event_file.rs",
12  "line_number": 32,
13  "span": {
14    "span_attr_1": "span_attr_1",
15    "span_attr_2": "span_attr_2",
16    "name": "span_name"
17  },
18  "spans": [
19    {
20      "span_attr_1": "span_attr_1",
21      "span_attr_2": "span_attr_2",
22      "name": "span_name"
23    }
24  ],
25  "threadName": "main",
26  "threadId": "ThreadId(1)"
27}

display_current_span

Events may also output information about the span that they are raised in, which is useful to log attributes attached to the span for a particular request.

To log span information, set the telemetry.exporters.logging.stdout.format.json.display_current_span option to true:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: 
7           json:
8             display_current_span: true

Example output with span information:

JSON
1{
2  "timestamp": "2023-10-30T14:09:34.771388Z",
3  "level": "INFO",
4  "fields": {
5    "event_attr_1": "event_attr_1",
6    "event_attr_2": "event_attr_2"
7  },
8  "target": "event_target",
9  "span": {
10    "span_attr_1": "span_attr_1",
11    "span_attr_2": "span_attr_2",
12    "name": "span_name"
13  }
14}

display_span_list

The telemetry.exporters.logging.stdout.format.json.display_span_list option is like display_current_span but instead of outputting information for the current span, display_span_list outputs information for all spans that an event was raised in.

For example, if you have a custom trace_id from a request header, as long as the attribute is configured on the router span it will appear on all log events associated with the request.

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: 
7           json:
8             display_span_list: true

Example output with a list of spans:

JSON
1{
2  "timestamp": "2023-10-30T14:09:34.771388Z",
3  "level": "INFO",
4  "fields": {
5    "event_attr_1": "event_attr_1",
6    "event_attr_2": "event_attr_2"
7  },
8  "target": "event_target",
9  "spans": [
10    {
11      "span_attr_1": "span_attr_1",
12      "span_attr_2": "span_attr_2",
13      "name": "span_name"
14    }
15  ]
16}

span_attributes

The telemetry.exporters.logging.stdout.format.json.span_attributes option allows you to display a subset of all span attributes. It takes as input an array of span attribute names to log.

When span_attributes is specified, the router searches for the first attribute in the list of span attributes from the root span to the current span and attaches it to the outermost JSON object for the log event.

If you set the same attribute name for different spans at different levels, the router chooses the attributes of child spans before the attributes of parent spans.

For example, you can display just the span_attr_1 span attribute:

YAML
router.yaml
1telemetry:
2  exporters:
3     logging:
4       stdout:
5         enabled: true
6         format: 
7           json:
8             display_span_list: false
9             span_attributes:
10             - span_attr_1

Example output with a list of spans:

JSON
1{
2  "timestamp": "2023-10-30T14:09:34.771388Z",
3  "level": "INFO",
4  "fields": {
5    "event_attr_1": "event_attr_1",
6    "event_attr_2": "event_attr_2"
7  },
8  "target": "event_target",
9  "span_attr_1": "span_attr_1"
10}

display_resource

The telemetry.logging.stdout.format.json.display_resource option configures whether resources configured in router.yaml are displayed in log messages. By default, display_resource is true.

For example, given a router.yaml with display_resource: true and a configured resource, log messages will display the resource:

YAML
router.yaml
1telemetry:
2  exporters:
3    logging:
4      stdout:
5        format: 
6          json:
7            display_resource: true
8      common:
9        service_name: bryn-router
10        resource:
11          test.resource: test
Text
"resource":{"test.resource":"test","service.name":"bryn-router"}

json configuration reference

OptionValuesDefaultEvent FieldDescription
display_current_spantrue|falsefalsespanThe span in which the event was raised and all of its' attributes.
display_filenametrue|falsefalsefilenameThe filename where the log event was raised.
display_leveltrue|falsetruelevelThe level of the log event, e.g. INFO, WARN, ERROR, TRACE.
display_line_numbertrue|falsefalseline_numberThe line number where the event was raised.
display_targettrue|falsetruetargetThe module name where the event was raised.
display_thread_idtrue|falsefalsethread_idThe id of the thread where the event was raised.
display_thread_nametrue|falsefalsethread_nameThe name of the thread where the event was raised.
display_timestamptrue|falsetruetimestampThe timestamp of when the event was raised.
display_span_listtrue|falsetruespansA list of all spans to root in which the event was raised and all of their attributes.
display_resourcetrue|falsetrueresourceThe resource as configured in tracing common.
display_trace_idtrue|false|open_telemetry|hexadecimal|decimal|datadog|uuidThe trace id of the span in which the event was raised.
display_span_idtrue|falsetruespan_idThe span id of the span in which the event was raised.
span_attributes[string][]*List of span attributes to attach to the JSON log object.
Feedback

Edit on GitHub

Forums