mash-playbook/docs/services/promtail.md

Promtail

Promtail agent is a log aggregation system designed to store and query logs from all your applications and infrastructure. It integrates nicely with Grafana Loki.

Dependencies

This service requires the following other services:

• Grafana Loki - a log-storage server where you'd be sending the logs
• (optional) Traefik - a reverse-proxy server, if you're exposing Promtail's metrics or API

Configuration

To enable this service, add the following configuration to your vars.yml file and re-run the installation process:

########################################################################
#                                                                      #
# promtail                                                             #
#                                                                      #
########################################################################

promtail_enabled: true

# See "Configuring scrapers" below.
# You need to enable at least one scraper to have Promtail do anything.

# If you haven't enabled Grafana Loki on the same server, you will need
# to define some clients to push logs to.
# See "Configuring clients" below.

########################################################################
#                                                                      #
# /promtail                                                            #
#                                                                      #
########################################################################

Configuring scrapers

No scrapers are enabled by default. As such, Promtail does not do anything in its default configuration.

Below, we show you a few built-in scrapers you can easily enable, as well as how to create your own custom ones.

Scraping systemd-journald logs

To scrape the systemd Journal, enable the already-prepared scraper for this with this additional vars.yml configuration:

# Some distros only store a non-persistent (in-memory) journal in a path like in `/run/log/journal`.
# Others may be using a path different than `/var/log/journal`.
# Adjust accordingly.
promtail_journald_scraper_enabled: true
promtail_journald_scraper_host_path: /var/log/journal

Scraping textual log files (/var/log, etc.)

A lot of distros dump textual log files in /var/log. To scrape them, enable the already-prepared scraper for this with this additional vars.yml configuration:

promtail_varlog_scraper_enabled: true
# Consider adjusting this if you'd like to scrape a different path
# promtail_varlog_scraper_host_path: /var/log

You can see the configuration for this scraper in the promtail_varlog_scraper_config variable in the defaults/main.yml file of the ansible-role-promtail Ansible role.

When using this scraper, beware that log-rotation may lead to double-ingestion as described here in the official documentation:

If you are rotating logs, be careful when using a wildcard pattern like *.log, and make sure it doesn’t match the rotated log file. For example, if you move your logs from server.log to server.01-01-1970.log in the same directory every night, a static config with a wildcard search pattern like *.log will pick up that new file and read it, effectively causing the entire days logs to be re-ingested.

To work around it, you may wish to adjust promtail_varlog_scraper_config_labels_path_suffix which defaults to /**/*log.

Scraping other directories

Besides the predefined scrapers described above, you can also define your own additional ones with the help of these variables:

• promtail_container_additional_mounts_custom, to mount additional paths into the Promtail container
• promtail_config_scrape_configs_custom, to inject additional jobs into Promtail's scrape_configs configuration. See promtail_journald_scraper_config and promtail_varlog_scraper_config for an example

Here's an example for scraping some hypothethical SSH logs stored somewhere:

promtail_container_additional_mounts_custom:
  - "type=bind,source=</path/to/ssh/logs>,target=/data/ssh,readonly"

promtail_config_scrape_configs_custom:
  - job_name: ssh
    static_configs:
    - localhost
      __path__: /data/ssh
      labels:
        job: ssh

Scraping syslog

The following example demonstrates the use of rsyslog and promtail to scrape syslog logs.

Prerequisites: Edit your rsyslog configuration in order to send logs to promtail.*`` This could be done by creating a /etc/rsyslog.d/00-promtail-relay.conf` file with the following content:

*.* action(type="omfwd" protocol="tcp" target="<promtail_host>" port="<promtail_port>" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on")

The port is a port number that you come up with yourself (e.g. 1234).

First, you need a custom scrape configuration which tells Promtail to listen on this port (replace SOME_PORT_NUMBER_IN_CONTAINER with your port number of choice):

promtail_config_scrape_configs_custom:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:SOME_PORT_NUMBER_IN_CONTAINER
      labels:
        job: syslog
    relabel_configs:
      - source_labels: [__syslog_message_hostname]
        target_label: host
      - source_labels: [__syslog_message_hostname]
        target_label: hostname
      - source_labels: [__syslog_message_severity]
        target_label: level
      - source_labels: [__syslog_message_app_name]
        target_label: application
      - source_labels: [__syslog_message_facility]
        target_label: facility
      - source_labels: [__syslog_connection_hostname]
        target_label: connection_hostname

You'd then need to expose this TCP port outside of the container, so that the local host (or remote host) can reach it.

To expose it on the loopback interface (reachable only from the same machine), use a configuration like this:

promtail_container_extra_arguments_custom:
  - "-p 127.0.0.1:1234:1234"

Configuring clients

If you've also enabled Grafana Loki on the same server, Promtail will automatically be configured to push logs to it.

Otherwise, you will need to extend the Promtail configuration by specifying clients to push to. Add something like this to your vars.yml configuration:

promtail_config_clients_custom:
  # Note the double /loki/loki.
  # This assumes Loki is installed at a `/loki` path-prefix.
  - url: https://mash.example.com/loki/loki/api/v1/push
    tenant_id: some-tenant-id-here

For more information about configuring clients, see the Promtail clients configuration reference.

Exposing the web interface

There are 2 reasons to expose Promtail to the public web:

1. So that you can scrape its Prometheus-compatible /metrics endpoint or observe its current /targets via API
2. So that you can use loki_push_api and push logs to Promtail (so that it can forward them onto its clients). This feature likely needs to be enabled explicitly.

To expose Promtail to the web, you need to assign a hostname in promtail_hostname and optionally a path-prefix.

You can then decide whether you'd like to expose Promtail's whole API via promtail_container_labels_api_enabled or just its metrics endpoint via promtail_container_labels_metrics_enabled.

Consult the defaults/main.yml file for variables related to these.

When exposing metrics, and especially the whole API, it's important to protected them. The Promtail Ansible role has variables that let you easily set up HTTP Basic Authentication via promtail_container_labels_api_traefik_middleware_basic_auth_* and promtail_container_labels_metrics_traefik_middleware_basic_auth_* variables.

Recommended other services

• Grafana Loki - a storage server for your logs compatible with Promtail
• Grafana - a web-based tool for visualizing your Promtail logs (stored in Grafana Loki or elsewhere)