172 lines
7.3 KiB
Markdown
172 lines
7.3 KiB
Markdown
|
# Promtail
|
||
|
|
|
||
|
|
[Promtail](https://grafana.com/oss/promtail/) agent is a log aggregation system designed to store and query logs from all your applications and infrastructure. It integrates nicely with [Grafana Loki](./grafana-loki.md).
|
||
|
|
|
||
|
|
|
||
|
|
## Dependencies
|
||
|
|
|
||
|
|
This service requires the following other services:
|
||
|
|
|
||
|
|
- [Grafana Loki](grafana-loki.md) - a log-storage server where you'd be sending the logs
|
||
|
|
- (optional) [Traefik](traefik.md) - a reverse-proxy server, if you're exposing Promtail's metrics or API
|
||
|
|
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
To enable this service, add the following configuration to your `vars.yml` file and re-run the [installation](../installing.md) process:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
########################################################################
|
||
|
|
# #
|
||
|
|
# promtail #
|
||
|
|
# #
|
||
|
|
########################################################################
|
||
|
|
|
||
|
|
promtail_enabled: true
|
||
|
|
|
||
|
|
# See "Configuring scrapers" below.
|
||
|
|
# You need to enable at least one scraper to have Promtail do anything.
|
||
|
|
|
||
|
|
# If you haven't enabled Grafana Loki on the same server, you will need
|
||
|
|
# to define some clients to push logs to.
|
||
|
|
# See "Configuring clients" below.
|
||
|
|
|
||
|
|
########################################################################
|
||
|
|
# #
|
||
|
|
# /promtail #
|
||
|
|
# #
|
||
|
|
########################################################################
|
||
|
|
```
|
||
|
|
|
||
|
|
### Configuring scrapers
|
||
|
|
|
||
|
|
**No scrapers are enabled by default**. As such, Promtail does not do anything in its default configuration.
|
||
|
|
|
||
|
|
Below, we show you a few built-in scrapers you can easily enable, as well as how to create your own custom ones.
|
||
|
|
|
||
|
|
#### Scraping systemd-journald logs
|
||
|
|
|
||
|
|
To scrape the [systemd Journal](https://wiki.archlinux.org/title/Systemd/Journal), enable the already-prepared scraper for this with this additional `vars.yml` configuration:
|
||
|
|
|
||
|
|
```yml
|
||
|
|
# Some distros only store a non-persistent (in-memory) journal in a path like in `/run/log/journal`.
|
||
|
|
# Others may be using a path different than `/var/log/journal`.
|
||
|
|
# Adjust accordingly.
|
||
|
|
promtail_journald_scraper_enabled: true
|
||
|
|
promtail_journald_scraper_host_path: /var/log/journal
|
||
|
|
```
|
||
|
|
|
||
|
|
#### Scraping textual log files (/var/log, etc.)
|
||
|
|
|
||
|
|
A lot of distros dump textual log files in `/var/log`. To scrape them, enable the already-prepared scraper for this with this additional `vars.yml` configuration:
|
||
|
|
|
||
|
|
```yml
|
||
|
|
promtail_varlog_scraper_enabled: true
|
||
|
|
# Consider adjusting this if you'd like to scrape a different path
|
||
|
|
# promtail_varlog_scraper_host_path: /var/log
|
||
|
|
```
|
||
|
|
|
||
|
|
#### Scraping other directories
|
||
|
|
|
||
|
|
Besides the predefined scrapers described above, you can also define your own additional ones with the help of these variables:
|
||
|
|
|
||
|
|
- `promtail_container_additional_mounts_custom`, to mount additional paths into the Promtail container
|
||
|
|
- `promtail_config_scrape_configs_custom`, to inject additional jobs into Promtail's `scrape_configs` configuration. See `promtail_journald_scraper_config` and `promtail_varlog_scraper_config` for an example
|
||
|
|
|
||
|
|
Here's an example for scraping some hypothethical SSH logs stored somewhere:
|
||
|
|
|
||
|
|
```yml
|
||
|
|
promtail_container_additional_mounts_custom:
|
||
|
|
- "type=bind,source=</path/to/ssh/logs>,target=/data/ssh,readonly"
|
||
|
|
|
||
|
|
|
||
|
|
promtail_config_scrape_configs_custom:
|
||
|
|
- job_name: ssh
|
||
|
|
static_configs:
|
||
|
|
- localhost
|
||
|
|
__path__: /data/ssh
|
||
|
|
labels:
|
||
|
|
job: ssh
|
||
|
|
```
|
||
|
|
|
||
|
|
##### Scraping syslog
|
||
|
|
|
||
|
|
The following example demonstrates the use of rsyslog and promtail to scrape syslog logs.
|
||
|
|
|
||
|
|
**Prerequisites**: Edit your rsyslog configuration in order to send logs to `promtail.*``
|
||
|
|
This could be done by creating a `/etc/rsyslog.d/00-promtail-relay.conf` file with the following content:
|
||
|
|
|
||
|
|
```
|
||
|
|
*.* action(type="omfwd" protocol="tcp" target="<promtail_host>" port="<promtail_port>" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on")
|
||
|
|
```
|
||
|
|
|
||
|
|
The port is a port number that you come up with yourself (e.g. `1234`).
|
||
|
|
|
||
|
|
First, you need a custom scrape configuration which tells Promtail to listen on this port (replace `SOME_PORT_NUMBER_IN_CONTAINER` with your port number of choice):
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
promtail_config_scrape_configs_custom:
|
||
|
|
- job_name: syslog
|
||
|
|
syslog:
|
||
|
|
listen_address: 0.0.0.0:SOME_PORT_NUMBER_IN_CONTAINER
|
||
|
|
labels:
|
||
|
|
job: syslog
|
||
|
|
relabel_configs:
|
||
|
|
- source_labels: [__syslog_message_hostname]
|
||
|
|
target_label: host
|
||
|
|
- source_labels: [__syslog_message_hostname]
|
||
|
|
target_label: hostname
|
||
|
|
- source_labels: [__syslog_message_severity]
|
||
|
|
target_label: level
|
||
|
|
- source_labels: [__syslog_message_app_name]
|
||
|
|
target_label: application
|
||
|
|
- source_labels: [__syslog_message_facility]
|
||
|
|
target_label: facility
|
||
|
|
- source_labels: [__syslog_connection_hostname]
|
||
|
|
target_label: connection_hostname
|
||
|
|
```
|
||
|
|
|
||
|
|
You'd then need to expose this TCP port outside of the container, so that the local host (or remote host) can reach it.
|
||
|
|
|
||
|
|
To expose it on the loopback interface (reachable only from the same machine), use a configuration like this:
|
||
|
|
```yaml
|
||
|
|
promtail_container_extra_arguments_custom:
|
||
|
|
- "-p 127.0.0.1:1234:1234"
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
### Configuring clients
|
||
|
|
|
||
|
|
If you've also enabled [Grafana Loki](./grafana-loki.md) on the same server, Promtail will automatically be configured to push logs to it.
|
||
|
|
|
||
|
|
Otherwise, you will need to extend the Promtail configuration by specifying clients to push to. Add something like this to your `vars.yml` configuration:
|
||
|
|
|
||
|
|
```yml
|
||
|
|
promtail_config_clients_custom:
|
||
|
|
# Note the double /loki/loki.
|
||
|
|
# This assumes Loki is installed at a `/loki` path-prefix.
|
||
|
|
- url: https://mash.example.com/loki/loki/api/v1/push
|
||
|
|
```
|
||
|
|
|
||
|
|
### Exposing the web interface
|
||
|
|
|
||
|
|
There are 2 reasons to expose Promtail to the public web:
|
||
|
|
|
||
|
|
1. So that you can scrape its Prometheus-compatible `/metrics` endpoint or observe its current `/targets` via API
|
||
|
|
2. So that you can use [loki_push_api](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#loki_push_api) and push logs to Promtail (so that it can forward them onto its [clients](#configuring-clients)). This feature likely needs to be enabled explicitly.
|
||
|
|
|
||
|
|
To expose Promtail to the web, you need to assign a hostname in `promtail_hostname` and optionally a path-prefix.
|
||
|
|
|
||
|
|
You can then decide whether you'd like to expose Promtail's whole API via `promtail_container_labels_api_enabled` or just its metrics endpoint via `promtail_container_labels_metrics_enabled`.
|
||
|
|
|
||
|
|
Consult the `defaults/main.yml` file for variables related to these.
|
||
|
|
|
||
|
|
When exposing metrics, and especially the whole API, it's important to protected them. The Promtail Ansible role has variables that let you easily set up [HTTP Basic Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication) via `promtail_container_labels_api_traefik_middleware_basic_auth_*` and `promtail_container_labels_metrics_traefik_middleware_basic_auth_*` variables.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Recommended other services
|
||
|
|
|
||
|
|
- [Grafana Loki](grafana-loki.md) - a storage server for your logs compatible with Promtail
|
||
|
|
- [Grafana](grafana.md) - a web-based tool for visualizing your Promtail logs (stored in [Grafana Loki](grafana-loki.md) or elsewhere)
|