use chapters instead of parts to allow sub-headings
This commit is contained in:
parent
13b756b09c
commit
5a3e072f34
9 changed files with 118 additions and 127 deletions
|
@ -1,7 +0,0 @@
|
||||||
# Requests Error Budget Alert {#blocks-monitoring-budget-alerts}
|
|
||||||
|
|
||||||
This alert will fire when the ratio between number of requests getting a 5XX response from a service
|
|
||||||
and the total requests to that service exceeds 1%.
|
|
||||||
|
|
||||||
![](./assets/alert_rules_5xx_1.png)
|
|
||||||
![](./assets/alert_rules_5xx_2.png)
|
|
|
@ -1,64 +0,0 @@
|
||||||
# Configuration {#blocks-monitoring-configuration}
|
|
||||||
|
|
||||||
```nix
|
|
||||||
shb.monitoring = {
|
|
||||||
enable = true;
|
|
||||||
subdomain = "grafana";
|
|
||||||
inherit domain;
|
|
||||||
contactPoints = [ "me@example.com" ];
|
|
||||||
adminPasswordFile = config.sops.secrets."monitoring/admin_password".path;
|
|
||||||
secretKeyFile = config.sops.secrets."monitoring/secret_key".path;
|
|
||||||
};
|
|
||||||
|
|
||||||
sops.secrets."monitoring/admin_password" = {
|
|
||||||
sopsFile = ./secrets.yaml;
|
|
||||||
mode = "0400";
|
|
||||||
owner = "grafana";
|
|
||||||
group = "grafana";
|
|
||||||
restartUnits = [ "grafana.service" ];
|
|
||||||
};
|
|
||||||
sops.secrets."monitoring/secret_key" = {
|
|
||||||
sopsFile = ./secrets.yaml;
|
|
||||||
mode = "0400";
|
|
||||||
owner = "grafana";
|
|
||||||
group = "grafana";
|
|
||||||
restartUnits = [ "grafana.service" ];
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
With that, Grafana, Prometheus, Loki and Promtail are setup! You can access `Grafana` at
|
|
||||||
`grafana.example.com` with user `admin` and password ``.
|
|
||||||
|
|
||||||
I recommend adding a STMP server configuration so you receive alerts by email:
|
|
||||||
|
|
||||||
```nix
|
|
||||||
shb.monitoring.smtp = {
|
|
||||||
from_address = "grafana@$example.com";
|
|
||||||
from_name = "Grafana";
|
|
||||||
host = "smtp.mailgun.org";
|
|
||||||
port = 587;
|
|
||||||
username = "postmaster@mg.example.com";
|
|
||||||
passwordFile = config.sops.secrets."monitoring/smtp".path;
|
|
||||||
};
|
|
||||||
|
|
||||||
sops.secrets."monitoring/secret_key" = {
|
|
||||||
sopsFile = ./secrets.yaml;
|
|
||||||
mode = "0400";
|
|
||||||
owner = "grafana";
|
|
||||||
group = "grafana";
|
|
||||||
restartUnits = [ "grafana.service" ];
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
Since all logs are now stored in Loki, you can probably reduce the systemd journal retention
|
|
||||||
time with:
|
|
||||||
|
|
||||||
```nix
|
|
||||||
# See https://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMaxUse=
|
|
||||||
services.journald.extraConfig = ''
|
|
||||||
SystemMaxUse=2G
|
|
||||||
SystemKeepFree=4G
|
|
||||||
SystemMaxFileSize=100M
|
|
||||||
MaxFileSec=day
|
|
||||||
'';
|
|
||||||
```
|
|
|
@ -1,9 +0,0 @@
|
||||||
# Errors Dashboard {#blocks-monitoring-error-dashboard}
|
|
||||||
|
|
||||||
This dashboard is meant to be the first stop to understand why a service is misbehaving.
|
|
||||||
|
|
||||||
![](./assets/dashboards_Errors_1.png)
|
|
||||||
![](./assets/dashboards_Errors_2.png)
|
|
||||||
|
|
||||||
The yellow and red dashed vertical bars correspond to the [Requests Error Budget
|
|
||||||
Alert](#blocks-monitoring-budget-alerts) firing.
|
|
|
@ -1,6 +0,0 @@
|
||||||
# Performance Dashboard {#blocks-monitoring-performance-dashboard}
|
|
||||||
|
|
||||||
This dashboard is meant to be the first stop to understand why a service is performing poorly.
|
|
||||||
|
|
||||||
![Performance Dashboard Top Part](./assets/dashboards_Performance_1.png)
|
|
||||||
![Performance Dashboard Bottom Part](./assets/dashboards_Performance_2.png)
|
|
|
@ -8,10 +8,113 @@ This block sets up the monitoring stack for Self Host Blocks. It is composed of:
|
||||||
- Prometheus as the database for metrics.
|
- Prometheus as the database for metrics.
|
||||||
- Loki as the database for logs.
|
- Loki as the database for logs.
|
||||||
|
|
||||||
```{=include=} parts
|
## Configuration {#blocks-monitoring-configuration}
|
||||||
configuration.md
|
|
||||||
provisioning.md
|
```nix
|
||||||
dashboard-errors.md
|
shb.monitoring = {
|
||||||
dashboard-performance.md
|
enable = true;
|
||||||
alerts-requests-error-budger.md
|
subdomain = "grafana";
|
||||||
|
inherit domain;
|
||||||
|
contactPoints = [ "me@example.com" ];
|
||||||
|
adminPasswordFile = config.sops.secrets."monitoring/admin_password".path;
|
||||||
|
secretKeyFile = config.sops.secrets."monitoring/secret_key".path;
|
||||||
|
};
|
||||||
|
|
||||||
|
sops.secrets."monitoring/admin_password" = {
|
||||||
|
sopsFile = ./secrets.yaml;
|
||||||
|
mode = "0400";
|
||||||
|
owner = "grafana";
|
||||||
|
group = "grafana";
|
||||||
|
restartUnits = [ "grafana.service" ];
|
||||||
|
};
|
||||||
|
sops.secrets."monitoring/secret_key" = {
|
||||||
|
sopsFile = ./secrets.yaml;
|
||||||
|
mode = "0400";
|
||||||
|
owner = "grafana";
|
||||||
|
group = "grafana";
|
||||||
|
restartUnits = [ "grafana.service" ];
|
||||||
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
|
With that, Grafana, Prometheus, Loki and Promtail are setup! You can access `Grafana` at
|
||||||
|
`grafana.example.com` with user `admin` and password ``.
|
||||||
|
|
||||||
|
I recommend adding a STMP server configuration so you receive alerts by email:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
shb.monitoring.smtp = {
|
||||||
|
from_address = "grafana@$example.com";
|
||||||
|
from_name = "Grafana";
|
||||||
|
host = "smtp.mailgun.org";
|
||||||
|
port = 587;
|
||||||
|
username = "postmaster@mg.example.com";
|
||||||
|
passwordFile = config.sops.secrets."monitoring/smtp".path;
|
||||||
|
};
|
||||||
|
|
||||||
|
sops.secrets."monitoring/secret_key" = {
|
||||||
|
sopsFile = ./secrets.yaml;
|
||||||
|
mode = "0400";
|
||||||
|
owner = "grafana";
|
||||||
|
group = "grafana";
|
||||||
|
restartUnits = [ "grafana.service" ];
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Since all logs are now stored in Loki, you can probably reduce the systemd journal retention
|
||||||
|
time with:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# See https://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMaxUse=
|
||||||
|
services.journald.extraConfig = ''
|
||||||
|
SystemMaxUse=2G
|
||||||
|
SystemKeepFree=4G
|
||||||
|
SystemMaxFileSize=100M
|
||||||
|
MaxFileSec=day
|
||||||
|
'';
|
||||||
|
```
|
||||||
|
|
||||||
|
## Provisioning {#blocks-monitoring-provisioning}
|
||||||
|
|
||||||
|
Self Host Blocks will create automatically the following resources:
|
||||||
|
|
||||||
|
- For Grafana:
|
||||||
|
- datasources
|
||||||
|
- dashboards
|
||||||
|
- contact points
|
||||||
|
- notification policies
|
||||||
|
- alerts
|
||||||
|
- For Prometheus, the following exporters and related scrapers:
|
||||||
|
- node
|
||||||
|
- smartctl
|
||||||
|
- nginx
|
||||||
|
- For Loki, the following exporters and related scrapers:
|
||||||
|
- systemd
|
||||||
|
|
||||||
|
Those resources are namespaced as appropriate under the Self Host Blocks namespace:
|
||||||
|
|
||||||
|
[](./assets/folder.png)
|
||||||
|
|
||||||
|
## Errors Dashboard {#blocks-monitoring-error-dashboard}
|
||||||
|
|
||||||
|
This dashboard is meant to be the first stop to understand why a service is misbehaving.
|
||||||
|
|
||||||
|
![](./assets/dashboards_Errors_1.png)
|
||||||
|
![](./assets/dashboards_Errors_2.png)
|
||||||
|
|
||||||
|
The yellow and red dashed vertical bars correspond to the [Requests Error Budget
|
||||||
|
Alert](#blocks-monitoring-budget-alerts) firing.
|
||||||
|
|
||||||
|
## Performance Dashboard {#blocks-monitoring-performance-dashboard}
|
||||||
|
|
||||||
|
This dashboard is meant to be the first stop to understand why a service is performing poorly.
|
||||||
|
|
||||||
|
![Performance Dashboard Top Part](./assets/dashboards_Performance_1.png)
|
||||||
|
![Performance Dashboard Bottom Part](./assets/dashboards_Performance_2.png)
|
||||||
|
|
||||||
|
## Requests Error Budget Alert {#blocks-monitoring-budget-alerts}
|
||||||
|
|
||||||
|
This alert will fire when the ratio between number of requests getting a 5XX response from a service
|
||||||
|
and the total requests to that service exceeds 1%.
|
||||||
|
|
||||||
|
![](./assets/alert_rules_5xx_1.png)
|
||||||
|
![](./assets/alert_rules_5xx_2.png)
|
||||||
|
|
|
@ -1,20 +0,0 @@
|
||||||
# Provisioning {#blocks-monitoring-provisioning}
|
|
||||||
|
|
||||||
Self Host Blocks will create automatically the following resources:
|
|
||||||
|
|
||||||
- For Grafana:
|
|
||||||
- datasources
|
|
||||||
- dashboards
|
|
||||||
- contact points
|
|
||||||
- notification policies
|
|
||||||
- alerts
|
|
||||||
- For Prometheus, the following exporters and related scrapers:
|
|
||||||
- node
|
|
||||||
- smartctl
|
|
||||||
- nginx
|
|
||||||
- For Loki, the following exporters and related scrapers:
|
|
||||||
- systemd
|
|
||||||
|
|
||||||
Those resources are namespaced as appropriate under the Self Host Blocks namespace:
|
|
||||||
|
|
||||||
[](./assets/folder.png)
|
|
|
@ -96,12 +96,12 @@ in stdenv.mkDerivation {
|
||||||
${optionsDocs.optionsJSON}/share/doc/nixos/options.json
|
${optionsDocs.optionsJSON}/share/doc/nixos/options.json
|
||||||
|
|
||||||
find . -name "*.md" -print0 | \
|
find . -name "*.md" -print0 | \
|
||||||
while IFS= read -r -d ''' f; do
|
while IFS= read -r -d ''' f; do
|
||||||
substituteInPlace "''${f}" \
|
substituteInPlace "''${f}" \
|
||||||
--replace \
|
--replace \
|
||||||
'@REPO@' \
|
'@REPO@' \
|
||||||
"${lib.debug.traceVal ghRoot}"
|
"${ghRoot}"
|
||||||
done
|
done
|
||||||
|
|
||||||
nixos-render-docs manual html \
|
nixos-render-docs manual html \
|
||||||
--manpage-urls ${manpage-urls} \
|
--manpage-urls ${manpage-urls} \
|
||||||
|
|
|
@ -7,18 +7,10 @@
|
||||||
preface.md
|
preface.md
|
||||||
```
|
```
|
||||||
|
|
||||||
```{=include=} parts html:into-file=//blocks-monitoring.html
|
```{=include=} chapters html:into-file=//blocks-monitoring.html
|
||||||
blocks/monitoring/default.md
|
blocks/monitoring/default.md
|
||||||
```
|
```
|
||||||
|
|
||||||
```{=include=} appendix html:into-file=//options.html
|
```{=include=} appendix html:into-file=//options.html
|
||||||
options.md
|
options.md
|
||||||
```
|
```
|
||||||
|
|
||||||
<!-- ```{=include=} appendix html:into-file=//nixos-options.html -->
|
|
||||||
<!-- nixos-options.md -->
|
|
||||||
<!-- ``` -->
|
|
||||||
|
|
||||||
<!-- ```{=include=} appendix html:into-file=//nix-darwin-options.html -->
|
|
||||||
<!-- nix-darwin-options.md -->
|
|
||||||
<!-- ``` -->
|
|
||||||
|
|
|
@ -1,6 +1,8 @@
|
||||||
# Preface {#preface}
|
# Preface {#preface}
|
||||||
|
|
||||||
|
::: {.note}
|
||||||
This document is the complete manual for Self Host Blocks, the building blocks for self-hosting with battery included.
|
This document is the complete manual for Self Host Blocks, the building blocks for self-hosting with battery included.
|
||||||
|
:::
|
||||||
|
|
||||||
Self Host Blocks is hosted on [GitHub](https://github.com/ibizaman/selfhostblocks). If you encounter
|
Self Host Blocks is hosted on [GitHub](https://github.com/ibizaman/selfhostblocks). If you encounter
|
||||||
problems or bugs then please report them on the [issue
|
problems or bugs then please report them on the [issue
|
||||||
|
|
Loading…
Reference in a new issue