1
0
Fork 0

distinguish building blocks and provided services

I want to show how composable this project is. For example, you could
just use the Authelia module to add SSO to any service, not just those
provided here.
This commit is contained in:
Pierre Penninckx 2023-11-20 22:20:19 -08:00 committed by GitHub
parent 2373d054c2
commit eae5eade56
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
22 changed files with 191 additions and 138 deletions

288
README.md
View file

@ -3,43 +3,33 @@
*Building blocks for self-hosting with battery included.* *Building blocks for self-hosting with battery included.*
SHB's (Self Host Blocks) goal is to provide a lower entry-bar for self-hosting. I intend to achieve SHB's (Self Host Blocks) goal is to provide a lower entry-bar for self-hosting. I intend to achieve
this by providing opinionated building blocks fitting together to self-host a wide range of this by providing opinionated [building blocks](#building-blocks) fitting together to self-host any service
services. Also, the design will be extendable to allow users to add services not provided by SHB. you'd want. Some [common services](#provided-services) are provided out of the box.
For each service, I intend to provide turn-key Nix options to setup: The building blocks allow you to easily setup:
- access through a subdomain, - Access through a subdomain ([Nginx](https://www.nginx.com/)).
- HTTPS access, - HTTPS access ([Nginx](https://www.nginx.com/) + [Letsencrypt](https://letsencrypt.org/)).
- backup, - Backup ([Borgmatic](https://torsion.org/borgmatic/) and/or [Restic](https://restic.net/)).
- single sign-on, - Single sign-on ([Authelia](https://www.authelia.com/)).
- LDAP user management, - LDAP user management ([LLDAP](https://github.com/lldap/lldap)).
- and metrics and logs monitoring and alerting. - Metrics, logs and alerting ([Grafana](https://grafana.com/) + [Prometheus](https://prometheus.io/) + [Loki](https://grafana.com/oss/loki/) + [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) + [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)).
- Database setup (Only [Postgresql](https://www.postgresql.org/) so far).
- VPN tunnels with optional proxys ([OpenVPN](https://openvpn.net/) with [Tinyproxy](http://tinyproxy.github.io/)).
The provided services will have all those integrated. Progress is detailed in the [Supported Features](#supported-features) section.
You should know that although I am using everything in this repo for my personal production server, this is
really just a one person effort for now and there are most certainly bugs that I didn't discover yet.
## TOC ## TOC
<!--toc:start--> <!--toc:start-->
- [Supported Features](#supported-features) - [Supported Features](#supported-features)
- [Usage](#usage) - [Building Blocks](#building-blocks)
- [Provided Services](#provided-services)
- [Demos](#demos) - [Demos](#demos)
- [Examples](#examples) - [Import selfhostblocks](#import-selfhostblocks)
- [Add SSL configuration](#add-ssl-configuration)
- [Add LDAP and Authelia services](#add-ldap-and-authelia-services)
- [Deploy the full Grafana, Prometheus and Loki suite](#deploy-the-full-grafana-prometheus-and-loki-suite)
- [Deploy a Nextcloud Instance](#deploy-a-nextcloud-instance)
- [Enable verbose Nginx logging](#enable-verbose-nginx-logging)
- [Deploy an hledger Instance with LDAP and SSO support](#deploy-an-hledger-instance-with-ldap-and-sso-support)
- [Deploy a Jellyfin instance with LDAP and SSO support](#deploy-a-jellyfin-instance-with-ldap-and-sso-support)
- [Deploy a Home Assistant instance with LDAP support](#deploy-a-home-assistant-instance-with-ldap-support)
- [Set up network tunnel with VPN and Proxy](#set-up-network-tunnel-with-vpn-and-proxy)
- [Tips](#tips) - [Tips](#tips)
- [Run tests](#run-tests)
- [Deploy using colmena](#deploy-using-colmena)
- [Use a local version of selfhostblocks](#use-a-local-version-of-selfhostblocks)
- [Diff changes](#diff-changes)
- [What is deployed](#what-is-deployed)
- [What will get deployed](#what-will-get-deployed)
- [Get the full diff](#get-the-full-diff)
- [Get version bumps](#get-version-bumps)
- [Generate random secret](#generate-random-secret)
- [TODOs](#todos) - [TODOs](#todos)
- [Links that helped](#links-that-helped) - [Links that helped](#links-that-helped)
- [License](#license) - [License](#license)
@ -75,8 +65,9 @@ Currently supported services and features are:
- [X] LDAP auth, unfortunately we need to configure this manually. - [X] LDAP auth, unfortunately we need to configure this manually.
- [ ] Declarative setup. - [ ] Declarative setup.
- [ ] SSO auth. - [ ] SSO auth.
- [ ] Declarative setup.
- [X] Backup support. - [X] Backup support.
- [x] Optional tracing debug. - [X] Optional tracing debug.
- [ ] Export traces to Prometheus. - [ ] Export traces to Prometheus.
- [ ] Export metrics to Prometheus. - [ ] Export metrics to Prometheus.
- [X] Home Assistant. - [X] Home Assistant.
@ -105,80 +96,40 @@ Currently supported services and features are:
- [ ] Gitea to deploy - [ ] Gitea to deploy
- [ ] Scrutiny to monitor hard drives health - [ ] Scrutiny to monitor hard drives health
- [ ] Export metrics to Prometheus. - [ ] Export metrics to Prometheus.
- [x] Tests - [x] QoL
- [x] Unit tests for modules. - [x] Unit tests for modules.
- [x] Running in CI. - [x] Running in CI.
- [ ] Integration tests with real nodes. - [ ] Integration tests with real nodes.
- [ ] Self published documentation for options.
- [ ] Examples for all building blocks.
## Usage ## Building Blocks
The top-level `flake.nix` just outputs a nixos module that gathers all other modules from the [`modules/`](./modules/) directory. The building blocks are the foundation selfhostblocks intend to provide to allow you to self host
easily and with best practices any service of your choosing. Some services are already provided out of
the box but you might not want to use those if for example you want to integrate with existing
services or slowly transition to NixOS.
Some provided modules are low-level and some are high-level that re-use those low-level ones. For Following somewhat the Unix principle, each block has one goal and does it correctly. They also are
example, the nextcloud module re-uses the backup and nginx ones. independent of each other, you can use only one or combine them to your liking.
You want to use this repo as a flake input to your own repo. The `inputs` field of your `flake.nix` Although these blocks provide options that encourage best practices, these are just NixOS modules that
file in your repo should look like so: configure other modules provided by nixpkgs. Would you need to make tweaks, you can always
access those underlying modules directly, like for any NixOS module.
```nix - [`authelia.nix`](./modules/blocks/authelia.nix) for Single Sign On.
inputs = { - [`backup.nix`](./modules/blocks/backup.nix).
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; - [`ldap.nix`](./modules/blocks/ldap.nix) for user management.
sops-nix.url = "github:Mic92/sops-nix"; - [`monitoring.nix`](./modules/blocks/monitoring.nix) for dashboards, logs and alerts.
- [`nginx.nix`](./modules/blocks/nginx.nix) for reverse proxy with SSL termination.
- [`postgresql.nix`](./modules/blocks/postgresql.nix) for database setup.
- [`ssl.nix`](./modules/blocks/ssl.nix) for maintaining SSL certificates provided by letsencrypt.
- [`tinyproxy.nix`](./modules/blocks/tinyproxy.nix) to forward traffic to a VPN tunnel.
- [`vpn.nix`](./modules/blocks/vpn.nix) to setup a VPN tunnel.
selfhostblocks.url = "github:ibizaman/selfhostblocks"; The best way for now to understand how to use those modules is to read the code linked above and see
selfhostblocks.inputs.nixpkgs.follows = "nixpkgs"; how they are used in the [provided services](#provided-services) and in the [demos](#demos). Also, here are a
selfhostblocks.inputs.sops-nix.follows = "sops-nix"; few examples taken from my personal usage of selfhostblocks.
};
```
`sops-nix` is used to setup passwords and secrets. Currently `selfhostblocks` has a strong
dependency on it but I'm working on removing that so you could use any secret provider.
The snippet above makes `selfhostblocks`' inputs follow yours. This is not maintainable though
because options that `selfhostblocks` rely on can change or disappear and you have no control on
that. Later, I intend to make `selfhostblocks` provide its own `nixpkgs` input and update it myself
through CI.
How you actually deploy using selfhostblocks depends on what system you choose. If you use
[colmena](https://colmena.cli.rs), this is what your `outputs` field could look like:
```nix
outputs = inputs@{ self, nixpkgs, ... }: {
colmena = {
meta = {
nixpkgs = import inputs.nixpkgs {
system = "x86_64-linux";
};
specialArgs = inputs;
};
myserver = import ./machines/myserver.nix;
};
}
```
Now, what goes inside this `./machines/myserver.nix` file? First, import `selfhostblocks` and
`sops-nix`:
```nix
imports = [
selfhostblocks.nixosModules.x86_64-linux.default
sops-nix.nixosModules.default
]
```
For how to configure the services, check the sections below.
## Demos
Demos that start and deploy on a Virtual Machine on your computer are located under the
[demo](./demo/) folder. These show the onboarding experience you would get if you deployed
selfhostblocks on your own server.
## Examples
I plan to have documentation for all options provided by selfhostblocks and more examples. For now,
I have a few examples:
### Add SSL configuration ### Add SSL configuration
@ -412,6 +363,69 @@ MaxFileSec=day
''; '';
``` ```
### Set up network tunnel with VPN and Proxy
```nix
shb.vpn.nordvpnus = {
enable = true;
# Only "nordvpn" supported for now.
provider = "nordvpn";
dev = "tun1";
# Must be unique per VPN instance.
routingNumber = 10;
# Change to the one you want to connect to
remoteServerIP = "1.2.3.4";
sopsFile = ./secrets/vpn.yaml;
proxyPort = 12000;
};
```
This sets up a tunnel interface `tun1` that connects to the VPN provider, here NordVPN. Also, if the
`proxyPort` option is not null, this will spin up a `tinyproxy` instance that listens on the given
port and redirects all traffic through that VPN.
```bash
$ curl 'https://api.ipify.org?format=json'
{"ip":"107.21.107.115"}
$ curl --interface tun1 'https://api.ipify.org?format=json'
{"ip":"46.12.123.113"}
$ curl --proxy 127.0.0.1:12000 'https://api.ipify.org?format=json'
{"ip":"46.12.123.113"}
```
## Provided Services
- [`arr.nix`](./modules/services/arr.nix) for finding media https://wiki.servarr.com/.
- [`deluge.nix`](./modules/services/deluge.nix) for downloading linux isos https://deluge-torrent.org/.
- [`hledger.nix`](./modules/services/hledger.nix) for managing finances https://hledger.org/.
- [`home-assistant.nix`](./modules/services/home-assistant.nix) for private IoT https://www.home-assistant.io/.
- [`jellyfin.nix`](./modules/services/jellyfin.nix) for watching media https://jellyfin.org/.
- [`nextcloud-server.nix`](./modules/services/nextcloud-server.nix) for private documents, contacts, calendar, etc https://nextcloud.com.
- [`vaultwarden.nix`](./modules/services/vaultwarden.nix) for passwords https://github.com/dani-garcia/vaultwarden.
The services above are those I am using myself. I intend to add more.
The best way for now to understand how to use those modules is to read the code linked above and see
how they are used in the demos. Also, here are a
few examples taken from my personal usage of selfhostblocks.
### Common Options
Some common options are provided for all services.
- `enable` (bool). Set to true to deploy and run the service.
- `subdomain` (string). Subdomain under which to serve the service.
- `domain` (string). Domain under which to server the service.
Some other common options are the following. I am not satisfied with how those are expressed so those will most certainly change.
- LDAP and OIDC options for SSO, authentication and authorization.
- Secrets.
- Backups.
Note that for backups, every service exposes what directory should be backed up, you must merely choose when those backups will take place and where they will be stored.
### Deploy a Nextcloud Instance ### Deploy a Nextcloud Instance
```nix ```nix
@ -597,38 +611,70 @@ home-assistant: |
longitude_home: "-0.01234567890123" longitude_home: "-0.01234567890123"
``` ```
### Set up network tunnel with VPN and Proxy ## Demos
Demos that start and deploy a service on a Virtual Machine on your computer are located under the
[demo](./demo/) folder. These show the onboarding experience you would get if you deployed
one of the services on your own server.
## Import selfhostblocks
Ready to start using selfhostblocks? Thank you for trusting selfhostblocks. Please raise any
question you have or hurdle you encounter by creating an issue.
The top-level `flake.nix` just outputs a nixos module that gathers all other modules from
the [`modules/`](./modules/) directory. Use this repo as a flake input to your own repo.
The `inputs` field of your `flake.nix` file in your repo should look like so:
```nix ```nix
shb.vpn.nordvpnus = { inputs = {
enable = true; nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
# Only "nordvpn" supported for now. sops-nix.url = "github:Mic92/sops-nix";
provider = "nordvpn";
dev = "tun1"; selfhostblocks.url = "github:ibizaman/selfhostblocks";
# Must be unique per VPN instance. selfhostblocks.inputs.nixpkgs.follows = "nixpkgs";
routingNumber = 10; selfhostblocks.inputs.sops-nix.follows = "sops-nix";
# Change to the one you want to connect to
remoteServerIP = "1.2.3.4";
sopsFile = ./secrets/vpn.yaml;
proxyPort = 12000;
}; };
``` ```
This sets up a tunnel interface `tun1` that connects to the VPN provider, here NordVPN. Also, if the `sops-nix` is used to setup passwords and secrets. Currently `selfhostblocks` has a strong
`proxyPort` option is not null, this will spin up a `tinyproxy` instance that listens on the given dependency on it but I'm working on removing that so you could use any secret provider.
port and redirects all traffic through that VPN.
```bash The snippet above makes `selfhostblocks`' inputs follow yours. This is not maintainable though
$ curl 'https://api.ipify.org?format=json' because options that `selfhostblocks` rely on can change or disappear and you have no control on
{"ip":"107.21.107.115"} that. Later, I intend to make `selfhostblocks` provide its own `nixpkgs` input and update it myself
through CI.
$ curl --interface tun1 'https://api.ipify.org?format=json' How you actually deploy using selfhostblocks depends on what system you choose. If you use
{"ip":"46.12.123.113"} [colmena](https://colmena.cli.rs), this is what your `outputs` field could look like:
$ curl --proxy 127.0.0.1:12000 'https://api.ipify.org?format=json' ```nix
{"ip":"46.12.123.113"} outputs = inputs@{ self, nixpkgs, ... }: {
colmena = {
meta = {
nixpkgs = import inputs.nixpkgs {
system = "x86_64-linux";
};
specialArgs = inputs;
};
myserver = import ./machines/myserver.nix;
};
}
``` ```
Now, what goes inside this `./machines/myserver.nix` file? First, import `selfhostblocks` and
`sops-nix`:
```nix
imports = [
selfhostblocks.nixosModules.x86_64-linux.default
sops-nix.nixosModules.default
]
```
For the rest, see the above [building blocks](#building-blocks), [provided services](#provided-services) and [demos](#demos) sections.
## Tips ## Tips
### Run tests ### Run tests
@ -654,7 +700,13 @@ selfhostblocks.url = "/home/me/projects/selfhostblocks";
Or override on the command line: Or override on the command line:
```bash ```bash
$ nix run nixpkgs#colmena --override-input selfhostblocks ../selfhostblocks -- apply $ nix flake lock --override-input selfhostblocks ../selfhostblocks
```
I usually combine the override snippet above with deploying:
```bash
$ nix flake lock --override-input selfhostblocks ../selfhostblocks && nix run nixpkgs#colmena -- apply
``` ```
### Diff changes ### Diff changes

View file

@ -15,23 +15,24 @@
{ {
nixosModules.default = { config, ... }: { nixosModules.default = { config, ... }: {
imports = [ imports = [
modules/arr.nix modules/blocks/authelia.nix
modules/authelia.nix modules/blocks/backup.nix
modules/backup.nix modules/blocks/davfs.nix
modules/deluge.nix modules/blocks/ldap.nix
modules/davfs.nix modules/blocks/monitoring.nix
modules/hledger.nix modules/blocks/nginx.nix
modules/home-assistant.nix modules/blocks/postgresql.nix
modules/jellyfin.nix modules/blocks/ssl.nix
modules/ldap.nix modules/blocks/tinyproxy.nix
modules/monitoring.nix modules/blocks/vpn.nix
modules/nextcloud-server.nix
modules/nginx.nix modules/services/arr.nix
modules/postgresql.nix modules/services/deluge.nix
modules/ssl.nix modules/services/hledger.nix
modules/tinyproxy.nix modules/services/home-assistant.nix
modules/vaultwarden.nix modules/services/jellyfin.nix
modules/vpn.nix modules/services/nextcloud-server.nix
modules/services/vaultwarden.nix
]; ];
}; };

View file

@ -24,7 +24,7 @@ let
services.sonarr = anyOpt {}; services.sonarr = anyOpt {};
}; };
} }
../../modules/arr.nix ../../modules/services/arr.nix
m m
]; ];
}).config; }).config;

View file

@ -16,7 +16,7 @@ let
services = anyOpt {}; services = anyOpt {};
}; };
} }
../../modules/davfs.nix ../../modules/blocks/davfs.nix
m m
]; ];
}).config; }).config;

View file

@ -15,7 +15,7 @@ let
systemd = anyOpt {}; systemd = anyOpt {};
}; };
} }
../../modules/postgresql.nix ../../modules/blocks/postgresql.nix
m m
]; ];
}).config; }).config;