diff --git a/README.md b/README.md index b6eefc6..d37cb75 100644 --- a/README.md +++ b/README.md @@ -3,43 +3,33 @@ *Building blocks for self-hosting with battery included.* SHB's (Self Host Blocks) goal is to provide a lower entry-bar for self-hosting. I intend to achieve -this by providing opinionated building blocks fitting together to self-host a wide range of -services. Also, the design will be extendable to allow users to add services not provided by SHB. +this by providing opinionated [building blocks](#building-blocks) fitting together to self-host any service +you'd want. Some [common services](#provided-services) are provided out of the box. -For each service, I intend to provide turn-key Nix options to setup: -- access through a subdomain, -- HTTPS access, -- backup, -- single sign-on, -- LDAP user management, -- and metrics and logs monitoring and alerting. +The building blocks allow you to easily setup: +- Access through a subdomain ([Nginx](https://www.nginx.com/)). +- HTTPS access ([Nginx](https://www.nginx.com/) + [Letsencrypt](https://letsencrypt.org/)). +- Backup ([Borgmatic](https://torsion.org/borgmatic/) and/or [Restic](https://restic.net/)). +- Single sign-on ([Authelia](https://www.authelia.com/)). +- LDAP user management ([LLDAP](https://github.com/lldap/lldap)). +- Metrics, logs and alerting ([Grafana](https://grafana.com/) + [Prometheus](https://prometheus.io/) + [Loki](https://grafana.com/oss/loki/) + [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) + [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)). +- Database setup (Only [Postgresql](https://www.postgresql.org/) so far). +- VPN tunnels with optional proxys ([OpenVPN](https://openvpn.net/) with [Tinyproxy](http://tinyproxy.github.io/)). + +The provided services will have all those integrated. Progress is detailed in the [Supported Features](#supported-features) section. + +You should know that although I am using everything in this repo for my personal production server, this is +really just a one person effort for now and there are most certainly bugs that I didn't discover yet. ## TOC - [Supported Features](#supported-features) -- [Usage](#usage) +- [Building Blocks](#building-blocks) +- [Provided Services](#provided-services) - [Demos](#demos) -- [Examples](#examples) - - [Add SSL configuration](#add-ssl-configuration) - - [Add LDAP and Authelia services](#add-ldap-and-authelia-services) - - [Deploy the full Grafana, Prometheus and Loki suite](#deploy-the-full-grafana-prometheus-and-loki-suite) - - [Deploy a Nextcloud Instance](#deploy-a-nextcloud-instance) - - [Enable verbose Nginx logging](#enable-verbose-nginx-logging) - - [Deploy an hledger Instance with LDAP and SSO support](#deploy-an-hledger-instance-with-ldap-and-sso-support) - - [Deploy a Jellyfin instance with LDAP and SSO support](#deploy-a-jellyfin-instance-with-ldap-and-sso-support) - - [Deploy a Home Assistant instance with LDAP support](#deploy-a-home-assistant-instance-with-ldap-support) - - [Set up network tunnel with VPN and Proxy](#set-up-network-tunnel-with-vpn-and-proxy) +- [Import selfhostblocks](#import-selfhostblocks) - [Tips](#tips) - - [Run tests](#run-tests) - - [Deploy using colmena](#deploy-using-colmena) - - [Use a local version of selfhostblocks](#use-a-local-version-of-selfhostblocks) - - [Diff changes](#diff-changes) - - [What is deployed](#what-is-deployed) - - [What will get deployed](#what-will-get-deployed) - - [Get the full diff](#get-the-full-diff) - - [Get version bumps](#get-version-bumps) - - [Generate random secret](#generate-random-secret) - [TODOs](#todos) - [Links that helped](#links-that-helped) - [License](#license) @@ -75,8 +65,9 @@ Currently supported services and features are: - [X] LDAP auth, unfortunately we need to configure this manually. - [ ] Declarative setup. - [ ] SSO auth. + - [ ] Declarative setup. - [X] Backup support. - - [x] Optional tracing debug. + - [X] Optional tracing debug. - [ ] Export traces to Prometheus. - [ ] Export metrics to Prometheus. - [X] Home Assistant. @@ -105,80 +96,40 @@ Currently supported services and features are: - [ ] Gitea to deploy - [ ] Scrutiny to monitor hard drives health - [ ] Export metrics to Prometheus. -- [x] Tests +- [x] QoL - [x] Unit tests for modules. - [x] Running in CI. - [ ] Integration tests with real nodes. + - [ ] Self published documentation for options. + - [ ] Examples for all building blocks. -## Usage +## Building Blocks -The top-level `flake.nix` just outputs a nixos module that gathers all other modules from the [`modules/`](./modules/) directory. +The building blocks are the foundation selfhostblocks intend to provide to allow you to self host +easily and with best practices any service of your choosing. Some services are already provided out of +the box but you might not want to use those if for example you want to integrate with existing +services or slowly transition to NixOS. -Some provided modules are low-level and some are high-level that re-use those low-level ones. For -example, the nextcloud module re-uses the backup and nginx ones. +Following somewhat the Unix principle, each block has one goal and does it correctly. They also are +independent of each other, you can use only one or combine them to your liking. -You want to use this repo as a flake input to your own repo. The `inputs` field of your `flake.nix` -file in your repo should look like so: +Although these blocks provide options that encourage best practices, these are just NixOS modules that +configure other modules provided by nixpkgs. Would you need to make tweaks, you can always +access those underlying modules directly, like for any NixOS module. -```nix -inputs = { - nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; - sops-nix.url = "github:Mic92/sops-nix"; +- [`authelia.nix`](./modules/blocks/authelia.nix) for Single Sign On. +- [`backup.nix`](./modules/blocks/backup.nix). +- [`ldap.nix`](./modules/blocks/ldap.nix) for user management. +- [`monitoring.nix`](./modules/blocks/monitoring.nix) for dashboards, logs and alerts. +- [`nginx.nix`](./modules/blocks/nginx.nix) for reverse proxy with SSL termination. +- [`postgresql.nix`](./modules/blocks/postgresql.nix) for database setup. +- [`ssl.nix`](./modules/blocks/ssl.nix) for maintaining SSL certificates provided by letsencrypt. +- [`tinyproxy.nix`](./modules/blocks/tinyproxy.nix) to forward traffic to a VPN tunnel. +- [`vpn.nix`](./modules/blocks/vpn.nix) to setup a VPN tunnel. - selfhostblocks.url = "github:ibizaman/selfhostblocks"; - selfhostblocks.inputs.nixpkgs.follows = "nixpkgs"; - selfhostblocks.inputs.sops-nix.follows = "sops-nix"; -}; -``` - -`sops-nix` is used to setup passwords and secrets. Currently `selfhostblocks` has a strong -dependency on it but I'm working on removing that so you could use any secret provider. - -The snippet above makes `selfhostblocks`' inputs follow yours. This is not maintainable though -because options that `selfhostblocks` rely on can change or disappear and you have no control on -that. Later, I intend to make `selfhostblocks` provide its own `nixpkgs` input and update it myself -through CI. - -How you actually deploy using selfhostblocks depends on what system you choose. If you use -[colmena](https://colmena.cli.rs), this is what your `outputs` field could look like: - -```nix -outputs = inputs@{ self, nixpkgs, ... }: { - colmena = { - meta = { - nixpkgs = import inputs.nixpkgs { - system = "x86_64-linux"; - }; - specialArgs = inputs; - }; - - myserver = import ./machines/myserver.nix; - }; -} -``` - -Now, what goes inside this `./machines/myserver.nix` file? First, import `selfhostblocks` and -`sops-nix`: - -```nix -imports = [ - selfhostblocks.nixosModules.x86_64-linux.default - sops-nix.nixosModules.default -] -``` - -For how to configure the services, check the sections below. - -## Demos - -Demos that start and deploy on a Virtual Machine on your computer are located under the -[demo](./demo/) folder. These show the onboarding experience you would get if you deployed -selfhostblocks on your own server. - -## Examples - -I plan to have documentation for all options provided by selfhostblocks and more examples. For now, -I have a few examples: +The best way for now to understand how to use those modules is to read the code linked above and see +how they are used in the [provided services](#provided-services) and in the [demos](#demos). Also, here are a +few examples taken from my personal usage of selfhostblocks. ### Add SSL configuration @@ -412,6 +363,69 @@ MaxFileSec=day ''; ``` +### Set up network tunnel with VPN and Proxy + +```nix +shb.vpn.nordvpnus = { + enable = true; + # Only "nordvpn" supported for now. + provider = "nordvpn"; + dev = "tun1"; + # Must be unique per VPN instance. + routingNumber = 10; + # Change to the one you want to connect to + remoteServerIP = "1.2.3.4"; + sopsFile = ./secrets/vpn.yaml; + proxyPort = 12000; +}; +``` + +This sets up a tunnel interface `tun1` that connects to the VPN provider, here NordVPN. Also, if the +`proxyPort` option is not null, this will spin up a `tinyproxy` instance that listens on the given +port and redirects all traffic through that VPN. + +```bash +$ curl 'https://api.ipify.org?format=json' +{"ip":"107.21.107.115"} + +$ curl --interface tun1 'https://api.ipify.org?format=json' +{"ip":"46.12.123.113"} + +$ curl --proxy 127.0.0.1:12000 'https://api.ipify.org?format=json' +{"ip":"46.12.123.113"} +``` + +## Provided Services + +- [`arr.nix`](./modules/services/arr.nix) for finding media https://wiki.servarr.com/. +- [`deluge.nix`](./modules/services/deluge.nix) for downloading linux isos https://deluge-torrent.org/. +- [`hledger.nix`](./modules/services/hledger.nix) for managing finances https://hledger.org/. +- [`home-assistant.nix`](./modules/services/home-assistant.nix) for private IoT https://www.home-assistant.io/. +- [`jellyfin.nix`](./modules/services/jellyfin.nix) for watching media https://jellyfin.org/. +- [`nextcloud-server.nix`](./modules/services/nextcloud-server.nix) for private documents, contacts, calendar, etc https://nextcloud.com. +- [`vaultwarden.nix`](./modules/services/vaultwarden.nix) for passwords https://github.com/dani-garcia/vaultwarden. + +The services above are those I am using myself. I intend to add more. + +The best way for now to understand how to use those modules is to read the code linked above and see +how they are used in the demos. Also, here are a +few examples taken from my personal usage of selfhostblocks. + +### Common Options + +Some common options are provided for all services. + +- `enable` (bool). Set to true to deploy and run the service. +- `subdomain` (string). Subdomain under which to serve the service. +- `domain` (string). Domain under which to server the service. + +Some other common options are the following. I am not satisfied with how those are expressed so those will most certainly change. +- LDAP and OIDC options for SSO, authentication and authorization. +- Secrets. +- Backups. + +Note that for backups, every service exposes what directory should be backed up, you must merely choose when those backups will take place and where they will be stored. + ### Deploy a Nextcloud Instance ```nix @@ -597,38 +611,70 @@ home-assistant: | longitude_home: "-0.01234567890123" ``` -### Set up network tunnel with VPN and Proxy +## Demos + +Demos that start and deploy a service on a Virtual Machine on your computer are located under the +[demo](./demo/) folder. These show the onboarding experience you would get if you deployed +one of the services on your own server. + +## Import selfhostblocks + +Ready to start using selfhostblocks? Thank you for trusting selfhostblocks. Please raise any +question you have or hurdle you encounter by creating an issue. + +The top-level `flake.nix` just outputs a nixos module that gathers all other modules from +the [`modules/`](./modules/) directory. Use this repo as a flake input to your own repo. +The `inputs` field of your `flake.nix` file in your repo should look like so: ```nix -shb.vpn.nordvpnus = { - enable = true; - # Only "nordvpn" supported for now. - provider = "nordvpn"; - dev = "tun1"; - # Must be unique per VPN instance. - routingNumber = 10; - # Change to the one you want to connect to - remoteServerIP = "1.2.3.4"; - sopsFile = ./secrets/vpn.yaml; - proxyPort = 12000; +inputs = { + nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; + sops-nix.url = "github:Mic92/sops-nix"; + + selfhostblocks.url = "github:ibizaman/selfhostblocks"; + selfhostblocks.inputs.nixpkgs.follows = "nixpkgs"; + selfhostblocks.inputs.sops-nix.follows = "sops-nix"; }; ``` -This sets up a tunnel interface `tun1` that connects to the VPN provider, here NordVPN. Also, if the -`proxyPort` option is not null, this will spin up a `tinyproxy` instance that listens on the given -port and redirects all traffic through that VPN. +`sops-nix` is used to setup passwords and secrets. Currently `selfhostblocks` has a strong +dependency on it but I'm working on removing that so you could use any secret provider. -```bash -$ curl 'https://api.ipify.org?format=json' -{"ip":"107.21.107.115"} +The snippet above makes `selfhostblocks`' inputs follow yours. This is not maintainable though +because options that `selfhostblocks` rely on can change or disappear and you have no control on +that. Later, I intend to make `selfhostblocks` provide its own `nixpkgs` input and update it myself +through CI. -$ curl --interface tun1 'https://api.ipify.org?format=json' -{"ip":"46.12.123.113"} +How you actually deploy using selfhostblocks depends on what system you choose. If you use +[colmena](https://colmena.cli.rs), this is what your `outputs` field could look like: -$ curl --proxy 127.0.0.1:12000 'https://api.ipify.org?format=json' -{"ip":"46.12.123.113"} +```nix +outputs = inputs@{ self, nixpkgs, ... }: { + colmena = { + meta = { + nixpkgs = import inputs.nixpkgs { + system = "x86_64-linux"; + }; + specialArgs = inputs; + }; + + myserver = import ./machines/myserver.nix; + }; +} ``` +Now, what goes inside this `./machines/myserver.nix` file? First, import `selfhostblocks` and +`sops-nix`: + +```nix +imports = [ + selfhostblocks.nixosModules.x86_64-linux.default + sops-nix.nixosModules.default +] +``` + +For the rest, see the above [building blocks](#building-blocks), [provided services](#provided-services) and [demos](#demos) sections. + ## Tips ### Run tests @@ -654,7 +700,13 @@ selfhostblocks.url = "/home/me/projects/selfhostblocks"; Or override on the command line: ```bash -$ nix run nixpkgs#colmena --override-input selfhostblocks ../selfhostblocks -- apply +$ nix flake lock --override-input selfhostblocks ../selfhostblocks +``` + +I usually combine the override snippet above with deploying: + +```bash +$ nix flake lock --override-input selfhostblocks ../selfhostblocks && nix run nixpkgs#colmena -- apply ``` ### Diff changes diff --git a/flake.nix b/flake.nix index 1593c8b..c428fb0 100644 --- a/flake.nix +++ b/flake.nix @@ -15,23 +15,24 @@ { nixosModules.default = { config, ... }: { imports = [ - modules/arr.nix - modules/authelia.nix - modules/backup.nix - modules/deluge.nix - modules/davfs.nix - modules/hledger.nix - modules/home-assistant.nix - modules/jellyfin.nix - modules/ldap.nix - modules/monitoring.nix - modules/nextcloud-server.nix - modules/nginx.nix - modules/postgresql.nix - modules/ssl.nix - modules/tinyproxy.nix - modules/vaultwarden.nix - modules/vpn.nix + modules/blocks/authelia.nix + modules/blocks/backup.nix + modules/blocks/davfs.nix + modules/blocks/ldap.nix + modules/blocks/monitoring.nix + modules/blocks/nginx.nix + modules/blocks/postgresql.nix + modules/blocks/ssl.nix + modules/blocks/tinyproxy.nix + modules/blocks/vpn.nix + + modules/services/arr.nix + modules/services/deluge.nix + modules/services/hledger.nix + modules/services/home-assistant.nix + modules/services/jellyfin.nix + modules/services/nextcloud-server.nix + modules/services/vaultwarden.nix ]; }; diff --git a/modules/authelia.nix b/modules/blocks/authelia.nix similarity index 100% rename from modules/authelia.nix rename to modules/blocks/authelia.nix diff --git a/modules/backup.nix b/modules/blocks/backup.nix similarity index 100% rename from modules/backup.nix rename to modules/blocks/backup.nix diff --git a/modules/davfs.nix b/modules/blocks/davfs.nix similarity index 100% rename from modules/davfs.nix rename to modules/blocks/davfs.nix diff --git a/modules/ldap.nix b/modules/blocks/ldap.nix similarity index 100% rename from modules/ldap.nix rename to modules/blocks/ldap.nix diff --git a/modules/monitoring.nix b/modules/blocks/monitoring.nix similarity index 100% rename from modules/monitoring.nix rename to modules/blocks/monitoring.nix diff --git a/modules/nginx.nix b/modules/blocks/nginx.nix similarity index 100% rename from modules/nginx.nix rename to modules/blocks/nginx.nix diff --git a/modules/postgresql.nix b/modules/blocks/postgresql.nix similarity index 100% rename from modules/postgresql.nix rename to modules/blocks/postgresql.nix diff --git a/modules/ssl.nix b/modules/blocks/ssl.nix similarity index 100% rename from modules/ssl.nix rename to modules/blocks/ssl.nix diff --git a/modules/tinyproxy.nix b/modules/blocks/tinyproxy.nix similarity index 100% rename from modules/tinyproxy.nix rename to modules/blocks/tinyproxy.nix diff --git a/modules/vpn.nix b/modules/blocks/vpn.nix similarity index 100% rename from modules/vpn.nix rename to modules/blocks/vpn.nix diff --git a/modules/arr.nix b/modules/services/arr.nix similarity index 100% rename from modules/arr.nix rename to modules/services/arr.nix diff --git a/modules/deluge.nix b/modules/services/deluge.nix similarity index 100% rename from modules/deluge.nix rename to modules/services/deluge.nix diff --git a/modules/hledger.nix b/modules/services/hledger.nix similarity index 100% rename from modules/hledger.nix rename to modules/services/hledger.nix diff --git a/modules/home-assistant.nix b/modules/services/home-assistant.nix similarity index 100% rename from modules/home-assistant.nix rename to modules/services/home-assistant.nix diff --git a/modules/jellyfin.nix b/modules/services/jellyfin.nix similarity index 100% rename from modules/jellyfin.nix rename to modules/services/jellyfin.nix diff --git a/modules/nextcloud-server.nix b/modules/services/nextcloud-server.nix similarity index 100% rename from modules/nextcloud-server.nix rename to modules/services/nextcloud-server.nix diff --git a/modules/vaultwarden.nix b/modules/services/vaultwarden.nix similarity index 100% rename from modules/vaultwarden.nix rename to modules/services/vaultwarden.nix diff --git a/test/modules/arr.nix b/test/modules/arr.nix index 972f2dc..b999f38 100644 --- a/test/modules/arr.nix +++ b/test/modules/arr.nix @@ -24,7 +24,7 @@ let services.sonarr = anyOpt {}; }; } - ../../modules/arr.nix + ../../modules/services/arr.nix m ]; }).config; diff --git a/test/modules/davfs.nix b/test/modules/davfs.nix index bcbb9e8..12709b3 100644 --- a/test/modules/davfs.nix +++ b/test/modules/davfs.nix @@ -16,7 +16,7 @@ let services = anyOpt {}; }; } - ../../modules/davfs.nix + ../../modules/blocks/davfs.nix m ]; }).config; diff --git a/test/modules/postgresql.nix b/test/modules/postgresql.nix index 5adb6ad..132df68 100644 --- a/test/modules/postgresql.nix +++ b/test/modules/postgresql.nix @@ -15,7 +15,7 @@ let systemd = anyOpt {}; }; } - ../../modules/postgresql.nix + ../../modules/blocks/postgresql.nix m ]; }).config;