diff --git a/README.md b/README.md index faaca68..0ba814c 100644 --- a/README.md +++ b/README.md @@ -5,14 +5,20 @@ [![Documentation](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml/badge.svg)](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml) [![Tests](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Fgarnix.io%2Fapi%2Fbadges%2Fibizaman%2Fselfhostblocks%3Fbranch%3Dmain)](https://garnix.io) (using Garnix) -SHB's (Self Host Blocks) is yet another server management tool whose goal is to provide a lower -entry-bar for self-hosting. SHB provides opinionated [building blocks](#available-blocks) fitting -together to self-host any service you'd want. Some [common services](#provided-services) are -provided out of the box. +SHB's (Self Host Blocks) is yet another server management tool whose goal is to provide better +building blocks for self-hosting. Indeed, SHB provides opinionated [building +blocks](#available-blocks) fitting together to self-host any service you'd want. Some [common +services](#provided-services) are provided out of the box. -To achieve this, SHB is using the full power of NixOS modules. Indeed, each building block and each -service is a NixOS module and uses the modules defined in -[Nixpkgs](https://github.com/NixOS/nixpkgs/). +SHB's goal is to make these building blocks plug-and-play. To achieve this, SHB pioneers +[contracts](https://shb.skarabox.com/usage.html) which allows you, the final user, to be more in +control of which pieces go where. The promise here is to let you choose, for example, any reverse +proxy you want or any database you want, without requiring work from maintainers of the services you +want to self host. + +To achieve all this, SHB is using the full power of NixOS modules and NixOS VM tests. Indeed, each +building block and each service is a NixOS module using modules defined in +[Nixpkgs](https://github.com/NixOS/nixpkgs/) and they are tested using full VMs on every commit. ## TOC @@ -39,16 +45,18 @@ Self Host Blocks is available as a flake. To use it in your project, add the fol inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks"; ``` -See [the manual](https://shb.skarabox.com/usage.html) for more information about installing Self -Host Blocks. +This is not quite enough though and more information is provided in [the +manual](https://shb.skarabox.com/usage.html). - You are new to self hosting and want pre-configured services to deploy easily. Look at the [services section](https://shb.skarabox.com/services.html). - You are a seasoned self-hoster but want to enhance some services you deploy already. Go to the [blocks section](https://shb.skarabox.com/blocks.html). -- You are a user of Self Host Blocks but would like to use your own implementation for a block. Head - over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) to talk about it - (this is WIP). +- You are a user of Self Host Blocks but would like to use your own implementation for a block. Go + to the [contracts section](https://shb.skarabox.com/contracts.html). + +Head over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) for any remaining +question, or just to say hi :) ## Why yet another self hosting tool? @@ -59,21 +67,24 @@ specifically: - atomic configuration rollbacks; - real programming language to define configurations; - user-defined abstractions (create your own functions or NixOS modules on top of SHB!); -- integration with the rest of nixpkgs. +- integration with the rest of nixpkgs; +- much fewer "works on my machine" type of issues. In no particular order, here are some aspects of SHB which I find interesting and differentiates it from other server management projects: - SHB intends to be a library, not a framework. You can either go all in and use SHB provided services directly or use just one block in your existing infrastructure. -- SHB introduces contracts to allow you to swap implementation for each self-hosting need. - For example, you should be able to use the reverse proxy you want without modifying any services - depending on it. +- SHB introduces [contracts](https://shb.skarabox.com/contracts.html) to allow you to swap + implementation for each self-hosting need. For example, you should be able to use the reverse + proxy you want without modifying any services depending on it. - SHB contracts also allows you to use your own custom implementation instead of the provided one, as long as it follows the contract and passes the tests. -- SHB provides at least one implementation for each self-hosting need like backups, SSL - certificates, reverse proxy, VPN, etc. Those are called blocks here. They are documented in [the +- SHB provides at least one implementation for each contract like backups, SSL certificates, reverse + proxy, VPN, etc. Those are called blocks here and are documented in [the manual](https://shb.skarabox.com/blocks.html). +- SHB provides several services out of the box fully using the blocks provided. Those can also be + found in [the manual](https://shb.skarabox.com/services.html). - SHB follows nixpkgs unstable branch closely. There is a GitHub action running daily that updates the `nixpkgs` input in the root `flakes.nix`, runs the tests and merges a PR with the new input if the tests pass. @@ -82,9 +93,9 @@ from other server management projects: The manual can be found at [shb.skarabox.com](https://shb.skarabox.com/). -Currently, only some services and blocks are documented. For the rest, unfortunately the source code -is the best place to read about them. [Here](./modules/services) for services and -[here](./modules/blocks) for blocks. +Work is in progress to document everything in the manual but I'm not there yet. For what's not yet +documented, unfortunately the source code is the best place to read about them. +[Here](./modules/services) for services and [here](./modules/blocks) for blocks. ## Roadmap @@ -95,9 +106,11 @@ contracts. Upstreaming changes is also on the roadmap. -Check [the issues](https://github.com/ibizaman/selfhostblocks/issues) to see planned works. +Check [the issues](https://github.com/ibizaman/selfhostblocks/issues) to see planned works. Feel +free to add more! -That being said, I am personally using all the blocks and services in this project, so they do work. +That being said, I am personally using all the blocks and services in this project, so they do work +to some extent. ## Available Blocks diff --git a/docs/assets/contracts_after.png b/docs/assets/contracts_after.png new file mode 100644 index 0000000..3b23bb2 Binary files /dev/null and b/docs/assets/contracts_after.png differ diff --git a/docs/assets/contracts_before.png b/docs/assets/contracts_before.png new file mode 100644 index 0000000..7a39a38 Binary files /dev/null and b/docs/assets/contracts_before.png differ diff --git a/docs/assets/contracts_separationofconcerns.png b/docs/assets/contracts_separationofconcerns.png new file mode 100644 index 0000000..cd3bc13 Binary files /dev/null and b/docs/assets/contracts_separationofconcerns.png differ diff --git a/docs/blocks.md b/docs/blocks.md index 44dcfea..3849fd2 100644 --- a/docs/blocks.md +++ b/docs/blocks.md @@ -1,12 +1,12 @@ # Blocks {#blocks} -Blocks help you self-host apps or services. They define and implement a specific function like -backup or secure access through a subdomain. Each block is designed to be usable on its own and to -fit nicely with others. +Blocks help you self-host apps or services. They implement a specific function like backup or secure +access through a subdomain. Each block is designed to be usable on its own and to fit nicely with +others. -In practice, a block defines a contract that must be followed to implement a specific self-hosting -function. It also comes with a unit test and NixOS VM test suite to ensure any implementation -follows the contract. +In practice, a block implements a [contract](contracts.html) that must be followed to implement a +specific self-hosting function. It also comes with a unit test and NixOS VM test suite to ensure any +implementation follows the contract. As an example, let's take the HTTPS access block which allows for a service to be accessible through a specific subdomain. In Nix terms, this block defines at minimum the inputs: diff --git a/docs/contracts.md b/docs/contracts.md new file mode 100644 index 0000000..25ed211 --- /dev/null +++ b/docs/contracts.md @@ -0,0 +1,91 @@ +# Contracts {#contracts} + +A contract decouples modules that use a functionality from modules that provide it. A first +intuition for contracts is they are generally related to accessing a shared resource. + +A few examples of contracts are generating SSL certificates, creating a user or knowing which files +and folders to backup. Indeed, when generating certificates, the service using those do not care how +they were created. They just need to know where the certificate files are located. + +In practice, a contract is a set of options that any user of a contract expects to exist. Also, the +values of these options dictate the behavior of the implementation. This is enforced with NixOS VM +tests. + +## Provided contracts {#contracts-provided} + +Self Host Blocks is a proving ground of contracts. This repository adds a layer on top of services +available in nixpkgs to make them work using contracts. In time, we hope to upstream as much of this +as possible, reducing the quite thick layer that it is now. + +Provided contracts are: + +- [SSL generator contract](contracts-ssl.html) to generate SSL certificates. Two implementations are provided: self-signed and Let's Encrypt. + +```{=include=} chapters html:into-file=//contracts-ssl.html +modules/contracts/ssl/docs/default.md +``` + +## Why do we need this new concept? {#contracts-why} + +Currently in nixpkgs, every module needing access to a shared resource must implement the logic +needed to setup that resource themselves. Similarly, if the module is mature enough to let the user +select a particular implementation, the code lives inside that module. + +![](./assets/contracts_before.png "A module composed of a core logic and a lot of peripheral logic.") + +This has a few disadvantages: + +- This leads to a lot of **duplicated code**. If a module wants to support a new implementation of a +contract, the maintainers of that module must write code to make that happen. +- This also leads to **tight coupling**. The code written by the maintainers cannot be reused in + other modules, apart from copy pasting. +- There is also a **lack of separation of concerns**. The maintainers of a service must be experts + in all implementations they let the users choose from. +- Finally, this is **not extensible**. If you, the user of the module, want to use another + implementation that is not supported, you are out of luck. You can always dive into the module's + code and extend it, but that is not an optimal experience. + +We do believe that the decoupling contracts provides helps alleviate all the issues outlined above +which makes it an essential step towards more adoption of Nix, if only in the self hosting scene. + +![](./assets/contracts_after.png "A module containing only logic using peripheral logic through contracts.") + +Indeed, contracts allow: + +- **Reuse of code**. Since the implementation of a contract lives outside of modules using it, using + that implementation elsewhere is trivial. +- **Loose coupling**. Modules that use a contract do not care how they are implemented, as long as + the implementation follows the behavior outlined by the contract. +- Full **separation of concerns** (see diagram below). Now, each party's concern is separated with a + clear boundary. The maintainer of a module using a contract can be different from the maintainers + of the implementation, allowing them to be experts in their own respective fields. But more + importantly, the contracts themselves can be created and maintained by the community. +- Full **extensibility**. The final user themselves can choose an implementation, even new custom + implementations not available in nixpkgs, without changing existing code. + +![](./assets/contracts_separationofconcerns.png "Separation of concerns thanks to contracts.") + +Thanks to NixOS VM test, we can even go one step further by ensuring each implementation of a +contract provides required options and behaves as the contract requires. + +## Are there contracts in nixpkgs already? {#contracts-nixpkgs} + +Actually yes, there are some ubiquitous options in nixpkgs. Those I found are: + +- `services..enable` +- `services..package` +- `services..openFirewall` +- `services..user` +- `services..group` + +What makes those nearly contracts are: + +- Pretty much every service provides them. +- Users of a service expects them to exist and expects a consistent type and behavior from them. + Indeed, everyone knows what happens if you set `enable = true`. +- Maintainers of a service knows that users expects those options. They also know what behavior the + user expects when setting those options. +- The name of the options is the same everywhere. + +The only thing missing to make these explicit contracts is, well, the contracts themselves. +Currently, they are conventions and not contracts. diff --git a/docs/default.nix b/docs/default.nix index c2df40d..b1af2d6 100644 --- a/docs/default.nix +++ b/docs/default.nix @@ -144,6 +144,11 @@ in stdenv.mkDerivation { '@OPTIONS_JSON@' \ ${individualModuleOptionsDocs ../modules/services/nextcloud-server.nix}/share/doc/nixos/options.json + substituteInPlace ./modules/contracts/ssl/docs/default.md \ + --replace \ + '@OPTIONS_JSON@' \ + ${individualModuleOptionsDocs ../modules/contracts/ssl/dummyModule.nix}/share/doc/nixos/options.json + find . -name "*.md" -print0 | \ while IFS= read -r -d ''' f; do substituteInPlace "''${f}" \ diff --git a/docs/manual.md b/docs/manual.md index 869f26f..08a9bb3 100644 --- a/docs/manual.md +++ b/docs/manual.md @@ -15,6 +15,10 @@ usage.md services.md ``` +```{=include=} chapters html:into-file=//contracts.html +contracts.md +``` + ```{=include=} chapters html:into-file=//blocks.html blocks.md ``` diff --git a/flake.nix b/flake.nix index 3c82ff2..cdb503b 100644 --- a/flake.nix +++ b/flake.nix @@ -49,6 +49,11 @@ modules/services/nextcloud-server.nix modules/services/vaultwarden.nix ]; + + # Only used for documentation. + contractDummyModules = [ + modules/contracts/ssl/dummyModule.nix + ]; in { nixosModules.default = { config, ... }: { @@ -56,7 +61,8 @@ }; packages.manualHtml = pkgs.callPackage ./docs { - inherit allModules nmdsrc; + inherit nmdsrc; + allModules = allModules ++ contractDummyModules; release = "0.0.1"; }; diff --git a/modules/blocks/monitoring/docs/default.md b/modules/blocks/monitoring/docs/default.md index 52a4c34..17240e7 100644 --- a/modules/blocks/monitoring/docs/default.md +++ b/modules/blocks/monitoring/docs/default.md @@ -92,7 +92,7 @@ Self Host Blocks will create automatically the following resources: Those resources are namespaced as appropriate under the Self Host Blocks namespace: -[](./assets/folder.png) +![](./assets/folder.png) ## Errors Dashboard {#blocks-monitoring-error-dashboard} diff --git a/modules/blocks/ssl.nix b/modules/blocks/ssl.nix index b667138..ea88024 100644 --- a/modules/blocks/ssl.nix +++ b/modules/blocks/ssl.nix @@ -32,7 +32,7 @@ in description = '' Paths where CA certs will be located. - This option is the contract output of the `shb.certs.cas` SSL block. + This option implements the SSL Generator contract. ''; type = contracts.ssl.certs-paths; default = rec { @@ -42,7 +42,11 @@ in }; systemdService = lib.mkOption { - description = "Systemd oneshot service used to generate the certs."; + description = '' + Systemd oneshot service used to generate the certs. + + This option implements the SSL Generator contract. + ''; type = lib.types.str; default = "shb-certs-ca-${config._module.args.name}.service"; }; @@ -100,7 +104,7 @@ in description = '' Paths where certs will be located. - This option is the contract output of the `shb.certs.certs` SSL block. + This option implements the SSL Generator contract. ''; type = contracts.ssl.certs-paths; default = rec { @@ -110,7 +114,11 @@ in }; systemdService = lib.mkOption { - description = "Systemd oneshot service used to generate the certs."; + description = '' + Systemd oneshot service used to generate the certs. + + This option implements the SSL Generator contract. + ''; type = lib.types.str; default = "shb-certs-cert-selfsigned-${config._module.args.name}.service"; }; @@ -159,7 +167,7 @@ in description = '' Paths where certs will be located. - This option is the contract output of the `shb.certs.certs` SSL block. + This option implements the SSL Generator contract. ''; type = contracts.ssl.certs-paths; default = { @@ -178,7 +186,11 @@ in }; systemdService = lib.mkOption { - description = "Systemd oneshot service used to generate the certs."; + description = '' + Systemd oneshot service used to generate the certs. + + This option implements the SSL Generator contract. + ''; type = lib.types.str; default = "shb-certs-cert-letsencrypt-${config._module.args.name}.service"; }; diff --git a/modules/blocks/ssl/docs/default.md b/modules/blocks/ssl/docs/default.md index 1e9784c..68090a8 100644 --- a/modules/blocks/ssl/docs/default.md +++ b/modules/blocks/ssl/docs/default.md @@ -1,6 +1,6 @@ -# SSL Block {#ssl-block} +# SSL Generator Block {#ssl-block} -This NixOS module is a block that provides a contract to generate TLS certificates. +This NixOS module is a block that implements the [SSL certificate generator](contracts-ssl.html) contract. It is implemented by: - [`shb.certs.cas.selfsigned`][10] and [`shb.certs.certs.selfsigned`][11]: Generates self-signed certificates, @@ -14,27 +14,7 @@ It is implemented by: [11]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.selfsigned [12]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.letsencrypt -## Contract {#ssl-block-contract} - -The contract for this block is defined in [`/modules/contracts/ssl.nix`](@REPO@/modules/contracts/ssl.nix). - -Every module implementing this contract provides the following options: - -- `domain`: Domain to generate the certificate for. -- `extraDomains`: Other domains the certificate should be generated for. -- `group`: The unix group owning this certificate. -- `reloadServices`: Systemd services to reload when the certificate gets renewed. -- `paths.cert`: Path to the cert file. -- `paths.key`: Path to the key file. -- `systemdService`: Systemd oneshot service used to generate the certificate. - The Systemd service file name must include the `.service` suffix. Downstream users of the - certificate can use this option to wait for the certificate to be generated. - -## Implementations {#ssl-block-impl} - -This sections explains how to generate certificates using the SSL block implementations. - -### Self-Signed Certificates {#ssl-block-impl-self-signed} +## Self-Signed Certificates {#ssl-block-impl-self-signed} Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix). @@ -72,7 +52,7 @@ shb.certs.certs.selfsigned = { The group has been chosen to be `nginx` to be consistent with the examples further down in this document. -### Let's Encrypt {#ssl-block-impl-lets-encrypt} +## Let's Encrypt {#ssl-block-impl-lets-encrypt} Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix). @@ -109,6 +89,7 @@ where the certificate and the private key are located: ```nix config.shb.certs.certs...paths.cert config.shb.certs.certs...paths.key +config.shb.certs.certs...systemdService ``` For example: @@ -116,50 +97,27 @@ For example: ```nix config.shb.certs.certs.selfsigned."example.com".paths.cert config.shb.certs.certs.selfsigned."example.com".paths.key -``` -We can then configure Nginx to use those certificates: - -```nix -services.nginx.virtualHosts."example.com" = - let - cert = config.shb.certs.certs.selfsigned."example.com"; - in - { - onlySSL = true; - sslCertificate = cert.paths.cert; - sslCertificateKey = cert.paths.key; - - locations."/".extraConfig = '' - add_header Content-Type text/plain; - return 200 'It works!'; - ''; - }; +config.shb.certs.certs.selfsigned."example.com".systemdService ``` -To make sure the Nginx webserver can find the generated file, we will make it wait for the -certificate to the generated: - -```nix -systemd.services.nginx = { - after = [ config.shb.certs.certs.selfsigned."example.com".systemdService ]; - requires = [ config.shb.certs.certs.selfsigned."example.com".systemdService ]; -}; -``` - -If needed, we can also wait on the CA bundle to be generated by waiting for the Systemd service: +The full CA bundle is generated by the following Systemd service, running after each individual +generator finished: ```nix config.shb.certs.systemdService ``` +See also the [SSL certificate generator usage](contracts-ssl.html#ssl-contract-usage) for a more detailed usage +example. + ## Debug {#ssl-block-debug} -Each CA and Cert is generated by a systemd service whose name can be seen in `systemdService` -options below. You can then see the latest errors messages using `journalctl`. +Each CA and Cert is generated by a systemd service whose name can be seen in the `systemdService` +option. You can then see the latest errors messages using `journalctl`. ## Tests {#ssl-block-tests} -This block is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix). +The self-signed implementation is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix). ## Options Reference {#ssl-block-options} diff --git a/modules/contracts/ssl.nix b/modules/contracts/ssl.nix index 7bbdb4a..33e2da6 100644 --- a/modules/contracts/ssl.nix +++ b/modules/contracts/ssl.nix @@ -1,4 +1,4 @@ -{ lib }: +{ lib, ... }: rec { certs-paths = lib.types.submodule { freeformType = lib.types.anything; @@ -28,8 +28,14 @@ rec { }; systemdService = lib.mkOption { - description = "Systemd oneshot service used to generate the CA."; + description = '' + Systemd oneshot service used to generate the CA. Ends with the `.service` suffix. + + Use this if downstream services must wait for the certificates to be generated before + starting. + ''; type = lib.types.str; + example = "ca-generator.service"; }; }; }; @@ -48,10 +54,13 @@ rec { systemdService = lib.mkOption { description = '' - Systemd oneshot service used to generate the certificate. The name must include the - `.service` suffix. + Systemd oneshot service used to generate the certificate. Ends with the `.service` suffix. + + Use this if downstream services must wait for the certificates to be generated before + starting. ''; type = lib.types.str; + example = "cert-generator.service"; }; }; }; diff --git a/modules/contracts/ssl/docs/default.md b/modules/contracts/ssl/docs/default.md new file mode 100644 index 0000000..db30337 --- /dev/null +++ b/modules/contracts/ssl/docs/default.md @@ -0,0 +1,119 @@ +# SSL Generator Contract {#ssl-contract} + +This NixOS contract represents an SSL certificate generator. This contract is used to decouple +generating an SSL certificate from using it. In practice, you can swap generators without updating +modules depending on it. + +## Contract Reference {#ssl-contract-options} + +These are all the options that are expected to exist for this contract to be respected. + +```{=include=} options +id-prefix: contracts-ssl-options- +list-id: selfhostblocks-options +source: @OPTIONS_JSON@ +``` + +## Usage {#ssl-contract-usage} + +Let's assume a module implementing this contract is available under the `ssl` variable: + +```nix +let + ssl = <...>; +in +``` + +To use this module, we can reference the path where the certificate and the private key are located with: + +```nix +ssl.paths.cert +ssl.paths.key +``` + +We can then configure Nginx to use those certificates: + +```nix +services.nginx.virtualHosts."example.com" = { + onlySSL = true; + sslCertificate = ssl.paths.cert; + sslCertificateKey = ssl.paths.key; + + locations."/".extraConfig = '' + add_header Content-Type text/plain; + return 200 'It works!'; + ''; +}; +``` + +To make sure the Nginx webserver can find the generated file, we will make it wait for the +certificate to the generated: + +```nix +systemd.services.nginx = { + after = [ ssl.systemdService ]; + requires = [ ssl.systemdService ]; +}; +``` + +## Provided Implementations {#ssl-contract-impl-shb} + +Multiple implementation are provided out of the box at [SSL block](blocks-ssl.html). + +## Custom Implementation {#ssl-contract-impl-custom} + +To implement this contract, you must create a module that respects this contract. The following +snippet shows an example. + +```nix +{ lib, ... }: +{ + options.my.generator = { + paths = lib.mkOption { + description = '' + Paths where certs will be located. + + This option implements the SSL Generator contract. + ''; + type = contracts.ssl.certs-paths; + default = { + key = "/var/lib/my_generator/key.pem"; + cert = "/var/lib/my_generator/cert.pem"; + }; + }; + + systemdService = lib.mkOption { + description = '' + Systemd oneshot service used to generate the certs. + + This option implements the SSL Generator contract. + ''; + type = lib.types.str; + default = "my-generator.service"; + }; + + # Other options needed for this implementation + }; + + config = { + # custom implementation goes here + }; +} +``` + +You can then create an instance of this generator: + +```nix +{ + my.generator = ...; +} +``` + +And use it whenever a module expects something implementing this SSL generator contract: + +```nix +{ config, ... }: +{ + my.service.ssl = config.my.generator; +} +``` diff --git a/modules/contracts/ssl/dummyModule.nix b/modules/contracts/ssl/dummyModule.nix new file mode 100644 index 0000000..f78912e --- /dev/null +++ b/modules/contracts/ssl/dummyModule.nix @@ -0,0 +1,10 @@ +{ pkgs, lib, ... }: +let + contracts = pkgs.callPackage ../. {}; +in +{ + options.shb.contracts.ssl = lib.mkOption { + description = "Contract for SSL Certificate generator."; + type = contracts.ssl.certs; + }; +}