add contract documentation (#225)

README.md

@@ -5,14 +5,20 @@
 [![Documentation](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml/badge.svg)](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml)
 [![Tests](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Fgarnix.io%2Fapi%2Fbadges%2Fibizaman%2Fselfhostblocks%3Fbranch%3Dmain)](https://garnix.io) (using Garnix)
 
-SHB's (Self Host Blocks) is yet another server management tool whose goal is to provide a lower
+SHB's (Self Host Blocks) is yet another server management tool whose goal is to provide better
-entry-bar for self-hosting. SHB provides opinionated [building blocks](#available-blocks) fitting
+building blocks for self-hosting. Indeed, SHB provides opinionated [building
-together to self-host any service you'd want. Some [common services](#provided-services) are
+blocks](#available-blocks) fitting together to self-host any service you'd want. Some [common
-provided out of the box.
+services](#provided-services) are provided out of the box.
 
-To achieve this, SHB is using the full power of NixOS modules. Indeed, each building block and each
+SHB's goal is to make these building blocks plug-and-play. To achieve this, SHB pioneers
-service is a NixOS module and uses the modules defined in
+[contracts](https://shb.skarabox.com/usage.html) which allows you, the final user, to be more in
-[Nixpkgs](https://github.com/NixOS/nixpkgs/).
+control of which pieces go where. The promise here is to let you choose, for example, any reverse
+proxy you want or any database you want, without requiring work from maintainers of the services you
+want to self host.
+
+To achieve all this, SHB is using the full power of NixOS modules and NixOS VM tests. Indeed, each
+building block and each service is a NixOS module using modules defined in
+[Nixpkgs](https://github.com/NixOS/nixpkgs/) and they are tested using full VMs on every commit.
 
 ## TOC
 
@@ -39,16 +45,18 @@ Self Host Blocks is available as a flake. To use it in your project, add the fol
 inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
 ```
 
-See [the manual](https://shb.skarabox.com/usage.html) for more information about installing Self
+This is not quite enough though and more information is provided in [the
-Host Blocks.
+manual](https://shb.skarabox.com/usage.html).
 
 - You are new to self hosting and want pre-configured services to deploy easily. Look at the
   [services section](https://shb.skarabox.com/services.html).
 - You are a seasoned self-hoster but want to enhance some services you deploy already. Go to the
   [blocks section](https://shb.skarabox.com/blocks.html).
-- You are a user of Self Host Blocks but would like to use your own implementation for a block. Head
+- You are a user of Self Host Blocks but would like to use your own implementation for a block. Go
-  over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) to talk about it
+  to the [contracts section](https://shb.skarabox.com/contracts.html).
-  (this is WIP).
+
+Head over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) for any remaining
+question, or just to say hi :)
 
 ## Why yet another self hosting tool?
 
@@ -59,21 +67,24 @@ specifically:
 - atomic configuration rollbacks;
 - real programming language to define configurations;
 - user-defined abstractions (create your own functions or NixOS modules on top of SHB!);
-- integration with the rest of nixpkgs.
+- integration with the rest of nixpkgs;
+- much fewer "works on my machine" type of issues.
 
 In no particular order, here are some aspects of SHB which I find interesting and differentiates it
 from other server management projects:
 
 - SHB intends to be a library, not a framework. You can either go all in and use SHB provided
   services directly or use just one block in your existing infrastructure.
-- SHB introduces contracts to allow you to swap implementation for each self-hosting need.
+- SHB introduces [contracts](https://shb.skarabox.com/contracts.html) to allow you to swap
-  For example, you should be able to use the reverse proxy you want without modifying any services
+  implementation for each self-hosting need. For example, you should be able to use the reverse
-  depending on it.
+  proxy you want without modifying any services depending on it.
 - SHB contracts also allows you to use your own custom implementation instead of the provided one,
   as long as it follows the contract and passes the tests.
-- SHB provides at least one implementation for each self-hosting need like backups, SSL
+- SHB provides at least one implementation for each contract like backups, SSL certificates, reverse
-  certificates, reverse proxy, VPN, etc. Those are called blocks here. They are documented in [the
+  proxy, VPN, etc. Those are called blocks here and are documented in [the
   manual](https://shb.skarabox.com/blocks.html).
+- SHB provides several services out of the box fully using the blocks provided. Those can also be
+  found in [the manual](https://shb.skarabox.com/services.html).
 - SHB follows nixpkgs unstable branch closely. There is a GitHub action running daily that updates
   the `nixpkgs` input in the root `flakes.nix`, runs the tests and merges a PR with the new input if
   the tests pass.
@@ -82,9 +93,9 @@ from other server management projects:
 
 The manual can be found at [shb.skarabox.com](https://shb.skarabox.com/).
 
-Currently, only some services and blocks are documented. For the rest, unfortunately the source code
+Work is in progress to document everything in the manual but I'm not there yet. For what's not yet
-is the best place to read about them. [Here](./modules/services) for services and
+documented, unfortunately the source code is the best place to read about them.
-[here](./modules/blocks) for blocks.
+[Here](./modules/services) for services and [here](./modules/blocks) for blocks.
 
 ## Roadmap
 
@@ -95,9 +106,11 @@ contracts.
 
 Upstreaming changes is also on the roadmap.
 
-Check [the issues](https://github.com/ibizaman/selfhostblocks/issues) to see planned works.
+Check [the issues](https://github.com/ibizaman/selfhostblocks/issues) to see planned works. Feel
+free to add more!
 
-That being said, I am personally using all the blocks and services in this project, so they do work.
+That being said, I am personally using all the blocks and services in this project, so they do work
+to some extent.
 
 ## Available Blocks
 

docs/blocks.md

@@ -1,12 +1,12 @@
 # Blocks {#blocks}
 
-Blocks help you self-host apps or services. They define and implement a specific function like
+Blocks help you self-host apps or services. They implement a specific function like backup or secure
-backup or secure access through a subdomain. Each block is designed to be usable on its own and to
+access through a subdomain. Each block is designed to be usable on its own and to fit nicely with
-fit nicely with others.
+others.
 
-In practice, a block defines a contract that must be followed to implement a specific self-hosting
+In practice, a block implements a [contract](contracts.html) that must be followed to implement a
-function. It also comes with a unit test and NixOS VM test suite to ensure any implementation
+specific self-hosting function. It also comes with a unit test and NixOS VM test suite to ensure any
-follows the contract.
+implementation follows the contract.
 
 As an example, let's take the HTTPS access block which allows for a service to be accessible through
 a specific subdomain. In Nix terms, this block defines at minimum the inputs:

docs/contracts.md

@@ -0,0 +1,91 @@
+# Contracts {#contracts}
+
+A contract decouples modules that use a functionality from modules that provide it. A first
+intuition for contracts is they are generally related to accessing a shared resource.
+
+A few examples of contracts are generating SSL certificates, creating a user or knowing which files
+and folders to backup. Indeed, when generating certificates, the service using those do not care how
+they were created. They just need to know where the certificate files are located.
+
+In practice, a contract is a set of options that any user of a contract expects to exist. Also, the
+values of these options dictate the behavior of the implementation. This is enforced with NixOS VM
+tests.
+
+## Provided contracts {#contracts-provided}
+
+Self Host Blocks is a proving ground of contracts. This repository adds a layer on top of services
+available in nixpkgs to make them work using contracts. In time, we hope to upstream as much of this
+as possible, reducing the quite thick layer that it is now.
+
+Provided contracts are:
+
+- [SSL generator contract](contracts-ssl.html) to generate SSL certificates. Two implementations are provided: self-signed and Let's Encrypt.
+
+```{=include=} chapters html:into-file=//contracts-ssl.html
+modules/contracts/ssl/docs/default.md
+```
+
+## Why do we need this new concept? {#contracts-why}
+
+Currently in nixpkgs, every module needing access to a shared resource must implement the logic
+needed to setup that resource themselves. Similarly, if the module is mature enough to let the user
+select a particular implementation, the code lives inside that module.
+
+![](./assets/contracts_before.png "A module composed of a core logic and a lot of peripheral logic.")
+
+This has a few disadvantages:
+
+- This leads to a lot of **duplicated code**. If a module wants to support a new implementation of a
+contract, the maintainers of that module must write code to make that happen.
+- This also leads to **tight coupling**. The code written by the maintainers cannot be reused in
+  other modules, apart from copy pasting.
+- There is also a **lack of separation of concerns**. The maintainers of a service must be experts
+  in all implementations they let the users choose from.
+- Finally, this is **not extensible**. If you, the user of the module, want to use another
+  implementation that is not supported, you are out of luck. You can always dive into the module's
+  code and extend it, but that is not an optimal experience.
+
+We do believe that the decoupling contracts provides helps alleviate all the issues outlined above
+which makes it an essential step towards more adoption of Nix, if only in the self hosting scene.
+
+![](./assets/contracts_after.png "A module containing only logic using peripheral logic through contracts.")
+
+Indeed, contracts allow:
+
+- **Reuse of code**. Since the implementation of a contract lives outside of modules using it, using
+  that implementation elsewhere is trivial.
+- **Loose coupling**. Modules that use a contract do not care how they are implemented, as long as
+  the implementation follows the behavior outlined by the contract.
+- Full **separation of concerns** (see diagram below). Now, each party's concern is separated with a
+  clear boundary. The maintainer of a module using a contract can be different from the maintainers
+  of the implementation, allowing them to be experts in their own respective fields. But more
+  importantly, the contracts themselves can be created and maintained by the community.
+- Full **extensibility**. The final user themselves can choose an implementation, even new custom
+  implementations not available in nixpkgs, without changing existing code.
+
+![](./assets/contracts_separationofconcerns.png "Separation of concerns thanks to contracts.")
+
+Thanks to NixOS VM test, we can even go one step further by ensuring each implementation of a
+contract provides required options and behaves as the contract requires.
+
+## Are there contracts in nixpkgs already? {#contracts-nixpkgs}
+
+Actually yes, there are some ubiquitous options in nixpkgs. Those I found are:
+
+- `services.<name>.enable`
+- `services.<name>.package`
+- `services.<name>.openFirewall`
+- `services.<name>.user`
+- `services.<name>.group`
+
+What makes those nearly contracts are:
+
+- Pretty much every service provides them.
+- Users of a service expects them to exist and expects a consistent type and behavior from them.
+  Indeed, everyone knows what happens if you set `enable = true`.
+- Maintainers of a service knows that users expects those options. They also know what behavior the
+  user expects when setting those options.
+- The name of the options is the same everywhere.
+
+The only thing missing to make these explicit contracts is, well, the contracts themselves.
+Currently, they are conventions and not contracts.

docs/default.nix

@@ -144,6 +144,11 @@ in stdenv.mkDerivation {
         '@OPTIONS_JSON@' \
        ${individualModuleOptionsDocs ../modules/services/nextcloud-server.nix}/share/doc/nixos/options.json
 
+    substituteInPlace ./modules/contracts/ssl/docs/default.md \
+      --replace \
+        '@OPTIONS_JSON@' \
+       ${individualModuleOptionsDocs ../modules/contracts/ssl/dummyModule.nix}/share/doc/nixos/options.json
+
     find . -name "*.md" -print0 | \
       while IFS= read -r -d ''' f; do
         substituteInPlace "''${f}" \

docs/manual.md

@@ -15,6 +15,10 @@ usage.md
 services.md
 ```
 
+```{=include=} chapters html:into-file=//contracts.html
+contracts.md
+```
+
 ```{=include=} chapters html:into-file=//blocks.html
 blocks.md
 ```

flake.nix

@@ -49,6 +49,11 @@
         modules/services/nextcloud-server.nix
         modules/services/vaultwarden.nix
       ];
+
+      # Only used for documentation.
+      contractDummyModules = [
+        modules/contracts/ssl/dummyModule.nix
+      ];
     in
       {
         nixosModules.default = { config, ... }: {
@@ -56,7 +61,8 @@
         };
 
         packages.manualHtml = pkgs.callPackage ./docs {
-          inherit allModules nmdsrc;
+          inherit nmdsrc;
+          allModules = allModules ++ contractDummyModules;
           release = "0.0.1";
         };
 

modules/blocks/monitoring/docs/default.md

@@ -92,7 +92,7 @@ Self Host Blocks will create automatically the following resources:
 
 Those resources are namespaced as appropriate under the Self Host Blocks namespace:
 
-[](./assets/folder.png)
+![](./assets/folder.png)
 
 ## Errors Dashboard {#blocks-monitoring-error-dashboard}
 

modules/blocks/ssl.nix

@@ -32,7 +32,7 @@ in
             description = ''
               Paths where CA certs will be located.
 
-              This option is the contract output of the `shb.certs.cas` SSL block.
+              This option implements the SSL Generator contract.
             '';
             type = contracts.ssl.certs-paths;
             default = rec {
@@ -42,7 +42,11 @@ in
           };
 
           systemdService = lib.mkOption {
-            description = "Systemd oneshot service used to generate the certs.";
+            description = ''
+              Systemd oneshot service used to generate the certs.
+
+              This option implements the SSL Generator contract.
+            '';
             type = lib.types.str;
             default = "shb-certs-ca-${config._module.args.name}.service";
           };
@@ -100,7 +104,7 @@ in
             description = ''
               Paths where certs will be located.
 
-              This option is the contract output of the `shb.certs.certs` SSL block.
+              This option implements the SSL Generator contract.
             '';
             type = contracts.ssl.certs-paths;
             default = rec {
@@ -110,7 +114,11 @@ in
           };
 
           systemdService = lib.mkOption {
-            description = "Systemd oneshot service used to generate the certs.";
+            description = ''
+              Systemd oneshot service used to generate the certs.
+
+              This option implements the SSL Generator contract.
+            '';
             type = lib.types.str;
             default = "shb-certs-cert-selfsigned-${config._module.args.name}.service";
           };
@@ -159,7 +167,7 @@ in
             description = ''
               Paths where certs will be located.
 
-              This option is the contract output of the `shb.certs.certs` SSL block.
+              This option implements the SSL Generator contract.
             '';
             type = contracts.ssl.certs-paths;
             default = {
@@ -178,7 +186,11 @@ in
           };
 
           systemdService = lib.mkOption {
-            description = "Systemd oneshot service used to generate the certs.";
+            description = ''
+              Systemd oneshot service used to generate the certs.
+
+              This option implements the SSL Generator contract.
+            '';
             type = lib.types.str;
             default = "shb-certs-cert-letsencrypt-${config._module.args.name}.service";
           };

modules/blocks/ssl/docs/default.md

@@ -1,6 +1,6 @@
-# SSL Block {#ssl-block}
+# SSL Generator Block {#ssl-block}
 
-This NixOS module is a block that provides a contract to generate TLS certificates.
+This NixOS module is a block that implements the [SSL certificate generator](contracts-ssl.html) contract.
 
 It is implemented by:
 - [`shb.certs.cas.selfsigned`][10] and [`shb.certs.certs.selfsigned`][11]: Generates self-signed certificates,
@@ -14,27 +14,7 @@ It is implemented by:
 [11]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.selfsigned
 [12]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.letsencrypt
 
-## Contract {#ssl-block-contract}
+## Self-Signed Certificates {#ssl-block-impl-self-signed}
-
-The contract for this block is defined in [`/modules/contracts/ssl.nix`](@REPO@/modules/contracts/ssl.nix).
-
-Every module implementing this contract provides the following options:
-
-- `domain`: Domain to generate the certificate for.
-- `extraDomains`: Other domains the certificate should be generated for.
-- `group`: The unix group owning this certificate.
-- `reloadServices`: Systemd services to reload when the certificate gets renewed.
-- `paths.cert`: Path to the cert file.
-- `paths.key`: Path to the key file.
-- `systemdService`: Systemd oneshot service used to generate the certificate.
-  The Systemd service file name must include the `.service` suffix. Downstream users of the
-  certificate can use this option to wait for the certificate to be generated.
-
-## Implementations {#ssl-block-impl}
-
-This sections explains how to generate certificates using the SSL block implementations.
-
-### Self-Signed Certificates {#ssl-block-impl-self-signed}
 
 Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix).
 
@@ -72,7 +52,7 @@ shb.certs.certs.selfsigned = {
 The group has been chosen to be `nginx` to be consistent with the examples further down in this
 document.
 
-### Let's Encrypt {#ssl-block-impl-lets-encrypt}
+## Let's Encrypt {#ssl-block-impl-lets-encrypt}
 
 Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix).
 
@@ -109,6 +89,7 @@ where the certificate and the private key are located:
 ```nix
 config.shb.certs.certs.<implementation>.<name>.paths.cert
 config.shb.certs.certs.<implementation>.<name>.paths.key
+config.shb.certs.certs.<implementation>.<name>.systemdService
 ```
 
 For example:
@@ -116,50 +97,27 @@ For example:
 ```nix
 config.shb.certs.certs.selfsigned."example.com".paths.cert
 config.shb.certs.certs.selfsigned."example.com".paths.key
-```
+config.shb.certs.certs.selfsigned."example.com".systemdService
-We can then configure Nginx to use those certificates:
-
-```nix
-services.nginx.virtualHosts."example.com" =
-  let
-    cert = config.shb.certs.certs.selfsigned."example.com";
-  in
-  {
-    onlySSL = true;
-    sslCertificate = cert.paths.cert;
-    sslCertificateKey = cert.paths.key;
-
-    locations."/".extraConfig = ''
-      add_header Content-Type text/plain;
-      return 200 'It works!';
-    '';
-  };
 ```
 
-To make sure the Nginx webserver can find the generated file, we will make it wait for the
+The full CA bundle is generated by the following Systemd service, running after each individual
-certificate to the generated:
+generator finished:
-
-```nix
-systemd.services.nginx = {
-  after = [ config.shb.certs.certs.selfsigned."example.com".systemdService ];
-  requires = [ config.shb.certs.certs.selfsigned."example.com".systemdService ];
-};
-```
-
-If needed, we can also wait on the CA bundle to be generated by waiting for the Systemd service:
 
 ```nix
 config.shb.certs.systemdService
 ```
 
+See also the [SSL certificate generator usage](contracts-ssl.html#ssl-contract-usage) for a more detailed usage
+example.
+
 ## Debug {#ssl-block-debug}
 
-Each CA and Cert is generated by a systemd service whose name can be seen in `systemdService`
+Each CA and Cert is generated by a systemd service whose name can be seen in the `systemdService`
-options below. You can then see the latest errors messages using `journalctl`.
+option. You can then see the latest errors messages using `journalctl`.
 
 ## Tests {#ssl-block-tests}
 
-This block is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix).
+The self-signed implementation is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix).
 
 ## Options Reference {#ssl-block-options}
 

modules/contracts/ssl.nix

@@ -1,4 +1,4 @@
-{ lib }:
+{ lib, ... }:
 rec {
   certs-paths = lib.types.submodule {
     freeformType = lib.types.anything;
@@ -28,8 +28,14 @@ rec {
       };
 
       systemdService = lib.mkOption {
-        description = "Systemd oneshot service used to generate the CA.";
+        description = ''
+          Systemd oneshot service used to generate the CA. Ends with the `.service` suffix.
+
+          Use this if downstream services must wait for the certificates to be generated before
+          starting.
+        '';
         type = lib.types.str;
+        example = "ca-generator.service";
       };
     };
   };
@@ -48,10 +54,13 @@ rec {
 
       systemdService = lib.mkOption {
         description = ''
-          Systemd oneshot service used to generate the certificate. The name must include the
+          Systemd oneshot service used to generate the certificate. Ends with the `.service` suffix.
-          `.service` suffix.
+
+          Use this if downstream services must wait for the certificates to be generated before
+          starting.
         '';
         type = lib.types.str;
+        example = "cert-generator.service";
       };
     };
   };

modules/contracts/ssl/docs/default.md

@@ -0,0 +1,119 @@
+# SSL Generator Contract {#ssl-contract}
+
+This NixOS contract represents an SSL certificate generator. This contract is used to decouple
+generating an SSL certificate from using it. In practice, you can swap generators without updating
+modules depending on it.
+
+## Contract Reference {#ssl-contract-options}
+
+These are all the options that are expected to exist for this contract to be respected.
+
+```{=include=} options
+id-prefix: contracts-ssl-options-
+list-id: selfhostblocks-options
+source: @OPTIONS_JSON@
+```
+
+## Usage {#ssl-contract-usage}
+
+Let's assume a module implementing this contract is available under the `ssl` variable:
+
+```nix
+let
+  ssl = <...>;
+in
+```
+
+To use this module, we can reference the path where the certificate and the private key are located with:
+
+```nix
+ssl.paths.cert
+ssl.paths.key
+```
+
+We can then configure Nginx to use those certificates:
+
+```nix
+services.nginx.virtualHosts."example.com" = {
+  onlySSL = true;
+  sslCertificate = ssl.paths.cert;
+  sslCertificateKey = ssl.paths.key;
+
+  locations."/".extraConfig = ''
+    add_header Content-Type text/plain;
+    return 200 'It works!';
+  '';
+};
+```
+
+To make sure the Nginx webserver can find the generated file, we will make it wait for the
+certificate to the generated:
+
+```nix
+systemd.services.nginx = {
+  after = [ ssl.systemdService ];
+  requires = [ ssl.systemdService ];
+};
+```
+
+## Provided Implementations {#ssl-contract-impl-shb}
+
+Multiple implementation are provided out of the box at [SSL block](blocks-ssl.html).
+
+## Custom Implementation {#ssl-contract-impl-custom}
+
+To implement this contract, you must create a module that respects this contract. The following
+snippet shows an example.
+
+```nix
+{ lib, ... }:
+{
+  options.my.generator = {
+    paths = lib.mkOption {
+      description = ''
+        Paths where certs will be located.
+
+        This option implements the SSL Generator contract.
+      '';
+      type = contracts.ssl.certs-paths;
+      default = {
+        key = "/var/lib/my_generator/key.pem";
+        cert = "/var/lib/my_generator/cert.pem";
+      };
+    };
+
+    systemdService = lib.mkOption {
+      description = ''
+        Systemd oneshot service used to generate the certs.
+
+        This option implements the SSL Generator contract.
+      '';
+      type = lib.types.str;
+      default = "my-generator.service";
+    };
+
+    # Other options needed for this implementation
+  };
+
+  config = {
+    # custom implementation goes here
+  };
+}
+```
+
+You can then create an instance of this generator:
+
+```nix
+{
+  my.generator = ...;
+}
+```
+
+And use it whenever a module expects something implementing this SSL generator contract:
+
+```nix
+{ config, ... }:
+{
+  my.service.ssl = config.my.generator;
+}
+```

modules/contracts/ssl/dummyModule.nix

@@ -0,0 +1,10 @@
+{ pkgs, lib, ... }:
+let
+  contracts = pkgs.callPackage ../. {};
+in
+{
+  options.shb.contracts.ssl = lib.mkOption {
+    description = "Contract for SSL Certificate generator.";
+    type = contracts.ssl.certs;
+  };
+}