hosting/selfhostblocks
1
0
Fork 0
Code Issues Activity

add contract documentation (#225)

Browse source
This commit is contained in:
Pierre Penninckx 2024-04-14 15:21:20 -07:00 committed by GitHub
parent 26f406db5f
commit 43f19a871a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
15 changed files with 324 additions and 97 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

59
README.md
View file

@ -5,14 +5,20 @@
[![Documentation](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml/badge.svg)](https://github.com/ibizaman/selfhostblocks/actions/workflows/pages.yml)
[![Tests](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Fgarnix.io%2Fapi%2Fbadges%2Fibizaman%2Fselfhostblocks%3Fbranch%3Dmain)](https://garnix.io) (using Garnix)
SHB's (Self Host Blocks) is yet another server management tool whose goal is to provide a lower
entry-bar for self-hosting. SHB provides opinionated [building blocks](#available-blocks) fitting
together to self-host any service you'd want. Some [common services](#provided-services) are
provided out of the box.
SHB's (Self Host Blocks) is yet another server management tool whose goal is to provide better
building blocks for self-hosting. Indeed, SHB provides opinionated [building
blocks](#available-blocks) fitting together to self-host any service you'd want. Some [common
services](#provided-services) are provided out of the box.
To achieve this, SHB is using the full power of NixOS modules. Indeed, each building block and each
service is a NixOS module and uses the modules defined in
[Nixpkgs](https://github.com/NixOS/nixpkgs/).
SHB's goal is to make these building blocks plug-and-play. To achieve this, SHB pioneers
[contracts](https://shb.skarabox.com/usage.html) which allows you, the final user, to be more in
control of which pieces go where. The promise here is to let you choose, for example, any reverse
proxy you want or any database you want, without requiring work from maintainers of the services you
want to self host.
To achieve all this, SHB is using the full power of NixOS modules and NixOS VM tests. Indeed, each
building block and each service is a NixOS module using modules defined in
[Nixpkgs](https://github.com/NixOS/nixpkgs/) and they are tested using full VMs on every commit.
## TOC
@ -39,16 +45,18 @@ Self Host Blocks is available as a flake. To use it in your project, add the fol
inputs.selfhostblocks.url = "github:ibizaman/selfhostblocks";
```
See [the manual](https://shb.skarabox.com/usage.html) for more information about installing Self
Host Blocks.
This is not quite enough though and more information is provided in [the
manual](https://shb.skarabox.com/usage.html).
- You are new to self hosting and want pre-configured services to deploy easily. Look at the
[services section](https://shb.skarabox.com/services.html).
- You are a seasoned self-hoster but want to enhance some services you deploy already. Go to the
[blocks section](https://shb.skarabox.com/blocks.html).
- You are a user of Self Host Blocks but would like to use your own implementation for a block. Head
over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) to talk about it
(this is WIP).
- You are a user of Self Host Blocks but would like to use your own implementation for a block. Go
to the [contracts section](https://shb.skarabox.com/contracts.html).
Head over to the [matrix channel](https://matrix.to/#/#selfhostblocks:matrix.org) for any remaining
question, or just to say hi :)
## Why yet another self hosting tool?
@ -59,21 +67,24 @@ specifically:
- atomic configuration rollbacks;
- real programming language to define configurations;
- user-defined abstractions (create your own functions or NixOS modules on top of SHB!);
- integration with the rest of nixpkgs.
- integration with the rest of nixpkgs;
- much fewer "works on my machine" type of issues.
In no particular order, here are some aspects of SHB which I find interesting and differentiates it
from other server management projects:
- SHB intends to be a library, not a framework. You can either go all in and use SHB provided
services directly or use just one block in your existing infrastructure.
- SHB introduces contracts to allow you to swap implementation for each self-hosting need.
For example, you should be able to use the reverse proxy you want without modifying any services
depending on it.
- SHB introduces [contracts](https://shb.skarabox.com/contracts.html) to allow you to swap
implementation for each self-hosting need. For example, you should be able to use the reverse
proxy you want without modifying any services depending on it.
- SHB contracts also allows you to use your own custom implementation instead of the provided one,
as long as it follows the contract and passes the tests.
- SHB provides at least one implementation for each self-hosting need like backups, SSL
certificates, reverse proxy, VPN, etc. Those are called blocks here. They are documented in [the
- SHB provides at least one implementation for each contract like backups, SSL certificates, reverse
proxy, VPN, etc. Those are called blocks here and are documented in [the
manual](https://shb.skarabox.com/blocks.html).
- SHB provides several services out of the box fully using the blocks provided. Those can also be
found in [the manual](https://shb.skarabox.com/services.html).
- SHB follows nixpkgs unstable branch closely. There is a GitHub action running daily that updates
the `nixpkgs` input in the root `flakes.nix`, runs the tests and merges a PR with the new input if
the tests pass.
@ -82,9 +93,9 @@ from other server management projects:
The manual can be found at [shb.skarabox.com](https://shb.skarabox.com/).
Currently, only some services and blocks are documented. For the rest, unfortunately the source code
is the best place to read about them. [Here](./modules/services) for services and
[here](./modules/blocks) for blocks.
Work is in progress to document everything in the manual but I'm not there yet. For what's not yet
documented, unfortunately the source code is the best place to read about them.
[Here](./modules/services) for services and [here](./modules/blocks) for blocks.
## Roadmap
@ -95,9 +106,11 @@ contracts.
Upstreaming changes is also on the roadmap.
Check [the issues](https://github.com/ibizaman/selfhostblocks/issues) to see planned works.
Check [the issues](https://github.com/ibizaman/selfhostblocks/issues) to see planned works. Feel
free to add more!
That being said, I am personally using all the blocks and services in this project, so they do work.
That being said, I am personally using all the blocks and services in this project, so they do work
to some extent.
## Available Blocks

BIN
docs/assets/contracts_after.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 69 KiB

BIN
docs/assets/contracts_before.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
docs/assets/contracts_separationofconcerns.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 77 KiB

12
docs/blocks.md
View file

@ -1,12 +1,12 @@
# Blocks {#blocks}
Blocks help you self-host apps or services. They define and implement a specific function like
backup or secure access through a subdomain. Each block is designed to be usable on its own and to
fit nicely with others.
Blocks help you self-host apps or services. They implement a specific function like backup or secure
access through a subdomain. Each block is designed to be usable on its own and to fit nicely with
others.
In practice, a block defines a contract that must be followed to implement a specific self-hosting
function. It also comes with a unit test and NixOS VM test suite to ensure any implementation
follows the contract.
In practice, a block implements a [contract](contracts.html) that must be followed to implement a
specific self-hosting function. It also comes with a unit test and NixOS VM test suite to ensure any
implementation follows the contract.
As an example, let's take the HTTPS access block which allows for a service to be accessible through
a specific subdomain. In Nix terms, this block defines at minimum the inputs:

91
docs/contracts.md Normal file
View file

@ -0,0 +1,91 @@
# Contracts {#contracts}
A contract decouples modules that use a functionality from modules that provide it. A first
intuition for contracts is they are generally related to accessing a shared resource.
A few examples of contracts are generating SSL certificates, creating a user or knowing which files
and folders to backup. Indeed, when generating certificates, the service using those do not care how
they were created. They just need to know where the certificate files are located.
In practice, a contract is a set of options that any user of a contract expects to exist. Also, the
values of these options dictate the behavior of the implementation. This is enforced with NixOS VM
tests.
## Provided contracts {#contracts-provided}
Self Host Blocks is a proving ground of contracts. This repository adds a layer on top of services
available in nixpkgs to make them work using contracts. In time, we hope to upstream as much of this
as possible, reducing the quite thick layer that it is now.
Provided contracts are:
- [SSL generator contract](contracts-ssl.html) to generate SSL certificates. Two implementations are provided: self-signed and Let's Encrypt.
```{=include=} chapters html:into-file=//contracts-ssl.html
modules/contracts/ssl/docs/default.md
```
## Why do we need this new concept? {#contracts-why}
Currently in nixpkgs, every module needing access to a shared resource must implement the logic
needed to setup that resource themselves. Similarly, if the module is mature enough to let the user
select a particular implementation, the code lives inside that module.
![](./assets/contracts_before.png "A module composed of a core logic and a lot of peripheral logic.")
This has a few disadvantages:
- This leads to a lot of **duplicated code**. If a module wants to support a new implementation of a
contract, the maintainers of that module must write code to make that happen.
- This also leads to **tight coupling**. The code written by the maintainers cannot be reused in
other modules, apart from copy pasting.
- There is also a **lack of separation of concerns**. The maintainers of a service must be experts
in all implementations they let the users choose from.
- Finally, this is **not extensible**. If you, the user of the module, want to use another
implementation that is not supported, you are out of luck. You can always dive into the module's
code and extend it, but that is not an optimal experience.
We do believe that the decoupling contracts provides helps alleviate all the issues outlined above
which makes it an essential step towards more adoption of Nix, if only in the self hosting scene.
![](./assets/contracts_after.png "A module containing only logic using peripheral logic through contracts.")
Indeed, contracts allow:
- **Reuse of code**. Since the implementation of a contract lives outside of modules using it, using
that implementation elsewhere is trivial.
- **Loose coupling**. Modules that use a contract do not care how they are implemented, as long as
the implementation follows the behavior outlined by the contract.
- Full **separation of concerns** (see diagram below). Now, each party's concern is separated with a
clear boundary. The maintainer of a module using a contract can be different from the maintainers
of the implementation, allowing them to be experts in their own respective fields. But more
importantly, the contracts themselves can be created and maintained by the community.
- Full **extensibility**. The final user themselves can choose an implementation, even new custom
implementations not available in nixpkgs, without changing existing code.
![](./assets/contracts_separationofconcerns.png "Separation of concerns thanks to contracts.")
Thanks to NixOS VM test, we can even go one step further by ensuring each implementation of a
contract provides required options and behaves as the contract requires.
## Are there contracts in nixpkgs already? {#contracts-nixpkgs}
Actually yes, there are some ubiquitous options in nixpkgs. Those I found are:
- `services.<name>.enable`
- `services.<name>.package`
- `services.<name>.openFirewall`
- `services.<name>.user`
- `services.<name>.group`
What makes those nearly contracts are:
- Pretty much every service provides them.
- Users of a service expects them to exist and expects a consistent type and behavior from them.
Indeed, everyone knows what happens if you set `enable = true`.
- Maintainers of a service knows that users expects those options. They also know what behavior the
user expects when setting those options.
- The name of the options is the same everywhere.
The only thing missing to make these explicit contracts is, well, the contracts themselves.
Currently, they are conventions and not contracts.

5
docs/default.nix
View file

@ -144,6 +144,11 @@ in stdenv.mkDerivation {
'@OPTIONS_JSON@' \
${individualModuleOptionsDocs ../modules/services/nextcloud-server.nix}/share/doc/nixos/options.json
substituteInPlace ./modules/contracts/ssl/docs/default.md \
--replace \
'@OPTIONS_JSON@' \
${individualModuleOptionsDocs ../modules/contracts/ssl/dummyModule.nix}/share/doc/nixos/options.json
find . -name "*.md" -print0 | \
while IFS= read -r -d ''' f; do
substituteInPlace "''${f}" \

4
docs/manual.md
View file

@ -15,6 +15,10 @@ usage.md
services.md
```
```{=include=} chapters html:into-file=//contracts.html
contracts.md
```
```{=include=} chapters html:into-file=//blocks.html
blocks.md
```

8
flake.nix
View file

@ -49,6 +49,11 @@
modules/services/nextcloud-server.nix
modules/services/vaultwarden.nix
];
# Only used for documentation.
contractDummyModules = [
modules/contracts/ssl/dummyModule.nix
];
in
{
nixosModules.default = { config, ... }: {
@ -56,7 +61,8 @@
};
packages.manualHtml = pkgs.callPackage ./docs {
inherit allModules nmdsrc;
inherit nmdsrc;
allModules = allModules ++ contractDummyModules;
release = "0.0.1";
};

2
modules/blocks/monitoring/docs/default.md
View file

@ -92,7 +92,7 @@ Self Host Blocks will create automatically the following resources:
Those resources are namespaced as appropriate under the Self Host Blocks namespace:
[](./assets/folder.png)
![](./assets/folder.png)
## Errors Dashboard {#blocks-monitoring-error-dashboard}

24
modules/blocks/ssl.nix
View file

@ -32,7 +32,7 @@ in
description = ''
Paths where CA certs will be located.
This option is the contract output of the `shb.certs.cas` SSL block.
This option implements the SSL Generator contract.
'';
type = contracts.ssl.certs-paths;
default = rec {
@ -42,7 +42,11 @@ in
};
systemdService = lib.mkOption {
description = "Systemd oneshot service used to generate the certs.";
description = ''
Systemd oneshot service used to generate the certs.
This option implements the SSL Generator contract.
'';
type = lib.types.str;
default = "shb-certs-ca-${config._module.args.name}.service";
};
@ -100,7 +104,7 @@ in
description = ''
Paths where certs will be located.
This option is the contract output of the `shb.certs.certs` SSL block.
This option implements the SSL Generator contract.
'';
type = contracts.ssl.certs-paths;
default = rec {
@ -110,7 +114,11 @@ in
};
systemdService = lib.mkOption {
description = "Systemd oneshot service used to generate the certs.";
description = ''
Systemd oneshot service used to generate the certs.
This option implements the SSL Generator contract.
'';
type = lib.types.str;
default = "shb-certs-cert-selfsigned-${config._module.args.name}.service";
};
@ -159,7 +167,7 @@ in
description = ''
Paths where certs will be located.
This option is the contract output of the `shb.certs.certs` SSL block.
This option implements the SSL Generator contract.
'';
type = contracts.ssl.certs-paths;
default = {
@ -178,7 +186,11 @@ in
};
systemdService = lib.mkOption {
description = "Systemd oneshot service used to generate the certs.";
description = ''
Systemd oneshot service used to generate the certs.
This option implements the SSL Generator contract.
'';
type = lib.types.str;
default = "shb-certs-cert-letsencrypt-${config._module.args.name}.service";
};

70
modules/blocks/ssl/docs/default.md
View file

@ -1,6 +1,6 @@
# SSL Block {#ssl-block}
# SSL Generator Block {#ssl-block}
This NixOS module is a block that provides a contract to generate TLS certificates.
This NixOS module is a block that implements the [SSL certificate generator](contracts-ssl.html) contract.
It is implemented by:
- [`shb.certs.cas.selfsigned`][10] and [`shb.certs.certs.selfsigned`][11]: Generates self-signed certificates,
@ -14,27 +14,7 @@ It is implemented by:
[11]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.selfsigned
[12]: blocks-ssl.html#blocks-ssl-options-shb.certs.certs.letsencrypt
## Contract {#ssl-block-contract}
The contract for this block is defined in [`/modules/contracts/ssl.nix`](@REPO@/modules/contracts/ssl.nix).
Every module implementing this contract provides the following options:
- `domain`: Domain to generate the certificate for.
- `extraDomains`: Other domains the certificate should be generated for.
- `group`: The unix group owning this certificate.
- `reloadServices`: Systemd services to reload when the certificate gets renewed.
- `paths.cert`: Path to the cert file.
- `paths.key`: Path to the key file.
- `systemdService`: Systemd oneshot service used to generate the certificate.
The Systemd service file name must include the `.service` suffix. Downstream users of the
certificate can use this option to wait for the certificate to be generated.
## Implementations {#ssl-block-impl}
This sections explains how to generate certificates using the SSL block implementations.
### Self-Signed Certificates {#ssl-block-impl-self-signed}
## Self-Signed Certificates {#ssl-block-impl-self-signed}
Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix).
@ -72,7 +52,7 @@ shb.certs.certs.selfsigned = {
The group has been chosen to be `nginx` to be consistent with the examples further down in this
document.
### Let's Encrypt {#ssl-block-impl-lets-encrypt}
## Let's Encrypt {#ssl-block-impl-lets-encrypt}
Defined in [`/modules/blocks/ssl.nix`](@REPO@/modules/blocks/ssl.nix).
@ -109,6 +89,7 @@ where the certificate and the private key are located:
```nix
config.shb.certs.certs.<implementation>.<name>.paths.cert
config.shb.certs.certs.<implementation>.<name>.paths.key
config.shb.certs.certs.<implementation>.<name>.systemdService
```
For example:
@ -116,50 +97,27 @@ For example:
```nix
config.shb.certs.certs.selfsigned."example.com".paths.cert
config.shb.certs.certs.selfsigned."example.com".paths.key
```
We can then configure Nginx to use those certificates:
```nix
services.nginx.virtualHosts."example.com" =
let
cert = config.shb.certs.certs.selfsigned."example.com";
in
{
onlySSL = true;
sslCertificate = cert.paths.cert;
sslCertificateKey = cert.paths.key;
locations."/".extraConfig = ''
add_header Content-Type text/plain;
return 200 'It works!';
'';
};
config.shb.certs.certs.selfsigned."example.com".systemdService
```
To make sure the Nginx webserver can find the generated file, we will make it wait for the
certificate to the generated:
```nix
systemd.services.nginx = {
after = [ config.shb.certs.certs.selfsigned."example.com".systemdService ];
requires = [ config.shb.certs.certs.selfsigned."example.com".systemdService ];
};
```
If needed, we can also wait on the CA bundle to be generated by waiting for the Systemd service:
The full CA bundle is generated by the following Systemd service, running after each individual
generator finished:
```nix
config.shb.certs.systemdService
```
See also the [SSL certificate generator usage](contracts-ssl.html#ssl-contract-usage) for a more detailed usage
example.
## Debug {#ssl-block-debug}
Each CA and Cert is generated by a systemd service whose name can be seen in `systemdService`
options below. You can then see the latest errors messages using `journalctl`.
Each CA and Cert is generated by a systemd service whose name can be seen in the `systemdService`
option. You can then see the latest errors messages using `journalctl`.
## Tests {#ssl-block-tests}
This block is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix).
The self-signed implementation is tested in [`/tests/vm/ssl.nix`](@REPO@/tests/vm/ssl.nix).
## Options Reference {#ssl-block-options}

17
modules/contracts/ssl.nix
View file

@ -1,4 +1,4 @@
{ lib }:
{ lib, ... }:
rec {
certs-paths = lib.types.submodule {
freeformType = lib.types.anything;
@ -28,8 +28,14 @@ rec {
};
systemdService = lib.mkOption {
description = "Systemd oneshot service used to generate the CA.";
description = ''
Systemd oneshot service used to generate the CA. Ends with the `.service` suffix.
Use this if downstream services must wait for the certificates to be generated before
starting.
'';
type = lib.types.str;
example = "ca-generator.service";
};
};
};
@ -48,10 +54,13 @@ rec {
systemdService = lib.mkOption {
description = ''
Systemd oneshot service used to generate the certificate. The name must include the
`.service` suffix.
Systemd oneshot service used to generate the certificate. Ends with the `.service` suffix.
Use this if downstream services must wait for the certificates to be generated before
starting.
'';
type = lib.types.str;
example = "cert-generator.service";
};
};
};

119
modules/contracts/ssl/docs/default.md Normal file
View file

@ -0,0 +1,119 @@
# SSL Generator Contract {#ssl-contract}
This NixOS contract represents an SSL certificate generator. This contract is used to decouple
generating an SSL certificate from using it. In practice, you can swap generators without updating
modules depending on it.
## Contract Reference {#ssl-contract-options}
These are all the options that are expected to exist for this contract to be respected.
```{=include=} options
id-prefix: contracts-ssl-options-
list-id: selfhostblocks-options
source: @OPTIONS_JSON@
```
## Usage {#ssl-contract-usage}
Let's assume a module implementing this contract is available under the `ssl` variable:
```nix
let
ssl = <...>;
in
```
To use this module, we can reference the path where the certificate and the private key are located with:
```nix
ssl.paths.cert
ssl.paths.key
```
We can then configure Nginx to use those certificates:
```nix
services.nginx.virtualHosts."example.com" = {
onlySSL = true;
sslCertificate = ssl.paths.cert;
sslCertificateKey = ssl.paths.key;
locations."/".extraConfig = ''
add_header Content-Type text/plain;
return 200 'It works!';
'';
};
```
To make sure the Nginx webserver can find the generated file, we will make it wait for the
certificate to the generated:
```nix
systemd.services.nginx = {
after = [ ssl.systemdService ];
requires = [ ssl.systemdService ];
};
```
## Provided Implementations {#ssl-contract-impl-shb}
Multiple implementation are provided out of the box at [SSL block](blocks-ssl.html).
## Custom Implementation {#ssl-contract-impl-custom}
To implement this contract, you must create a module that respects this contract. The following
snippet shows an example.
```nix
{ lib, ... }:
{
options.my.generator = {
paths = lib.mkOption {
description = ''
Paths where certs will be located.
This option implements the SSL Generator contract.
'';
type = contracts.ssl.certs-paths;
default = {
key = "/var/lib/my_generator/key.pem";
cert = "/var/lib/my_generator/cert.pem";
};
};
systemdService = lib.mkOption {
description = ''
Systemd oneshot service used to generate the certs.
This option implements the SSL Generator contract.
'';
type = lib.types.str;
default = "my-generator.service";
};
# Other options needed for this implementation
};
config = {
# custom implementation goes here
};
}
```
You can then create an instance of this generator:
```nix
{
my.generator = ...;
}
```
And use it whenever a module expects something implementing this SSL generator contract:
```nix
{ config, ... }:
{
my.service.ssl = config.my.generator;
}
```

10
modules/contracts/ssl/dummyModule.nix Normal file
View file

@ -0,0 +1,10 @@
{ pkgs, lib, ... }:
let
contracts = pkgs.callPackage ../. {};
in
{
options.shb.contracts.ssl = lib.mkOption {
description = "Contract for SSL Certificate generator.";
type = contracts.ssl.certs;
};
}