This patch contains migrations that require that there are no follow records.
If you have any, the migration will (hopefully) fail and you'll need to
manually delete any follow records you have. In the next patch I'll try to add
automatic following on the pseudo-client side by running both e.g. createNoteC
and followC in the same POST request handler.
Here's how it works:
- When Vervis starts, it writes a config file and it writes post-receive hooks
into all the repos it manages
- When a git push is accepted, git runs the post-receive hook, which is a
trivial shell script that executes the actual Haskell program implementing
the hook logic
- The Haskell hook program generates a Push JSON object and HTTP POSTs it to
Vervis running on localhost
- Vervis currently responds with an error, the next step is to implement the
actual publishing of ForgeFed Push activities
Currently it's a paged Collection where the items are merely URIs. This could
be changed to have actual Commit objects as items; for that we need to examine
the whole thing with the LogEntry type and the Patch type and have an
AP-friendly log item representation, but without commit diffs.
FedURIs, until now, have been requiring HTTPS, and no port number, and DNS
internet domain names. This works just fine on the forge fediverse, but it
makes local dev builds much less useful.
This patch introduces URI types that have a type tag specifying one of 2 modes:
- `Dev`: Works with URIs like `http://localhost:3000/s/fr33`
- `Fed`: Works with URIs like `https://dev.community/s/fr33`
This should allow even to run multiple federating instances for development,
without needing TLS or reverse proxies or editing the hosts files or anything
like that.
This patch also disables the ability to specify deps when creating a ticket,
because those deps won't be in the ticket object anymore. Instead of coding a
workaround and getting complications later, I just disabled that thing. It
wasn't really being used by anyone anyway.
aeson-pretty implements by formatting using a text Builder, and the ByteString
is encoded from that. So instead of decoding the ByteString to produce Text or
Builder, use the Builder as the starting point, to match how aeson-pretty works
and save computation and weird backwards-decoding stuff.
highligher2 doesn't have a JSON syntax and the JS lexer seems to be failing,
not sure exactly why yet. To have an alternative, I'm adding a Skylighting
option.
This patch doesn't just add the handler code, it also does lots of refactoring
and moves around pieces of code that are used in multiple places. There is
still lots of refactoring to make though. In this patch I tried to make minimal
changes to the existing Note handler to avoid breaking it. In later patches
I'll do some more serious refactoring, hopefully resulting with less mess in
the code.
`updateGet` isn't atomic by default. In PostgreSQL the default isolation level
if committed read, and an `update` followed by a `get` doesn't guarantee you
get the same value you sent. However I'm making a patch for `persistent` to
make `updateGet` atomic for PostgreSQL.
- The data returned from activity authentication has nicer types now, and no
mess of big tuples.
- Activity authentication code has its own module now, Vervis.Federation.Auth.
- The sharer inbox handler can now handle and store activities by a local
project actor, forwarded from a remote actor. This isn't in use right now,
but once projects start publishing Accept activities, or other things, it may
be needed.
CRITICAL: Due to the requirement that each new ticket points to its Offer
activity, ticket creation has been disabled! The next patches should implement
C2S submission of Offer Ticket, and then ticket creation will work again. Sorry
for that.
Since MonadSite now requires MonadIO, and not MonadUnliftIO, to allow for more
instances, the MonadUnliftIO constraint may need to be added manually
sometimes.
This allows to browse via e.g. localhost:3000 even if the instance host is
something else and the rendered URLs don't have a port number. It still makes
many things impossible or inconvenient, but at least you can launch Vervis
locally for development and see pages. Right now even CSS doesn't work because
of the URLs not matching the actual localhost:3000 access. Maybe gradually I'll
figure it out.
- Fork and async are no longer class methods, which simplifies things a lot and
allows for many more trivial instances, much like with MonadHandler. Fork and
async are still available, but instead of unnecessarily being class methods,
they are now provided as follows: You can fork and async a worker (no more
fork/async for handler, because I never actually need that, and not sure
there's ever a need for that in general), and you can do that from any
MonadSite. So, you can fork or async a worker from a Handler, from a Worker,
from a ReaderT on top of them e.g. inside runDB, and so on.
- Following the simplification, new MonadSite instances are provided, so far
just the ones in actual use in the code. ReaderT, ExceptT and lazy RWST. More
can be added easily. Oh, and WidgetFor got an instance too.
In particular, this change means there's no usage of `forkHandler` anymore, at
all. I wonder if it ever makes a difference to `forkWorker` versus
`forkHandler`. Like, does it cause memory leaks or anything. I guess could
check why `forkResource` etc. is good for in `forkHandler` implementation. I
suppose if needed, I could fix possible memory leaks in `forkWorker`.
* No more full URIs, all terms are used as short non-prefixed names
* Some terms support parsing full URI form for compatibility with objects in DB
* No more @context checking when parsing
* Use the new ForgeFed context URI specified in the spec draft
* Use an extension context URI for all custom properties not specific to forges
* Rename "events" property to "history", thanks cjslep for suggesting this name
* Have a project team collection, content is the same as ticket team (but
potentially ticket team allows people to opt out of updates on specific
tickets, while project team isn't tied to any specific ticket or other child
object)
* Have a project followers collection, and address it in ticket comments in
addition to the already used recipients (project, ticket team, ticket
followers)
This allows the inbox system to be separate from Person, allowing other kinds
of objects to have inboxes too. Much like there's FollowerSet which works
separately from Tickets, and will allow to have follower sets for projects,
users, etc. too.
Inboxes are made independent from Person users because I'm going to give
Projects inboxes too.
There used to be project roles and repo roles, and they were separate. A while
ago I merged them, and there has been a single role system, used with both
repos and projects. However the table names were still "ProjectRole" and things
like that. This patch renames some tables to just refer to a "Role" because
there's only one kind of role system.
A thing still missing there is that it sets empty audience for comments on
remote tickets, but that's fine because dev.angeley.es doesn't have such
comments in the database.
I added a migration that creates an ugly fake OutboxItem for messages that
don't have one. I'll try to turn it into a real one. And then very possibly
remove the whole ugly migration, replacing it with addFielfRefRequiredEmpty,
which should work for empty instances.
- Allow client to specify recipients that don't need to be delivered to
- When fetching recipient, recognize collections and don't try to deliver to
them
- Remember collections in DB, and use that to skip HTTP delivery
Worker is enough and seems much simpler. forkHandler does stuff with
forkResourceT and more stuff that I don't exactly understand and which may
involve more resource allocation. I guess forkWorker would generally be the
preferred approach, and there are bugs with delivery leading to sudden
CPU/memory peaks forcing me to kill the process. Maybe not related, just
mentioning it ^_^
In the new inbox forwarding scheme, we use an additional special HTTP signature
to indicate that we allow or expect forwarding, and to allow that forwarding to
later be verified. When delivering a comment on a remote ticket, we'd like the
project to do inbox forwarding. Based on the URI alone, it's impossible to tell
which recipient is the project, and I guess there are various tricks we could
use here, but for now a very simple solution is used: Enable forwarding for all
remote recipients whose host is the same as the ticket's host.
Until now, there were some simple host checks when verifying the HTTP sig,
meant to forbid hosts that are IP addresses, local hosts, and maybe other weird
cases. These checks moved to Network.FedURI, so now FedURIs in general aren't
allowed to have such hosts. The host type is still `Text` though, for now.
This patch does a small simple change, however at the cost of the request body
not being available for display in the latest activity list, unless processing
succeeds. I'll fix this situation in a separate patch.
It runs checks against all the relevant tables, but ultimately just inserts the
activity into the recipient's inbox and nothing more, leaving the RemoteMessage
creation and inbox forwarding to the project inbox handler.
Inbox post is disabled but in the next patches I'll code and integrate a fixed
complete one, hopefully finally getting ticket comment federation ready for
testing.
I'm making this change because if an actor receives an activity due to being
addressed in bto, ot bcc, or being listed in some remote collection, the server
doesn't have a way to tell which actor(s) are the intended recipients, without
having an individual inbox URL for each actor. I could use a different hack for
this, but it wouldn't be compatible with other AP servers (unless the whole
fediverse agrees on a method).
I wasn't using sharedInbox anyway, and it's an optimization either way.
Each ticket has a single discussion ID, and each ticket has a unique one, so,
given an inner join of tickets and discussions, I think there should be exactly
1 way select a (ticket, discussion) pair given any of these.
But for some reason, PostgreSQL started complaining. Not sure what changed.
Anyway, for now, I switched the groupBy from discussion.id to ticket.id, which
is essentially the same, but for some reason makes PostgreSQL happy. It can't
tell that given a discussion ID, there's exactly 1 way to choose the ticket. Or
something like that. I wonder if I messed up something in DB migrations.
Before this patch, the shared fetch used plain insert, because it relied on
being the only place in the codebase where new RemoteActors get inserted. I was
hoping for that to be the case, but while I tweak things and handle fetching
URIs that can be an actor or a public key (for which ActorFetchShare isn't
sufficient without some smart modification), I'd like concurrent insertions to
be safe, without getting in the way of ActorFetchShare.
With this patch, it now uses insertBy', which doesn't mind concurrent
insertions.
I wrote a function handleOutboxNote that's supposed to do the whole outbox POST
handler process. There's an outbox item table in the DB now, I adapted things
in various source files. Ticket comment federation work is still in progress.
The custom module provides a parametric wrapper, allowing any specific
FromJSON/ToJSON instance to be used. It's a standalone module though, and not a
wrapper of persistent-postgresql, because persistent-postgresql uses aeson
Value and it prevents using toEncoding to get from the value directly to a
string.
* Adapt DB related code to return the InstanceId and RemoteSharerId
* Previously, when fetching a known shared key, we were running a DB
check/update for the shared usage record. I noticed - and hopefully I
correctly noticed - that this check already runs when we discover the keyId
points to a shared key we already know. So, after successful sig
verification, there's no need to run the check again. So I removed it.
- Exclude hosts without periods, so things like localhost and IPv6 are rejected
- Exclude hosts without letters, so things like IPv4 are rejected
- Exclude the instance's own host, just in case somehow some fake activity
slips in and gets approved, maybe even accidentally when delivered by another
server
Before, things worked like this:
* Only signatures of Ed25519 keys could be verified
* Key encoding placed the plain binary Ed25519 key in the PEM, instead of the
key's ASN1 encoding
With this patch it now works like this:
* Ed25519 signatures are supported as before
* RSA keys are now supported too, assuming RSA-SHA256 signatures
* Both Ed25519 and RSA keys are encoded and decoded using actual PEM with ASN1
When we verify an HTTP signature,
* If we know the key, check in the DB whether we know the actor lists it. If it
doesn't, and there's room left for keys, HTTP GET the actor and update the DB
accordingly.
* If we know the key but had to update it, do the same, check usage in DB and
update DB if needed
* If we don't know the key, record usage in DB
However,
* If we're GETing a key and discovering it's a shared key, we GET the actor to
verify it lists the key. When we don't know the key at all yet, that's fine
(can be further optimized but it's marginal), but if it's a key we do know,
it means we already know the actor and for now it's enough for us to rely
only on the DB to test usage.
Previously, when verifying an HTTP signature and we fetched the key and
discovered it's shared, we'd fetch the actor and make sure it lists the key URI
in the `publicKey` field. But if we already knew the key, had it cached in our
DB, we wouldn't check the actor at all, despite not knowing whether it lists
the key.
With this patch, we now always GET the actor when the key is shared,
determining the actor URI from the `ActivityPub-Actor` request header, and we
verify that the actor lists the key URI. We do that regardless of whether or
not we have the key in the DB, although these two cases and handled in
different parts of the code right now (for a new key, it's in Web.ActivityPub
fetchKey; for a known key, it's in Vervis.Foundation httpVerifySig).
Previously, when verifying an HTTP signature and we find out we have the
provided keyid in the DB, and this key is a personal key, we would just grab
the key owner from the DB and ignore the ActivityPub-Actor header.
This patch adds a check: If we find the key in the DB and it's a personal key,
do grab the owner from that DB row, but also check the actor header: If it's
provided, it has to be identical to the key owner ID URI.
If the key we fetched is a shared key, the only way to determine the actor to
which the signature applies is to read the HTTP header ActivityPub-Actor. But
if it's a personal key, we can detect the actor by checking the key's owner
field. Still, if that actor header is provided, we now compare it to the key
owner and make sure they're identical.
When fetching a key that is embedded in the actor document, we were already
comparing the actor ID with the actor header, so that part didn't require
changes.
When a local user wants to publish an activity, we were always GETing the
recipient actor, so that we could determine their inbox and POST the activity
to it. But now, instead, whenever we GET an actor (whether it's for the key sig
verification or for determining inbox URI), we keep their inbox URI in the
database, and we don't need to GET it again next time.
Using a dedicated type allows to record in the type the guarantees that we
provide, such as scheme being HTTPS and authority being present. Allows to
replace ugly `fromJust` and such with direct field access.
Before, there was a single key used as a personal key for all actors. Now,
things work like this:
- There are 2 keys, each time one is rotated, this way the old key remains
valid and we can freely rotate without a risk of race conditions on other
servers and end up with our posts being rejected
- The keys are explicitly instance-scope keys, all actors refer to them
- We add the ActivityPub-Actor header to all activity POSTs we send, to declare
for which specific actor our signature applies. Activities and otherwise
different payloads may have varying ways to specify attribution; using this
header will be a standard uniform way to specify the actor, regardless of
payload format. Of course, servers should make sure the actual activity is
attributed to the same actor we specified in the header. (This is important
with instance-scope keys; for personal keys it's not critical)
