To be honest, this is a huge patch that changes tons of stuff and probably
should have been broken up into small changes. But I already had the codebase
not building, so... just did all of this at once :P
Basically this patch does the following:
- DB migrations for ticket dependency related tables, e.g. allowing a remote
author and a remote child
- Allowing S2S handlers to provide an async continued processing function,
which is executed and the result then added to the debug page
- Most UI and functionality related to ticket deps is disabled, new
implementation being added gradually via ActivityPub
- Improvements to AP tools, e.g. allow to specify multiple hosts for approved
forwarding when sending out an activity, and allow to specify audience of
software-authored activities using a convenient human-friendly structure
- Implementation of S2S sharerOfferDepF which creates a dependency under a
sharer-hosted ticket/patch and sends back an Accept
- Sharer-patch was already working due to the shared DB tabled
- Repo-patch support added
- Fixed a bug in following project-tickets: It allowed to follow a
sharer-ticket by sending the Follow to the project. Now, the project allows
to follow only the project-tickets, and refuses to handle a sharer-ticket.
Also fixed a bug in which trying to follow a ticket with nonexistent
ltkhid/talkhid would result with 404 as if the actor inbox is nonexistent. Now,
there's a friendly message reported.
The examples in the security vocabulary's spec use "Key" but the JSON-LD
context doesn't define that term. From now on, just in case, recognizing both
"Key" and "CryptographicKey" as indication that the object is a key.
zPlus, thanks for finding this bug!
This is the first step preparing for patches and merge requests.
The work-item aspect of MRs will reuse the Ticket related tables, except MRs
will live under repos. So, the context of tickets will no longer be just
projects, but will also be repos.
So, TicketProjectLocal turns into TicketContextLocal, and there are 2 new
tables that refer to it: TicketProjectLocal and TicketRepoLocal. Tickets will
have the former, MRs will have the latter.
Removed its usage in pseudo-client when publishing a comment, and removed it
from inbox forwarding when handling a remote comment.
Very possibly, the ticket team collection will be entirely removed. For now
leaving it there as-is. Just not using it for addressing in activities.
The implementation felt quite weird, had to add an extra field to Fetched and
to VerifKeyDetail. Should probably figure out the whole mess in that code, have
something clean there. Easily add fields. Easily and safely re-fetch an actor
or key.
Now it's much clearer when looking at the code, that these routes are about
project-hosted tickets, and it's easier to see where the author-hosted
equivalents are missing.
IMPORTANT: Since a lot of ticket code still doesn't use TicketUnderProject,
creating tickets now appears to be failing. Usage of this patch as is, is at
your own risk ^_^ the next patches will update the ticket handlers to fix this
problem.
Delivery of an activity into local inboxes is being done using custom local
functions. Each C2S or S2S handler has its own specific variant for this.
As part of the ongoing refactoring and evolution of the federation code, I
implemented a general-purpose local delivery function: It takes a
LocalRecipientSet and simply delivers to everyone, no handler-specific
assumptions or limitations.
To limit the recipient set according to handler specific rules, just
filter/adapt/edit it before passing to the delivery function.
The function isn't exported yet, but the existing 'deliverLocal' that delivers
only to actors and to author's followers is now implemented via the new
general-purpose function. I hope that's a step towards doing all the local
delivery using this one function, simplifying the complicated federation
code.
I'll use this for C2S to allow client to state who the tracker actor is. It's
still possible to do without it, by HTTP GETing the ticket's context and
checking whether we got an actor, or a non-actor with ticketsTrackedBy. Tbh I'm
adding createTarget simply because it's easier for coding, no need for a custom
variant of actor fetching :P
This allows the context to be specified even when replies/followers/deps/etc.
aren't. This is needed for Create-ing a Ticket. Also, it allows a ticket's
context to be on a different host than where it's hosted, which is also needed
for the Create flow.
A row in this table will be required for local-project-local-author tickets
hosted under the project, and non-existence of a row will be required for such
tickets hosted by the author. So I'll need to CAREFULLY update all the ticket
route handler code and all the ticket related AP code. The latter includes C2S
and S2S for tickets, ticket deps, ticket discussion... everything that is under
tickets.
That's because with the Create flow added, the activity that reports a ticket
can be either Create or Offer, maybe later Announce too.
The old TAL unique name mentioned in the migration has what may look like a
typo, "Locale" instead of "Local". That's because I made a typo in migration
115, and now needed to specify the typoed name I used then. I verified in dev
DB and on dev.angeley.es DB that the typo is reflected in the PostgreSQL
database side and fixed by the new migrations.
Everywhere Ticket is found, a matching LocalTicket is now expected to be found
too. Ticket doesn't point at LocalTicket because there will be remote cached
tickets too. Also, ticket URLs are going to switch the khid from Ticket to
LocalTicket (much like it's already the case for MessageR).
This is a step preparing for the Create flow for tickets. Each Ticket now gets
a matching LocalTicket that points to it. But otherwise the LocalTicket isn't
in use yet.
- WorkflowField now has a color, it's a simple `Maybe Int` for now. Valid
values are only 1-4
- That color is used for displaying ticket class params a.k.a labels in ticket
list view
- Ticket list now also serves a paged OrderedCollection
I tried to use a single SQL query to grab the tickets along with their labels,
but couldn't figure out a way to aggregate tuples/rows into an array (it seems
only single values are supported in Esqueleto). Instead of doing manual SQL or
adding Esqueleto functions, I just switched from 1 query to O(n) queries: Each
ticket has its own query selecting its labels. I guess it's slower, but also,
ticket list is paged now with fixed page size so it's really O(1) ^_^
I'm not sure this will improve much, because the error messages come from
attoparsec, but at least the message text won't be constant, which was the
previous situation.
Before, Push activities were being ignored by all inboxes. I just forgot to add
code to handle them. Now, person inboxes accept them if they're about a
relevant repo (i.e. a repo of which the user is a remote follower; remote
collaboration would be relevant too, but it's not implemented yet).
Follows used to be added automatically, without a Follow activity sent by the
client. They aren't added automatically anymore, so there's no need for those
"manual" boolean fields.
Before this patch, if you ran more than 1 instance as the same OS user, they'd
use the same config file path and overwrite it and cause post hooks to have
errors due to wrong config being used.
In Darcs, any command can have a post hook (and a pre hook), and the hook
command can be set using a command-line option to the darcs command that you
run. So, in the Vervis SSH server, if we add a --posthook option when running
`darcs apply` to apply remotely received patches, we get a chance to process
the patch data much like in the git post-receive hook.
The setup this patch creates is similar to the git one: It writes a
_darcs/prefs/defaults file to all Darcs repos, and that defaults file sets the
posthook line for `darcs apply`. The posthook line simply executes the actual
hook program written in Haskell.
The current hook program is a one-liner that prints a line to stdout, so every
time you `darcs push` you can tell the hook got executed. The next step is to
implement the actual hook logic, by reading patch data from the environment
variable in which Darcs puts it.
This patch contains migrations that require that there are no follow records.
If you have any, the migration will (hopefully) fail and you'll need to
manually delete any follow records you have. In the next patch I'll try to add
automatic following on the pseudo-client side by running both e.g. createNoteC
and followC in the same POST request handler.
Here's how it works:
- When Vervis starts, it writes a config file and it writes post-receive hooks
into all the repos it manages
- When a git push is accepted, git runs the post-receive hook, which is a
trivial shell script that executes the actual Haskell program implementing
the hook logic
- The Haskell hook program generates a Push JSON object and HTTP POSTs it to
Vervis running on localhost
- Vervis currently responds with an error, the next step is to implement the
actual publishing of ForgeFed Push activities
Currently it's a paged Collection where the items are merely URIs. This could
be changed to have actual Commit objects as items; for that we need to examine
the whole thing with the LogEntry type and the Patch type and have an
AP-friendly log item representation, but without commit diffs.
FedURIs, until now, have been requiring HTTPS, and no port number, and DNS
internet domain names. This works just fine on the forge fediverse, but it
makes local dev builds much less useful.
This patch introduces URI types that have a type tag specifying one of 2 modes:
- `Dev`: Works with URIs like `http://localhost:3000/s/fr33`
- `Fed`: Works with URIs like `https://dev.community/s/fr33`
This should allow even to run multiple federating instances for development,
without needing TLS or reverse proxies or editing the hosts files or anything
like that.
This patch also disables the ability to specify deps when creating a ticket,
because those deps won't be in the ticket object anymore. Instead of coding a
workaround and getting complications later, I just disabled that thing. It
wasn't really being used by anyone anyway.
aeson-pretty implements by formatting using a text Builder, and the ByteString
is encoded from that. So instead of decoding the ByteString to produce Text or
Builder, use the Builder as the starting point, to match how aeson-pretty works
and save computation and weird backwards-decoding stuff.
highligher2 doesn't have a JSON syntax and the JS lexer seems to be failing,
not sure exactly why yet. To have an alternative, I'm adding a Skylighting
option.
This patch doesn't just add the handler code, it also does lots of refactoring
and moves around pieces of code that are used in multiple places. There is
still lots of refactoring to make though. In this patch I tried to make minimal
changes to the existing Note handler to avoid breaking it. In later patches
I'll do some more serious refactoring, hopefully resulting with less mess in
the code.
`updateGet` isn't atomic by default. In PostgreSQL the default isolation level
if committed read, and an `update` followed by a `get` doesn't guarantee you
get the same value you sent. However I'm making a patch for `persistent` to
make `updateGet` atomic for PostgreSQL.
- The data returned from activity authentication has nicer types now, and no
mess of big tuples.
- Activity authentication code has its own module now, Vervis.Federation.Auth.
- The sharer inbox handler can now handle and store activities by a local
project actor, forwarded from a remote actor. This isn't in use right now,
but once projects start publishing Accept activities, or other things, it may
be needed.
CRITICAL: Due to the requirement that each new ticket points to its Offer
activity, ticket creation has been disabled! The next patches should implement
C2S submission of Offer Ticket, and then ticket creation will work again. Sorry
for that.
Since MonadSite now requires MonadIO, and not MonadUnliftIO, to allow for more
instances, the MonadUnliftIO constraint may need to be added manually
sometimes.
This allows to browse via e.g. localhost:3000 even if the instance host is
something else and the rendered URLs don't have a port number. It still makes
many things impossible or inconvenient, but at least you can launch Vervis
locally for development and see pages. Right now even CSS doesn't work because
of the URLs not matching the actual localhost:3000 access. Maybe gradually I'll
figure it out.
- Fork and async are no longer class methods, which simplifies things a lot and
allows for many more trivial instances, much like with MonadHandler. Fork and
async are still available, but instead of unnecessarily being class methods,
they are now provided as follows: You can fork and async a worker (no more
fork/async for handler, because I never actually need that, and not sure
there's ever a need for that in general), and you can do that from any
MonadSite. So, you can fork or async a worker from a Handler, from a Worker,
from a ReaderT on top of them e.g. inside runDB, and so on.
- Following the simplification, new MonadSite instances are provided, so far
just the ones in actual use in the code. ReaderT, ExceptT and lazy RWST. More
can be added easily. Oh, and WidgetFor got an instance too.
In particular, this change means there's no usage of `forkHandler` anymore, at
all. I wonder if it ever makes a difference to `forkWorker` versus
`forkHandler`. Like, does it cause memory leaks or anything. I guess could
check why `forkResource` etc. is good for in `forkHandler` implementation. I
suppose if needed, I could fix possible memory leaks in `forkWorker`.
* No more full URIs, all terms are used as short non-prefixed names
* Some terms support parsing full URI form for compatibility with objects in DB
* No more @context checking when parsing
* Use the new ForgeFed context URI specified in the spec draft
* Use an extension context URI for all custom properties not specific to forges
* Rename "events" property to "history", thanks cjslep for suggesting this name
* Have a project team collection, content is the same as ticket team (but
potentially ticket team allows people to opt out of updates on specific
tickets, while project team isn't tied to any specific ticket or other child
object)
* Have a project followers collection, and address it in ticket comments in
addition to the already used recipients (project, ticket team, ticket
followers)
This allows the inbox system to be separate from Person, allowing other kinds
of objects to have inboxes too. Much like there's FollowerSet which works
separately from Tickets, and will allow to have follower sets for projects,
users, etc. too.
Inboxes are made independent from Person users because I'm going to give
Projects inboxes too.
There used to be project roles and repo roles, and they were separate. A while
ago I merged them, and there has been a single role system, used with both
repos and projects. However the table names were still "ProjectRole" and things
like that. This patch renames some tables to just refer to a "Role" because
there's only one kind of role system.
A thing still missing there is that it sets empty audience for comments on
remote tickets, but that's fine because dev.angeley.es doesn't have such
comments in the database.
I added a migration that creates an ugly fake OutboxItem for messages that
don't have one. I'll try to turn it into a real one. And then very possibly
remove the whole ugly migration, replacing it with addFielfRefRequiredEmpty,
which should work for empty instances.
- Allow client to specify recipients that don't need to be delivered to
- When fetching recipient, recognize collections and don't try to deliver to
them
- Remember collections in DB, and use that to skip HTTP delivery
Worker is enough and seems much simpler. forkHandler does stuff with
forkResourceT and more stuff that I don't exactly understand and which may
involve more resource allocation. I guess forkWorker would generally be the
preferred approach, and there are bugs with delivery leading to sudden
CPU/memory peaks forcing me to kill the process. Maybe not related, just
mentioning it ^_^
In the new inbox forwarding scheme, we use an additional special HTTP signature
to indicate that we allow or expect forwarding, and to allow that forwarding to
later be verified. When delivering a comment on a remote ticket, we'd like the
project to do inbox forwarding. Based on the URI alone, it's impossible to tell
which recipient is the project, and I guess there are various tricks we could
use here, but for now a very simple solution is used: Enable forwarding for all
remote recipients whose host is the same as the ticket's host.
Until now, there were some simple host checks when verifying the HTTP sig,
meant to forbid hosts that are IP addresses, local hosts, and maybe other weird
cases. These checks moved to Network.FedURI, so now FedURIs in general aren't
allowed to have such hosts. The host type is still `Text` though, for now.
This patch does a small simple change, however at the cost of the request body
not being available for display in the latest activity list, unless processing
succeeds. I'll fix this situation in a separate patch.
It runs checks against all the relevant tables, but ultimately just inserts the
activity into the recipient's inbox and nothing more, leaving the RemoteMessage
creation and inbox forwarding to the project inbox handler.
Inbox post is disabled but in the next patches I'll code and integrate a fixed
complete one, hopefully finally getting ticket comment federation ready for
testing.
I'm making this change because if an actor receives an activity due to being
addressed in bto, ot bcc, or being listed in some remote collection, the server
doesn't have a way to tell which actor(s) are the intended recipients, without
having an individual inbox URL for each actor. I could use a different hack for
this, but it wouldn't be compatible with other AP servers (unless the whole
fediverse agrees on a method).
I wasn't using sharedInbox anyway, and it's an optimization either way.
Each ticket has a single discussion ID, and each ticket has a unique one, so,
given an inner join of tickets and discussions, I think there should be exactly
1 way select a (ticket, discussion) pair given any of these.
But for some reason, PostgreSQL started complaining. Not sure what changed.
Anyway, for now, I switched the groupBy from discussion.id to ticket.id, which
is essentially the same, but for some reason makes PostgreSQL happy. It can't
tell that given a discussion ID, there's exactly 1 way to choose the ticket. Or
something like that. I wonder if I messed up something in DB migrations.
Before this patch, the shared fetch used plain insert, because it relied on
being the only place in the codebase where new RemoteActors get inserted. I was
hoping for that to be the case, but while I tweak things and handle fetching
URIs that can be an actor or a public key (for which ActorFetchShare isn't
sufficient without some smart modification), I'd like concurrent insertions to
be safe, without getting in the way of ActorFetchShare.
With this patch, it now uses insertBy', which doesn't mind concurrent
insertions.
I wrote a function handleOutboxNote that's supposed to do the whole outbox POST
handler process. There's an outbox item table in the DB now, I adapted things
in various source files. Ticket comment federation work is still in progress.
The custom module provides a parametric wrapper, allowing any specific
FromJSON/ToJSON instance to be used. It's a standalone module though, and not a
wrapper of persistent-postgresql, because persistent-postgresql uses aeson
Value and it prevents using toEncoding to get from the value directly to a
string.
* Adapt DB related code to return the InstanceId and RemoteSharerId
* Previously, when fetching a known shared key, we were running a DB
check/update for the shared usage record. I noticed - and hopefully I
correctly noticed - that this check already runs when we discover the keyId
points to a shared key we already know. So, after successful sig
verification, there's no need to run the check again. So I removed it.
- Exclude hosts without periods, so things like localhost and IPv6 are rejected
- Exclude hosts without letters, so things like IPv4 are rejected
- Exclude the instance's own host, just in case somehow some fake activity
slips in and gets approved, maybe even accidentally when delivered by another
server
Before, things worked like this:
* Only signatures of Ed25519 keys could be verified
* Key encoding placed the plain binary Ed25519 key in the PEM, instead of the
key's ASN1 encoding
With this patch it now works like this:
* Ed25519 signatures are supported as before
* RSA keys are now supported too, assuming RSA-SHA256 signatures
* Both Ed25519 and RSA keys are encoded and decoded using actual PEM with ASN1
When we verify an HTTP signature,
* If we know the key, check in the DB whether we know the actor lists it. If it
doesn't, and there's room left for keys, HTTP GET the actor and update the DB
accordingly.
* If we know the key but had to update it, do the same, check usage in DB and
update DB if needed
* If we don't know the key, record usage in DB
However,
* If we're GETing a key and discovering it's a shared key, we GET the actor to
verify it lists the key. When we don't know the key at all yet, that's fine
(can be further optimized but it's marginal), but if it's a key we do know,
it means we already know the actor and for now it's enough for us to rely
only on the DB to test usage.
Previously, when verifying an HTTP signature and we fetched the key and
discovered it's shared, we'd fetch the actor and make sure it lists the key URI
in the `publicKey` field. But if we already knew the key, had it cached in our
DB, we wouldn't check the actor at all, despite not knowing whether it lists
the key.
With this patch, we now always GET the actor when the key is shared,
determining the actor URI from the `ActivityPub-Actor` request header, and we
verify that the actor lists the key URI. We do that regardless of whether or
not we have the key in the DB, although these two cases and handled in
different parts of the code right now (for a new key, it's in Web.ActivityPub
fetchKey; for a known key, it's in Vervis.Foundation httpVerifySig).
Previously, when verifying an HTTP signature and we find out we have the
provided keyid in the DB, and this key is a personal key, we would just grab
the key owner from the DB and ignore the ActivityPub-Actor header.
This patch adds a check: If we find the key in the DB and it's a personal key,
do grab the owner from that DB row, but also check the actor header: If it's
provided, it has to be identical to the key owner ID URI.
If the key we fetched is a shared key, the only way to determine the actor to
which the signature applies is to read the HTTP header ActivityPub-Actor. But
if it's a personal key, we can detect the actor by checking the key's owner
field. Still, if that actor header is provided, we now compare it to the key
owner and make sure they're identical.
When fetching a key that is embedded in the actor document, we were already
comparing the actor ID with the actor header, so that part didn't require
changes.
When a local user wants to publish an activity, we were always GETing the
recipient actor, so that we could determine their inbox and POST the activity
to it. But now, instead, whenever we GET an actor (whether it's for the key sig
verification or for determining inbox URI), we keep their inbox URI in the
database, and we don't need to GET it again next time.
Using a dedicated type allows to record in the type the guarantees that we
provide, such as scheme being HTTPS and authority being present. Allows to
replace ugly `fromJust` and such with direct field access.
Before, there was a single key used as a personal key for all actors. Now,
things work like this:
- There are 2 keys, each time one is rotated, this way the old key remains
valid and we can freely rotate without a risk of race conditions on other
servers and end up with our posts being rejected
- The keys are explicitly instance-scope keys, all actors refer to them
- We add the ActivityPub-Actor header to all activity POSTs we send, to declare
for which specific actor our signature applies. Activities and otherwise
different payloads may have varying ways to specify attribution; using this
header will be a standard uniform way to specify the actor, regardless of
payload format. Of course, servers should make sure the actual activity is
attributed to the same actor we specified in the header. (This is important
with instance-scope keys; for personal keys it's not critical)
Allow keys to specify expiration time using w3c security vocabulary. If a key
has expired, we treat it like sig validation failure and re-fetch the key from
the other server. And we never accept a sig, even a valid sig, if the key has
expired.
Since servers keep actors and keys in the DB, expiration can be a nice way to
ask that keys aren't used more than we want them to. The security vocab spec
also recommends to set expiration time on keys, so it's nice to support this
feature.
It's now possible for activities we be attributed to actors that have more than
one key. We allow up to 2 keys. We also store in the DB. Scaling to support any
number of keys is trivial, but I'm limiting to 2 to avoid potential trouble and
because 2 is the actual number we need.
By having 2 keys, and replacing only one of them in each rotation, we avoid
race conditions. With 1 key, the following can happen:
1. We send an activity to another server
2. We rotate our key
3. The server reaches the activity in its processing queue, tries to verify our
request signature, but fails because it can't fetch the key. It's the old
key and we discarded it already, replaced it with the new one
When we use 2 keys, the previous key remains available and other servers have
time to finish processing our requests signed with that key. We can safely
rotate, without worrying about whether the user sent anything right before the
rotation time.
Caveat: With this feature, we allow OTHER servers to rotate freely. It's safe
because it's optional, but it's just Vervis right now. Once Vervis itself
starts using 2 keys, it will be able to rotate freely without race condition
risk, but probably Mastodon etc. won't accept its signatures because of the use
of 2 keys and because they're server-scope keys.
Maybe I can get these features adopted by the fediverse?
Shared key means the key is used for multiple actors. I'm not sure explicitly
specifying this will be necessary, but I prefer to have it in place to help
with debugging in case something unexpected comes from other servers, or my
format overlaps with stuff used in other software and encodes a different
meaning.
Each public key can specify whether it's shared or personal, and this patch
checks for that when verifying a request signature. It rejects shared keys,
accepting valid sigs only from personal keys.
Very soon I'll add shared key support.
* Repo collab now supports basic default roles developer/user/guest like
project collab does
* User/Anon collab for repos and projects are now stored as fields instead of
in dedicated tables, there was never a need for dedicated tables but I didn't
see that before
* Repo push op is now part of `ProjectOperation`
* `RepoRole` and related code has been entirely removed, only project roles
remain and they're used for both repos and projects
* This is the first not-totally-trivial DB migration in Vervis, it's automatic
but please be careful and report errors
* When adding collaborators, you don't need a custom role. If you don't choose
one, a basic default "developer" role will be used
* If you don't assign a `ProjectCollabUser` role, a default "user" role is
assumed for logged in users, otherwise a "guest" role
* The "guest" role currently has no access at all
* Theoretically there may also be a "maintainer" role allowing project
sharers/maintainers to give maintainer-level access to more people, but right
now maintainer role would be the same as developer so I haven't added it yet
It already had one, but it didn't have a public key and it was using the old
mess of the Vervis.ActivityStreams module, which I'll possibly remove soon.
It's hopefully more elegant now.
This patch includes some ugliness and commented out code. Sorry for that. I'll
clean it up soon.
Basically there's a TVar holding a Vector of at most 10 AP activities. You can
freely POST stuff to /inbox, and then GET /inbox and see what you posted, or an
error description saying why your activity was rejected.
The actor key will be used for all actors on the server. It's held in a `TVar`
so that it can always be safely updated and safely retrieved (technically there
is a single writer so IORef and MVar could work, but they require extra care
while TVar is by design suited for this sort of thing).
In Haskell by default if a thread has an exception, the main thread isn't
notified at all. This patch changes service thread launching to re-throw their
exceptions in the main thread, so that their failure is noticed.
I suppose there's no performance difference in using one, but it requires
`http-conduit` as a build dependency, so potentially we may be reducing build
time by removing unnecessary deps.
Git pull uses a POST request, which is treated as a write request and the CSRF
token is checked. However, no modification to the server is made by git pulls,
as far as I know (actually I'm not sure why it uses a POST). The entire
response is handled by the git command, and the client side is usually the git
command running in the terminal, there's no session and no cookies (as far as I
know). So I'm just disabling CSRF token checking for this route.
The sharer and repo were being taken and used as is to check push permissions,
which is how it's supposed to be, *but* they were also being used as is to
build the repo path! So sharer and repo names that aren't all lowercase were
getting "No such repository" errors when trying to push.
I changed `RepoSpec` to hold `ShrIdent` and `RpIdent` instead of plain `Text`,
to avoid confusions like that and be clear and explicit about the
representation, and failures to find a repo after verifying it against the DB
are now logged as errors to help with debugging.
I hope this fixes the problem.
