I'm not sure this will improve much, because the error messages come from
attoparsec, but at least the message text won't be constant, which was the
previous situation.
Before, Push activities were being ignored by all inboxes. I just forgot to add
code to handle them. Now, person inboxes accept them if they're about a
relevant repo (i.e. a repo of which the user is a remote follower; remote
collaboration would be relevant too, but it's not implemented yet).
Follows used to be added automatically, without a Follow activity sent by the
client. They aren't added automatically anymore, so there's no need for those
"manual" boolean fields.
Before this patch, if you ran more than 1 instance as the same OS user, they'd
use the same config file path and overwrite it and cause post hooks to have
errors due to wrong config being used.
In Darcs, any command can have a post hook (and a pre hook), and the hook
command can be set using a command-line option to the darcs command that you
run. So, in the Vervis SSH server, if we add a --posthook option when running
`darcs apply` to apply remotely received patches, we get a chance to process
the patch data much like in the git post-receive hook.
The setup this patch creates is similar to the git one: It writes a
_darcs/prefs/defaults file to all Darcs repos, and that defaults file sets the
posthook line for `darcs apply`. The posthook line simply executes the actual
hook program written in Haskell.
The current hook program is a one-liner that prints a line to stdout, so every
time you `darcs push` you can tell the hook got executed. The next step is to
implement the actual hook logic, by reading patch data from the environment
variable in which Darcs puts it.
This patch contains migrations that require that there are no follow records.
If you have any, the migration will (hopefully) fail and you'll need to
manually delete any follow records you have. In the next patch I'll try to add
automatic following on the pseudo-client side by running both e.g. createNoteC
and followC in the same POST request handler.
Here's how it works:
- When Vervis starts, it writes a config file and it writes post-receive hooks
into all the repos it manages
- When a git push is accepted, git runs the post-receive hook, which is a
trivial shell script that executes the actual Haskell program implementing
the hook logic
- The Haskell hook program generates a Push JSON object and HTTP POSTs it to
Vervis running on localhost
- Vervis currently responds with an error, the next step is to implement the
actual publishing of ForgeFed Push activities
Currently it's a paged Collection where the items are merely URIs. This could
be changed to have actual Commit objects as items; for that we need to examine
the whole thing with the LogEntry type and the Patch type and have an
AP-friendly log item representation, but without commit diffs.
FedURIs, until now, have been requiring HTTPS, and no port number, and DNS
internet domain names. This works just fine on the forge fediverse, but it
makes local dev builds much less useful.
This patch introduces URI types that have a type tag specifying one of 2 modes:
- `Dev`: Works with URIs like `http://localhost:3000/s/fr33`
- `Fed`: Works with URIs like `https://dev.community/s/fr33`
This should allow even to run multiple federating instances for development,
without needing TLS or reverse proxies or editing the hosts files or anything
like that.
This patch also disables the ability to specify deps when creating a ticket,
because those deps won't be in the ticket object anymore. Instead of coding a
workaround and getting complications later, I just disabled that thing. It
wasn't really being used by anyone anyway.
aeson-pretty implements by formatting using a text Builder, and the ByteString
is encoded from that. So instead of decoding the ByteString to produce Text or
Builder, use the Builder as the starting point, to match how aeson-pretty works
and save computation and weird backwards-decoding stuff.
highligher2 doesn't have a JSON syntax and the JS lexer seems to be failing,
not sure exactly why yet. To have an alternative, I'm adding a Skylighting
option.
This patch doesn't just add the handler code, it also does lots of refactoring
and moves around pieces of code that are used in multiple places. There is
still lots of refactoring to make though. In this patch I tried to make minimal
changes to the existing Note handler to avoid breaking it. In later patches
I'll do some more serious refactoring, hopefully resulting with less mess in
the code.
`updateGet` isn't atomic by default. In PostgreSQL the default isolation level
if committed read, and an `update` followed by a `get` doesn't guarantee you
get the same value you sent. However I'm making a patch for `persistent` to
make `updateGet` atomic for PostgreSQL.
- The data returned from activity authentication has nicer types now, and no
mess of big tuples.
- Activity authentication code has its own module now, Vervis.Federation.Auth.
- The sharer inbox handler can now handle and store activities by a local
project actor, forwarded from a remote actor. This isn't in use right now,
but once projects start publishing Accept activities, or other things, it may
be needed.
CRITICAL: Due to the requirement that each new ticket points to its Offer
activity, ticket creation has been disabled! The next patches should implement
C2S submission of Offer Ticket, and then ticket creation will work again. Sorry
for that.
Since MonadSite now requires MonadIO, and not MonadUnliftIO, to allow for more
instances, the MonadUnliftIO constraint may need to be added manually
sometimes.
This allows to browse via e.g. localhost:3000 even if the instance host is
something else and the rendered URLs don't have a port number. It still makes
many things impossible or inconvenient, but at least you can launch Vervis
locally for development and see pages. Right now even CSS doesn't work because
of the URLs not matching the actual localhost:3000 access. Maybe gradually I'll
figure it out.
- Fork and async are no longer class methods, which simplifies things a lot and
allows for many more trivial instances, much like with MonadHandler. Fork and
async are still available, but instead of unnecessarily being class methods,
they are now provided as follows: You can fork and async a worker (no more
fork/async for handler, because I never actually need that, and not sure
there's ever a need for that in general), and you can do that from any
MonadSite. So, you can fork or async a worker from a Handler, from a Worker,
from a ReaderT on top of them e.g. inside runDB, and so on.
- Following the simplification, new MonadSite instances are provided, so far
just the ones in actual use in the code. ReaderT, ExceptT and lazy RWST. More
can be added easily. Oh, and WidgetFor got an instance too.
In particular, this change means there's no usage of `forkHandler` anymore, at
all. I wonder if it ever makes a difference to `forkWorker` versus
`forkHandler`. Like, does it cause memory leaks or anything. I guess could
check why `forkResource` etc. is good for in `forkHandler` implementation. I
suppose if needed, I could fix possible memory leaks in `forkWorker`.
* No more full URIs, all terms are used as short non-prefixed names
* Some terms support parsing full URI form for compatibility with objects in DB
* No more @context checking when parsing
* Use the new ForgeFed context URI specified in the spec draft
* Use an extension context URI for all custom properties not specific to forges
* Rename "events" property to "history", thanks cjslep for suggesting this name
* Have a project team collection, content is the same as ticket team (but
potentially ticket team allows people to opt out of updates on specific
tickets, while project team isn't tied to any specific ticket or other child
object)
* Have a project followers collection, and address it in ticket comments in
addition to the already used recipients (project, ticket team, ticket
followers)
This allows the inbox system to be separate from Person, allowing other kinds
of objects to have inboxes too. Much like there's FollowerSet which works
separately from Tickets, and will allow to have follower sets for projects,
users, etc. too.
Inboxes are made independent from Person users because I'm going to give
Projects inboxes too.
There used to be project roles and repo roles, and they were separate. A while
ago I merged them, and there has been a single role system, used with both
repos and projects. However the table names were still "ProjectRole" and things
like that. This patch renames some tables to just refer to a "Role" because
there's only one kind of role system.
A thing still missing there is that it sets empty audience for comments on
remote tickets, but that's fine because dev.angeley.es doesn't have such
comments in the database.
I added a migration that creates an ugly fake OutboxItem for messages that
don't have one. I'll try to turn it into a real one. And then very possibly
remove the whole ugly migration, replacing it with addFielfRefRequiredEmpty,
which should work for empty instances.
- Allow client to specify recipients that don't need to be delivered to
- When fetching recipient, recognize collections and don't try to deliver to
them
- Remember collections in DB, and use that to skip HTTP delivery
Worker is enough and seems much simpler. forkHandler does stuff with
forkResourceT and more stuff that I don't exactly understand and which may
involve more resource allocation. I guess forkWorker would generally be the
preferred approach, and there are bugs with delivery leading to sudden
CPU/memory peaks forcing me to kill the process. Maybe not related, just
mentioning it ^_^
In the new inbox forwarding scheme, we use an additional special HTTP signature
to indicate that we allow or expect forwarding, and to allow that forwarding to
later be verified. When delivering a comment on a remote ticket, we'd like the
project to do inbox forwarding. Based on the URI alone, it's impossible to tell
which recipient is the project, and I guess there are various tricks we could
use here, but for now a very simple solution is used: Enable forwarding for all
remote recipients whose host is the same as the ticket's host.
Until now, there were some simple host checks when verifying the HTTP sig,
meant to forbid hosts that are IP addresses, local hosts, and maybe other weird
cases. These checks moved to Network.FedURI, so now FedURIs in general aren't
allowed to have such hosts. The host type is still `Text` though, for now.
This patch does a small simple change, however at the cost of the request body
not being available for display in the latest activity list, unless processing
succeeds. I'll fix this situation in a separate patch.
It runs checks against all the relevant tables, but ultimately just inserts the
activity into the recipient's inbox and nothing more, leaving the RemoteMessage
creation and inbox forwarding to the project inbox handler.
Inbox post is disabled but in the next patches I'll code and integrate a fixed
complete one, hopefully finally getting ticket comment federation ready for
testing.
I'm making this change because if an actor receives an activity due to being
addressed in bto, ot bcc, or being listed in some remote collection, the server
doesn't have a way to tell which actor(s) are the intended recipients, without
having an individual inbox URL for each actor. I could use a different hack for
this, but it wouldn't be compatible with other AP servers (unless the whole
fediverse agrees on a method).
I wasn't using sharedInbox anyway, and it's an optimization either way.
Each ticket has a single discussion ID, and each ticket has a unique one, so,
given an inner join of tickets and discussions, I think there should be exactly
1 way select a (ticket, discussion) pair given any of these.
But for some reason, PostgreSQL started complaining. Not sure what changed.
Anyway, for now, I switched the groupBy from discussion.id to ticket.id, which
is essentially the same, but for some reason makes PostgreSQL happy. It can't
tell that given a discussion ID, there's exactly 1 way to choose the ticket. Or
something like that. I wonder if I messed up something in DB migrations.
Before this patch, the shared fetch used plain insert, because it relied on
being the only place in the codebase where new RemoteActors get inserted. I was
hoping for that to be the case, but while I tweak things and handle fetching
URIs that can be an actor or a public key (for which ActorFetchShare isn't
sufficient without some smart modification), I'd like concurrent insertions to
be safe, without getting in the way of ActorFetchShare.
With this patch, it now uses insertBy', which doesn't mind concurrent
insertions.
I wrote a function handleOutboxNote that's supposed to do the whole outbox POST
handler process. There's an outbox item table in the DB now, I adapted things
in various source files. Ticket comment federation work is still in progress.
The custom module provides a parametric wrapper, allowing any specific
FromJSON/ToJSON instance to be used. It's a standalone module though, and not a
wrapper of persistent-postgresql, because persistent-postgresql uses aeson
Value and it prevents using toEncoding to get from the value directly to a
string.
* Adapt DB related code to return the InstanceId and RemoteSharerId
* Previously, when fetching a known shared key, we were running a DB
check/update for the shared usage record. I noticed - and hopefully I
correctly noticed - that this check already runs when we discover the keyId
points to a shared key we already know. So, after successful sig
verification, there's no need to run the check again. So I removed it.
- Exclude hosts without periods, so things like localhost and IPv6 are rejected
- Exclude hosts without letters, so things like IPv4 are rejected
- Exclude the instance's own host, just in case somehow some fake activity
slips in and gets approved, maybe even accidentally when delivered by another
server
Before, things worked like this:
* Only signatures of Ed25519 keys could be verified
* Key encoding placed the plain binary Ed25519 key in the PEM, instead of the
key's ASN1 encoding
With this patch it now works like this:
* Ed25519 signatures are supported as before
* RSA keys are now supported too, assuming RSA-SHA256 signatures
* Both Ed25519 and RSA keys are encoded and decoded using actual PEM with ASN1
