- When pushing to a repo, a Push activity is now automatically published
- The 'actor' is now the repo, and 'attributedTo' specifies the person who
pushed
- No need for 'context' in the Push anymore, since it's always the 'actor'
- 'target' now specifies the branch as a Branch object rather than URI (since
Vervis doesn't keep AS2 objects for branches anymore)
- I deleted 'pushCommitsC' (from Vervis.API) because the code for preparing and
pushing an activity is so simple with the new delivery API, doesn't need a
dedicated pushCommitsC function
- The generated Push activity does generate an HTML summary, unlike all other
generated activities (in which I removed the summary generating code); I'm
still unsure whether to bring back those summaries (extra code to write, for
a problematic feature that may become useless when the new UI comes)
When looking up a specfic actor record for a given ActorId, you're pretty much
guaranteed to find the actor if it exists, because there's 1 function in the
codebase that handles this. Whenever a new actor type is added, which is a rare
event, that function gets updated.
But when mass-selecting actors using Esqueleto? Then, you need to LeftOuterJoin
by yourself on each actor type. This is both ugly and error prone, because all
those places in the codebase need to be updated when adding an actor type. The
only downside is that it means O(n) DB queries instead of O(1).
Perhaps there's some elegant way to "add" the specific-actor Joins to a given
Esqueleto query. Something to do some other time, as an optimization, if the
need arises.
Per-actor keys are now fully supported in Vervis! Caveats:
- The HTTP Signature keys produced by Vervis are Ed25519 keys; software that
expects only RSA keys will fail here
- Like instance keys, per-actor keys are currently served in separate
documents, not embedded in the actor document; so software that expects
embedded keys will fail here
New iteration of the ActivityPub delivery implementation and interface.
Advantages over previous interface:
* When sending a ByteString body, the sender is explicitly passed as a
parameter instead of JSON-parsing it out of the ByteString
* Clear 3 operations provided: Send, Resend and Forward
* Support for per-actor keys
* Actor-type-specific functions (e.g. deliverRemoteDB_D) removed
* Only the most high-level API is exposed to Activity handler code, making
handler code more concise and clear
Also added in this patch:
* Foundation for per-actor key support
* 1 key per actor allowed in DB
* Disabled C2S and S2S handlers now un-exported for clarity
* Audience and capability parsing automatically done for all C2S handlers
* Audience and activity composition automatically done for Vervis.Client
builder functions
Caveats:
* Actor documents still don't link to their per-actor keys; that should be the
last piece to complete per-actor key support
* No moderation and anti-spam tools yet
* Delivery API doesn't yet have good integration of persistence layer, e.g.
activity is separately encoded into bytestring for DB and for HTTP; this will
be improved in the next iteration
* Periodic delivery now done in 3 separate steps, running sequentially; it
simplifies the code, but may be changed for efficiency/robustness in the next
iterations
* Periodic delivery collects per-actor keys in a
1-DB-transaction-for-each-delivery fashion, rather than grabbing them in the
big Esqueleto query (or keeping the signed output in the DB; this isn't done
currently to allow for smooth actor key renewal)
* No support yet in the API for delivery where the actor key has already been
fetched, rather than doing a DB transaction to grab it; such support would be
just an optimization, so it's low-priority, but will be added in later
iterations
It causes first login after verification to redirect to registration page,
which is weird, and the default Yesod falue is False, so, switching back to
False
Repo/Loom/Deck created one in their migrations, but CollabTopicAccept isn't
itself a topic, it's just a helper, so it needs to *find* the relevant
CollabTopicLocal rather than create a new one.
Both Git and Darcs are supported
- Darcs implementation applies right on the bare repo, I haven't tested to make
sure it works right (federated MR demo is going to be only for Git)
- Git implementation clones to temporary repo, runs `git am` on it to apply,
then pushes to the real bare repo (because `git am` doesn't work on bare
repos; I haven't tested yet to see how it handles conflicts; cloning and
pushing should be efficient since the refs are just hardlinked rather than
copied)
For now it's implemented only for Git:
If tracker is a local loom, and a (local or remote) origin repo is specified,
but no patches are provided, then generate them ourselves!
* Clone the (local) target repo
* Add the (local or remote) origin repo as a git remote
* Make sure target branch is an ancestor of the origin branch
* Generate patches for the commits that origin adds on top of target
* Insert them into our DB
In offerTicketC it can be used for fetching commits from the remote origin
repo, by knowing its clone URI
Only HTTP clone URIs are supported for now, because it's enough for finishing
the federated MR implementation. Apparently user@host:path isn't a valid URI
and I'll later add a parser for that
What's missing:
- Match patch VCS, origin repo VCS and target repo VCS
- Hook into postPersonInboxR
- If only origin is provided, generate patches / otherwise somehow remember the
commits proposed
RepoSourceR, for a repo that doesn't have a loom, lists looms that want to
serve that repo with buttons for bidirectionally linking the repo to a loom
Once linked, the repo navbar has a Patches/MRs link pointing to the LoomClothsR
of the linked Loom
Person inbox handler:
- Invite: Parse and insert to inbox
- Grant: Parse and insert to inbox
Repo/Deck/Loom inbox handler:
- Invite: Parse and remember as Collab record in DB for later
- Accept: Send a Grant (and remember it in DB)
Along with inviteC and acceptC, the Invite->Accept->Grant flow is now fully
federated, yay!
What's missing is UI for actually using it. Coming soon.
Giving access now starts with an Invite activity, followed by Accept from the
Invite's recipient. Finally, the resource sends a Grant, which is the actual
OCap.
I was going to link the matching Accept tables to them, but then switched to
the Invite-Accept-Grant model and going to implement it in the next patches. So
I'm committing these new tables just in case I decide to revert to the current
model.
The steps are:
- Parse activity ID and match with the authenticated sender
- For local activity (we got via forwarding), find in DB
- For remote activity, cache in DB
- Insert activity to recipient's inbox
What's not there yet is the actual logic of handling specific activities.
Previously there was just CollabTopicAccept, which worked only for local topics
but pretended to apply to both, due to directly pointing to Collab, thus
possible to insert rows even if there's a CollabTopicRemote.
The new situation is a new CollabTopicLocal table to which the local topic
things point, thus keeping the local and remote data separate and difficult to
confuse.
Since that table being added, a Collab without a CollabTopicAccept is
considered an open proposal/invitation waiting for completion. So inserting the
CollabTopicAccept is now required for making the Grant valid and active.
This is such a huge patch, it's probably impossible to tell what it does by
looking at the code. One thing is clear: It changes *everything* :P so here's
an overview:
- There are now 5 types of actors, each having its own top-level route
- So projects, repos, etc. are no longer "under" sharers
- Actor routes are now based on their KeyHashid, there are no "idents" anymore,
i.e. URLs look random and don't contain user or repo names
- No sharers anymore; people and groups are distinct entities not sharing a
common namespace or anything like that
- Project has been renamed to Deck and it simply means a ticket tracker; repos
are no longer "under" projects
- In addition to Person, Group, Repo and Deck, there's a new actor type Loom,
which is a patch tracker; i.e. Repo actors don't manage MRs anymore
- All C2S and S2S is temporarily disabled, because huge changes to the whole
code are required and I'll do them gradually in the next patches
- Since form-based actions are implemented using C2S, they're disabled as well,
so Vervis is now essentially read-only
- Some views have been temporarily removed, e.g. repo history and commit view
- A huge set of DB migrations has been added to adapt the DB to these changes;
I haven't tested them yet on a read DB so there may be errors there; I'll fix
them in the next patches if I find any (probably going to test on the main
instance where Vervis itself is hosted...)
- Some modules got tech upgrades, e.g. LocalActor became a higher-kinded type
and a similar pattern is probably relevant for several other types
- There's an 'Actor' entity in the DB schema now, and all 5 actor types use it
for common things like inbox and outbox
- Although inbox and outbox are used only by Actor, so essentially could be
removed, I haven't removed them; that's because I wonder if at some point
users can have a tree of inboxes much like in email; I don't have an excuse
for Outbox, but anyway, leaving them as is for now
- Workflows, roles and collaborators are partially removed/unused until I
figure out a sane federated way to provide these features
- Since repo routes don't contain a "sharer" anymore, SSH URIs are now simpler,
they already look like user@host/repo regardless of who "controls" that repo
* Publish a Create activity and respond with a Grant activity
* postProjectsR reuses that code
* No automatic following at the moment
* Workflow and role specified in new project form are ignored for now
* Can't create tracker under a group yet, just under the user
Ugh, that module is such a horrible mess... I hope to turn it soon into
something sane. Is there some generic non-clumsy way restructure the AP
parser/encoder API?
For now, making these ugly changes to support the represenation of
Create {TicketTracker}, which I'm about to implement.
I realized I never intend to leave patterns (e.g. pattern matching in a 'case'
clause) incomplete, i.e. some cases left missing. When I do that it means I
forgot, and I'd like GHC to highlight it by raising an error instead of just
warning. Vervis has lots of warnings so it's hard to detect among them.
I suppose in other kinds of software people sometimes leave incomplete patterns
intentionally / are okay with a runtime exception being thrown? In a web
application, I definitely want to handle all cases, and be in control of how
errors are handled and displayed in UI.
This will allow to use this representation for Offer and Create activities.
When creating a new MR, the inner Offer's 'object' is a 'Patch' object. When
serving an existing hosted MR, the inner Offer's 'object' is just a URI
pointing to the patch.
If sharer receives Accept on an Offer/Dep where the sharer hosts the child
ticket, it records a RemoteTicketDependency and runs inbox forwarding to ticket
followers. But this relies on a TicketDependencyOffer record already existing.
I'll take care of that in the next patches.
sharerAcceptF and sharerRejectF now use the insertToInbox from
Vervis.Federation.Util instead of their own copies of it, which were identical
anyway. Perhaps gradually all the inbox insertion in all S2S handlers will
switch to using that function.
To be honest, this is a huge patch that changes tons of stuff and probably
should have been broken up into small changes. But I already had the codebase
not building, so... just did all of this at once :P
Basically this patch does the following:
- DB migrations for ticket dependency related tables, e.g. allowing a remote
author and a remote child
- Allowing S2S handlers to provide an async continued processing function,
which is executed and the result then added to the debug page
- Most UI and functionality related to ticket deps is disabled, new
implementation being added gradually via ActivityPub
- Improvements to AP tools, e.g. allow to specify multiple hosts for approved
forwarding when sending out an activity, and allow to specify audience of
software-authored activities using a convenient human-friendly structure
- Implementation of S2S sharerOfferDepF which creates a dependency under a
sharer-hosted ticket/patch and sends back an Accept
- Sharer-patch was already working due to the shared DB tabled
- Repo-patch support added
- Fixed a bug in following project-tickets: It allowed to follow a
sharer-ticket by sending the Follow to the project. Now, the project allows
to follow only the project-tickets, and refuses to handle a sharer-ticket.
Also fixed a bug in which trying to follow a ticket with nonexistent
ltkhid/talkhid would result with 404 as if the actor inbox is nonexistent. Now,
there's a friendly message reported.
The examples in the security vocabulary's spec use "Key" but the JSON-LD
context doesn't define that term. From now on, just in case, recognizing both
"Key" and "CryptographicKey" as indication that the object is a key.
zPlus, thanks for finding this bug!
This is the first step preparing for patches and merge requests.
The work-item aspect of MRs will reuse the Ticket related tables, except MRs
will live under repos. So, the context of tickets will no longer be just
projects, but will also be repos.
So, TicketProjectLocal turns into TicketContextLocal, and there are 2 new
tables that refer to it: TicketProjectLocal and TicketRepoLocal. Tickets will
have the former, MRs will have the latter.
Removed its usage in pseudo-client when publishing a comment, and removed it
from inbox forwarding when handling a remote comment.
Very possibly, the ticket team collection will be entirely removed. For now
leaving it there as-is. Just not using it for addressing in activities.
The implementation felt quite weird, had to add an extra field to Fetched and
to VerifKeyDetail. Should probably figure out the whole mess in that code, have
something clean there. Easily add fields. Easily and safely re-fetch an actor
or key.
Now it's much clearer when looking at the code, that these routes are about
project-hosted tickets, and it's easier to see where the author-hosted
equivalents are missing.
IMPORTANT: Since a lot of ticket code still doesn't use TicketUnderProject,
creating tickets now appears to be failing. Usage of this patch as is, is at
your own risk ^_^ the next patches will update the ticket handlers to fix this
problem.
Delivery of an activity into local inboxes is being done using custom local
functions. Each C2S or S2S handler has its own specific variant for this.
As part of the ongoing refactoring and evolution of the federation code, I
implemented a general-purpose local delivery function: It takes a
LocalRecipientSet and simply delivers to everyone, no handler-specific
assumptions or limitations.
To limit the recipient set according to handler specific rules, just
filter/adapt/edit it before passing to the delivery function.
The function isn't exported yet, but the existing 'deliverLocal' that delivers
only to actors and to author's followers is now implemented via the new
general-purpose function. I hope that's a step towards doing all the local
delivery using this one function, simplifying the complicated federation
code.
I'll use this for C2S to allow client to state who the tracker actor is. It's
still possible to do without it, by HTTP GETing the ticket's context and
checking whether we got an actor, or a non-actor with ticketsTrackedBy. Tbh I'm
adding createTarget simply because it's easier for coding, no need for a custom
variant of actor fetching :P
This allows the context to be specified even when replies/followers/deps/etc.
aren't. This is needed for Create-ing a Ticket. Also, it allows a ticket's
context to be on a different host than where it's hosted, which is also needed
for the Create flow.
A row in this table will be required for local-project-local-author tickets
hosted under the project, and non-existence of a row will be required for such
tickets hosted by the author. So I'll need to CAREFULLY update all the ticket
route handler code and all the ticket related AP code. The latter includes C2S
and S2S for tickets, ticket deps, ticket discussion... everything that is under
tickets.
That's because with the Create flow added, the activity that reports a ticket
can be either Create or Offer, maybe later Announce too.
The old TAL unique name mentioned in the migration has what may look like a
typo, "Locale" instead of "Local". That's because I made a typo in migration
115, and now needed to specify the typoed name I used then. I verified in dev
DB and on dev.angeley.es DB that the typo is reflected in the PostgreSQL
database side and fixed by the new migrations.
Everywhere Ticket is found, a matching LocalTicket is now expected to be found
too. Ticket doesn't point at LocalTicket because there will be remote cached
tickets too. Also, ticket URLs are going to switch the khid from Ticket to
LocalTicket (much like it's already the case for MessageR).
This is a step preparing for the Create flow for tickets. Each Ticket now gets
a matching LocalTicket that points to it. But otherwise the LocalTicket isn't
in use yet.
- WorkflowField now has a color, it's a simple `Maybe Int` for now. Valid
values are only 1-4
- That color is used for displaying ticket class params a.k.a labels in ticket
list view
- Ticket list now also serves a paged OrderedCollection
I tried to use a single SQL query to grab the tickets along with their labels,
but couldn't figure out a way to aggregate tuples/rows into an array (it seems
only single values are supported in Esqueleto). Instead of doing manual SQL or
adding Esqueleto functions, I just switched from 1 query to O(n) queries: Each
ticket has its own query selecting its labels. I guess it's slower, but also,
ticket list is paged now with fixed page size so it's really O(1) ^_^
I'm not sure this will improve much, because the error messages come from
attoparsec, but at least the message text won't be constant, which was the
previous situation.
Before, Push activities were being ignored by all inboxes. I just forgot to add
code to handle them. Now, person inboxes accept them if they're about a
relevant repo (i.e. a repo of which the user is a remote follower; remote
collaboration would be relevant too, but it's not implemented yet).
Follows used to be added automatically, without a Follow activity sent by the
client. They aren't added automatically anymore, so there's no need for those
"manual" boolean fields.
Before this patch, if you ran more than 1 instance as the same OS user, they'd
use the same config file path and overwrite it and cause post hooks to have
errors due to wrong config being used.
In Darcs, any command can have a post hook (and a pre hook), and the hook
command can be set using a command-line option to the darcs command that you
run. So, in the Vervis SSH server, if we add a --posthook option when running
`darcs apply` to apply remotely received patches, we get a chance to process
the patch data much like in the git post-receive hook.
The setup this patch creates is similar to the git one: It writes a
_darcs/prefs/defaults file to all Darcs repos, and that defaults file sets the
posthook line for `darcs apply`. The posthook line simply executes the actual
hook program written in Haskell.
The current hook program is a one-liner that prints a line to stdout, so every
time you `darcs push` you can tell the hook got executed. The next step is to
implement the actual hook logic, by reading patch data from the environment
variable in which Darcs puts it.
This patch contains migrations that require that there are no follow records.
If you have any, the migration will (hopefully) fail and you'll need to
manually delete any follow records you have. In the next patch I'll try to add
automatic following on the pseudo-client side by running both e.g. createNoteC
and followC in the same POST request handler.
Here's how it works:
- When Vervis starts, it writes a config file and it writes post-receive hooks
into all the repos it manages
- When a git push is accepted, git runs the post-receive hook, which is a
trivial shell script that executes the actual Haskell program implementing
the hook logic
- The Haskell hook program generates a Push JSON object and HTTP POSTs it to
Vervis running on localhost
- Vervis currently responds with an error, the next step is to implement the
actual publishing of ForgeFed Push activities
Currently it's a paged Collection where the items are merely URIs. This could
be changed to have actual Commit objects as items; for that we need to examine
the whole thing with the LogEntry type and the Patch type and have an
AP-friendly log item representation, but without commit diffs.
FedURIs, until now, have been requiring HTTPS, and no port number, and DNS
internet domain names. This works just fine on the forge fediverse, but it
makes local dev builds much less useful.
This patch introduces URI types that have a type tag specifying one of 2 modes:
- `Dev`: Works with URIs like `http://localhost:3000/s/fr33`
- `Fed`: Works with URIs like `https://dev.community/s/fr33`
This should allow even to run multiple federating instances for development,
without needing TLS or reverse proxies or editing the hosts files or anything
like that.
