There's recently been much discussion on the Fediverse about ActivityPub (AP) testing and associated test suites. The original ActivityPub test suite, originally hosted at activitypub.rocks, is frequently mentioned. The test suite hasn't been running for years and it's been difficult to find a maintainer because of the implementation programming language, Guile. Because of that situation, few people have seen it run or know about its capabilities.
Several developers have tried to run the Guile software, but there are issues with missing libraries and there was nobody with the time, expertise, or motivation to resolve the issues. I'm not a Guile programmer, but I programmed in LISP for years (many years ago) so I could at least read the code. I decided to port the implementation to Python for several reasons. It would be a good demonstration of the test suite functionality which I believe is not sufficient for ActivityPub testing purposes. It would make the code more accessible to a larger group of Python programmers.
There's a good overview of the test suite at SocialHub. It supports testing AP clients, the server side of the Client-to-Server (C2S) protocol, and the Server-to-Server (S2S) protocols used for federation. The original author describes some of the quirks in that forum post:
Christine Lemmer Webber, SocialHub
- It was done in a rush to meet the standards deadline. The main purpose was to collect implementation reports, and given the rush some things are funny about it.
- Technically it’s two test suites. Scratch that, it’s a test suite and a questionaire:
- There’s a test suite for the client to server protocol, which really does resemble a test suite; it makes many requests against your server and sees if they work right. (It’s buggy, though.)
- There’s a mostly-questionaire (but it runs a couple tests, I think) for the server-to-server protocol. Why? Because a) performing a test suite on an asynchronous protocol is already hard enough (when does the test complete? it can be done with timeouts, but it’s gross) and b) many of the requirements required that some authentication or authorization take place, but due to process reasons we couldn’t specify what that was, and c) at that point we already had an unusual amount of interop, so we decided that if multiple implementations could federate on a feature and could confirm that in the questionaire, we could consider that sufficient. (Even weirder, some of the questions would involve the server doing something to your server, but then still having a question for the user to confirm if the thing worked.)
- A proper test suite wouldn’t work like this. It would be fully automated.
The main screen is shown in the following screenshot. This is where you select which tests you want to run.
The client and S2S "tests" are really just questionnaires where developers can self-report the capabilities of their software. The C2S tests are automated but most servers don't implement C2S for client access.
At the end of the test session, you can download a JSON report of the results of the tests. Originally, these could be submitted as implementation reports. For now, it's just a demonstration.
At the moment, I'm hosting the demonstration of the test suite at https://aptestsuite.stevebate.dev. I don't know how long I'll run it there, but it's easy to self-host on a local network (no SSL or public address required). The source code is here with instructions about how to run it locally.