Testing Fediverse Servers

Are there any ActivityPub testing tools? — Fediverse developer

It is a simple question, but the answer is not. There are several styles or methodologies for testing a Fediverse (ActivityPub) server, and they differ in various dimensions: how much planning is required, is custom test-related coding required, how much the tester can observe and steer, and how much of the work is automated. Because each strategy involves different tradeoffs, none of the techniques replaces the others. Often, a server is tested with more than one.

What's usually needed are Fediverse Protocol test tools rather than tools to test ActivityPub itself. Most of what's required to interoperate in the Fediverse is not defined by the ActivityPub Recommendation. Furthermore, this article only addresses federated server testing. I have discussed ActivityPub client testing elsewhere.

Manual Testing

Manual testing involves the developer running testing tools to explore server behavior and then interpreting the results to determine if the behavior is correct or not. The effectiveness of this methodology depends on the developer's understanding of the expected behavior.

Several tools exist or are being developed for this purpose.

ToolDescription
BrowserPubRenders the object behind a URI or a handle, so a broken actor document reads as a document rather than as a failed assertion.
Fedify debuggerRecords the activities an application sends and receives as a trace, including the signature check on each one.
DrFedFrom the Fedify maintainers, under development at the time this article was written and not yet released: object lookup, an inbox monitor, failure diagnosis that names the stage that broke, WebFinger and NodeInfo testers, a JSON-LD toolkit that compacts, expands, and compares how implementations read the same document, an HTTP signatures debugger, and an activity builder in one platform.
ActivityPub.AcademyA modified Mastodon instance that displayed activities as they passed. It is offline, and the style it served continues in the inspector tools. At the time of this writing, it's currently not clear to me if this tool is still being developed or will be replaced by DrFed.
ASML ActivityPub FuzzerReplays document shapes collected by the Fediverse Schema Observatory, which has observed many projects and software versions, into a local server

Ad-hoc manual testing

In this manual testing style, a developer explores server behavior with little or no preplanning or structure to the testing activities. Sometimes this style of testing is used as a "smoke test" (does the software work at all) prior to more advanced testing.

Ad-hoc testing has good observability when the server is local. The tester can stop a delivery in a debugger, read a queue, and query storage after a POST to see what the server kept. The cost is repeatability: a failure found this way is easy to see and hard to repeat, nothing accumulates between sessions, and nothing runs again next week to check that a fix held or that a new feature broke existing code (a behavior "regression").

Manual testing with a plan

A test plan turns exploration into a procedure. Scenarios are written down with their expected outcomes. The tester works through them by hand, and the results are recorded as pass or fail. The plan can be run again after a change, handed to another tester, or attached to a release.

It does not use automation. However, the repeatability is only as strong as the discipline of whoever runs it, and a plan that doesn't track the server's development goes stale quietly.

The original activitypub.rocks server-side suite was a questionnaire: it asked developers whether they implemented a feature and recorded the answer. However, self-report measured belief rather than behavior. The suite is not maintained and hasn't been updated for years.

The go-fed/testsuite was partially automated as a hosted web application. However, it's considered alpha stage software and has not been maintained. The tested server's details are re-entered on every run, individual tests cannot be re-run, and interactive steps wait on browser reloads. The automation covers the presentation of results, not the execution of tests.

Unit testing

A unit test exercises one component in isolation, in process: a function, or a message handler, with the rest of the system simulated. The definition of a component can vary from the function or object-level to small clusters of interacting objects.

A test that parses a document, validates a signature, or computes delivery targets without starting the server is a unit test. When a test starts requiring network connections, the test has usually crossed into integration testing (discussed below).

The protocol-specific helpers are few and recent. activitypub-nock installs interceptors that emulate a remote server: actors, collections, WebFinger, inboxes, and HTTP signatures, so delivery code runs without a peer. Fedify's testing package records the activities a handler sends and provides a mock queue for background delivery. Most ActivityPub code is still tested with ordinary framework tests over captured fixtures.

ToolLanguageWhat it substitutes for
activitypub-nockJavaScriptIntercepts HTTP with nock and answers as a remote server: actor, collection, WebFinger, inbox, and signature responses.
@fedify/testingTypeScriptThe federation layer, inside the test process: a test Federation that records the activities a handler sends and holds queued delivery open for inspection.

An effective unit test is fast and precise. Ideally, a server can have thousands of unit tests and run them in seconds. When a test fails, it is relatively easy to diagnose the problem because of the limited scope of the test. With integration testing (or full-server manual testing), the source of a failure is more difficult to pinpoint.

Integration testing

Integration testing runs the full server, typically through its external interfaces. This article discusses three variants of this methodology and the second party in the test tells them apart: a fixed set of expectations (implemented as a test fixture), a second copy of the same implementation, or another implementation entirely.

Baseline Interoperability Testing

A baseline suite uses the ActivityPub Recommendation and common Fediverse behaviors to define what a conforming server does. It then measures each server against that statement. The theory is that, if the suite represents the interoperability requirements properly, each server that implements the required behavior should interoperate with each other.

The output is a report on one server against one fixed set of expectations. The peer in every test is simulated inside the framework by a test fixture. A common approach is to build a server-specific test suite. This duplicates work because most ActivityPub tests are very similar for every server.

The activitypub-testsuite is an example of a reusable test suite. It defines an abstract interface for a Fediverse server that allows full introspection and control of test inputs and server responses. This allows the test suite to not only test server behavior for known good input ("sunny day scenarios"), but also for invalid input. The abstract interface implementations are server-specific, but the tests are not. Two components make that work: the tests, and a Server Abstraction Layer (SAL), the analogue of a hardware abstraction layer, that normalizes access to the server under test. The SAL is per-server code: it creates and configures actors, posts to the inbox and the outbox, reads collections, simulates remote servers, and resets server state. For servers that implement the standard client API, much of the SAL is reused from the library; for servers with proprietary APIs, it is bespoke code, and writing it is the non-trivial cost of the strategy. However, once the SAL code is written, the benefits accrue for the duration of the software project.

The framework is designed around two considerations. The first is how the suite communicates with the server: over the full network stack, with the server in a subprocess, which works for any language and suits servers that start quickly; or through simulated network communication in process, which is faster and needs more SAL code.

The second is how server state is reset between tests: an in-memory store, an external store that is cleared, or a test API added to the server are some options. Reset is the mechanism of test isolation: each test starts from a clean server, so one result cannot contaminate the next. It also prevents a test from having side-effects that change the behavior of the next run of that test. A server that depends on several external processes, such as a database, a cache, a task queue, and full-text indexing, typically needs a container set or a VM, and a reset mechanism for that scenario.

Most of the ActivityPub Recommendation describes either fully optional behavior or conditional requirements that are, in effect, optional if the conditions don't hold. The activitypub-testsuite software uses a capability profile to automatically skip unsupported server behavior (with an optional explanation) and define expected values (like HTTP status codes). The framework can generate a comprehensive report describing tests that have passed, failed or been skipped.

Although the test framework is written in Python, the framework does not require the server under test to be written in Python. It's been used to test servers written in Javascript, Go, and Rust for example.

The advantage of this testing approach is that the full server functionality is tested. The tests are automated so they can be re-run after the software is modified to prevent software behavior regressions. The disadvantage is that the tests run relatively slowly and it can be complicated to implement test isolation depending on the design of the server.

Multi-server Interoperability testing

Multi-server automated interoperability testing runs servers against each other. This contrasts with the baseline interoperability testing in that for N pair-wise servers, full testing will require N*(N-1) test permutations. This can also be extended to more than pairs, which increases the number of permutations dramatically.

An example of this kind of test suite is FediTest. The framework assembles a constellation of nodes, drives each through a server-specific adapter, and checks that an activity created on one produces the expected effect on another. Its Fediverse suite covers WebFinger conformance and server-to-server scenarios across one and two hops. The benefit is realism, but the disadvantage, compared to baseline testing, is observability and control. A peer is a black box: its queue, its logs, and its errors are invisible, and some failure cases cannot be induced at all. A baseline suite can reset state, inspect what the server stored, and provoke a failure on demand. That loss is the central difficulty of federation testing, not a defect in the tool.

Implementation Self-Testing

Self-federation runs two instances of the same software, on separate ports or in separate containers, and checks that an activity created in one appears in the other. This approach is typically also manual rather than automated.

This type of testing can also include server-specific user interfaces or APIs. Those control and observability surfaces become a part of the test infrastructure.

What the strategy proves is that a server agrees with itself. An implementation that fails when run against itself will probably fail against a peer. The converse does not follow, because two copies of one implementation share the same dialect, the same assumptions, and the same bugs. It is the manual form of peer testing. Therefore, it can also be categorized as an ad-hoc or planned testing methodology, as described earlier.

Technical challenges

Although not methodology per se, some of the most difficult aspects of integration testing are technical. How will the server under test be locally deployed (and repeatedly re-deployed). How will test isolation be implemented. And how will required features like TLS/SSL certificates be supported?

Each of these challenges could be a separate blog article, but here's an incomplete list of ideas to get started.

Common server deployment techniques

  • Containers and container stacks
  • Virtual machines
  • Bare metal with server management scripts

Techniques for resetting server state

  • Container or VM restarts
  • Server admin APIs
  • Seed data files
  • Direct database updates

Techniques for SSL requirements (assuming valid certificates are required)

  • Wildcard subdomain certificate and a local proxy (LAN only)
  • Cloudflare Tunnels or similar with TLS termination (exposed to Internet)
  • Disable by server configuration (if allowed)

Specification challenges

No general strategy can assert more than the specification and common Fediverse practice permits. However, the ActivityPub specification permits a great deal. Requirements are few, optional parts are many, and several requirements are open to interpretation in ways that decide what a test can assert.

For example, the inbox stream is defined to contain all activities received by the actor, and the specification says the server SHOULD filter that content according to the requester's permission. A SHOULD is not a MUST, and both degenerate readings are conformant: an implementation that filters nothing, showing private activities to anyone who asks, and an implementation that shows the inbox to no one, not even its owner. Mastodon implements the second, with every inbox GET answering 404, and a family of Mastodon-inspired servers behaves the same way. A test that posts an activity and then reads the inbox collection has nothing to assert against a server where the collection endpoint answers 404 (or 401 or 403).

The same looseness runs elsewhere. Most HTTP status codes are left open, and an implementer can answer an error with a success code for reasons of security or privacy. Authorization requirements are too vague to test mechanically; in the absence of a definition, a test suite must make many assumptions based on common Fediverse practices. For example, it may assume visibility from the as:Public audience, the recipient fields, and object attribution.

An activity that references its object rather than embedding it, or partially embeds an object, is within the specification. An implementation that only processes embedded objects fails against a peer that sends references. Most servers assume one object per activity and one type per object.

Asynchronous processing removes feedback: a server may accept a POST, hand the activity to a work queue, and reject it later with no signal to the publisher, so a test (or human) must poll a collection until the effect appears or a timeout decides the activity was dropped.

Combining methodologies

The methodologies can complement each other, and a Fediverse server project uses more than one methodology. Automated unit tests, if they exist, tend to be relatively numerous, run quickly, and are good at locating the source of issues. However, they may not cover higher-level server behavior that arises from the interaction of several or many areas of the implementation.

Automated integration tests address this limitation, but are much slower. They are also less precise tests, and, typically, fewer of them are required to cover the server requirements.

Finally, there is manual testing. It's the most labor intensive form of testing. However, it may be necessary to diagnose issues that are not yet covered by unit or integration tests. A good practice is to add tests in these cases so manual testing is not required in the future for the same kind of problem.

These types of testing can be viewed as a pyramid. The automated unit tests provide the foundation. They are run frequently to verify that most server behavior is correct. A good unit test suite allows the code to be modified in significant ways with less fear of breaking something. They are often run at least on every code commit, if not more frequently.

As tests accumulate (and your testing improves), you gain confidence in the behavior of the system. As you refine the design, more and more changes become possible. — Kent Beck, Test-Driven Development: By Example.

But the biggest benefit isn't about merely avoiding production bugs, it's about the confidence that you get to make changes to the system. — Martin Fowler, Self Testing Code

The integration tests will usually be slower (maybe minutes instead of seconds). They may also run asynchronously in a Continuous Integration (CI) pipeline. It's a slower feedback loop, but still very valuable. The need for manual testing should be relatively rare and cover the other cases.

The following describes various methodologies, their purpose, and supporting tools.

MethodologyQuestion it answersTools
Manual, ad hocWhat the server is sending, and what one activity doesBrowserPub, Fedify debugger, DrFed, ActivityPub.Academy, ASML ActivityPub Fuzzer
Manual, with a test planWhether the server passes a written set of scenariosactivitypub.rocks questionnaire, go-fed/testsuite
UnitWhether a handler handles a given documentactivitypub-nock, Fedify testing
Integration, against a baselineWhether the server meets a fixed set of expectationsactivitypub-testsuite
Integration, same implementationWhether two copies of one server agreeTwo containers
Integration, many implementationsWhether servers interoperateFediTest

Being a Good Citizen

The W3C wiki has suggestions for testing ActivityPub software. Some of these suggestions are intended to protect the Fediverse network. For example,

  • Use test accounts rather than real accounts
  • Avoid bulk activity, because a flood of created notes can overwhelm other servers
  • Treat a server as a production system once its objects are stored across the network.

In general, it's not a good practice to use the live Fediverse network as a testbed.

In Summary

Hopefully, it's clear why the question that opened this article has no single answer. Testing a Fediverse server has no single shape.

Manual exploration, planned scenarios, unit tests, and integration testing each trade planning, coding effort, observability, and automation differently, and each reveals a different class of defect. Which tradeoffs are worth paying depends on the behavior a project needs to confirm. A developer asking about ActivityPub testing on the Fediverse gets a better answer by describing that behavior.

The good practice behind any specific tool is to define the expected behavior, and assert it automatically. The result is that we, both developers and users, benefit from less stress and a high quality Fediverse.

Leave a Comment

Mastodon