Testing asynchronous systems
I'm spending these days kicking off end-to-end tests. By end-to-end tests, I mean tests that interact with a set of IT systems as if the set was a big black box. For example, one test sends a REST API request to system A, waits until the data propagates through systems B and C to system D, and checks the result in system D via another REST API call. Now, how to implement the waiting part for this asynchronous test?
The first option is to wait for a fixed period of time, such as 10 seconds or 1 minute. But with the period being too long, it's cumbersome to develop such tests; I want fast feedback loops when developing. On the other hand, with the period being too short, the tests can fail; and I need tests to be stable to trust them.
Another option is to set up a service that provides visibility into the asynchronous request. So we can periodically poll the status of the request from tests. However, building such a service can be a mound of engineering work across multiple systems. Such a service could not be even feasible to build.
The third option is to write tests that check periodically that the expectation is met until a given timeout is reached. For example, we can check every second that the result in system D is present; and we can keep checking up until one minute. I find the third option to be the sweet spot: minimal engineering work, minimal waiting time, and fast feedback loops.