Consider nemeses used in Foundation DB

[ligurio]

FoundationDB nemeses:

• Laggy communication between nodes
• Network routing errors
• Swizzle clogging*
• Simulated software bugs**
• Upgrades from old server/file versions
• Incomplete writes to disk (see Implement filesystem fault-injections #75)
• Corrupted writes to disk (see Implement filesystem fault-injections #75)
• Disk drive runs out of space (see Implement filesystem fault-injections #75)
• Single node in a cluster terminates (done in Add Kill nemesis #16)
• Network partitions between nodes (done in Add partition nemesis #17)
• Processes freeze for random durations (done in Add pause nemesis #21)

"For a while, there was an informal competition within the engineering team to design failures that found the toughest bugs and issues the most easily. After a period of one-upsmanship, the reigning champion is called “swizzle-clogging”. To swizzle-clog, you first pick a random subset of nodes in the cluster. Then, you “clog” (stop) each of their network connections one by one over a few seconds. Finally, you unclog them in a random order, again one by one, until they are all up. This pattern seems to be particularly good at finding deep issues that only happen in the rarest real-world cases." https://apple.github.io/foundationdb/testing.html