Skip to content

Enable multi node testing #98

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Totktonada opened this issue Nov 14, 2021 · 0 comments
Open

Enable multi node testing #98

Totktonada opened this issue Nov 14, 2021 · 0 comments
Labels
bug Something isn't working

Comments

@Totktonada
Copy link
Member

I tried to run multi node testing (jepsen-cluster and jepsen-cluster-txm workflows in tarantool) and found that it does not work.

There are a lot warnings of this kind:

WARN [2021-11-14 14:45:49,060] jepsen node 146.185.243.54 - jepsen.control Encountered error with conn [:control "146.185.243.54"]; reopening
java.lang.InterruptedException: sleep interrupted

That finally ends with:

CMake Error at cmake/atomic.cmake:46 (message):
  C atomics not supported

Which points me to tarantool/tarantool#2088 and, it seems, means that those retries somehow lead to miss of the git submodule update --init --recursive command and/or incomplete cmake <...> commands.

The code that builds tarantool is the same for single node and multi node testing, so my guess is that it is a synchronization problem in the ssh connector implementation. There were relevant fixes in recent Jepsen versions, so we can try to update it and look, whether the problem will gone. See #30.

Full logs and artifacts:

Full logs from successful (single node) testing:

Tarantool's commit on which I run CI and got those logs.


As I see from tarantool/tarantool#5736 multi node testing was not enabled to save machine resources. I think we should enable it anyway, maybe just run rarely. Otherwise we'll meet surprises like this one without understanding what actually occurs.

@Totktonada Totktonada added the bug Something isn't working label Nov 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant