Explore adding a reproducibility test to rust test infrastructure.

[biabbas]

Fixes #75362
Trying to add a reproducibility check on rust infrastructure itself to detect reproducibility issues.
These were the issues encountered till now:

1. On ci only nightly build is supported because of an issue here: https://github.com/rust-lang/rust/blob/master/src/build_helper/src/git.rs#L143
2. Because this is a nightly build, llvm is built from sources. So space was an issue. This was resolved by deleting the copy of source and all build contents except for stage1 artifacts.

Questions:

1. The check takes about 5hours to complete for now. Is this sustainable for each pr?
2. Can we add this as a post merge workflow?
   This is also related to issue: Rust reproducibility issue - Finding the proper fix #134589

[rustbot]

r? @marcoieni

rustbot has assigned @marcoieni.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[biabbas]

@rustbot label -T-bootstrap

[biabbas]

@rustbot label +T-infra

[marcoieni]

2. space was an issue. This was resolved by deleting the copy of source and all build contents except for stage1 artifacts.

please write this as a comment in the code 👍

[marcoieni]

The PR looks good overall, the comments are great, thanks!
I left a few more minor comments.

I want to involve @Kobzol to ask him if he agrees with the overall strategy here. I.e.:

• do we want to add this workflow?
• does it solve the problem in the right way?

[marcoieni]

Suggested change

- reproducible

this is a leftover from the test, right?

[Kobzol]

While it would be great to have some reproducibility checks on CI, I don't think that the current approach used by this PR is the right one, for a few reasons.

• If we do any tests post-merge, it will be quite difficult to act upon them. We should IMO either do this on a best-effort basis out-of-tree (or only in PR CI), or make it actually blocking for merging PRs and run it on auto builds.
• When a reproducibility test fails, it can be quite difficult to understand what went wrong. We should first invest in at least some basic tooling that will display some useful form of a diff between the binary artifacts so that we can actually figure out what went wrong.
• A related aspect to that is reproducibility of the test itself. If the test will be implemented in a separate CI workflow, it will be impossible to run it locally. It would be great to have the option to run the test locally, likely either as a bootstrap test or a run-make test. Both would require some code changes and test infra improvements to make this possible, probably.
• I don't think that (re)building the compiler twice from scratch, even including (sccached) LLVM builds has to take 5 hours. Building the stage 2 compiler on x86_64-gnu, which runs on the free 4-core runner, take ~2000s, so doing that twice should take at most 1.5 hours, if we have proper caching configured. This is also why I'd like to do this in our normal CI workflow, not in a separate workflow that doesn't have any of our supporting infrastructure. I think that we can also start with something simpler, as was proposed in CI for deterministic / reproducible builds #75362, e.g. do the test only for stdlib as a start. Or, as an alternative, maybe we could do a stage 2 build, then move the sources to a different location, do a stage3 build, and just compare stage 2 and stage 3? There are likely several approaches that we can try.

I think that this is something that we should ideally discuss with t-infra first to gather consensus and come up with some plan, before starting with the implementation.