yesod-static binaries differ from filesystem readdir order

[bmwiedemann]

Bug Reports

• Your OS name and version: openSUSE-Tumbleweed-20200629

• The versions of tools you're using (e.g. stack, yesod ghc): yesod-1.6.0.1, ghc-8.10.1

• Finally, if possible, please reproduce the error in a small script, or if necessary create a new Github repo with the smallest possible reproducing case: I would, if I knew Haskell.

While working on reproducible builds for openSUSE, I found that our git-annex package varied across builds and a discussion at
https://git-annex.branchable.com/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/ made me conclude that yesod-static causes its .o file to contain static files in random filesystem readdir order. Some (sort fs) in the right places might be needed.

See https://reproducible-builds.org/ for why this matters.

[snoyberg]

Thanks for the report. I think this is best solved at a lower level (the file-embed library), so that it solves the problem more broadly. I've pushed a commit (snoyberg/file-embed@a3c2430) that I believe solves this. Would you be able to test if this change addresses your issue before I make a release? If that's too difficult, let me know and I'll make a release of file-embed to Hackage instead.

[bmwiedemann]

I did a double-build with that patch but it seems there are still ordering issues left in git-annex-8.20200617/dist/build/git-annex/git-annex-tmp/Assistant/WebApp/Types.o

https://build.opensuse.org/package/show/home:bmwiedemann:branches:devel:languages:haskell/ghc-file-embed is my branch used for the semi-manual git-annex build.

[bmwiedemann]

output from filterdiff strings /path/to/builddir*/.../Types.o using my filterdiff
Types.o.strings.diff.txt

[snoyberg]

I should have asked in advance what APIs git-annex was using, my mistake. I think the newest push to this repo (084a3ce) should address the other possible sources of this. Can you test? And @joeyh would you mind letting me know which API calls you're using in yesod-static?

[joeyh]

@snoyberg import Yesod.Static (embed, Static, publicFiles)

[bmwiedemann]

I'm still seeing order variations with those 2 patches applied.

[snoyberg]

Thanks @joeyh, it may have been the usage of publicFiles (the API is much larger than I remember). @bmwiedemann can you test out 7964967? I'm sorry this is taking so many steps, I just don't have a way to repro this myself.

[bmwiedemann]

With the 3 patches git-annex now builds fully reproducibly :-)

Thanks a lot @snoyberg - I know that walking in the dark is not easy.

[snoyberg]

Thank you for confirming. New versions of packages released.

And if it's not clear: I'm really appreciative of the work going on by yourself and others to make reproducible builds happen, thank you.