Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building: Bad interface file/Cannot continue after interface file error #328

Open
Vekhir opened this issue Dec 4, 2024 · 20 comments
Open

Comments

@Vekhir
Copy link

Vekhir commented Dec 4, 2024

Hi, I'm coming from Arch Linux and working on getting GHC 9.4 running. The rebuilds require happy to compile.

Building happy 2.0.2 with GHC 9.4.8 leads to this error:

Declaration for lalrBackendDataDir
Unfolding of lalrBackendDataDir:
  Bad interface file: /usr/lib/ghc-9.4.8/site-local/happy-lib-2.0.2/Paths_happy_lib.dyn_hi
      Something is amiss; requested module  happy-lib-2.0.2-6uhwNS4KElfEeCW2LxXwlE-backend-lalr:Paths_happy_lib differs from name found in the interface file happy-lib-2.0.2-CsmyxCqWSYG1DawVd57fhe-backend-glr:Paths_happy_lib (if these names look the same, try again with -dppr-debug)

<no location info>: error:
    Cannot continue after interface file error

Investigation

The same happens with happy 2.1.3. Superficially, the error looks similar to #315. One difference is that, for bootstrapping reasons, we use the GHC included runhaskell script instead of cabal install.

Building with GHC 9.2 and Cabal 3.6 emits the same Bad interface file [...] message, but does not fail to build. See error log below for details.

Conclusion

For the time being, is there anything we can do to workaround this issue?

If you need anything else from me or want me to check something, feel free to ask.
-- Vekhir.

Appendix

System information:

OS: Arch Linux
Kernel: Linux 6.12.1-arch1-1
GHC: 9.4.8
Cabal: 3.8.1.0
happy-lib: 2.0.2 and 2.1.3 (PKGBUILD)
happy: 2.0.2 and 2.1.3 (PKGBUILD)

Error log

GHC 9.4:

GHC 9.2:

@sgraf812
Copy link
Collaborator

sgraf812 commented Dec 4, 2024

Thanks for the bug report. Unfortunately, I'm out of my depth here, because I have never used the Setup.hs interface directly. #315 does sound related.

Could you provide a self-contained script for reproduction? I tried the following for happy-2.1.3 on GHC 9.4.8 and Cabal 3.10, but it failed to reproduce:

$ cd lib
$ runhaskell --ghc-arg='-package Cabal' Setup configure --prefix=/tmp/happy
$ runhaskell --ghc-arg='-package Cabal' Setup build
$ runhaskell --ghc-arg='-package Cabal' Setup install
$ cd ..
$ runhaskell --ghc-arg='-package Cabal' Setup configure --prefix=/tmp/happy
$ runhaskell --ghc-arg='-package Cabal' Setup build
<success>

It would help to know what exactly triggers the error.

@Vekhir
Copy link
Author

Vekhir commented Dec 4, 2024

Thanks for your quick action!

Could you provide a self-contained script for reproduction?

The PKGBUILD contains the necessary information. The interesting instructions for our specific case:

  runhaskell Setup configure -O --enable-shared --enable-executable-dynamic --disable-library-vanilla \
    --prefix=/usr --docdir=/usr/share/doc/happy --datasubdir=happy --enable-tests \
    --dynlibdir=/usr/lib --libsubdir=\$compiler/site-local/\$pkgid \
    --ghc-option=-optl-Wl\,-z\,relro\,-z\,now \
    --ghc-option='-pie'
  runhaskell Setup build

To run tests:

PATH="$PWD/dist/build/happy:$PATH" HC="ghc -dynamic" runhaskell Setup test

Edit: The issue also occurs with runhaskell Setup configure --enable-shared --enable-executable-dynamic. I can't
remove those options without rebuilding everything statically. I don't think the extra options are the cause. Can you check with Cabal 3.8 if the issue is fixed in later versions? I'm gonna try build with GHC 9.6 and Cabal 3.10 if that makes a difference.

Note: The Haskell packages on Arch are built as shared libraries, which is reflected in above instructions. You might need to remove the related options, but I included them for full transparency.

@sgraf812
Copy link
Collaborator

sgraf812 commented Dec 5, 2024

The PKGBUILD contains the necessary information.

Thanks, I had seen that file, but I don't know what to do with it, nor what precise commands you executed.

I cannot reproduce with the following commands, using the setup from my last message:

runhaskell --ghc-arg='-package Cabal' Setup configure  --enable-shared --enable-executable-dynamic
runhaskell --ghc-arg='-package Cabal' Setup build

Perhaps you can try a newer Cabal and see if it works? Sadly I don't know how to install an older version of cabal locally...

@Vekhir
Copy link
Author

Vekhir commented Dec 6, 2024

Same error with GHC 9.6 and Cabal 3.10.

It's interesting that the issue gets ignored in Cabal 3.6, but leads to failure in 3.8 and 3.10.

I can't test GHC 9.8, as that would need happy to build. Not that it's likely to be any different.

My build environment

As for the PKGBUILD, it's basically a bash script executed in a newly created systemd container. When the dependencies are installed and the source archive extracted, the build() function is called. After the cd $pkgname-$pkgver call, the script is inside the root directory of the archive (i.e. of this repo).
Some environment flags are set (see also the verbose build log below), but $MAKEFLAGS is actually empty here. No special commands or anything.

The only command I execute personally is makechrootpkg. Everything else is automated, so no deviation possible. See also Arch Wiki: Building in a clean chroot.
A more comprehensive explanation is documented in the Arch Wiki.

More logs

happy-2.1.3-1-x86_64-build.log

Here is another log with verbose=2 set:
build log: verbose=2
The most interesting part might be the command used to build happy:
Running: /usr/bin/ghc --make -no-link -fbuilding-cabal-package -O -dynamic -fPIC -osuf dyn_o -hisuf dyn_hi -outputdir dist/build/happy/happy-tmp -odir dist/build/happy/happy-tmp -hidir dist/build/happy/happy-tmp -stubdir dist/build/happy/happy-tmp -i -idist/build/happy/happy-tmp -iapp -idist/build/happy/autogen -idist/build/global-autogen -Idist/build/happy/autogen -Idist/build/global-autogen -Idist/build/happy/happy-tmp -optP-include -optPdist/build/happy/autogen/cabal_macros.h -hide-all-packages -Wmissing-home-modules -no-user-package-db -package-db dist/package.conf.inplace -package-id array-0.5.6.0 -package-id base-4.18.2.1 -package-id containers-0.6.7 -package-id happy-lib-2.1.3-3jmHOsCpWQN1H41MAybsH1 -package-id mtl-2.3.1 -XHaskell98 -XCPP -XMagicHash -XFlexibleContexts -XNamedFieldPuns app/Main.lhs Paths_happy -Wall -Wno-incomplete-uni-patterns -optl-Wl,-z,relro,-z,now -pie

Full build log with verbose=3

@Vekhir
Copy link
Author

Vekhir commented Dec 6, 2024

Just to check, I've created a PKGBUILD to do the normal install (without polluting my system):
PKGBUILD

The important observation is that cabal v1-install happy with GHC 9.4.8 and Cabal 3.12.1.0 fails with the same error as previously reported:
happy-bug-328-testscript-2.1.3-1-x86_64-build.log
In that it is different from the linked issue. With cabal install happy, it works fine though.
Edit: With GHC 9.8.2, I experience the error in the linked issue.

So likely a Cabal issue like the other one, even if potentially not the same one.
I think that's the end of the road for me, unless you have another idea to try?

Edit: For my own convenience, here is the latest version of the testscript:
testscript: cabal v1-install

@sgraf812
Copy link
Collaborator

sgraf812 commented Dec 6, 2024

I think that's the end of the road for me, unless you have another idea to try?

It's a bit sad that I can't really help here, but I think you are correct. Is it possible for you to simply use cabal v2-install to install happy?

@Vekhir
Copy link
Author

Vekhir commented Dec 6, 2024

Is it possible for you to simply use cabal v2-install to install happy?

Not really, because the container would need to be set up for that which means at least redownloading the package list every time. And then it's questionable if it would even work with the system-provided GHC.

For now, I've resorted to downgrading happy to continue building. That's not a long-term solution. The packaging changes in happy 2.0.2 did provide some advantages, like finally enabling the tests.

@sgraf812
Copy link
Collaborator

sgraf812 commented Dec 6, 2024

Yes, I think downgrading happy is the correct solution until this is fixed in Cabal.

It is not unlikely that GHC 9.14 (due in 2025) will require happy-2.1.3, though. Let's hope that Cabal has an answer to this problem (haskell/cabal#8733) until that is the case; otherwise we should consider rolling back on using private sub-libraries.

@Vekhir
Copy link
Author

Vekhir commented Dec 16, 2024

The issue with private sublibraries is probably very old. But it was never a problem (in terms of compilation), because the relevant function (forkM_maybe in compiler/GHC/Tc/Utils/Monad.hs) just returned Nothing. This changed in 61a203ba, so that it now fails with the error message above: compiler/GHC/Tc/Utils/Monad.hs#L2275
Unfortunately, the change does not seem easily revertible. However, the commit message explicitly states:

Instead we now panic so these bad situations which should never arise can be identified.

Maybe an issue could be opened there to find a solution. The error is initially raised in compiler/GHC/Iface/Load.hs#L912.

Another point for context: runhaskell is independent of Cabal, but likely uses the same interfaces in GHC as v1-install. I don't know exactly what v2-install does differently in that regard. But I don't think that Cabal can really do anything about this.

This is about as far as I got. @sgraf812 Can you talk to the people over at GHC? Perhaps you can figure out a way together.

-- Vekhir

@sgraf812
Copy link
Collaborator

Sorry for the silence; I was on holiday and then busy. I see you already opened https://gitlab.haskell.org/ghc/ghc/-/issues/25603. That's great, especially because I don't have a clue as to what is going on.

Another point for context: runhaskell is independent of Cabal, but likely uses the same interfaces in GHC as v1-install. I don't know exactly what v2-install does differently in that regard. But I don't think that Cabal can really do anything about this.

Note that in your workflow, you call runhaskell on Setup.hs, which is a local "script" calling into the Cabal library. You are likely correct that it uses the same interface as v1-install, at least the similarity of the error suggests so. I do not know how to replicate the v2-install workflow using the Setup.hs script.

I'm not sure if it's actually a bug in GHC; the commit you link sounds reasonable. If you force GHC to load a library defining a type T with a different hash than is expected by client code, you should better get a type error because the type definition might be completely different. The actual issue is: Why do we get a different hash in the first place?

@sgraf812
Copy link
Collaborator

One bit I'm continually puzzled about: How do you build happy-lib? You only provided the instructions for happy, but I think the crucial part is how you built happy-lib in the first place.

If I do

runhaskell --ghc-arg='-package Cabal' Setup configure --enable-shared --enable-executable-dynamic

at the top-level (to build happy), I get

Configuring happy-2.1.3...
Error: Setup: Encountered missing or private dependencies:
happy-lib ==2.1.3

Run the 'configure' command first.

If I run

cd lib/
runhaskell --ghc-arg='-package Cabal' Setup configure --enable-shared --enable-executable-dynamic

to build happy-lib, I get

<no location info>: error: [GHC-82272]
    module ‘Setup’ cannot be found locally
*** Exception: ExitFailure 1

which makes sense, because there is no Setup.hs in lib/. It appears I'm still failing to reproduce your issue, other than with v1-install (which might be different).

@sgraf812
Copy link
Collaborator

Using GHC 9.6 and Cabal 3.10.3, I can confirm that the following instructions successfully build happy:

cp Setup.hs lib
cd lib
runhaskell --ghc-arg='-package Cabal' Setup configure --enable-shared --enable-executable-dynamic
runhaskell --ghc-arg='-package Cabal' Setup build
runhaskell --ghc-arg='-package Cabal' Setup configure --enable-shared --enable-executable-dynamic --package-db=lib/dist/package.conf.inplace
runhaskell --ghc-arg='-package Cabal' Setup build
dist/build/happy/happy

Can you try this locally on a clean checkout?

Note the --package-db=lib/dist/package.conf.inplace. Without it, Cabal is unable to find happy-lib on my machine. You will need to provide a similar path, I think. Perhaps in your case some other version of happy-lib from a global package DB is polluting the environment?

See also Section 5.3 of the Cabal user guide which has a few remarks about internal sublibrary components. I'm not sure if it's relevant to you.

@Vekhir
Copy link
Author

Vekhir commented Dec 24, 2024

Sorry for the silence; I was on holiday and then busy

No problem, happy holidays!

I see you already opened https://gitlab.haskell.org/ghc/ghc/-/issues/25603. That's great, especially because I don't have a clue as to what is going on.

That makes two of us.

I'm not sure if it's actually a bug in GHC; the commit you link sounds reasonable.

Part of it is just involving any people that might be able to bring some clarity into the situation. The commit is fine; but it's the reason why GHC 9.2 works and 9.4 doesn't. And this issue is apparently something that they might have encountered previously? At least, they say it should never happen, so no idea how Cabal can produce this level of garbage.

At this stage, my best guess is that the Paths_happy_lib.dyn.hi file is corrupted; It seems to be generated from an (also generated) Paths_happy_lib.hs which is the same with v1-build and v2-build (which I've checked out in the meantime). So, the issue probably lies in the step Paths_happy_lib.hs -> Paths_happy_lib.dyn.hi, which I'd assume only GHC can do anything about.


One bit I'm continually puzzled about: How do you build happy-lib? You only provided the instructions for happy, but I think the crucial part is how you built happy-lib in the first place.

We follow the structure on Hackage, i.e. happy-lib is a separate package with its own PKGBUILD. I've linked it in the OP, but didn't receive much attention. It is then used as a dependency in happy, where it is installed in the global namespace under /usr in the usual place.

The missing Setup.hs file is a common issue. I don't know exactly why people stopped providing it, but I think it's because it's not required with v2-stuff. And in any case, it's always the same, so might as well.
Anyway, we generate it on the fly using the gen-setup utility to create a default Setup.hs.


I'll try out your instructions and report back the results.

@Vekhir
Copy link
Author

Vekhir commented Dec 24, 2024

OK, I turned your example into this PKGBUILD:
PKGBUILD.txt

It clones the github repo and switches to it, then executes the instructions as above. I had to add --disable-library-vanilla, as otherwise I'd get this error:

Configuring happy-lib-2.1.3...
Preprocessing library 'grammar' for happy-lib-2.1.3..
Building library 'grammar' for happy-lib-2.1.3..
[1 of 2] Compiling Happy.Grammar.ExpressionWithHole ( grammar/src/Happy/Grammar/ExpressionWithHole.hs, dist/build/grammar/Happy/Grammar/ExpressionWithHole.o, dist/build/grammar/Happy/Grammar/ExpressionWithHole.dyn_o )

grammar/src/Happy/Grammar/ExpressionWithHole.hs:1:8: error:
    Could not find module ‘Prelude’
    There are files missing in the ‘base-4.16.4.0’ package,
    try running 'ghc-pkg check'.
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
1 | module Happy.Grammar.ExpressionWithHole where
  |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

With that, building happy-lib succeeds, but then this error occurs:

ghc-9.2.8: can't find a package database at lib/dist/package.conf.inplace

Well, we need to switch back to the top-level :)

Interestingly, there is no error message at all - previously, it would complain, but nothing happens. I then confirmed with GHC 9.6.6 and it actually works!
Not quite sure what's going on. We usually install the packages via register.sh, maybe something there is going off?
Edit: I added all our usual flags and it still works.

Full build log (GHC 9.2.8)
register.sh for happy-lib

@Vekhir
Copy link
Author

Vekhir commented Dec 24, 2024

Hm, tests are failing with

happy: /usr/local/share/x86_64-linux-ghc-9.2.8/happy-lib-2.1.3/HappyTemplate.hs: openFile: does not exist (No such file or directory)

since we don't install happy-lib. I'm going to try building GHC.

@sgraf812
Copy link
Collaborator

Indeed, in the end you will need to run copy and register. I'm just trying to come up with an isolated script for reproduction, because as yet I haven't been able to reproduce. I cannot run the PKGBUILD file directly, for example, and any attempt of mine to replicate it is guess work. In an attempt to delta debug, I gave a script that seems to work. Now my question for you is: What is the minimal edits I need to do to the script such that it does not work? Surely it is related to setting up some package database, registering happy-lib after copying it and then trying to build happy. But what are the steps exactly?

@sgraf812
Copy link
Collaborator

Here is a tweakable script that successfully build, packages, installs and tests happy-lib and happy.
Note that I got rid of the --force tweak to the registering scripts, which were likely hiding errors while building happy-lib.

  • Can you verify it works on your end?
  • Does this help to fix your packaging script?
  • If not, can you tweak this script such that it reproduces the bug you are experiencing? I will also need the build log for happy-lib, which is likely broken.

Thanks!

#! /usr/bin/env bash

TMP=${TMP:-/tmp}

DIST_DIR=$TMP/happy-lib-dist
DB=$TMP/db
INSTALL_DIR=$TMP/usr/local
CONFIGURE_FLAGS="-O --enable-shared --enable-executable-dynamic --disable-library-vanilla \
  --prefix=$INSTALL_DIR --package-db=$DB --enable-tests \
  --ghc-option=-optl-Wl,-z,relro,-z,now \
  --ghc-option=-pie"

rm -rf $DB $DIST_DIR $INSTALL_DIR

ghc-pkg init $DB
git clone --depth=1 https://github.com/haskell/happy $TMP/happy
cd $TMP/happy
git reset --hard HEAD
git clean -fxd
cp Setup.hs lib

# Build happy-lib
cd $TMP/happy/lib
runhaskell --ghc-arg='-package Cabal' Setup configure $CONFIGURE_FLAGS --prefix=$INSTALL_DIR
runhaskell --ghc-arg='-package Cabal' Setup build

# Package happy-lib
runhaskell --ghc-arg='-package Cabal' Setup copy --destdir=$DIST_DIR
runhaskell --ghc-arg='-package Cabal' Setup register --gen-script
runhaskell --ghc-arg='-package Cabal' Setup unregister --gen-script
install -D -m744 register.sh unregister.sh $DIST_DIR/$INSTALL_DIR
# Now we'd create the tarball for $DIST_DIR.

# Install happy-lib
# The following line emulates unpacking said tarball, which installs into
# INSTALL_DIR:
(cd $DIST_DIR; cp -r * /)

# Now call the register script for INSTALL_DIR:
$INSTALL_DIR/register.sh --package-db $DB

# Build happy
cd $TMP/happy
runhaskell --ghc-arg='-package Cabal' Setup configure $CONFIGURE_FLAGS
runhaskell --ghc-arg='-package Cabal' Setup build
runhaskell --ghc-arg='-package Cabal' Setup test

# Package happy
runhaskell --ghc-arg='-package Cabal' Setup copy --destdir=$DIST_DIR
runhaskell --ghc-arg='-package Cabal' Setup register --gen-script
runhaskell --ghc-arg='-package Cabal' Setup unregister --gen-script
install -D -m744 register.sh unregister.sh $DIST_DIR/$INSTALL_DIR

# Install happy
(cd $DIST_DIR; cp -r * /)
$INSTALL_DIR/register.sh --package-db $DB

# Run happy on a test, verifying that distribution was successlful
$INSTALL_DIR/bin/happy $TMP/happy/tests/issue91.y
echo success!

@Vekhir
Copy link
Author

Vekhir commented Dec 27, 2024

Please try out appending --libsubdir=\$compiler/site-local/\$pkgid to the CONFIGURE_FLAGS. This is the smallest change that leads to failure on my side. On the flip-side, removing that from the official PKGBUILD leads to successful build and all tests passing.
Short note, I'm not considering this a solution.

With the success case in hand I was able to compare the build results at each stage. I then noticed that up to runhaskell Setup copy ... everything was identical, in particular the directory structure. I somehow assumed that copy would just do that - copy. But after the command, the two variants differed in folder and file count.
You can actually look at the file structure in the official package at https://archlinux.org/packages/extra/x86_64/haskell-happy-lib/ (click on View the file list for haskell-happy-lib). Searching for Paths_happy_lib.dyn_hi only yields 1 result in the happy-lib-2.0.2 folder, but the success case has two, one for each backend. The above flag is the most obvious choice affecting this stage, and it did.
Setting --libsubdir to anything at all (including "") leads to the wrong file structure. I also ran the copy command with --verbose, the result is appended below. It also includes the directory structure via the tree command. In short, the Paths_happy_lib file for each backend gets written to the same location, so the second one overwrites the first. This just happens to be the one from the glr backend as confirmed by the sha256sums.

I also checked out your latest example, after some tweaking I got it to work. Unsurprisingly, it too doesn't feature --libsubdir.

Log (haskell-happy-lib)

with --libsubdir=\$compiler/site-local/\$pkgid

without --libsubdir

@Vekhir
Copy link
Author

Vekhir commented Jan 4, 2025

@sgraf812 Happy new year! Have you tried out my suggestion above?

Since then, I've opened an issue in Cabal: haskell/cabal#10687
The difference between working and failing seems to be that $libname needs to be used in the libsubdir to distinguish the two Paths_happy_lib files that are auto-generated for each backend.

When trying to make both backends depend on the same sublibrary (let's say common-paths), then under appropriate circumstances it produces a Duplicate modules in library error. It seems that this check isn't done for auto-generated modules. Anyway, I couldn't get this to work.

The best solution (although not pretty) until the issue is fixed in Cabal is probably to manually merge the two backends, so they only generate one Paths_happy_lib module. It would look something like this:

library backend-lalr-glr
  import: common-stanza
  hs-source-dirs:      backend-lalr/src,
                       backend-glr/src,
  exposed-modules:     Happy.Backend.LALR,
                       Happy.Backend.LALR.ProduceCode,
                       Happy.Backend.GLR,
                       Happy.Backend.GLR.ProduceCode
  build-depends:       base < 5, array, grammar, tabular
  other-modules:       Paths_happy_lib
  autogen-modules:     Paths_happy_lib

(compare full cabal file below)
I've tested this and it avoids the issue. If you are fine with this as a temporary measure, then I'd be happy with that.

happy-lib.cabal

@sgraf812
Copy link
Collaborator

sgraf812 commented Jan 6, 2025

Thanks for digging deeper. Indeed, I can reproduce with --libsubdir=\$compiler/site-local/\$pkgid. I don't particularly like your workaround, but I think we should adopt it if we don't find a better one. I'm currently trying to move the autogenerated module to the tabular sublib.

Is it no option for you to have $libname occur in libsubdir?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants