Debugging binary caches not working on macOS

emdash · June 16, 2024, 11:32am

I’ve just started trying out IHP on macOS, and it seems the binary caches are not working: initial setup spent a long time compiling haskell packages (132 of them, if I read correctly?), and the same happens when I add any haskell packages to the nix flake.

I’m not really sure how to debug the binary cache, so I’m hoping for some help/tips.

Running ihp-new initially edited ~/.config/nix/nix.conf and updated these lines:

substituters = https://cache.nixos.org https://ghcide-nix.cachix.org https://digitallyinduced.cachix.org
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ghcide-nix.cachix.org-1:ibAY5FD+XWLzbLr8fxK6n8fL9zZe7jS+gYeyxyWYK5c= digitallyinduced.cachix.org-1:y+wQvrnxQ+PdEsCt91rmvv39qRCYzEgGQaldK26hCKE=

I also tried running cachix authtoken digitallyinduced followed by cachix use digitallyinduced, with no major change (it may have changed the specific data in those files – I didn’t check before doing it).

I also added those lines to /etc/nix/nix.conf, with no change.

I think I should check whether the packages that are being built are present in the digitallyinduced cache, but I don’t know how to find out the store paths for the packages being built. (Maybe there’s a nix command I can run to find the store paths for the flake? That seems logical.)

I also saw some advice in a couple of places about restarting the nix-daemon, but I don’t know how to do this on macOS – I can’t actually tell whether the daemon is running (e.g. ps aux | rg nix doesn’t return anything for the daemon, launchctl list | rg nix shows nothing), but I assume it is running because I can install things fine with nix-env.

emdash · June 16, 2024, 11:39am

Well, it looks like nix flake show should be the command I need:

This command shows the output attributes provided by the flake specified by flake reference flake-url. These are the top-level attributes in the outputs of the flake, as well as lower-level attributes for some standard outputs (e.g. packages or checks).

However, trying to run this for my IHP project gives me a devenv error I’ve run into before when trying to get nix develop working:

warning: Git tree '/Users/Emily/Work/Pannal Village Hall (no git)/ihp-alternative' is dirty
git+file:///Users/Emily/Work/Pannal%20Village%20Hall%20%28no%20git%29/ihp-alternative
├───devShells
│   ├───aarch64-darwin
│   │   └───default omitted (use '--all-systems' to show)
│   ├───aarch64-linux
│   │   └───default omitted (use '--all-systems' to show)
│   ├───x86_64-darwin
error:
       … while evaluating the attribute 'optionalValue.value'
         at /nix/store/sc2fmza4wlzi16jjjmkg6rmmmf75jxmr-source/lib/modules.nix:856:5:
          855|
          856|     optionalValue =
             |     ^
          857|       if isDefined then { value = mergedValue; }

       … while evaluating a branch condition
         at /nix/store/sc2fmza4wlzi16jjjmkg6rmmmf75jxmr-source/lib/modules.nix:857:7:
          856|     optionalValue =
          857|       if isDefined then { value = mergedValue; }
             |       ^
          858|       else {};

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: Failed assertions:
       - devenv was not able to determine the current directory.

         See https://devenv.sh/guides/using-with-flakes/ how to use it with flakes.

So it seems I might need to know more about devenv to figure this out?

Note: because my project is currently building (as the binary cache isn’t working, it’s building pandoc from source in another terminal), I’m trying to run my nix commands outside of the project directory so direnv doesn’t try to build it a second time. So the command I actually ran was this (maybe this doesn’t work? It does seem to find the flake ok though):

nix flake show ihp-alternative/

emdash · June 16, 2024, 11:50am

Ok, I’ve taken another tack. I can see while direnv is building pandoc that it’s building doclayout-0.4.0.1, so I found a nix store path for that package:

ls /nix/store | rg doclayout-0.4.0.1

One of the results from that (the only one without a suffix of some kind) was wsp5zmdyszavqha49cy6qm57ps356xjz-doclayout-0.4.0.1, so I checked if that’s in the digitally induced cache following instructions on the nixos wiki:

curl https://digitallyinduced.cachix.org/wsp5zmdyszavqha49cy6qm57ps356xjz-doclayout-0.4.0.1.narinfo

This gave me a 404.

But, I don’t have a lot of certainty that this is a conclusive answer: how do I know I’m checking for the right store path? Also, it seems more relevant for the top-level dependencies (pandoc, in this case) rather than the transitive dependencies.

Once pandoc has finished building I’ll try using that.

I suppose I could also check the store path for the version of ghc that’s installed when I’m in the repository. (Do those vary according to the packages you have installed, though? I.e. now that I’ve installed pandoc will that be a different hash that wouldn’t be in the cache anyway?)

emdash · June 16, 2024, 11:57am

I realised that my cache check was slightly incorrect: I should only be using the hash part of the store path, not the package as well:

curl https://digitallyinduced.cachix.org/wsp5zmdyszavqha49cy6qm57ps356xjz.narinfo

However, this still gives me a 404.

I figured I’d check for something more normal in the cache to try and see a cache hit – maybe GHC?

Unfortunately the versions of ghc I have in my nix store appear not to be in the cache:

Emily@Peach /n/v/l/n/drvs> ls /nix/store/ | rg "ghc-[0-9.]*\$"
118q8w172i7dyckl3gifj6cmyzkgfxby-ghc-9.6.4
5f5f79raa0vy02qxffr671qbjyliaqnp-ghc-9.4.8
p07zdrglgsl584y4ii7cg4flinwmlzr7-ghc-9.8.2
Emily@Peach /n/v/l/n/drvs> curl https://digitallyinduced.cachix.org/5f5f79raa0vy02qxffr671qbjyliaqnp.narinfo
404 - Not Found
Emily@Peach /n/v/l/n/drvs> curl https://digitallyinduced.cachix.org/118q8w172i7dyckl3gifj6cmyzkgfxby.narinfo
404 - Not Found
Emily@Peach /n/v/l/n/drvs> curl https://digitallyinduced.cachix.org/p07zdrglgsl584y4ii7cg4flinwmlzr7.narinfo
404 - Not Found

emdash · June 16, 2024, 12:03pm

The other approach mentioned on the nixos wiki is using nix path-info, so I tried that:

Emily@Peach /n/v/l/n/drvs> nix path-info -r /nix/store/p07zdrglgsl584y4ii7cg4flinwmlzr7-ghc-9.8.2 --store https://digitallyinduced.cachix.org
these 37 paths will be fetched (350.03 MiB download, 2884.03 MiB unpacked):
  /nix/store/blrfnaa900q44i0zjlpn6xlvgdzcfxx7-Libsystem-1238.60.2
  /nix/store/dcd018g56mxjpa6vq8hm0d7xvwbxsjqx-bash-5.2p26
  /nix/store/yqn8zbqbmv62fnwqrsq66i3qqjwy4jvy-binutils-2.41
  /nix/store/95y326cgv1yw85iavav89x4r86fyhvmc-binutils-2.41-lib
  /nix/store/iinf7ibm4flf5srlkny3nzli3zvz75ds-cctools-binutils-darwin-16.0.6-973.0.1
  /nix/store/4nykd9layri8m95pbxah0mxfln9g47nh-cctools-binutils-darwin-wrapper-16.0.6-973.0.1
  /nix/store/zcsi3abhx8vlrg9kv423k4rj2c7rq763-cctools-llvm-16.0.6-973.0.1
  /nix/store/ykab48f941z0lspl6wjrmnhfrdawdmml-cctools-port-973.0.1
  /nix/store/452idxpydykikx6q7nk7vjpcr6g0xgxg-clang-16.0.6
  /nix/store/17mgi0a1zwxl38fcrmk93hl28j6g2wk7-clang-16.0.6-lib
  /nix/store/8qkc9bnray3ymq1hyqc3fn7qqcgbqzdx-clang-wrapper-16.0.6
  /nix/store/kjmxmp997rf1djih957gv4r5dl7mk9qf-compiler-rt-libc-16.0.6
  /nix/store/4pnmhw08vvv94lvxbgiplpl27xrf5cw9-compiler-rt-libc-16.0.6-dev
  /nix/store/kr9dzg0zdim23lm212ycir6cd4f50q61-coreutils-9.5
  /nix/store/yvg470x1bddx3ccz152p2jbs70dmlydd-expand-response-params
  /nix/store/h9px1psvwxbvh6i731gf827xvphkjr90-gettext-0.21.1
  /nix/store/p07zdrglgsl584y4ii7cg4flinwmlzr7-ghc-9.8.2
  /nix/store/2p94r9sp433nrqj8qq17b559sq62zddn-ghc-9.8.2-doc
  /nix/store/a5gcmpykx542bzcmks6q62i408wmkw5l-gmp-with-cxx-6.3.0
  /nix/store/m12vhipc2j4wgh2rbq6awsgcim6xq07b-gmp-with-cxx-6.3.0-dev
  /nix/store/32hn0zxdbkhr6025qd4dfw544s55dyjr-gnugrep-3.11
  /nix/store/br3pb7wpiij1kpk4krf8l188rdr7w04h-libcxx-16.0.6
  /nix/store/8kv4c66q885l6bd3z72s55r2fm7bkzwa-libcxx-16.0.6-dev
  /nix/store/8xfd2a98plijd2ww72sqk3g12hd9bbzl-libffi-3.4.6
  /nix/store/yijqb3i0vbfdipcglz8mb4w5akj27flr-libffi-3.4.6-dev
  /nix/store/myc79009xi0hzi8q7dgz2986bwrpmsy8-libiconv-50
  /nix/store/sf1awz08r88376szcaxanrx7jgbkpr0w-libtapi-1100.0.11
  /nix/store/jfpcp3sznyrwnwvjw8pinr13a8f4924a-libxml2-2.12.6
  /nix/store/l1050d03j5h90lh0y8mxknqrdaxllajl-lld-16.0.6
  /nix/store/xnjyj5yx8kkm3mvirgm1hykd755l4hn0-llvm-16.0.6
  /nix/store/nwd17cl7b70kyk4l04gv18qzjzwbllb0-llvm-16.0.6-lib
  /nix/store/qvx88s469b31cgs4rh2z3msc8cv94260-llvm-binutils-16.0.6
  /nix/store/lci741mvipbb1jyxvvy5h14rqf2g7ykp-ncurses-6.4
  /nix/store/06a1isg9pvrg37lkrrmlfj04mgrng1yg-ncurses-6.4-dev
  /nix/store/2w3b6mam9vjx8fsdjsg9067hc2picg2j-ncurses-6.4-man
  /nix/store/qnzd5s1svhc39gr8mhd3vw70g4gmh3da-pcre2-10.43
  /nix/store/79akhfrhg2lvsr4g9rdkcwh2lm681yg3-zlib-1.3.1
error: path '/nix/store/p07zdrglgsl584y4ii7cg4flinwmlzr7-ghc-9.8.2' is not valid

This gives me something new at least: the warning about the path not being valid.

I remember seeing this in the cachix FAQs:

I get InvalidPath error from Nix when invoking Cachix

There are two cases under which the error is raised:

Garbage Collection kicks in while Cachix is trying to push a path. To confirm if that’s the case, check if GC timestamps correlate with when the error was raised

Store paths weren’t built yet. Some Nix commands return store paths that would exist if Nix has built them.

This store path definitely exists (or at least, it does in my local nix store – is that what that means?), so I’m guessing the other bullet point is the relevant one. But… I thought nix garbage collection only runs when you manually run nix-collect-garbage? I haven’t run that today. I also don’t know where to check for GC timestamps.

emdash · June 16, 2024, 12:28pm

Pandoc has finished building! Querying the digitallyinduced cache for the resulting store path gives a slightly different result from before:

Emily@Peach ~/W/P/ihp-alternative (main)> ls /nix/store/ | rg "pandoc-[0-9.]*\$"
adfl2zimmi1ikhzqnxsb439jrc735gva-commonmark-pandoc-0.2.2.1
jz25z7k6gz6kgl41fxnzp3la8v78v9y9-pandoc-3.1.11.1

Emily@Peach /n/v/l/n/drvs [1]> nix path-info -r /nix/store/jz25z7k6gz6kgl41fxnzp3la8v78v9y9-pandoc-3.1.11.1 --store https://digitallyinduced.cachix.org
don't know how to build these paths:
  /nix/store/jz25z7k6gz6kgl41fxnzp3la8v78v9y9-pandoc-3.1.11.1
error: path '/nix/store/jz25z7k6gz6kgl41fxnzp3la8v78v9y9-pandoc-3.1.11.1' is not valid

Emily@Peach /n/v/l/n/drvs [1]> curl https://digitallyinduced.cachix.org/jz25z7k6gz6kgl41fxnzp3la8v78v9y9.narinfo
404 - Not Found

I’m not sure what to make of this!

amitaibu · June 16, 2024, 12:45pm

It sounds like maybe cachix isn’t being used - so it builds all the packages.

What do you get when you execute the script from here

curl --silent https://raw.githubusercontent.com/digitallyinduced/ihp/master/Troubleshoot/ihp-troubleshoot | python3

emdash · June 16, 2024, 12:54pm

Ah, thanks for the suggestion! Here’s the output:

Emily@Peach ~/W/P/ihp-alternative (main)> curl --silent https://raw.githubusercontent.com/digitallyinduced/ihp/master/Troubleshoot/ihp-troubleshoot | python3
Checking that the current directory is an IHP project:
+ Found Main.hs
+ Found start script
+ Installed Nix version 2.21.2 is 2.4 or later
Checking direnv:
+ Found .envrc
+ direnv loads .envrc
- direnv denied .envrc, run `direnv allow` to allow .envrc
+ ghci is loaded from nix store
Checking .ghci:
+ Found .ghci
+ .ghci permissions are ok
Checking IHP:
- Symlink build/ihp-lib is missing. Try 'make build/ihp-lib' to fix this
- Symlink build/ihp-lib target directory does not exist. Try `nix-shell --run "make -B build/ihp-lib"` to fix this
+ IHP used from nix
Checking Cachix:
+ Cachix exists
+ digitallyinduced.cachix.org configured
+ No legacy cachix key found
Checking IHP files:
- The file `Makefile` was edited manually.
  Try reverting your changes to https://github.com/digitallyinduced/ihp-boilerplate/blob/master/Makefile to see whether that affects the issue
Debugging Details:
GHCI Output:
GHCi, version 9.8.2: https://www.haskell.org/ghc/  :? for help
package flags have changed, resetting and loading new packages...
package flags have changed, resetting and loading new packages...
[ 1 of 30] Compiling Application.Helper.Controller ( Application/Helper/Controller.hs, interpreted )
[ 2 of 30] Compiling Application.Helper.View ( Application/Helper/View.hs, interpreted )
[ 3 of 30] Compiling Config           ( Config/Config.hs, interpreted )
[ 4 of 30] Compiling Generated.Enums  ( build/Generated/Enums.hs, interpreted )
[ 5 of 30] Compiling Generated.ActualTypes ( build/Generated/ActualTypes.hs, interpreted )
[ 6 of 30] Compiling Generated.Event  ( build/Generated/Event.hs, interpreted )
[ 7 of 30] Compiling Generated.Types  ( build/Generated/Types.hs, interpreted )
[ 8 of 30] Compiling Admin.Types      ( Admin/Types.hs, interpreted )
[ 9 of 30] Compiling Admin.Routes     ( Admin/Routes.hs, interpreted )
[10 of 30] Compiling Admin.View.Layout ( Admin/View/Layout.hs, interpreted )
[11 of 30] Compiling Admin.View.Prelude ( Admin/View/Prelude.hs, interpreted )
[12 of 30] Compiling Admin.View.Static.Welcome ( Admin/View/Static/Welcome.hs, interpreted )
[13 of 30] Compiling Admin.Controller.Prelude ( Admin/Controller/Prelude.hs, interpreted )
[14 of 30] Compiling Admin.Controller.Static ( Admin/Controller/Static.hs, interpreted )
[15 of 30] Compiling Admin.FrontController ( Admin/FrontController.hs, interpreted )
[16 of 30] Compiling Web.Types        ( Web/Types.hs, interpreted )
[17 of 30] Compiling Web.Routes       ( Web/Routes.hs, interpreted )
[18 of 30] Compiling Web.Controller.Prelude ( Web/Controller/Prelude.hs, interpreted )
[19 of 30] Compiling Web.View.Layout  ( Web/View/Layout.hs, interpreted )
[20 of 30] Compiling Web.View.Prelude ( Web/View/Prelude.hs, interpreted )
[21 of 30] Compiling Web.View.Events.Show ( Web/View/Events/Show.hs, interpreted )
[22 of 30] Compiling Web.View.Events.New ( Web/View/Events/New.hs, interpreted )
[23 of 30] Compiling Web.View.Events.Index ( Web/View/Events/Index.hs, interpreted )
[24 of 30] Compiling Web.View.Events.Edit ( Web/View/Events/Edit.hs, interpreted )
[26 of 30] Compiling Web.View.Static.Welcome ( Web/View/Static/Welcome.hs, interpreted )
[27 of 30] Compiling Web.Controller.Static ( Web/Controller/Static.hs, interpreted )
Failed, 25 modules loaded.
Loaded GHCi configuration from /Users/Emily/Work/Pannal Village Hall (no git)/ihp-alternative/.ghci
IHP> ok
IHP> Leaving GHCi.

build/ihp-lib Target:
None
Direnv Output:
direnv exec path /Users/Emily/.nix-profile/bin/direnv
DIRENV_CONFIG /Users/Emily/.config/direnv
bash_path /nix/store/386xmpmnix11qhnn2l0f7lvv0dc70w8h-bash-5.2p26/bin/bash
disable_stdin false
warn_timeout 5s
whitelist.prefix []
whitelist.exact map[]
Loaded RC path /Users/Emily/Work/Pannal Village Hall (no git)/ihp-alternative/.envrc
Loaded watch: ".envrc" - 2024-06-15T14:42:51+01:00
Loaded watch: "../../../.local/share/direnv/allow/5734ee1efeb47251f6f407fbd25c1720c7f5d5c43cf6cf12844a416403143c31" - 2024-06-15T16:01:42+01:00
Loaded watch: "../../../.local/share/direnv/deny/00a4e16ffb4f24d4580bc3036b67ded88f550992f7845741c60efca9ed9c4069" - 1970-01-01T01:00:00+01:00
Loaded watch: "../../../.direnvrc" - 1970-01-01T01:00:00+01:00
Loaded watch: "../../../.config/direnv/direnvrc" - 1970-01-01T01:00:00+01:00
Loaded watch: "flake.nix" - 2024-06-16T11:49:07+01:00
Loaded watch: "flake.lock" - 2024-06-15T14:42:51+01:00
Loaded watch: "devshell.toml" - 1970-01-01T01:00:00+01:00
Loaded RC allowed 0
Loaded RC allowPath /Users/Emily/.local/share/direnv/allow/5734ee1efeb47251f6f407fbd25c1720c7f5d5c43cf6cf12844a416403143c31
Found RC path /Users/Emily/Work/Pannal Village Hall (no git)/ihp-alternative/.envrc
Found watch: ".envrc" - 2024-06-15T14:42:51+01:00
Found watch: "../../../.local/share/direnv/allow/5734ee1efeb47251f6f407fbd25c1720c7f5d5c43cf6cf12844a416403143c31" - 2024-06-15T16:01:42+01:00
Found watch: "../../../.local/share/direnv/deny/00a4e16ffb4f24d4580bc3036b67ded88f550992f7845741c60efca9ed9c4069" - 1970-01-01T01:00:00+01:00
Found RC allowed 0
Found RC allowPath /Users/Emily/.local/share/direnv/allow/5734ee1efeb47251f6f407fbd25c1720c7f5d5c43cf6cf12844a416403143c31

which direnv Output:
/nix/store/22wzsqsdcjk6w0ya3cf46hg6x4gx5m5c-ghc-9.8.2-with-packages/bin/ghci

default.nix:
# For backwards compatibility using flake.nix
(import
    (
        fetchTarball {
            url = "https://github.com/edolstra/flake-compat/archive/35bb57c0c8d8b62bbfd284272c928ceb64ddbde9.tar.gz";

OS:
macOS-12.7.4-x86_64-i386-64bit

I’m surprised that it says direnv denied .envrc, when it looks like that’s working? I have run direnv allow already, and running it again doesn’t change anything.

The Makefile edits were me adding some quotes to fix a couple of the commands to work in directories with spaces in their names:

commit f594137bb4bdaacc50b674c42ebb8a5974377fa3

diff --git a/Makefile b/Makefile
index a8624f1..a503354 100644
--- a/Makefile
+++ b/Makefile
@@ -17,3 +17,7 @@ include ${IHP}/Makefile.dist
 
 psql-fixed: ## Connects to the running postgresql server
        @psql -h "$$PWD"/build/db -d app
+
+dumpdb-fixed: dump_db-fixed ## Saves the current database state into the Fixtures.sql
+dump_db-fixed: ## Saves the current database state into the Fixtures.sql
+       pg_dump -a --inserts --column-inserts --disable-triggers -h "$$PWD"/build/db app | sed -e '/^--/d' > Application/Fixtures.sql

commit d4eb017c76056e5e862925d9d24878e6ac46f96d

diff --git a/Makefile b/Makefile
index 96fd66b..a8624f1 100644
--- a/Makefile
+++ b/Makefile
@@ -15,3 +15,5 @@ JS_FILES += ${IHP}/static/vendor/turbolinksMorphdom.js
 
 include ${IHP}/Makefile.dist
 
+psql-fixed: ## Connects to the running postgresql server
+       @psql -h "$$PWD"/build/db -d app

It was the same before this, so I doubt this has any impact.

I’ll try running that make build/ihp-lib command and see if that changes anything.

emdash · June 16, 2024, 12:56pm

Hmmm, it doesn’t look like that’s a valid rule name?

Emily@Peach ~/W/P/ihp-alternative (main)> make build/ihp-lib
make: *** No rule to make target 'build/ihp-lib'.  Stop.

marc · June 18, 2024, 4:48pm

build/ihp-lib has been removed in recent IHP versions. So that’s okay. (We should likely remove it from the troubleshoot process).

I’ve recently added a patch to the GHC that is used by IHP (macOS only). This triggered a rebuild of GHC on our CI system. This could have lead to a lot of binaries being pushed to cachix, which could have caused a garbage collection of older IHP builds.

Are you running v1.3 or latest master?

emdash · June 18, 2024, 7:35pm

Ah ok, that could be it! I think I’m using v1.3: inputs.ihp.url in flake.nix is github:digitallyinduced/ihp/v1.3. So changing to latest master and then seeing if the binary cache works would validate this theory, right?

marc · June 18, 2024, 8:28pm

Yes that’s a good way to test it