Unexplained Github CI cancellations recently

[inducer]

• Possibly first (?) observed in Base expression node types on dataclasses pymbolic#125 (comment)
• Facts collected (by @alexfikl) at Direct Solver: Recursive Skeletonization #164 (comment)

[connorjward]

Since last week Firedrake has started to encounter memory issues and CI cancellations. Is this likely to be the same thing as you seem to be experiencing here? If so I would be very interested to hear what your theories are about why this is happening.

[connorjward]

For us it appears that downgrading numpy to 1.24 makes the problem go away. We probably need to update our loopy fork since you have merged a fix.

[alexfikl]

For us it appears that downgrading numpy to 1.24 makes the problem go away. We probably need to update our loopy fork since you have merged a fix.

I don't think we have a good theory for why this started happening now. However, the loopy changes for the new numpy unfortunately just clean up some deprecations, so they shouldn't have an effect on this (haven't run anything with those changes included yet).

In our case, this seems to have started with inducer/pymbolic#125 before numpy 1.25 was released, but failures on other branches seem to have only appeared with numpy 1.25. Not sure if that means that there are two different issues here or that the pymbolic change just made it obvious sooner.

[connorjward]

Thanks for the reply. I think we will pin numpy for now.

[inducer]

I'll close this, to avoid spamming the folks on the Firedrake end. For us, the issue seems to be more along the lines of #213.