-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
join
fails because of an uninstructed cast from int to array[int, x] on 1.13.1
#19763
Comments
This should have been fixed by #19753, and we have made a new patch release available; take a look at |
join
fails because of an uninstructed cast from int to array[int, x] on 1.13.0join
fails because of an uninstructed cast from int to array[int, x] on 1.13.1
@alexander-beedie Unfortunately that patch doesn't fix the problem, and it still occurs exactly as described. |
Hmm, surprising! Are you able to create a small reproducible test-case that you can paste in to the Issue report? |
Did you by any chance happen to have exploded the I think I can produce the same error message with this - q = pl.LazyFrame().select(
pl.lit(pl.Series([[1, 1], [2, 2]], dtype=pl.Array(pl.Int64, 2)))
.explode()
.alias("k")
)
q = q.join(pl.LazyFrame({"k": [1, 2]}), on="k")
print(q.collect()) |
@nameexhaustion Let me know if you get stuck on this, I might be able to dig a bit deeper in our code to try and figure out the source of the data and order of operations that are applied that leads up to this. |
Checking... |
Checks
Reproducible example
Haven't been able to construct one yet. Working on it.
Log output
No response
Issue description
When joining two LazyFrames, one large, one smaller, the datatype of an int column in the larger frame suddenly turns into an array, causing a data type mismatch in the join.
The state of the two LazyFrames before the join:
The error of the join:
This error also occurs when optimizations are turned off:
Explicitly casting the columns to int32 before the operation seems to work:
Expected behavior
I expect the data type to not be cast to an array, and the join to succeed.
Installed versions
The text was updated successfully, but these errors were encountered: