You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When passing a "negative" timestamp (ie earlier than the epoch) to to_unix_timestamp, the result actually is off by 1. When looking through the code I noticed this snippet in GpuToUnixTImestampImproved (due to rapidsai/cudf#5166).
val longSecs = withResource(lhs.getBase.asTimestampSeconds()) { secs =>
secs.asLongs()
}
withResource(longSecs) { secs =>
val plusOne = withResource(Scalar.fromLong(1)) { one =>
secs.add(one)
}
withResource(plusOne) { plusOne =>
withResource(Scalar.fromLong(0)) { zero =>
withResource(secs.lessThan(zero)) { neg =>
neg.ifElse(plusOne, secs)
}
}
}
}
It looks like cuDF was updated at some point to "fix" this issue, and this plusOne logic might no longer be needed
More context here, it looks like this doesn't happen for all negative timestamp values. It's possible this could be a rounding issue that has been masked somehow in testing.
Yes, but let's just delete it. No one knows it exists. I added it because it was frustrating to have to jump through hoops to match what Spark was doing. But it was dumb to add it, because it really is just dead code.
Describe the bug
Tested this with 23.12 and 24.02
When passing a "negative" timestamp (ie earlier than the epoch) to
to_unix_timestamp
, the result actually is off by 1. When looking through the code I noticed this snippet inGpuToUnixTImestampImproved
(due to rapidsai/cudf#5166).It looks like cuDF was updated at some point to "fix" this issue, and this plusOne logic might no longer be needed
Steps/Code to reproduce bug
PySpark reproduce:
The text was updated successfully, but these errors were encountered: