-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix hashing step ids in loops #72
Merged
+102
−1
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
963886e
Fix hashing step ids in loops
albertchae 303e984
Use hash of stepId counts instead of looping
albertchae 43373c8
Use combination of hash and loop to find next unused stepId
albertchae 1fd5308
Handle edge case where user reuses step with stepNumber after it was …
albertchae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
45 changes: 45 additions & 0 deletions
45
...spring-boot-demo/src/main/java/com/inngest/springbootdemo/testfunctions/LoopFunction.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
package com.inngest.springbootdemo.testfunctions; | ||
|
||
import com.inngest.FunctionContext; | ||
import com.inngest.InngestFunction; | ||
import com.inngest.InngestFunctionConfigBuilder; | ||
import com.inngest.Step; | ||
import org.jetbrains.annotations.NotNull; | ||
|
||
public class LoopFunction extends InngestFunction { | ||
|
||
@NotNull | ||
@Override | ||
public InngestFunctionConfigBuilder config(InngestFunctionConfigBuilder builder) { | ||
return builder | ||
.id("loop-fn") | ||
.name("Loop Function") | ||
.triggerEvent("test/loop"); | ||
} | ||
|
||
|
||
@Override | ||
public Integer execute(FunctionContext ctx, Step step) { | ||
int runningCount = 10; | ||
|
||
// explicitly naming a step that the SDK will try to use in the loop shouldn't break the loop | ||
int effectivelyFinalVariableForLambda1 = runningCount; | ||
runningCount = step.run("add-num:3", () -> effectivelyFinalVariableForLambda1 + 50, Integer.class); | ||
|
||
for (int i = 0; i < 5; i++) { | ||
int effectivelyFinalVariableForLambda2 = runningCount; | ||
// The actual stepIds used will be add-num, add-num:1, add-num:2, add-num:4, add-num:5 | ||
runningCount = step.run("add-num", () -> effectivelyFinalVariableForLambda2 + 10, Integer.class); | ||
} | ||
|
||
// explicitly reusing step names that the SDK used during the loop should both execute | ||
// These will be modified to add-num:4:1 and add-num:4:2 respectively | ||
int effectivelyFinalVariableForLambda3 = runningCount; | ||
runningCount = step.run("add-num:4", () -> effectivelyFinalVariableForLambda3 + 30, Integer.class); | ||
int effectivelyFinalVariableForLambda4 = runningCount; | ||
runningCount = step.run("add-num:4", () -> effectivelyFinalVariableForLambda4 + 30, Integer.class); | ||
|
||
return runningCount; | ||
} | ||
} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
30 changes: 30 additions & 0 deletions
30
...pring-boot-demo/src/test/java/com/inngest/springbootdemo/LoopFunctionIntegrationTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
package com.inngest.springbootdemo; | ||
|
||
import com.inngest.Inngest; | ||
import org.junit.jupiter.api.Test; | ||
import org.junit.jupiter.api.parallel.Execution; | ||
import org.junit.jupiter.api.parallel.ExecutionMode; | ||
import org.springframework.beans.factory.annotation.Autowired; | ||
|
||
import static org.junit.jupiter.api.Assertions.assertEquals; | ||
|
||
@IntegrationTest | ||
@Execution(ExecutionMode.CONCURRENT) | ||
class LoopFunctionIntegrationTest { | ||
@Autowired | ||
private DevServerComponent devServer; | ||
|
||
@Autowired | ||
private Inngest client; | ||
|
||
@Test | ||
void testStepsInLoopExecuteCorrectly() throws Exception { | ||
String loopEvent = InngestFunctionTestHelpers.sendEvent(client, "test/loop").getIds()[0]; | ||
Thread.sleep(2000); | ||
|
||
RunEntry<Object> loopRun = devServer.runsByEvent(loopEvent).first(); | ||
assertEquals("Completed", loopRun.getStatus()); | ||
|
||
assertEquals(170, loopRun.getOutput()); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may also need a check here to see is this step ID already exists.
A user could do something like:
"my-step"
"my-step:1"
(user explicitly specifying this string)"my-step"
stepsI think here this would result in us redeclaring
"my-step:1"
in the first iteration of the loop even though we'd like to skip it and go straight to"my-step:2"
.These IDs just being strings means a user can accidentally stumble into our
[ID]:[count]
format and break some stuff. 😄There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK pushed a commit to combine both a hash for O(1) in most cases and a loop afterwards just in case a user used that stepId already. Correct me if I'm wrong but is this a bug in the Go SDK then https://github.com/inngest/inngestgo/blob/0a00daba0b2db68ff0f080f787cf63f0a63b44d8/internal/sdkrequest/manager.go#L123-L131 @darwin67 ?
This seems like it could potentially be worth reserving some delimiter characters for metadata if Inngest has other cases where it would want to modify the user provided stepId.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That works! Thank you.
Mm there's definitely some silly edge case that would be unlikely to hit where users are utilizing
*:n
step IDs explicitly, but that exists everywhere.Looks like that'd be a bug in Go too, aye. Long-term we can start to shift this over to something safer; it'd be great to not be directly influencing the ID internally for the hash, but requires a versioned change across SDKs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought of another edge case where the user could reuse a
my-step:n
name after a loop that used it, so I updated the test and logic to handle that too