Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TODO #1

Open
21 of 33 tasks
JulianKropp opened this issue Jul 2, 2024 · 1 comment
Open
21 of 33 tasks

TODO #1

JulianKropp opened this issue Jul 2, 2024 · 1 comment
Assignees

Comments

@JulianKropp
Copy link
Collaborator

JulianKropp commented Jul 2, 2024

pipeline

Issues

  • On error DP exits with "running": "True"
  • Improve gRPC to DP mapping by sending the id of the module
  • Remove set_order_tracker #24
  • FIRST_WINS: If a DP with an error races an DP with no error. The DP with no error will be a OUTDATED. (Fixed 5750f37)
  • DP with status exit or error will be callbacked later if using mode ORDER_BY_SEQUENCE or FIRST_WINS. They first have to wait (Fixed 5750f37)
  • Remove Outdated DP from Queue if not already processing #26 (will not implement this)
  • External Module doesnt know the difference between different pipelines and pipeline instances. It just do stuff. So It has to create an own object for each instance which wants to communicate with. And it needs a timeout which deletes this module object if it exists longer then x.

Future plans:

  • Error Handling #5
  • Scaling horizontal #8
  • ParallelModule #9
  • Mulithreading #3
  • Scaling vertical
  • Webside for Live view of the pipline and live edething (like order or adding/removing modules)
  • API #12
  • More Module Options (Queue mode. Oldest, newest, max=10)
  • PupSub between modules phases, controllers. Or a module can send data to an early module which then processes data differently
  • If a DP is faster and outhers are already processing. Stop them after a module is finished by setting the DP.status to outdated. New status RUNNING_OUTDATED. For tha add to OrderTracker self._processing_data_packages to set all older DPs to RUNNING_OUTDATED
@JulianKropp JulianKropp self-assigned this Jul 2, 2024
@JulianKropp
Copy link
Collaborator Author

JulianKropp commented Jul 7, 2024

Detailed Architecture Description

Pipeline

stateDiagram-v2
    [*] --> Pipeline: Start

    state Pipeline {

        [*] --> Controller0_NOT_PARALLEL

        state Controller0_NOT_PARALLEL {

            [*] --> Phase0_C0

            state Phase0_C0 {
                [*] --> Module0_P0_C0: Start Phase0
                Module0_P0_C0 --> Module1_P0_C0
                Module1_P0_C0 --> Module...n_P0_C0
                Module...n_P0_C0 --> [*]: End Phase0
            }

            Phase0_C0 --> Phase1_C0

            state Phase1_C0 {
                [*] --> Module0_P1_C0: Start Phase1
                Module0_P1_C0 --> Module1_P1_C0
                Module1_P1_C0 --> Module...n_P1_C0
                Module...n_P1_C0 --> [*]: End Phase1
            }

            Phase1_C0 --> Phase...n_C0

            state Phase...n_C0 {
                [*] --> Module0_P...n_C0: Start Phase...n
                Module0_P...n_C0 --> Module1_P...n_C0
                Module1_P...n_C0 --> Module...n_P...n_C0
                Module...n_P...n_C0 --> [*]: End Phase...n
            }

            Phase...n_C0 --> [*]
        }

        Controller0_NOT_PARALLEL --> Controller1_NO_ORDER

        state Controller1_NO_ORDER {

            state fork_state_C0 <<fork>>
            [*] --> fork_state_C0
            fork_state_C0 --> Phase0_C1: 0
            fork_state_C0 --> Phase0_C1: 1
            fork_state_C0 --> Phase0_C1: ...n

            state Phase0_C1 {
                [*] --> Module0_P0_C1: Start Phase0
                Module0_P0_C1 --> Module1_P0_C1
                Module1_P0_C1 --> Module...n_P0_C1
                Module...n_P0_C1 --> [*]: End Phase0
            }

            Phase0_C1 --> Phase1_C1: 0
            Phase0_C1 --> Phase1_C1: 1
            Phase0_C1 --> Phase1_C1: ...n

            state Phase1_C1 {
                [*] --> Module0_P1_C1: Start Phase1
                Module0_P1_C1 --> Module1_P1_C1
                Module1_P1_C1 --> Module...n_P1_C1
                Module...n_P1_C1 --> [*]: End Phase1
            }

            Phase1_C1 --> Phase...n_C1: 0
            Phase1_C1 --> Phase...n_C1: 1
            Phase1_C1 --> Phase...n_C1: ...n

            state Phase...n_C1 {
                [*] --> Module0_P...n_C1: Start Phase...n
                Module0_P...n_C1 --> Module1_P...n_C1
                Module1_P...n_C1 --> Module...n_P...n_C1
                Module...n_P...n_C1 --> [*]: End Phase...n
            }

            state join_state_C0 <<join>>
            Phase...n_C1 --> join_state_C0: 0
            Phase...n_C1 --> join_state_C0: 1
            Phase...n_C1 --> join_state_C0: ...n
            join_state_C0 --> [*]
        }

        Controller1_NO_ORDER --> Controller2_ORDER_BY_SEQUENCE

        state Controller2_ORDER_BY_SEQUENCE {

            [*] --> Add_Sequence_Number_C2

            state fork_state_C1 <<fork>>
            Add_Sequence_Number_C2 --> fork_state_C1
            fork_state_C1 --> Phase0_C2: 0
            fork_state_C1 --> Phase0_C2: 1
            fork_state_C1 --> Phase0_C2: ...n

            state Phase0_C2 {
                [*] --> Module0_P0_C2: Start Phase0
                Module0_P0_C2 --> Module1_P0_C2
                Module1_P0_C2 --> Module...n_P0_C2
                Module...n_P0_C2 --> [*]: End Phase0
            }

            Phase0_C2 --> Phase1_C2: 0
            Phase0_C2 --> Phase1_C2: 1
            Phase0_C2 --> Phase1_C2: ...n

            state Phase1_C2 {
                [*] --> Module0_P1_C2: Start Phase1
                Module0_P1_C2 --> Module1_P1_C2
                Module1_P1_C2 --> Module...n_P1_C2
                Module...n_P1_C2 --> [*]: End Phase1
            }

            Phase1_C2 --> Phase...n_C2: 0
            Phase1_C2 --> Phase...n_C2: 1
            Phase1_C2 --> Phase...n_C2: ...n

            state Phase...n_C2 {
                [*] --> Module0_P...n_C2: Start Phase...n
                Module0_P...n_C2 --> Module1_P...n_C2
                Module1_P...n_C2 --> Module...n_P...n_C2
                Module...n_P...n_C2 --> [*]: End Phase...n
            }

            state join_state_C1 <<join>>
            Phase...n_C2 --> join_state_C1: 0
            Phase...n_C2 --> join_state_C1: 1
            Phase...n_C2 --> join_state_C1: ...n
            join_state_C1 --> Order_By_Sequence_Number_C2
            Order_By_Sequence_Number_C2 --> [*]
        }

        Controller2_ORDER_BY_SEQUENCE --> Controller3_FIRST_WINS

        state Controller3_FIRST_WINS {

            [*] --> Add_Sequence_Number_C3

            state fork_state_C3 <<fork>>
            Add_Sequence_Number_C3 --> fork_state_C3
            fork_state_C3 --> Phase0_C3: 0
            fork_state_C3 --> Phase0_C3: 1
            fork_state_C3 --> Phase0_C3: ...n

            state Phase0_C3 {
                [*] --> Module0_P0_C3: Start Phase0
                Module0_P0_C3 --> Module1_P0_C3
                Module1_P0_C3 --> Module...n_P0_C3
                Module...n_P0_C3 --> [*]: End Phase0
            }

            Phase0_C3 --> Phase1_C3: 0
            Phase0_C3 --> Phase1_C3: 1
            Phase0_C3 --> Phase1_C3: ...n

            state Phase1_C3 {
                [*] --> Module0_P1_C3: Start Phase1
                Module0_P1_C3 --> Module1_P1_C3
                Module1_P1_C3 --> Module...n_P1_C3
                Module...n_P1_C3 --> [*]: End Phase1
            }

            Phase1_C3 --> Phase...n_C3: 0
            Phase1_C3 --> Phase...n_C3: 1
            Phase1_C3 --> Phase...n_C3: ...n

            state Phase...n_C3 {
                [*] --> Module0_P...n_C3: Start Phase...n
                Module0_P...n_C3 --> Module1_P...n_C3
                Module1_P...n_C3 --> Module...n_P...n_C3
                Module...n_P...n_C3 --> [*]: End Phase...n
            }

            state join_state_C2 <<join>>
            Phase...n_C3 --> join_state_C2: 0
            Phase...n_C3 --> join_state_C2: 1
            Phase...n_C3 --> join_state_C2: ...n
            join_state_C2 --> Remove_Older_Then_Last_Sequence_Number_C3

            Remove_Older_Then_Last_Sequence_Number_C3 --> [*]
        }

    }

    Pipeline --> [*]: End


Loading

Modules

ExecutionModule

Executes code to modify or process the data.

stateDiagram-v2
    [*] --> ExecutionModule
    state ExecutionModule {
        [*] --> ExecuteCode
        ExecuteCode --> [*]
    }
    ExecutionModule --> [*]
Loading

ConditionModule

Evaluates conditions and routes the data based on the result.

stateDiagram-v2
    [*] --> ConditionModule
    state ConditionModule {
        state if_state <<choice>>
        [*] --> ConditionCode
        ConditionCode --> if_state
        if_state --> TrueModule: If condition is true
        if_state --> FalseModule : If condition is false
        TrueModule --> [*]
        FalseModule --> [*]
    }
    ConditionModule --> [*]
Loading

CombinationModule

Combines multiple modules into one.

stateDiagram-v2
    [*] --> CombinationModule
    state CombinationModule {
        [*] --> Module0: Start Processing
        state Module0 {
            [*] --> ExecuteCode0
            ExecuteCode0 --> [*]
        }
        Module0 --> Module1
        state Module1 {
            [*] --> ExecuteCode1
            ExecuteCode1 --> [*]
        }
        Module1 --> Module...n
        state Module...n {
            [*] --> ExecuteCode...n
            ExecuteCode...n --> [*]
        }
        Module...n --> [*]
    }
    CombinationModule --> [*]
Loading

ExternalModule

Moves the execution of a module to an external server.

stateDiagram-v2
    state Server0 {
        state Pipeline0 {
            state Controller0 {
                state ExternalModule {
                    [*] --> StartExternalModule
                    EndExternalModule --> [*]
                }
            }
        }
    }
    state Server1 {
        Receiver --> Module
        Module --> Sender
    }
    StartExternalModule --> Receiver
    Sender --> EndExternalModule
Loading

@JulianKropp JulianKropp pinned this issue Jul 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant