Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Offline Pipeline] Batch processing #15

Open
sarahwooders opened this issue Dec 1, 2021 · 0 comments
Open

[Offline Pipeline] Batch processing #15

sarahwooders opened this issue Dec 1, 2021 · 0 comments

Comments

@sarahwooders
Copy link
Contributor

Offline processing can be made more efficient by batch-processing data, so we should offer an option to define a batch update function for the operators.

  • user-defined on_records(self, records: List[Record]) → List[Record]
  • set batch size for operators
class Operator: 

  def __init__(self, batch_size): 
    self.batch_size = batch_size

  # user defined transformation   
  def on_record(self, record: Record): 
      pass 

  # can be overwritten by user 
  def on_records(self, records: List[Record]): 
    for record in records: 
      self.on_record(record) 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant