Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: generate prompts from datasets #2

Merged
merged 20 commits into from
Feb 21, 2025
Merged

feat: generate prompts from datasets #2

merged 20 commits into from
Feb 21, 2025

Conversation

XuehengWang
Copy link
Collaborator

  1. Generate prompts from datasets, specified in datasets-config-file
  • We can load datasets from both local files and Hugging Face datasets.
  • Round-robin selection based on each dataset's selection ratio.
  • Requests can be randomly shuffled or maintain their original order.
  • Added some example configuration files in the datasets_config folder
  1. In SyntheticTraceGenerator, add random and Poisson-distributed timestamps

  2. Unit test for data_loader

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this file for?

"select_ratio": 6
},
"dataset_2": {
"file_name": "../datasets/GPT4_coding_sample.csv",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our repo don't actually have this file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me test loading from local files using a different approach :)

@future-xy future-xy merged commit b1a1526 into main Feb 21, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants