Commits on Feb 14, 2024

1. Add support for dataset load with multiple processes …

   Combining dataset loading (--pinecone-dataset) and multiple processes
   (--processes) does not currently work due to interactions between
   multithreading used by google.cloud.storage to download dataset files,
   and fork()ing done by locust to create multiple processes.
   
   Fix this by only performing Dataset downloading in the parent process
   when locust is first started ('init' event), and having the child
   processes only read the already-downloaded dataset files later when
   the test starts ('test_start' event). This also avoids any unnecessary
   / conflicting download of the same data multiple times.

   

   daverigby committed Feb 14, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 85847c5
   • Browse repository at this point

   Copy the full SHA

   85847c5 View commit details

   Browse the repository at this point in the history