Launching threads/async in DSPy 2.5.30+ without DSPy threading primitives

[italianconcerto]

import threading
import dspy
import os

lm = dspy.LM(
    "openrouter/meta-llama/llama-3.1-70b-instruct",
    api_base="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
    temperature=0.2,
    cache=False
)
dspy.settings.configure(lm=lm)

predict = dspy.Predict("question -> answer")

def run():
    response = predict("Paris.")
    print(response)

output = run()
print(output)

If I run this I get something like:

Exception in thread Thread-1 (run):
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/johnny_silicio/test/async_dspy.py", line 17, in run
    response = predict("Paris.")
               ^^^^^^^^^^^^^^^^^
  File "/Users/johnny_silicio/test/venv/lib/python3.12/site-packages/dspy/utils/callback.py", line 198, in wrapper
    callbacks = dspy.settings.get("callbacks", []) + getattr(instance, "callbacks", [])
                ^^^^^^^^^^^^^^^^^
  File "/Users/johnny_silicio/test/venv/lib/python3.12/site-packages/dsp/utils/settings.py", line 65, in __getattr__
    if hasattr(self.config, name):
               ^^^^^^^^^^^
  File "/Users/johnny_silicio/test/venv/lib/python3.12/site-packages/dsp/utils/settings.py", line 62, in config
    return self.stack_by_thread[thread_id][-1]
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 6185250816

It's like this with either python 3.10 and 3.12.

[okhat]

Hey @italianconcerto ! I don't think the code you posted will result in the error you posted.

However, if you're using threading or async then it may happen. Can you share more about how you're setting up threading or async?

[okhat]

I relaxed that error, so you'll get a warning now if you upgrade to DSPy 2.5.31 which is now released.

The supported way to create threads or async threads is to use launchers in dspy, like: dspy.asyncify, dspy.Parallel, dspy.Evaluate, program.batch, etc.

For async in particular, here are new docs on deployment with async/FastAPI: https://dspy.ai/tutorials/deployment/

[italianconcerto]

I confirm that the error came after I ran that script but I also confirm that I tried to run an multi-threaded program with dspy. I assume so that the first run impacted the second? Weird behaviour

[italianconcerto]

@okhat Thanks, I couldn't find it in the docs before

[okhat]

Thank you @italianconcerto ! Did this change resolve the issue?