Proposed alternate prompting scheme #241
Conversation
|
So this matches my intuition around how to put all of this together. While we have no way of testing for a regression, I'd be okay with running with something like this for now. The thing I'll note is that when @ahejlsberg tried something like this with the system prompt, he got slightly worse results. That said, he guessed part of this was probably tied to the schema falling further and further behind in history. So maybe something to alleviate this would be to instead do:
Basically, move the system prompt immediately before the user request.
No worries. To most quickly fix the build: cd TypeChat/python
pip install -e .[dev,examples]
pytest --snapshot-update
git add tests
git commit -m "Update snapshots." |
|
It probably also makes sense to make the repair prompt a |
|
One of the things I tried in https://github.com/microsoft/TypeChat/tree/systemPromptPython (ea0e0d8) was to take the approach here after fixing some issues. As @ahejlsberg mentioned, OpenAI unfortunately doesn't do a great job when you switch to the system prompts. Trying things out on the coffee shop example input file, it generally got more "wrong". One idea we could try is to expose the (very basic) logic we use to parse out JSON so that people can override translate and do whatever they want. But otherwise, the only other approaches I could advise to implement something like #238 is to experiment with preambles to talk about past state. |
|
Thanks for the feedback! Preamble issuesMy problem with the preamble is that it is prefixed to the prompt containing the schema, which makes the conversation a bit awkward, as it goes like this (ignoring system messages):
What I want is this (because my intuition tells me that it's a more natural dialog):
About system messages:My (rudimentary) understanding of system messages is that it's meant to set a behavior or personality that's hard(er) to subvert by users trying to be clever -- e.g. the system prompt would tell the model not to be racist. In TypeChat's case, I think that at best the prompt that sets the schema should be a system prompt, but the repair prompts should probably just be user prompts. The docs say a little bit about this. It makes me think that it's not worth experimenting too much with this in particular. I also just noticed "JSON Mode" in the docs, which may or may not be useful for TypeChat. |
|
@gvanrossum FWIW, a while back I did a multi-turn experiment where I didn't include previous replies in the prompt, but rather just repeated the request with the additional user input concatenated to the the previous user input (separated by newlines). So, your example would have a single It seemed to work quite well. Wonder if that would work in your case? |
The reason we aren't using JSON Mode is that there is no good way to know upfront if a model supports it, and, unfortunately, older models don't know to ignore it, but instead error on the request. Also, not getting JSON back from requests hasn't really been an issue. |
This is my proposal to address #39. It also fixes the issue I flagged in #240 about the user input being repeated twice. I tried it with my demo from #238 and that seems to work.
The prompt structure is as follows:
I don't understand how your test tools work (I'm a dinosaur, I don't know how hatch or toml or pipenv work, I can handle pytest though).
@DanielRosenwasser