[Bug]: Cannot reliably interface with local LLM when agent is built from source

[avi12]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Describe the bug and reproduction steps

openhands_2025-02-07.log
prompt_008.log

OpenHands Installation

Other

OpenHands Version

Built from source

Operating System

WSL2 on Windows 10

Logs, Errors, Screenshots, and Additional Context

No response

[enyst]

From the log, I don't see why does it immediately try to close the session it just opened

Ah, it was another session. Still weird, but differently weird. 😅

00:57:06 - openhands:DEBUG: agent_session.py:138 - Waiting for initialization to finish before closing session 556607007d764f6a910c42475d6947bf

LEVEL 0 LOCAL STEP 0 GLOBAL STEP 0

00:56:57 - openhands:INFO: standalone_conversation_manager.py:83 - Conversation 2f71d095de6d433b93ca58fc2e7539fb connected in 0.05562305450439453 seconds
00:56:57 - openhands:INFO: standalone_conversation_manager.py:64 - Reusing active conversation 2f71d095de6d433b93ca58fc2e7539fb
...
00:57:06 - openhands:DEBUG: agent_session.py:138 - Waiting for initialization to finish before closing session 556607007d764f6a910c42475d6947bf
...
00:57:28 - openhands:INFO: agent_controller.py:451 - [Agent Controller d7081bdd42c54dcda9606def5ca917e0] Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.ERROR

@avi12 does this happen if the only change you make is to use another model, like some hosted model? (you can use the link on the home page to get back to the last conversation)

[avi12]

I went back and forth from source to Docker commands and switched between versions so I don't have the original conversation
I tried to run from source with the config

[core]
workspace_base="/mnt/c/repositories/extensions/kimai-google-calendar"
debug=true

[llm]
model="lm_studio/qwen2.5-coder-7b-instruct"
base_url="http://host.internal.docker:1234/v1"
api_key="lm-studio"

https://huggingface.co/lmstudio-community/Qwen2.5-Coder-7B-Instruct-GGUF
It doesn't seem like it can communicate with the local server that LM Studio sets up
I also tried to change the base_url="http://127.0.0.1:1234/v1", and then it says that it cannot communicate with Docker

[avi12]

The result I got is

https://www.all-hands.dev/share?share_id=3b5b807947983fb811cedfc1b7c0035d378634e3402a9f7e8aae71cc1804fb53

[enyst]

Did you start the LM Studio server and is it on port 1234?

[avi12]

Literally when I try to run OH in 0.22.0 or 0.23.0 it can reliably plug into the LM Studio server
It's only when I build from its source that it cannot

[enyst]

Ah, okay! Can you try to enter base_url and the rest of the settings in the UI, regardless of whether they are in the toml too?

[avi12]

pretty much all set

[avi12]

When I manually try to set the "API key" as lm-studio, I get #6645

[enyst]

Could you please try to use the prefix "openai/" instead of lm_studio for the model name?

You need to set some api key in the UI, too, I believe, but I don't think it matters which. "lm-studio" should work.

[avi12]

Are you trying to make me use an OpenAI model with OH?

[avi12]

Interestingly, in the logs when running from the source I see

Message: 'litellm.APIError: APIError: Lm_studioException - Connection error.. Attempt #6 | You can customize retry values in the configuration.'
Arguments: ()

I wonder why this discrepancy exists, considering that this exact configuration works when I run the Docker command

[enyst]

Are you trying to make me use an OpenAI model with OH?

No, it doesn't mean openai models. "openai/" is a prefix that litellm recognizes for any models with the meaning openai-compatible format. ("this provider at this base_url serves model with an openai-compatible format")

Interestingly, in the logs when running from the source I see

Message: 'litellm.APIError: APIError: Lm_studioException - Connection error.. Attempt #6 | You can customize retry values in the configuration.'
Arguments: ()

I wonder why this discrepancy exists, considering that this exact configuration works when I run the Docker command

When running the docker command, the connection is from inside the docker app container. Now it's direct, please try to use localhost. Please check here (for ollama, but it should be similar for LM Studio):
https://docs.all-hands.dev/modules/usage/llms/local-llms

[enyst]

See for example, here:

• OpenAI-compatible endpoints: https://docs.litellm.ai/docs/providers/openai_compatible
• LM Studio documents their openai-compatibility: https://lmstudio.ai/docs/api/endpoints/openai

So you can call LMStudio LLMs with the prefix /openai, the rest of the model name as in LM Studio, the correct base_url, and litellm can figure it out.

[avi12]

Thus far I struggled with SmartManoj#258 (comment)

[enyst]

This becomes difficult to follow, sorry. Let's look at it this way: you said in that with docker run things worked. Please do try to take into the account the docs linked above, on local llms / Ollama, it highlights the differences in building from source vs docker run, in particular the actual URL. I know I set up LM-Studio in the past with the same process, it exposes similar functionality.

[avi12]

After some digging I discovered that the issue stems from me running Windows 10 (and that due to hardware limitations, I cannot run Windows 11)
This results in WSL2 being unable to send requests to localhost
My only workaround is to use a service like Ngrok

[Valve-too]

i get this issue too when using OpenAI (o3) with Openhands. so its not just related to local.

docker run -it --rm --pull=always `
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE="docker.all-hands.dev/all-hands-ai/runtime:main-nikolaik" `
    -e LOG_ALL_EVENTS=$true `
    -v "/var/run/docker.sock:/var/run/docker.sock" `
    -v "$env:USERPROFILE\.openhands:/home/openhands/.openhands" `
    -v /mnt/d/Github/projectrush:/opt/workspace_base `
    -p 3000:3000 `
    -p 9000:9000 `
    --add-host "host.docker.internal:host-gateway" `
    --name "openhands-app" `
    "docker.all-hands.dev/all-hands-ai/openhands:main"

[avi12]

It seems like the latest commit (1afe7f1) is also problematic, the agent can't get ready

[enyst]

It seems like the latest commit (1afe7f1) is also problematic, the agent can't get ready

That commit didn't cause anything on the local installations, though, it's only for the remote runtime which is not in use with a local docker. Maybe an older commit/issue.

Did you run normally with the development mode before updating to the newest commit?

[avi12]

I mean I already pulled the latest commit

[avi12]

If you really want, I can use git bisect to get to the latest working commit

[avi12]

Never mind, it was eventually able to start but it took its time, similar to #5813