Native Julia engine not working: "ERROR: Internal julia server error"

[mcanouil]

Description

Using the latest development version and a very simple document:

---
format: html
engine: julia
---

```{julia}
using Plots
plot(sin, 0, 2π)
```

The above produces:

Running [1/1] at line 6:  using Plots
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd:6
ArgumentError: Package Plots not found in current path.
- Run `import Pkg; Pkg.add("Plots")` to install the Plots package.
Stacktrace:
  [1] macro expansion
    @ ./loading.jl:1772 [inlined]
  [2] macro expansion
    @ ./lock.jl:267 [inlined]
  [3] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
  [4] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
  [5] invoke_in_world
    @ ./essentials.jl:923 [inlined]
  [6] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
  [7] eval
    @ ./boot.jl:385 [inlined]
  [8] (::Main.var"#14#28"{Module, Expr})()
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:309
  [9] (::Main.IOCapture.var"#4#7"{DataType, Main.var"#14#28"{Module, Expr}, IOContext{Base.PipeEndpoint}, IOContext{Base.PipeEndpoint}, IOStream, Base.PipeEndpoint})()
    @ Main.IOCapture ~/.julia/packages/IOCapture/Rzdxd/src/IOCapture.jl:161
 [10] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:515
 [11] with_logger
    @ ./logging.jl:627 [inlined]
 [12] capture(f::Main.var"#14#28"{Module, Expr}; rethrow::Type, color::Bool, passthrough::Bool, capture_buffer::IOBuffer)
    @ Main.IOCapture ~/.julia/packages/IOCapture/Rzdxd/src/IOCapture.jl:158
 [13] include_str(mod::Module, code::String; file::String, line::Int64)
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:293
 [14] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
 [15] invokelatest
    @ ./essentials.jl:889 [inlined]
 [16] #7
    @ ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:164 [inlined]
 [17] with_inline_display(f::Main.var"#7#24"{String, String, Int64}, cell_options::Dict{String, Any})
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:53
 [18] _render_thunk(thunk::Function, code::String, cell_options::Dict{String, Any})
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:184
 [19] render(code::String, file::String, line::Int64, cell_options::Dict{String, Any})
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:161
 [20] top-level scope
    @ none:1

ERROR: Internal julia server error

Stack trace:
    at writeJuliaCommand (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:663:13)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async executeJulia (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:511:20)
    at async Object.execute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:137:16)
    at async renderExecute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:232:25)
    at async renderFileInternal (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:592:35)
    at async renderFiles (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:325:9)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-shared.ts:102:18)
    at async renderForPreview (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:422:24)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:166:22)

The error is very cryptic as Plots is installed.

I am on macOS (latest).
Using Julia 1.10.2 (installed via Homebrew).

Quarto private environment:

~/Library/Caches/quarto/julia
├── Manifest.toml
├── Project.toml
└── julia_transport.txt

I don't see anything obvious here.

Even the following document does not work:

---
format: html
engine: julia
---

```{julia}
versioninfo()
```

Running [1/1] at line 6:  versioninfo()
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd:6
UndefVarError: `versioninfo` not defined
Stacktrace:
 [1] top-level scope
   @ ~/Projects/quarto/quarto-playground/index.qmd:7

ERROR: Internal julia server error

Stack trace:
    at writeJuliaCommand (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:663:13)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async executeJulia (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:511:20)
    at async Object.execute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:137:16)
    at async renderExecute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:232:25)
    at async renderFileInternal (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:592:35)
    at async renderFiles (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:325:9)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-shared.ts:102:18)
    at async renderForPreview (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:422:24)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:166:22)

While it works via Jupyter

Edit: I upgraded to 1.10.3 but I've got the exact same results.

[mcanouil]

@MichaelHatherly and @jkrumbiegel (cc @cscheid) if you have any pointers here.

[jkrumbiegel]

Two different things going on here:

I assume you have Plots installed in your main environment @v1.10. We have configured QuartoNotebookRunner to use the directory of the notebook as the default environment because this encourages using separate environments for separate projects. If you really want to use your main env from a notebook (which again is not advisable as you're bound to later make changes to it which break your notebook) you can set this in the frontmatter:

julia:
    exeflags: ["--project=@v#.#"]

The versioninfo thing is happening because Julia pulls in some stdlib functions only when it's used in interactive mode (with a REPL). The same thing happens when you directly run some code with the julia binary:

% julia -E 'versioninfo()'
ERROR: UndefVarError: `versioninfo` not defined
Stacktrace:
 [1] top-level scope
   @ none:1

In this case it's the module InteractiveUtils so you'd have to call using InteractiveUtils first:

julia> @which versioninfo
InteractiveUtils

[jkrumbiegel]

perhaps the timeouts aren't quite working as expected?

Could be, there are some tests on the QuartoNotebookRunner side but not integration tests with quarto

[jkrumbiegel]

DO you know how to kill that worker?

sorry, missed that question:

Using --execute-daemon-restart with the quarto render command or setting daemon: false for one invocation under the execute key in the frontmatter will restart the worker.

You cannot kill the main server process unless you find out its pid with --execute-debug and kill it yourself

[mcanouil]

Restarting the daemon did the trick, thanks!

I'll keep that in mind as some users might need this when troubleshooting issues with the Julia engine.

[mcanouil]

I'll probably make a PR to improve the documentation and ping you there for a review.

For the time being:

• Native Julia engine setup and documentation #9561

Thanks for the help!

[dmi3kno]

I am developing a Julia package and I want to use Plots in vignette (README.qmd), but my package does not need to depends on Plots. How do I make it explicit, that I am relying on system installation (@v1.10) of Plots, but the rest of the packages should be used from local package environment? Sort of like "Suggests" in R.

[cderv]

@dmi3kno I am not sure how this is relating to Quarto. Can you share more example ? You could open a new discussion as it seems quite specific and not directly related to this initial discussion.

We've recently updated our doc at https://quarto.org/docs/computations/julia.html#notebook-environments where --project flag can be set to indicate QuartoNotebookRunner.jl to use a specific project while running the Julia cells in a qmd document.

I would think setting that in your README.qmd would be enough to use a different environment.

Let's discuss in a new thread maybe.

[dmi3kno]

It's probably not related to Quarto, I agree. It is a question about Julia environments, potentially connected to Quarto by --project=... exeflag. It feels like I want to execute notebook in an environment where both Plots and my new package are installed, but, unfortunately, the way I do it now (running README.qmd from the root of the package) means that Project.toml has to contain Plots. Perhaps the best practice for package development is to create a new environment in, say, docs subfolder and add both my newly developed package and Plots there. I need to think a little about it. I will open an issue when I am ready to ask a meaningful question.