Julia native engine. Environment caching?

[dmi3kno]

Description

I am writing a small package which is supposed to extend Distributions.jl. The structure of my package is

Project.toml
Manifest.toml
README.qmd
src/
|- NewDist.jl

The content of my NewDist.jl

module NewDist

using Distributions

struct DistA{T<:Real} <: ContinuousUnivariateDistribution
    μ::T
    σ::T
    DistA{T}(μ::T, σ::T) where {T<:Real} = new{T}(μ,σ)
end

export DistA

end # module NewDist

The Project.toml is unremarkable

name = "NewDist"
uuid = "7682a1b4-220e-4e99-81b7-e147a8ccc904"
authors = ["MyNickName <[email protected]>"]
version = "0.1.1"

[deps]
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"

Manifest is auto-generated so I am not providing it.
I am writing a vignette in Quarto (content of README.qmd)

---
title: "Vignette"
engine: julia
julia:
  exeflags: ["--project=@."]
---

Here's my first distribution
```{julia}
using NewDist
d1 = DistA{Float64}(4.1,   5.2)
```

Before I render, I activate my local environment and make sure that Distributions is installed, and my package is installed

import Pkg; Pkg.activate(".")
Pkg.add("Distributions")
using NewDist

I am ready to render my README.qmd. Everything goes well and I get an output as expected.

Now, I feel excited and want to add another distribution. I modify the NewDist.jl to include another class. Modified NewDist.jl looks like this

module NewDist

using Distributions

struct DistA{T<:Real} <: ContinuousUnivariateDistribution
    μ::T
    σ::T
    DistA{T}(μ::T, σ::T) where {T<:Real} = new{T}(μ,σ)
end

struct DistB{T<:Real} <: ContinuousUnivariateDistribution
    μ::T
    β::T
    DistB{T}(μ::T, β::T) where {T<:Real} = new{T}(μ,β)
end

export DistA, DistB

end # module NewDist

I want to also include it in my vignette, so I modify README

---
title: "Vignette"
engine: julia
julia:
  exeflags: ["--project=@."]
---

Here's my first distribution
```{julia}
using NewDist

d1 = DistA{Float64}(4.1,   5.2)

```
And a second distribution

```{julia}
d2 = DistB{Float64}(4.1, 0.5)
```

Nothing has been changed in the YAML. I just added another Julia chunk. I try to render and I get an error

#>Error 1 of 1
#> @ /home/dm0737pe/Projects/NewDist/Readme.qmd:15
#> UndefVarError: `DistB` not defined

I start to panic. I try to install new version of the package in command line (using NewDist), I check the source code. I even run the Quarto file cell by cell. Everything works as expected. But Quarto rendering fails.

Now I modify only one option in YAML: exeflags: ["--project=."] and re-render, and everything works! In fact it does not matter what the option is --project=. or --project=@. I tried running with both. What matters is that it is changed. No changes in the document help (after all I added a new chunk), but the change in exeflags refreshes the status of my package and picks up the new class definition.

I think what is happening is that QuartoNotebookRunner.jl is using a session, where it caches the local (/NewDist) environment and does not refresh it until something as fundamental as exeflags are changed. Then QuartoNotebookRunner.jl reboots the cache and catches up to the current status of the local environment.

I dont know where QuartoNotebookRunner,jl keeps the cache, but I had a situation where the error kept referring me to non-existent line in the source code (which I know I deleted). Rebooting of Julia console in VSCode does not help. Only when I touched exeflags was the environment synchronized.

Sorry about lengthy description. I was trying to be very explicit in the steps I took. I understand that QuartoNotebookRunner,jl tries to reuse the existing session, but I think reproducibility of Quarto requires that the old session is dropped and the environment is compiled anew (even if that might cost a few extra seconds). Or perhaps there could be another flag which triggers the reuse of the old session to save compiling time (I guess my situation with active package development may be quite special).

[cderv]

I think what is happening is that QuartoNotebookRunner.jl is using a session, where it caches the local (/NewDist) environment and does not refresh it until something as fundamental as exeflags are changed. Then QuartoNotebookRunner.jl reboots the cache and catches up to the current status of the local environment.

@jkrumbiegel any insight on this ?

And any thoughts on the use case here ?

QuartoNotebookRunner.jl is using a session

@dmi3kno there is indeed a concept of worker that is started. You should be able to control the reuse of the worker, and even opt-out of it to always start fresh worker: https://quarto.org/docs/computations/julia.html#worker-process-reuse

It should work to start a new worker each time (though I found issue recently on windows with it : #10353)

I am thinking this worker restart could help your work case here.... Though @jkrumbiegel will tell us more on how QuartoNotebookRunner.jl is working.

By the way, thanks a lot for testing Quarto julia's engine ! We need more feedback like this to improve it and document it correctly, so this is really valuable !

[dmi3kno]

I think the behavior of the Julia daemon needs to be described more clearly in Quarto documentation. For example, changing UUID in my Project.toml also helps refresh the daemon (sync to the current state of the environment). I understand that every completed iteration of the package might need to have its version and UUID bumped, but this is a little too much to ask in active package development.

I really miss the Rstudio "Build" panel for Julia, with explicit "Check" and "Install" buttons. I guess in Julia it would be "Install to" with an option to select an environment to install the package into.

[dmi3kno]

I confirm that using

execute:
  daemon: false

helps on my Ubuntu 22.04 machine. Thanks for the tip! Sorry about verbose example. I realized I could make my reprex smaller.

[jkrumbiegel]

QuartoNotebookRunner only restarts workers forcefully if something fundamental like the exeflags changes, which cannot be adjusted (for example you cannot change the number of threads in a running Julia process). For code changes in dev'ed packages that you're loading in there you're expected to use Revise which is the canonical Julia solution to this problem. I think if Revise is on your load path (usually that means it's installed in your local env or in your global one), it should be loaded automatically. Could be that this is not documented, yet, in this case it should be added.

Here's one of our test files that does this with a local env, but I don't think Revise has to be in a local env, as I said above https://github.com/PumasAI/QuartoNotebookRunner.jl/blob/56794a34b99f954d6fdd422fb0419d53322d709f/test/examples/revise_integration.qmd

[cderv]

Thanks @jkrumbiegel !