Using SacessOptimizer on objectives with parameter priors leads to pickle error

[stephanmg]

Bug description
Using a benchmark model with parameter priors leads to pickle error, see log below.

Expected behavior
No pickle error and parameter estimation should start.

To reproduce
Pick for instance the Schwen_PONE2014 benchmark model try to load and estimate with scipy or sacess optimizer.

"Remedy": Remove parameter priors from parameters.tsv works, but obviously then runs parameter estimation without prior information which is not desirable.

Latest confirmed working version of pyPESTO was 0.4.0 for me, might be the starting point of a git bisect.

Environment

• Operating system: Unicorn cluster
• pypesto version: 0.5.3
• Python version: 3.11.1

Log:

main()
Traceback (most recent call last):
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 303, in <module>
    main()
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 286, in main
    problem, result = run_optimization(
                      ^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 112, in run_optimization
    result = optimizer.minimize(
             ^^^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/myvenv/lib/python3.11/site-packages/pypesto/optimize/ess/sacess.py", line 253, in minimize
    p.start()
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object '_prior_densities.<locals>.log_f'
Traceback (most recent call last):
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 303, in <module>
    main()
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 286, in main
    problem, result = run_optimization(
                      ^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/optimize_new.py", line 112, in run_optimization
    result = optimizer.minimize(
             ^^^^^^^^^^^^^^^^^^^
  File "/home/sgrein/optimizer_benchmark_sacess_reimpl/myvenv/lib/python3.11/site-packages/pypesto/optimize/ess/sacess.py", line 253, in minimize
    p.start()
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/ohpc/pub/spack/develop/opt/spack/linux-rocky8-zen2/gcc-12.2.0/python-3.11.1-efo4f5x5f4rnsqletfnwby5wslhnru2o/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object '_prior_densities.<locals>.log_f'

[dweindl]

This is specifically a problem with SacessOptimizer.

This problem was probably introduced in #1353 (pypesto 0.5.0).

As a quick workaround, you can try using SacessOptimizer(..., mp_start_method="fork") if supported in your setup. This might potentially cause other issues, but it worked for me in a simple test.

For a proper fix, SacessOptimizer needs to use cloudpickle instead of pickle, as is already the case in MultiProcessEngine.

[stephanmg]

Workaround works for me.