Bad draw error when running chunks

[gibsramen]

I ran into an issue when fitting multiple single features in a chunked fashion. The chains finish but there seems to be an error with the draw dimensions. I tried the errant feature individually and it worked as intended so there may be some file-system/HPC issue at hand. This issue does not occur for all chunks so my guess is it is not related to the BIRDMAn code directly.

for feature_num in range(start_num, stop_num):
    print(f"Feature: {feature_num}")
    if feature_num > num_feats:
        print("Over the table size")
        exit()
    feature_id = fids[feature_num]
    outdir = f"/panfs/grahman/birdman-analyses/speed/outdir/{feature_num}_{feature_id}"
    os.makedirs(outdir, exist_ok=True)

    model = NegativeBinomialSingle(...)
    model.compile_model()
    model.fit_model(sampler_args={"output_dir": outdir})

    inf = model.to_inference_object()
    print(inf.posterior)
    print(az.loo(inf, pointwise=True))

INFO:cmdstanpy:found newer exe file, not recompiling
INFO:cmdstanpy:compiled model file: /home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/birdman/templates/negative_binomial_single
INFO:cmdstanpy:start chain 1
INFO:cmdstanpy:start chain 2
INFO:cmdstanpy:start chain 3
INFO:cmdstanpy:start chain 4
INFO:cmdstanpy:finish chain 4
INFO:cmdstanpy:finish chain 3
INFO:cmdstanpy:finish chain 2
INFO:cmdstanpy:finish chain 1
Traceback (most recent call last):
  File "src/speed/run_birdman_chunked.py", line 46, in <module>
    model.fit_model(sampler_args={"output_dir": tmpdirname})
  File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/birdman/model_base.py", line 164, in fit_model
    **sampler_args
  File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/model.py", line 835, in sample
    mcmc = CmdStanMCMC(runset, validate_csv, logger=self._logger)
  File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/stanfit.py", line 433, in __init__
    self.validate_csv_files()
  File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/stanfit.py", line 700, in validate_csv_files
    thin=self._thin,
  File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 464, in check_sampler_csv
    meta = scan_sampler_csv(path, is_fixed_param)
  File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 520, in scan_sampler_csv
    lineno = scan_sampling_iters(fd, dict, lineno)
  File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 781, in scan_sampling_iters
    lineno, num_cols, len(line.split(','))
ValueError: line 80: bad draw, expecting 71969 items, found 73695
deleting tmpfiles dir: /tmp/tmptufa9zy5
done

cc @mortonjt