qproc failed on r4 on exposure 134116

[npadmana]

exp 134116 (flat) had an error in Qproc (on nightwatch) on r4. The exposure looks fine - the error looks like an error in processing (in qproc_compute_fiberflat). The error in the log appears to be in the following lines :

NFO:qfiberflat.py:49:qproc_compute_fiberflat: Starting...
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=20.0, retrying with 60.0</span>
<span style="color:orange;">WARNING:linalg.py:114:spline_fit: spline fit failed with resolution=5.0, retrying with 15.0</span>
<span style="color:red;">ERROR:linalg.py:117:spline_fit: spline fit failed</span>
<span style="color:black;"><mark>Traceback (most recent call last):</mark></span>
File "/software/datasystems/desiconda/20200924/code/desispec/master/py/desispec/linalg.py", line 109, in spline_fit
toto=scipy.interpolate.splrep(input_wave,input_flux,w=input_ivar,k=order,task=-1,t=knots)
File "/software/datasystems/desiconda/20200924/conda/lib/python3.8/site-packages/scipy/interpolate/fitpack.py", line 288, in splrep
res = _impl.splrep(x, y, w, xb, xe, k, task, s, t, full_output, per, quiet)
File "/software/datasystems/desiconda/20200924/conda/lib/python3.8/site-packages/scipy/interpolate/_fitpack_impl.py", line 516, in splrep
raise _iermess[ier][1](_iermess[ier][0])
ValueError: Error on input data

[sybenzvi]

Quickly confirming that this is not simply a processing glitch. The error was reproduced at KPNO and NERSC and during manual reprocessing.

[jose-bermejo]

It seems in line 110 of qfiberflat.py there's an iterative spline fit to smooth the flat in which the resolution is multiplied by 3 if the fit fails. According to the logfile the code wasn't able to fit it.

I've tried to change the limit to a higher resolution (e.g. 10 times the original resolution) and this way the error disappears. However, changing it would be a modification to desispec and I guess we want to be sure this fix wouldn't cause another problems.

[sbailey]

qfiberflat is not used by the offline pipeline so should be safe to update for nightwatch.

Please check the underlying data for why the spline fit is failing in the first place. This appears to be a relatively new problem and I'm not sure what changed to trigger it (or whether it has been happening for a long time and we just noticed).

[jose-bermejo]

I've pushed for the moment this branch which changes the maximum resolution factor from 3 to 5 (enough for processing this exposure). We can later try to figure out if there's something going on with the data.