Vario sound not working or freezing in flight.

[iglesiasmarianod]

I've experienced random disconnection issues with Variod and Sensord.
Sometimes, on startup, Variod does not connect to XCSoar, after restart, Variod connects but sensor data is unavailable.
I suspect Sensord not connecting to variod.
Sometimes, sensord disconnects from Xcsoar freezing vario sound on the last value received.
The only solution is to shut down and restart.
This behaviour seems to be random.

[OBrown92]

We've experienced the same issue twice yesterday.

Nothing shows up in the nmea monitor. Both times directly after switching the main power on. Restarting only xcsoar does not help, restarting the system from the menu solved it.

[iglesiasmarianod]

Hi guys, I'm still experiencing this issues even with the latest image built from master branch.
Vario sound freezes, monitor shows nothing or a couple of sentences.
Sometimes vario recovers after a while and it crashes again.
Is anybody else experiencing this? It renders variometer useles because it is very unreliable.

[iglesiasmarianod]

I've been testing yesterday and I found two Issues regarding this bug.
One is hardware related, the other software related.
What seems to happen in my case is that RF noise is picked up by the speaker wire creating a spike in the pressure readings and crashing sensord (or glitch correction runs away and never recovers, I can't tell right now). @hpmax?
XCSoar shows "no data" and vario sound stays on, fixed on the last valid value received.
RF noise in my case is produced by 132 to 134 mhz frequencies. Lower frecuencies does not seem to affect the sensors.
Unplugging the speaker makes the RF interference go away.
Sensord V 0.3.3 picks up the RF noise sendig the vario to 99+ climb but never crashes. It is annoying, but it's usable.
When the failure occurs, is pretty much the same every time:

1. Press PTT.
2. Vario goes wild.
3. Old sensord (0.3.3) recovers everytime, master branch sensord recovers 10% (or less) of the time.
4. Variod stops working, XCSoar monitor goes blank. After a couple of minutes one or two NMEA sentences arrive and it stops again.
5. Only possible recovery is restart.

@mihu-ov, Is it possible that you posted something about RF interference and speaker wiring some time ago?

[DanD222]

I had a similar issue wth an older vario (radio transmission would generate a vario reading, the magnitude would depend on the radio frequency).

The fix was to solder 1000 pF capacitors from every external connector to ground.

[mihu-ov]

MS5611 is known to be sensitive to RF interference, there´s nothing we can do about this on the sensor side other than trying to minimize the effects.

a) MS5611 power supply: each sensor has its own power supply with a Pi-filter after the supply. IIRC this was suggested from experience with MS5611 paraglider varios. Should be good in my opinion, but please crosscheck
b) Sensor cover: maybe a metal block over the sensors instead of plastics, or a metal shield over the plastic block could bring an improvement. I don´t know if anyone has tried that already
c) Reduce RF emissions: use good wiring practice in the glider (star point grounding, sepeate cables, shielding where required). Check for defective connectors and replace. There are +30yr old VHF antenna connectors out there bent a hundred times with broken shielding at the connector. Today I would solder a small class D amp directly to the OV speaker to avoid long wiring from the amp to the speaker
d) Improve gating / rejection of incorrect data in sensord to avoid runaway of the Kalman filter. Maybe detect runaway and reset the Kalman filter, or have another Kalman filter on delayed sensor data and switch to this in case of a runaway.

IMO most potential is in c) - Reduce RF emissions. My wiring is far from perfect but I have no issues with sensor data or Kalman filter runaway.

[DanD222]

Today I would solder a small class D amp directly to the OV speaker to avoid long wiring from the amp to the speaker

Not sure if I understand correctly, but I'm not sure if I'd do that myself - if you place the amp at the speaker, you'll be amplifying any interference that gets picked up along the way, plus you'd need to run power to the amp, which could also pick up interference.

Speaking of wiring, what I've been doing during re-wires is using a shielded 2-core cable, where the 2 cores are + and -, and the shield is connected to - at one end only (on the OV side, or to the star ground). This might help too.

[mihu-ov]

Yes, you´d need to run power to the amp, no real issue in my opinion. Yes, you could pick up interference but I doubt this is a real life issue. We are not talking Hi-Fi here, one could use a thin but shielded cable, and if required the amp input resistance could be reduced.
It´s the amp output to speaker wires that can cause RF interference at the vario, and these would be reduced to almost zero length.

[iglesiasmarianod]

Thanks guys for chiming in.
In my case all interference ceases when I unplug the speaker. I'll try to make the lead shorter.
What do you think about a capacitor in series or two ferrite beads in series on the wires to filter those frequencies?

On the software side, Variod seems to keep working but not receiving data.
If I'm not mistaken the kalman filter wasn't changed (release 0.3.3 works).
Max fixed some whitespace but no important algorithm change was performed.
The biggest change was made trying to compensate for glitches. @hpmax worked on it.
My suspicion is that glitch compensation runs away or cannot sense the end of the "glitch" event, thus crashing or entering an infinite loop somewhere.
I'm struggling trying to understand the code logic and the values used in the formulas to see if there's something I can do to fix it.

[DanD222]

What do you think about a capacitor in series or two ferrite beads in series on the wires to filter those frequencies?

Capacitors in series block DC / low frequencies from getting to the speaker, which is not what you're after; what you want are 'shunt' capacitors from signal to ground, to provide a path for any high frequency crap.

I would try this, plus the 2 core shielded cable, if you're going to do any re-wiring.

[iglesiasmarianod]

You are right about the cap!
I've used shielded wire with the shield connected to a ground post. It made no noticeable difference.
I did not understand which wire to ground through the cap. Shield or signal.
I'm using a 3 contact jack. I might be able to ground the shield there and make it part of the ground plane. Another thing I noticed is that the original sensorboard does not have a direct chassis ground. Do you think that might help?

[hpmax]

I'll be blunt and say I've lost all interest in supporting the MS5611. Every attempt I made to try to fix its problems in software seemed to generate new problems. Maybe could I do better starting from scratch, but maybe not.

The temperature compensation system is imperfect and it self heats. As a result temperature on the chip changes as a function of polling frequency. If the controller locks up (which it seems prone to do depending on software configuration) the temperature on the 5611 starts falling and then the next pressure reading is pretty far off. It then asymptomatically approaches a steady state value as normal polling resumes. It wouldn't shock me if there is RF sensitivity as well.

As Dan points out, putting a 1-1.5nF cap on all outward facing lines is probably the best solution. If the cap is larger than that the self resonant frequency is too low and it won't look like a capacitor at the VHF radio frequencies we use.

Dan and I's replacement board is a better built board (4 layer) with tons of via stitching, 1nF caps everywhere, and an ST sensor (the latest of which is probably higher performance than the 5611) which has a free running oscillator to drive the conversion.

As for a Class D. I agree that in theory if you have a Class D, you want it as close to the speaker as possible, but a Class AB will always give cleaner results.

[iglesiasmarianod]

Hi @hpmax. I understand you lost interest in supporting MS5611, but being the writer of the code, I can't think of anyone better to fix what's been broken.
In my opinion, I'd rather live with a working vario with blips than a blipless vario that dies midflight.

I find your code hard to understand, with a lot of constants that are not documented and a lot of math operations that are not documented either. This modification is carried out through several commits that change many unrelated things and I don't know what the effects of reverting them all would be.

You could make the blip correction optional through command line. This way, the use of it can be chosen and we can test without breaking the code.
There are a lot of old boards out there that might stop working because of this.

[DanD222]

I did not understand which wire to ground through the cap. Shield or signal.

Signal - you want a good path for high frequencies to ground, which the shield should already have by virtue of being grounded.

Another thing I noticed is that the original sensorboard does not have a direct chassis ground. Do you think that might help?

Could be worth a try. Is the case powder coated, and if so, is there a chassis connection to bare metal from circuit ground somewhere? Is the case shielding consistent throughout all of the case? That could also be something to check.

[iglesiasmarianod]

Thanks @DanD222, I'll try shorter wire and a 0.001uf cap in the signal line to ground.
My case is bare stainless steel.
All boards except sensorboard have connection between the case and their grounds through brass standoffs.
There's continuity between case and battery ground.
Sensorboard is fixed to the case with pressure fittings.
These fittings are screwed to the POM block isolating the board from the case.
The only ground connection it has is through the 10 wire flat cable.
I might be able to run a wire from any ground pin to the case.

[mihu-ov]

The amp output is differential, there is no signal or ground wire, just two speaker wires. Check the datasheet of the chip amp, maybe it has some information about reduction of RF emsissions.
Try a shielded cable, connect the shield to ground at OV side and don´t connect the shield anywhere on the speaker side. - Sorry, just noticed you have tried that already.

[DanD222]

The amp output is differential, there is no signal or ground wire, just two speaker wires

OK… I got confused by the Cinch connector :-/

In that case there should be a cap from both opamp output pins to ground, plus there should be a shield connection to case.

[mihu-ov]

Yes, the Cinch connector was a design fault in many ways ;-)

[hpmax]

@iglesiasmarianod

I'm not going to lie, I find my code hard to understand as well. I was essentially trying a series of experiments to get better performance. The idea was that once I got it working well I'd clean it up. But I found I was just adding patch over patch over patch and every attempt to make things work better would introduce new problems.

Some of the true killers were that the compensation scheme was dependent on pressure and possibly temperature. That means it's a 2 or 3 dimensional compensation scheme and likely something that couldn't be done on the ground. Another big problem was that it was hard to determine when the compensation is actually done and how to detect a retrigger event.

The intent was to beautify the code once I had something stable and working, but it was just one bandaid over another and really difficult to make clean. The end result was I felt we should abandon the original board. I can try to go through and rip out a lot of the bandaids I put in and clean it up some.

[iglesiasmarianod]

Hi @hpmax, I would skip the compensation altogether for now.
I think the best option is making the compensation optional through a command line argument.
That way all your work is not lost and we can test it if we feel to. It will take a lot less time to implement too.

[iglesiasmarianod]

Update. I started testing the least invasive option first. I added a ferrite core to the speaker cable near the jack with one turn around it. The wire I used is shielded microphone wire. The shield was not connected to ground. I also shortened the wire length to around 35/40 cm. I believe 1/4 wavelength of 125 mhz is around 60 cm so I shortened the wire to minimize de antenna effect.
So far so good. I tested on the ground with the frequencies that were creating interference and did not have a single vario glitch. I've flown for two hours testing various frequencies and the problem seems to be reduced or solved.
After this, I'll try to make sensord's glitch correction selectable.

[iglesiasmarianod]

I've been testing at home with my device and found some odd behaviour.
After vario freezes, trying to recalibrate sensors throws a "no eeprom" error.
The script stops Variod and Sensord before trying to communicate to sensorboard.
Trying to run sensorcal from command line throws the same error.
The only way I found to make it run again is to power off and on or to disconnect and reconnect sensorboard fisically.
I'm suspecting some kind of I2C blocking. This kind of disconnects started happening when OS changed from Armstrong.
The same happened to variod socket connection to XCSoar because Sock_err was not initialized.
Is anybody else having this issue?
Do you have any ideas on how to debug or sniff I2C communication?