Could not find appropriate gain for performing ADC self cal

[zhouzhiwen2000]

Issue Description

I am using UHD 4.7.0.0 for USRP X410. When loading the FPGA image, runtime error occurs: Could not find appropriate gain for performing ADC self cal.

Setup Details

| | Mboard: ni-x4xx-*********
| | module_pid: 42000
| | module_rev: 4
| | module_serial: *********
| | pid: 1040
| | rev: 6
| | rev_compat: 6
| | serial: *********
| | MPM Version: 5.3
| | FPGA Version: 8.3
| | FPGA git hash: c37b318.clean
| | Device DNA: 40020000014D9******************
| | RFNoC capable: Yes

Expected Behavior

No error occurs

Actual Behaviour

[INFO] [0/Radio#0] Clocking reconfigured, running ADC Self Cal on DB0...
[INFO] [0/Radio#0] Calibrating channel 0...
[INFO] [0/Radio#0] Calibrating channel 1...
[ERROR] [RFNOC::GRAPH] Error during initialization of block 0/Radio#0!
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: RuntimeError: Could not find appropriate gain for performing ADC self cal
Error: RuntimeError: Failure to create rfnoc_graph.

Steps to reproduce the problem

Flash the latest filesystem, reboot.
uhd_image_loader --args type=x4xx,addr=127.0.0.1,fpga=X4_200

Additional Information

[manderseck]

@zhouzhiwen2000 Thanks for reporting. This is not a generic issue with our code but it is specific to your device. Unfortunately, chances are high that you need to contact support for RMA, but let's check if this is an intermittent issue:
Shutdown the device and disconnect it from power for ~30 seconds, then turn it on and try again.
The expectation is that you see this issue with all UHD versions and FPGA images but you may of course still try with different versions.

Explanation:
The ADC self-cal tries to setup a convenient power level for saturating the ADC, so that the spur calibration can be performed properly. The cal signal is created by the DAC associated to the same channel (this is where the "self" comes from in the ADC self-cal). In X410 we increase the gain of that channel both on the TX and the RX side step by step until we reach a certain threshold. If we are unable to achieve this even at full gain, we throw the error you are seeing. Since in a healthy device we should always be able to achieve this we assume this is a hardware failure somewhere in the signal chain from the DAC via the daughterboard connector, a few switches, gain stages and back via the connector to the ADC. If you are interested to find out if the issue is rather on the TX or the RX side you may skip the self-cal for channel 1 (add cal_ch_list=023 to your args), live with a few spurs, and then send a signal on channel 1 and receive something to see if the TX is weak or the RX doesn't see a signal.

I'm closing this issue as it doesn't seem to be related to the UHD code. Please feel free to re-open if you come to different conclusions.

[zhouzhiwen2000]

Strangely, I rolled back to UHD_4.3.0.0-0-g1f8fd345, works without a problem.

UHD 4.7.0.0 works fine as well after flashing the FPGA and executing uhd_usrp_probe, it only occurs when loading the image.

Unfortunately, Labview loads FPGA image every time and this error is reported. I rolled back Labview 2022 and everything works, Labview 2024 doesn't.

So I still think it's on the software side.

[manderseck]

I think there's still a bit more to it: With UHD 4.5 we've changed the ADC self-cal strategy in our X4xx platform to accommodate the needs of X440 with its master clock rate flexibility. Before 4.5 we only performed the ADC self-cal silently during device startup. There used to be a service (service name: usrp-adc-self-cal) that was started immediately after MPM (service name: usrp-hwd). After the startup we considered the calibration to be "good enough" and didn't touch it anymore during operation. With X440 that was not possible anymore and so we started running the ADC self-cal after each clocking change, e.g. changed master clock rate or changed clock- or time-source. A bitfile update is considered as such a clocking change. Therefore, starting from UHD 4.5 you will see the ADC self-cal running here and there. For X440 this improves the spur surpression a lot, in X410 it still improves it a bit compared to the old approach.

What does that mean for you: The issue is most likely there in UHD 4.3, too, it is just hidden. You can check the log of the self-cal service in UHD 4.3 by running journalctl -b -u usrp-adc-self-cal on the device itself. I'm pretty sure you'll find the same error reported as in 4.7. If your device is still within warranty, get in touch with our support. If the device is out of warranty and the degraded performance of the channel impacted is an issue for you, get in touch with our support, too. If the device is out of warranty and the performance of the device overall is okay for you, you can continue using it, but you should keep the issue in mind when using the impacted channel. For your LabVIEW session you should be able to pass the cal_ch_list=023 argument (will only self-cal channels 0, 2 and 3, not the broken channel 1) which for your convenience you can also put into your uhd.conf file (see https://files.ettus.com/manual/page_configfiles.html).
I hope that helps.

[zhouzhiwen2000]

Thank you for replying, there is indeed a issue with the second channel, we are contacting out distributor for support.

[zhouzhiwen2000]

I think there's still a bit more to it: With UHD 4.5 we've changed the ADC self-cal strategy in our X4xx platform to accommodate the needs of X440 with its master clock rate flexibility. Before 4.5 we only performed the ADC self-cal silently during device startup. There used to be a service (service name: usrp-adc-self-cal) that was started immediately after MPM (service name: usrp-hwd). After the startup we considered the calibration to be "good enough" and didn't touch it anymore during operation. With X440 that was not possible anymore and so we started running the ADC self-cal after each clocking change, e.g. changed master clock rate or changed clock- or time-source. A bitfile update is considered as such a clocking change. Therefore, starting from UHD 4.5 you will see the ADC self-cal running here and there. For X440 this improves the spur surpression a lot, in X410 it still improves it a bit compared to the old approach.

What does that mean for you: The issue is most likely there in UHD 4.3, too, it is just hidden. You can check the log of the self-cal service in UHD 4.3 by running journalctl -b -u usrp-adc-self-cal on the device itself. I'm pretty sure you'll find the same error reported as in 4.7. If your device is still within warranty, get in touch with our support. If the device is out of warranty and the degraded performance of the channel impacted is an issue for you, get in touch with our support, too. If the device is out of warranty and the performance of the device overall is okay for you, you can continue using it, but you should keep the issue in mind when using the impacted channel. For your LabVIEW session you should be able to pass the cal_ch_list=023 argument (will only self-cal channels 0, 2 and 3, not the broken channel 1) which for your convenience you can also put into your uhd.conf file (see https://files.ettus.com/manual/page_configfiles.html). I hope that helps.

The repair is rather expensive, is there any suggestions on fixing it ourselves? Since the schematic is open https://files.ettus.com/schematics/x4x0/x410-zbx.pdf, and it seems to be a simple amplifier problem.