Add a checklist for synchronised operation of multiple HackRFs.

[martinling]

What would you like us to add to or change about the HackRF One documentation?

The discussion in #1340 made me think it would be useful to have a checklist in the documentation for troubleshooting setups that require synchronisation of multiple HackRFs.

Points to include:

• Are you running the latest firmware and host software versions?
  • There have been many bug fixes, so please use the latest release for both.
• Are you applying settings to the right target devices?
  • With more than one HackRF device connected, you always need to specify which one to talk to.
  • Use the -d option with a serial number on the command line to specify which device to target.
  • Check the documentation for GNU Radio blocks for how to select devices by serial number.
• Are all HackRFs sharing a clock via the external clock interface?
  • If not, frequencies and sample rates will not match exactly.
  • Is the clock input being detected? Check with hackrf_clock -i.
    • This requires 2022.09.1 or later host software.
    • The older way of checking using hackrf_debug will not work correctly on all hardware revisions.
  • If the clock input is not being detected:
    • If the clock source is CLKOUT of another HackRF, has it been enabled? Do this with hackrf_clock -o
    • Is the CLKIN waveform correct?
      • It should be a 10MHz square wave between 0V and 3-3.3V.
      • A sine wave is OK but may give greater phase noise.
      • An unbuffered TCXO output may not have sufficient voltage.
      • Some hardware revisions are less tolerant of out-of-spec clock input than others.
    • Is your hardware faulty? Some HackRF clones were sold with a non-functional CLKIN port.
• Are all HackRFs being started together using hardware triggering?.
  • If not, start times will not match exactly.
  • There is currently no support for hardware triggering via the Osmocom or Soapy blocks in GNU Radio.
    • The Osmocom block may look like it supports this but these options do nothing!
  • Use hackrf_transfer with the -H option to enable hardware triggering.
• Are any samples being lost due to USB throughput problems?
  • If RX samples are dropped or TX samples delayed, signals will become out of sync.
  • Are your HackRFs sharing a USB bus?
    • A single HackRF One at 20MHz sample rate uses practically all the bandwidth of a single USB 2.0 bus.
    • Unless using very low sample rates, each HackRF should be connected to its own bus.
  • Use hackrf_debug -S after running your software to check if there were throughput problems.
    • This requires 2022.09.1 or later host software.
    • If the shortfall count for a device is non-zero, some samples were dropped on RX or late for TX.
    • To force shortfalls to stop the device at runtime, use hackrf_debug -T 0 -R 0 to set zero tolerance for shortfalls.

[Saiqin]

Hi All. I try to the time synchronization for two hackrf one. The physical connections is shown as above.

Then command in terminal. (1st question here): why after "hackrf_clock -o 1", there is no clock signal detected?

I take another hackrf one to transmit the signal via gnuradio.

then command the "hackrf_transfer -r 001.bin -f 2400000000 -s 10000000 -d 000000000000000075b068dc324a0f07 -a 1 -H 1|hackrf_transfer -r 002.bin -f 2400000000 -s 10000000 -d 000000000000000057b068dc23760c63 -a 1 -H 1" to collect the receiving data.

the signal display in gunradio,

the signal displayed in maltab, you can see , there is still some delay for two hackrf one.

Thank you very much. this is the important implementation for me!!!

[martinling]

(1st question here): why after "hackrf_clock -o 1", there is no clock signal detected?

See in the checklist under this section:

Are you applying settings to the right target devices?

• With more than one HackRF device connected, you always need to specify which one to talk to.
• Use the -d option with a serial number on the command line to specify which device to target.

When you ran hackrf_clock -o and hackrf_clock -i, you didn't tell hackrf_clock which device to talk to. By default it will talk to the device that hackrf_info lists as index 0. So hackrf_clock -o 1 tells the first HackRF to turn its clock output on, and then hackrf_clock -i asks the same HackRF if it is receiving the clock input, which of course it isn't.

When you ran hackrf_debug -d 000000000000000075b068dc324a0f07 --si5351 -n 0 -r, you got the result [ 0] -> 0x01 which confirms that the second HackRF is receiving the clock input. If you ran hackrf_clock -i -d 000000000000000075b068dc324a0f07 you would get CLKIN status: clock signal detected.

[martinling]

I think your other problem is with the way you are setting up the hardware triggering.

There are two ways to do this:

1. Connect the trigger output of one HackRF to the trigger input of the second HackRF, and pass -H 1 to hackrf_transfer only for the HackRF that is receiving the trigger signal.
2. Connect the trigger inputs of both HackRFs to an external trigger signal from some other device, and pass -H 1 to hackrf_transfer for both HackRFs.

What you're doing is kind of a mix of both. You have the trigger output of one HackRF connected to the trigger inputs of both HackRFs, and you're passing -H 1 to both. I'm not convinced that will work.

[martinling]

Another problem is that you're starting both hackrf_transfer commands at the same time.

If you're using one HackRF to trigger the other, you need to start the hackrf_transfer ... -H 1 for the one that is going to be receiving the trigger signal first, because it takes a bit of time for it to get the HackRF into the state where it is waiting for the trigger. You should see a message saying Waiting for trigger...

Then you can start the other hackrf_transfer command (the one without -H 1), and the two devices should start receiving at the same time.

[Saiqin]

thank you very much! there is no clock signal detected after i take target device.
Hack Index 0--->Serial 59F (Tx); Hack Index 1--->Serial C63(Rx1); Hack Index 2--->Serial F07(Rx 2);

this is my connection attach as your convenience.

[Saiqin]

Another problem is that you're starting both hackrf_transfer commands at the same time.

If you're using one HackRF to trigger the other, you need to start the hackrf_transfer ... -H 1 for the one that is going to be receiving the trigger signal first, because it takes a bit of time for it to get the HackRF into the state where it is waiting for the trigger. You should see a message saying Waiting for trigger...

Then you can start the other hackrf_transfer command (the one without -H 1), and the two devices should start receiving at the same time.

thank you much for this , then everything make sense. some times it print wait trigger, i thought its a default message.

[martinling]

thank you very much! there is no clock signal detected after i take target device. Hack Index 0--->Serial 59F (Tx); Hack Index 1--->Serial C63(Rx1); Hack Index 2--->Serial F07(Rx 2);

You need to pass the index or serial number with the -d option.

hackrf_clock -d 1 -o 1 # Enable clock output on device 1
hackrf_clock -d 2 -i   # Check for clock input on device 2

[Saiqin]

Another problem is that you're starting both hackrf_transfer commands at the same time.

If you're using one HackRF to trigger the other, you need to start the hackrf_transfer ... -H 1 for the one that is going to be receiving the trigger signal first, because it takes a bit of time for it to get the HackRF into the state where it is waiting for the trigger. You should see a message saying Waiting for trigger...

Then you can start the other hackrf_transfer command (the one without -H 1), and the two devices should start receiving at the same time.

Thanks! i command as you said. There is still 15000 sampling delay(1500/10M=150 us) , Left is F07 (Waiting for trigger)， right is C63, maybe the delay is constant right?

[martinling]

How are you measuring that delay? Is there some unique feature in your transmitted signal that you're using to compare?

[Saiqin]

How are you measuring that delay? Is there some unique feature in your transmitted signal that you're using to compare?

The two hack receive the signal, why the second one shown above amplitude (first 15000 sampling) is lower than later, if they sampling at same time?
what i want is to use multiple Rx to measure to distance from the transmitter. Then use the TDOA method to localize the transmitter source.
Thank you very much!

[martinling]

If you're referring to the difference between the waveforms on the left and the ones on the right, this is because the one on the left has been waiting for the trigger so its RF path is already up and running when it starts receiving.

The one on the right is starting from cold, so the amplifier is being turned on just as reception starts, and it takes a couple of milliseconds to warm up. Once warmed up you should find that it's receiving the same signal as the other unit. You can just discard the first few milliseconds of data from both devices, everything following that should match up as expected.

[Saiqin]

If you're referring to the difference between the waveforms on the left and the ones on the right, this is because the one on the left has been waiting for the trigger so its RF path is already up and running when it starts receiving.

The one on the right is starting from cold, so the amplifier is being turned on just as reception starts, and it takes a couple of milliseconds to warm up. Once warmed up you should find that it's receiving the same signal as the other unit. You can just discard the first few milliseconds of data from both devices, everything following that should match up as expected.

thank you much much, you are my angel!

[martinling]

Glad I could help! 😇

[Saiqin]

Hi @martinling This is the two slave hackrf one to receive signal, do you know why the left is so different with right? right is perfect i think. Do you have any guess for this? Thank you very much.

[martinling]

@Saiqin

Those signals look well matched to me. There are two differences between them, both of which are to be expected:

1. Scaling. The range of the left one is around +/- 5 LSB wheras the one on the right is +/- 25 LSB. In terms of signal power that corresponds to a difference of about 14dB. There are several factors which can contribute to this:

   1. Positioning of the two receive antennas. Even if both are located at similar distance from the transmitting antenna, in an indoor environment the propagation of the transmitted signal will have peaks and troughs in space where the signal is stronger or weaker due to multipath reinforcement/cancelling effects.
   2. Gain of the two receive antennas. Even if both are of the same type there will be minor differences in performance.
   3. Receive performance of the two HackRF units. These can easily differ from each other by a few dB. If you want to reduce this difference, you can calibrate for it by measuring a reference signal at the frequency of interest, and then finding a scale factor for each of the receivers, or adjusting gain settings such that the two units read similarly. This should be done with a reference source cabled directly to the antenna port on each receiver, using the same cable setup for both. When doing so, take care not to exceed the maximum RX input power level of -5dBm.

2. Phase. Because the two receive antennas are located at different points in space, the transmitted wave passes each of them at a different point in its oscillation cycle. This results in the phase differences seen between the two received I/Q pairs. The two receivers are acting as a phased array. If the spacing between the two antennas is known, the phase differences can be analyzed to determine the angle of origin of the transmission, relative to the baseline between the two antennas. With further antennas a unique bearing can be established.

[Moji14]

Hi @martinling This is the two slave hackrf one to receive signal, do you know why the left is so different with right? right is perfect i think. Do you have any guess for this? Thank you very much.

Those signals look quite different. The left signal looks to have less resolution, although the sampling rate is the same (hence the same number of samples for each graph), the ADC quantizer seems to be repeating the same value for few samples of the left graph. Check out the ADC register values of each Hackrf units and compare them. Or, I'll ensure that both Hackrf units have exactly the same software. So a good idea will be to perform a software reset using the same files for both units.
And definetely if you are measuring/comparing the pahse diference between 2 devices you should ensure that your setup is adecuate for that purpose. Ensure that the input reference signal is the same for both receivers. Both pathways (cables and connectos) should have same attenuation and physical lenght (try to reach perfect simetry). Not only for the RF input but also for the clock reference input.

I'll be happy to check your new set up if you upload an image. Judging by the photos you've uploaded I'll do:

• If metallic enclousers close them to shield from external interferences. (Your Hackrf units don't have RF shields on the front ends, that will worsen the results as well).
• Receive from the SMA clock input port instead of using jump wires (That is a horrible noise picker) connected directly to the board. In your case it seems that you're only using jumping wires for the trigger signal. You can just go from one board to another avoiding the breadboard (you can also twist live and ground together).
• If it's possible to use a third Hackrf for example the TX unit then use that clock for feeding booth receivers.
• The clock input cables should have exactly the same size, use the same type of connector and length. Avoid having an adaptor on one cable and no adaptor on the other cable (as I've seen on previous photos).
• No antennas at all at the moment (mutipath propagation will definetely lead you to uncontrolable phase shifts), feed directly from the TX unit (using attenuators if needed). Once your system is configured, metered and calibrated then you can ensure that your pahse measurements are predictible with an X error margin.
• Also RX cables must be the same length and symetric.
• If you have to split signals use an adecuate 3dB splitter.

I good idea will also set the starting point of your measurements/experiments at much lower frequencies. I'd say I'll try to work on baseband and receive a 500KHz signal with a sampling rate of a 2MHz. This way you avoid samples dropping becuase of a busy USB connection near it's limits. Also setting up receiving frequency that low will avoid using the amplifiers, filters and frequency converter (mixer), taking out all those phase shiftings from the equation to make things easy to track. And from there you can start adding complexity.

[Saiqin]

@Saiqin

Those signals look well matched to me. There are two differences between them, both of which are to be expected:

1. Scaling. The range of the left one is around +/- 5 LSB wheras the one on the right is +/- 25 LSB. In terms of signal power that corresponds to a difference of about 14dB. There are several factors which can contribute to this:

   1. Positioning of the two receive antennas. Even if both are located at similar distance from the transmitting antenna, in an indoor environment the propagation of the transmitted signal will have peaks and troughs in space where the signal is stronger or weaker due to multipath reinforcement/cancelling effects.
   2. Gain of the two receive antennas. Even if both are of the same type there will be minor differences in performance.
   3. Receive performance of the two HackRF units. These can easily differ from each other by a few dB. If you want to reduce this difference, you can calibrate for it by measuring a reference signal at the frequency of interest, and then finding a scale factor for each of the receivers, or adjusting gain settings such that the two units read similarly. This should be done with a reference source cabled directly to the antenna port on each receiver, using the same cable setup for both. When doing so, take care not to exceed the maximum RX input power level of -5dBm.

2. Phase. Because the two receive antennas are located at different points in space, the transmitted wave passes each of them at a different point in its oscillation cycle. This results in the phase differences seen between the two received I/Q pairs. The two receivers are acting as a phased array. If the spacing between the two antennas is known, the phase differences can be analyzed to determine the angle of origin of the transmission, relative to the baseline between the two antennas. With further antennas a unique bearing can be established.

Thank you very much

[Saiqin]

If I just connect the p15 and p16 to make the synchronization, no connection between CLK_IN and CLK_OUT, there is just a sampling difference. So this is feasible for synchronization if I don't connect the CLK_IN and CLK_out (just using the jumper between p15 and p16)? thanks

[martinling]

If you connect just the hardware trigger signals and not the clocks, then the start times will be synchronised but the sampling rates will not match exactly. For short captures this might be good enough for your purposes. But over time, the differences in clock frequency between the two devices will add up until there is a noticeable offset between events.

[Saiqin]

If you connect just the hardware trigger signals and not the clocks, then the start times will be synchronised but the sampling rates will not match exactly. For short captures this might be good enough for your purposes. But over time, the differences in clock frequency between the two devices will add up until there is a noticeable offset between events.

Thanks, you are so kind!