[MAJOR] when new v1.0.2 application is made with US915 settings, no packets received in EU

[MedadRufus]

On ttnv3, when a new v1.0.2 application is made with US915 settings, no packets received in EU. Note that the same application can be created with EU868 settings and receive the data.

Changing region in teh interface to EU868 fails with the following error:

invalid value of field mac_state.current_parameters.rx2_data_rate_index

{
  "code": 3,
  "message": "error:pkg/networkserver:field_value (invalid value of field `mac_state.current_parameters.rx2_data_rate_index`)",
  "details": [
    {
      "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails",
      "namespace": "pkg/networkserver",
      "name": "field_value",
      "message_format": "invalid value of field `{field}`",
      "attributes": {
        "field": "mac_state.current_parameters.rx2_data_rate_index"
      },
      "correlation_id": "fc1143302d2e4fb7aac6a8d07da3afe4",
      "code": 3
    }
  ],
  "request_details": {
    "url": "/ns/applications/icss-lora-tracker/devices/icspace26-ttnv3-abp-us-v1-0-2",
    "method": "put",
    "stack_component": "ns"
  }
}

[MedadRufus]

Only solution may be to switch ABP keys for each region, and make a seperate ABP application for each region.

[MedadRufus]

Safest solution is to create ABP keys for each region. Each registration must have reset fcount enabled AT REGISTRATION. It apparently cannot be changed later.

Then when each region is selected, the keys should be changed along with it. Create look up table {region, network key, appsession key, id}

[MedadRufus]

Changing region after creating the application does not work, although the interface indicates it.

[MedadRufus]

This is the case for lorawan v1.0.3 as well

[MedadRufus]

Just make new keys for each region

[pimente]

Hello @MedadRufus
Congratulations for the success you have had so far.
This is quite impressing to have achieved circumnavigation with Lora only!

I am currently working on a similar project and have already achieved a successful flight in Europe with TTN3 and a Lora-E5 based tracker.
I am now trying to make the code work all over the world.
From the issues you have raised, it is not clear to me if we need to set up new ABP keys for each region.
This issue (the latest) seems to suggest that a key is needed for each region while issues 71 and 92 tend to suggest the opposite.

What final conclusion have you reached regarding this topic?

Your help is very much appreciated.

Cheers

[MedadRufus]

My conclusion is that separate ABP keys are needed for each region. On the Things Network V3 console, I register one ABP device for each region, and dynamically change the ABP keys(DevAddr, FNwkSIntKey, SNwkSIntKey, NwkSEncKey, AppSKey) depending on each region.

In TTNv2, it was possible register a single ABP end node, and it would work in all regions(US915, EU868, AS923). This has been tested in flight. There was no requirement to specify which region the device was registered to in the interface.

However, on TTNv3, when registering the device, its required to specify which region parameters the end node uses. It appears to me that the set of ABP credentials generated for each region will only work on those region frequencies. I have tested this by:

1. registering an end node with US915 region
2. transmitting on EU868 frequencies and receiving on an EU868 gateway.

I get no packets on the TTNv3 console. So I conclude that an end node registered to a region will not be received in any other region. Hence, I register a new device to each region(US, EU, AU, RU etc) and then switch the credentials on the end-node dynamically.

I have now abandoned this repo in favour of a fork off the Semtech reference code which can be viewed here. https://github.com/ImperialSpaceSociety/LoRaMac-node. I needed to use the Non-volatile memory to store all the keys for an OTAA activation, to avoid repeated joins after resets and this later version of the stack has this functionality. I am also getting the code to switch between the Things Network(ABP only) and The Helium Network(OTAA only) every 3 transmissions, to increase my coverage.

Below is a list of all my devices for each region.

[pimente]

Thank you for the very thorough answer!

I am using the mbed-os framework with mbed studio IDE. A lot of different hardware is supported by stm32customtargets and the example mbed-os-example-lorawan seemed much lighter than the Semtech LoRaWAN example which has a lot of boilerplate in my opinion. Not sure if mbed provides the same functionalities than the original Semtech code thought.

I did not know about the The Helium Network. It seems to have a great coverage. I now understand your need and reflexion around OTAA...
With your alternating transmission scheme, did you get a good sense of the Helium versus TTN coverage?
Does it really help to add the Helium Network?
If so, I'll give it a try.

From your screenshot, it seems that icspace26 is doing great with a flight longer than 110 days!? I can't wait for the flight report if you share it 😄

Cheers

[MedadRufus]

well ICSPACE26 is yet to be launched, only testing for 110 days 😄

I have looked very closely at the Semtech stack, and the only unnecessary part in there is the LoRaWAN Class B support, which I will never be using. Its not a big problem for me, as the compiled code size is 182 kb, less than the max of 192 kb for the STM32l072 processor I use. Ram is also under control, at around 11 kb, less than max of 20kb. I have chosen to use this stack because I know its well tested by the developers. I am less trusting of other ports of this code.

The helium network promises to increase coverage quite a bit. While there is significant overlap in coverage of The Things Network and Helium Network in the US/EU, in China, the Helium network has a massive improvement. In China, there is virtually no TTN coverage while the Helium network has much more coverage. Note that the Helium network only uses OTAA activation.

There is recent discussion about the helium networks vs Things Network on the picoballoon mailing list: https://groups.io/g/picoballoon/topic/85906023#4659

[MedadRufus]

With your alternating transmission scheme, did you get a good sense of the Helium versus TTN coverage?

I have yet to launch this tracker, with alternating helium/TTN transmissions. So I don't know. One thing I have seen is the helium coverage is much better than TTN. Outside my house in London, UK, I immediately managed to successfully join the Helium network in a few minutes, using gateways nearby. On the Things Network, I have almost never gotten public gateways to pickup packets.

[pimente]

well ICSPACE26 is yet to be launched, only testing for 110 days 😄

Finger crossed! Do you plan to launch soon?

I have looked very closely at the Semtech stack, and the only unnecessary part in there is the LoRaWAN Class B support, which I will never be using. Its not a big problem for me, as the compiled code size is 182 kb, less than the max of 192 kb for the STM32l072 processor I use. Ram is also under control, at around 11 kb, less than max of 20kb. I have chosen to use this stack because I know its well tested by the developers. I am less trusting of other ports of this code.

I agree that reliability is the most important criteria when using others' code for these flights. I've seen that your new fork introduces some UTs. That should help.

For the Helium Network, do you need multiple credentials for each region?
If not, and considering the coverage, I might move to the Helium Network instead of configuring TTN for each region...
Mbed implementation is lorawan 1.0.2. I see that you use version 1.0.3. Is it mandatory to connect to the Helium Network?

[MedadRufus]

For the Helium Network, do you need multiple credentials for each region?

Good question. I don't have the answer for this. To be safe, I have created registered 3 different end node(with unique keys) for the US, EU and CN. When registering a Helium network end node, I don't specify which region it will be used in. So one registered device might work in all regions around the world(US, EU, CN). However, I cannot test this as I don't have Helium Gateways for these 3 regions. So to be safe, I created 3.

[MedadRufus]

Unlike ABP devices, OTAA devices require a join request to be made, which includes a downlink from gateway to tracker. This is not reliable I discovered in previous flights. The end node may not receive the downlink from gateway, and cannot transmit data at all. So I want it to do only one join request in its lifetime. To do that, I save the keys and channel lists to non volatile memory so that after a reset, it does not have to rejoin.

[MedadRufus]

I agree that reliability is the most important criteria when using others' code for these flights. I've seen that your new fork introduces some UTs. That should help.

I need to document how to run the unittests. It uses CMAKE to build and CppUTest framework. I normally run it in VS code which has the test adaptor which makes running it very easy.

[MedadRufus]

Mbed implementation is lorawan 1.0.2. I see that you use version 1.0.3. Is it mandatory to connect to the Helium Network?

I don't have any information on which version to use. LoRaWAN version 1.0.3 is what I have used on the Things Network V2, and also works on TTNv3. And in testing, also works on the helium network.

[MedadRufus]

Finger crossed! Do you plan to launch soon?

Hoping to launch by the year end

[pimente]

One thing I have seen is the helium coverage is much better than TTN. Outside my house in London, UK, I immediately managed to successfully join the Helium network in a few minutes, using gateways nearby. On the Things Network, I have almost never gotten public gateways to pickup packets.

Following our discussion, I gave a try to the Helium Network and made a few tests here in Paris. I can confirm your observations. I successfully join in a few minutes using multiple gateways nearby. On TTN, I had to stick my device on a fishing rod and wait hours to get an uplink with ABP.

Unlike ABP devices, OTAA devices require a join request to be made, which includes a downlink from gateway to tracker. This is not reliable I discovered in previous flights. The end node may not receive the downlink from gateway, and cannot transmit data at all. So I want it to do only one join request in its lifetime. To do that, I save the keys and channel lists to non volatile memory so that after a reset, it does not have to rejoin.

This makes a lot of sense. I have also observed that I often need multiple Join Requests before the node actually receives the Join Accept. However, if I think about your other comment:

To be safe, I have created registered 3 different end node(with unique keys) for the US, EU and CN. When registering a Helium network end node, I don't specify which region it will be used in. So one registered device might work in all regions around the world(US, EU, CN). However, I cannot test this as I don't have Helium Gateways for these 3 regions. So to be safe, I created 3.

I understand that when you change region you will change credentials and make a new Join Request with the new region's credentials. Is that correct? Will you you try to Join with the new credentials if you can't transmit with the previous ones? I see a potential failure scenario here. When you change region:

• if you make a new Join Request with the new region's credentials, you risk of never receiving the Join Accept
• if you try to stick to the saved credentials, you need to confirm that the transmissions still work so you also need to receive downlinks.

Unless I am missing something here, in both scenarios, you are depending on a downlink message, which sounds risky.
If so, I think that by trying to be safer and having 3 credentials we might take more risk than if one credential was actually enough. With one credential and a unique Join request at the launch site, we are not depending on downlinks for the full flight.
If you agree with this, I think we should try to figure out if one credential is enough maybe by contacting someone outside Europe in the pico group and try with European credentials? That would be simpler and safer if is works.
Looking forward to your view on this.

[MedadRufus]

I understand that when you change region you will change credentials and make a new Join Request with the new region's credentials. Is that correct?

This is correct.

Will you you try to Join with the new credentials if you can't transmit with the previous ones?

No. For each region, I will try a join with the credentials for that region only. i.e. when over the US, I will try a join with one set of credentials(dev_eui, app_eui, app_key). When over the EU, I will use another set of credentials(dev_eui, app_eui, app_key).

Yes I do think the ideal scenario is to use only ONE set of OTAA credentials(dev_eui, app_eui, app_key) and do the join before launch. However, I see one issue. When you do an OTAA join, a channel frequency list is sent up(see picture below from LoRaWAN 1.0.3 spec)

The problem is, the CF list is going to be different for different regions(US, EU, CN). So at the moment, I am of the opinion that its safer to treat the tracker as 3 different devices with credentials for each region.

The requirement for downlink from the join request is a major issue. I am testing a method of completing the join request with a virtual device running on my laptop, and then saving those derived keys to my tracker's EEPROM. Then the tracker will not have to do the join at all. I am tracking progress in this issue: ImperialSpaceSociety/LoRaMac-node#113 (comment)

[MedadRufus]

The semtech LoRaMac-node implementation saves the Channel Frequency list(CF list) to eeprom, along with the region's keys(DevAddr, frame count etc). I am a bit reluctant to modify the list of parameters they save to EEPROM. I simply allocated 3 blocks of EEPROM, one block for each region, and make the tracker save all the keys, channel lists, frame count to the block that corresponds to that region.

[pimente]

Very thoughtful!
Do you know if the CF list if always the same for each region? If so, we could only change the CF list and keep the same set of OTAA credentials.

[MedadRufus]

I suspect the CF list will be different for each region. But I also suspect you can get the tracker to ignore the CF list, and only transmit on the default channels for that region(the EU has 3 default channels for example)

[MedadRufus]

But i have a feeling that we can use the same OTAA keys in all regions. This has to be tested, as you say, with somebody in another region of the world(e.g. US). I will try to test with the Virtual Lorawan device. I have gotten the virtual device to work on TTNv2 but not yet on helium network

[pimente]

That would do the trick! I've once forced a tracker to transmit only on the first channel, to have it work with a Single Channel Gateway. I had to watch out manually for duty cycle and it was not really lorawan compliant anymore.
But maybe just ignoring the CF list and transmitting on the default channels (not just the first one) would not break lorawan compliancy. Good luck on the virtual device. It will be useful in all scenarios even to test a single set of OTAA keys for all regions I suspect.

[pimente]

One side question: I am gearing up on the Helium Network and I am experiencing difficulties to always receive Join Answer. By looking at the Helium dashboard, I see a delay between the Join Request and Join Answer often around 2000ms which does not align with the default EU868_JOIN_ACCEPT_DELAY1 which is set to 5000ms. Did you have to tweak this parameter for the Helium Network?

[MedadRufus]

I have not tweaked any parameters for the join request(or any parameters at all). I have the same behavior on The Things Network V2 and V3 and the helium network during a join.

[MedadRufus]

The LoRaMac-node implementation uses a Join accept delay 1 of 5000 ms, as seen in the code: https://github.com/ImperialSpaceSociety/LoRaMac-node/blob/4027ac1ab855e449acf87140728d7d4d30bd71b3/src/mac/region/RegionCommon.h#L61-L64

[pimente]

You know why it does not work on TTN specifically or on both networks? Do you have an idea to fix? I can make some tests on my end tomorrow if that can help.

[MedadRufus]

I don't know yet why it does not work on both networks. What I know now it definitely does not work. I thought it was an endianness issue when hardcoding in the dev_eui but it does not seem to be the case.

Correction: I meant dev_addr not dev_eui

[pimente]

Ha ha! I exactly though the same but looking at how the dev_addr is read from the rx buffer. I tried inverting only the dev_addr without success and planning to try tomorrow by inverting all credentials... I vaguely remember that the lora packet is somehow inverted but have to look in the specs to really understand.

[MedadRufus]

The Things network console shows the dev_addr, nwk_skey, app_skey even for OTAA devices. I hard code them and try to join as an ABP device, but it does not work. No messages appear on console. Endianess should be correct. I copied the keys direct from the console. So it seems to me that dev_addr, nwk_skey, app_skey and frame count is not sufficient to trick the LoRaWAN network that its an OTAA device. Not sure why.

[MedadRufus]

OK I have finally figured out whats missing. The device must also store the AppEUI.

So to trick the network into thinking that an OTAA device is connected, the following pieces of information is needed(can be hardcoded):
1.DevAddr
2. AppEUI
3. NwkSKey
4. AppSKey
The Frame Count must also only increment(no decrement).

I can now get it working on The Things Network, registering as OTAA and hardcoding the DevAddr, AppEUI, NwkSKey, AppSKey and consistently incrementing fcount.

I got the clue from the LoRaWAN 1.0.3 spec.

[MedadRufus]

@pimente I am pleased to say that the Helium network will work around the world with a single set of credentials. This is how I tested it.

1. I used a Virtual Lorawan device "transmitting" on US915 frequencies to complete a join request. Then I extract the DevAddr, AppEUI, NwkSKey and AppSKey.
2. Then I hard code the above keys into the code for an ABP device on my physical tracker. Then I take it outside my house in London, UK, transmitting on European frequencies and immediately packets start appearing on my console. Success!

So since it worked across 2 regions, it will likely work around the world.

[pimente]

@MedadRufus Double win! This is a breakthrough for circumnavigation flight! Congrats! Great guess work. I will try to reproduce tomorrow from Paris, FR. Do you think there will be some king of expiration on theses credentials?

[MedadRufus]

Good question. I don't know if there is an expiry, when the credentials become invalid.
I must also point out, the device must be able to keep track of the frame count, so that its always incrementing.

[MedadRufus]

For reference, I link the instructions for running the virtual lorawan device for the helium network here: ImperialSpaceSociety/LoRaMac-node#113 (comment)

[pimente]

I have not been able to reproduce yet. In fact, I realised that I was already setting the AppEUI even with ABP. But I don't see it being used anywhere in the code with ABP...
This paragraph 6. 1 of LoRaWAN 1.0.3 spec is strange indeed, especially if you compare it to paragraph 6.3 which states that skeys are stored directly instead of DevEUI, AppEUI, and AppKey:

Quick question: Can you confirm that the DevAddr and AppEUI you generated with the virtual lorawan device and you use for ABP are the ones you see in the Helium console?

AppEUI in DEVICE DETAILS:

And DevAddr in the General tab of an Uplink event:

Thanks!

[MedadRufus]

@pimente I know the problem. Its the endianness of the dev_address. Try devaddr = 0x480002AC

[MedadRufus]

I know that the helium network uses dev_add values with 0x48xxxxxx. This is documented at the bottom of the docs: https://docs.helium.com/lorawan-on-helium/

[MedadRufus]

One more thing to note: every time you do a join, a new dev_addr, NwkSKey, AppSKey are generated. The old ones may become invalid.

[pimente]

The screenshot I have shared is a successful uplink just after the OTAA. I was suspecting from the Lora implementation's code that endianness was mixed up in the Helium console but the link you've shared just proves it. It is a bug in their interface.
The problem is that I have already tried with devaddr = 0x480002AC without success. I feel am missing something from your last discovery. Looking at the code, I don't see where AppEUI is being used with ABP. Have you understood that?

[pimente]

I'll try again a full procedure with OTAA Join followed by credentials extractions for ABP.

[MedadRufus]

I have not looked closely at the implementation difference between OTAA /ABP in the Semtech reference code. However, I am sure the AppEUI is required for OTAA. So even when using ABP, i think the AppEui must be used somewhere.

[MedadRufus]

Ill try with incorrect AppEUI and see if it will work.

[MedadRufus]

Correction from me. I switched around a few bytes of my hard coded AppEUI, and it still works. This proves to me that actually, the AppEUI is not necessary. I think you caught it @pimente

[MedadRufus]

The only keys that have to be correct I think are back to the DevAddr, AppEUI and NwkSKey. When I was testing yesterday, I keyed in the correct AppEUI, and immediately, it started working. It may have been some other cause that made it to work.

Correction. I meant the required keys are DevAddr, NwkSKey and AppSKey(not AppEui)

[MedadRufus]

I would recommend trying with the Things Network first, as it is a lot more transparent. It shows the DevAddr(with correct endian), NwkSKey and AppSKey on the console. Then if it works, the move on to the Helium Network. It helps to have a gateway at hand to test with.

[pimente]

This works for me as well! I suspect problem was coming from ADR but I still have to investigate this.
Switching to ABP, default DR was DR_5 which I though was not low enough to be received from the nearest gateway so I disabled ADR and tried different DR down to DR_0 without success.
Following our discussion, I joined again with OTAA, extracted DevAddr (watching out for the endianess), NwkSKey and AppSKey, reverted to all default parameters, so ADR as well and waited a bunch of uplinks on DR_5.
After 15, I got one which was received!

[pimente]

These launches will happen before the end of the year for sure 😄

[MedadRufus]

For a flying balloon, its best to disable ADR. ADR is useful if the device is static. It reduces the power level/air time to as low as possible, while maintaining the link.

[pimente]

Yes, ADR was disabled for my last flight in September. In fact I was using BlindADR which was a success until Turkey where it lost coverage. We should make more tests to understand the limits of this hack on the Helium network. Does is it work well with ADR disabled? Do the credential expire?... But this sounds like a huge step forward to achieve circumnavigation 👍

[MedadRufus]

I have always disabled ADR, and had no problem.

I am putting my thoughts and plan B for the expiring credentials issue here. Its somewhat seperate from the current issue: ImperialSpaceSociety/LoRaMac-node#214

[pimente]

Yep, we've been quite of course the initial issue in this thread 😄
I'll make a new test now with ADR to try to discharge this problem.

[MedadRufus]

I don't use ADR at all. I always use the highest possible power, and a fixed data rate. In the EU, I use DR_5 while in the rest of the world, I use DR_4. Note that in the US, the highest possible datarate is DR_4(not DR_5).

[pimente]

Yes, I've seen your flight reports. Thks for sharing btw.
I've used DR_4 and DR_2 once every 10 uplinks. DR_4 was cleary not necessary over Europe and reducing my transmit frequency to respect the TTN fair policy. Now with Helium, since we are just bound by the official duty cycle (much better than TTN fair policy) and we pay per packet data size, not packet time, I have to think about what would be the new optimum...
If next tests proves this hack reliable, I will go full Helium I think.

[pimente]

I confirm it works with ADR disabled. I must have been something else.

[MedadRufus]

For testing purposes, if you are relying on public gateways for testing, your best chance of getting received is to set it to transmit at max power(TX_POWER_0) and at the lowest datarate(DR_0). IT makes a big difference.

But during flight you can increase datarate to DR_5. my 700 km transmission to the gateway on top of the alps was I think on DR_4.