Skip to content

[TTN v3] Using MQTT to retrieve TTN aplication data on the fly

Jump to bottom
Sébastien Jean edited this page Feb 17, 2023 · 7 revisions

This section describes how to remotely retrieve live application data, connecting to TTN MQTT broker.

This requires to set up TTN MQTT integration, and either using a general-purpose MQTT client or writing an ad'hoc one.

Setting up MQTT integration

From application home page, select Integrations (from left-sidebar) and then MQTT:

TTN-MQTTIntegration-01

Then select Generate API Key (iut-valence-ubpe-hao@ttn/ API Key will further be used as MQTT client credentials):

TTN-MQTTIntegration-02

⚠️ Immediately copy and save the API Key since it couldn't be further retrieved!

To see all API keyx, select API Keys (from left-sidebar):

TTN-MQTTIntegration-03

Selcting an API key allows to give it a more friendly name, and to tune access rights (here limitedf to read application data):

TTN-MQTTIntegration-04

TTN-MQTTIntegration-05

Retrieving application devices data using a general purpose MQTT client

The client used here is MQTT Explorer, an open source MQTT client that can be found here.

Once launched, configure a new connection by filling fields with relevant information:

TTN-MQTTIntegration-06

⚠️ Certificate validation has to be disable and user/password must be filled with credentials obtained previously while configuration TTN MQTT integration.

Then, establish connection and wait for incoming messages to be retrieved:

TTN-MQTTIntegration-07

💡 Incoming device data are published on a topic whose name is like v3/username/devices/devide-id/up (here v3/iut-valence-ubpe-hao@ttn/devices/eui-0000000000000001/up), and retrieved as a JSON object:

{
   "end_device_ids":{
      "device_id":"eui-0000000000000001",
      "application_ids":{
         "application_id":"iut-valence-ubpe-hao"
      },
      "dev_eui":"0000000000000001",
      "dev_addr":"260B1D3D"
   },
   "correlation_ids":[
      "as:up:01GSCRFBNA0B950E6Q9NNR2013",
      "gs:conn:01GSAQRDR94P67SK26DBQ5PREA",
      "gs:up:host:01GSAQRDRCAF355XJT6ZK6M3Z3",
      "gs:uplink:01GSCRFBEW7JJEMDY1GWRC9233",
      "ns:uplink:01GSCRFBEWSRYW0GX07K6V249B",
      "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01GSCRFBEWJ756GYMNFAMG3MX5",
      "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01GSCRFBN9ERVQCPGVW9GVX4M1"
   ],
   "received_at":"2023-02-16T09:21:49.737776462Z",
   "uplink_message":{
      "f_port":1,
      "f_cnt":73,
      "frm_payload":"AgC4HoVBAAAoQg==",
      "decoded_payload":{
         "humidity":42,
         "sys_time":2,
         "temperature":16.6
      },
      "rx_metadata":[
         {
            "gateway_ids":{
               "gateway_id":"iut-valence-ttn-gw-02",
               "eui":"B827EBFFFEBB2805"
            },
            "time":"2023-02-16T09:21:49.525169362Z",
            "timestamp":3769747748,
            "rssi":-95,
            "channel_rssi":-95,
            "snr":7,
            "uplink_token":"CiMKIQoVaXV0LXZhbGVuY2UtdHRuLWd3LTAyEgi4J+v//rsoBRCkkseFDhoMCK3rt58GEMKT1/0BIKDp5rPbwA8=",
            "channel_index":2,
            "received_at":"2023-02-16T09:21:49.532007362Z"
         }
      ],
      "settings":{
         "data_rate":{
            "lora":{
               "bandwidth":125000,
               "spreading_factor":9,
               "coding_rate":"4/5"
            }
         },
         "frequency":"868500000",
         "timestamp":3769747748,
         "time":"2023-02-16T09:21:49.525169362Z"
      },
      "received_at":"2023-02-16T09:21:49.532956003Z",
      "consumed_airtime":"0.205824s",
      "network_ids":{
         "net_id":"000013",
         "tenant_id":"ttn",
         "cluster_id":"eu1",
         "cluster_address":"eu1.cloud.thethings.network"
      }
   }
}

⚠️ Only new MQTT messages can be retrieved, older are lost.

Retrieving application devices data using a Python script

💡 The following Python3 script acts as a MQTT client that connects to TTN MQTT server (with same credentials as previous) and subscribe to application devices uplink topic. Then, it saves timestamped raw JSON payloads in a file, and outputs application-level decoded payload (here, fake sensor values)

⚠️ This script relies on paho-mqtt library (that can be easily installed using pip)

# Dump all device uplink messages in a file

import paho.mqtt.client as mqtt

from datetime import datetime
import json
import os

MQTT_SERVER = "eu1.cloud.thethings.network"
MQTT_PORT = 1883
MQTT_CONNECTION_TIMEOUT = 60
MQTT_CLIENT_NAME = "ttn-client-dump"

#########################################################
# TO BE UPDATED WRT APP AND API KEY
#########################################################
TTN_APP_NAME = "iut-valence-ubpe-hao@ttn"
TTN_USER = "iut-valence-ubpe-hao@ttn"
TTN_PWD = "NNSXS.THIS-IS-NOT-THE-PWD-YOU-ARE-LOOKING-FOR"
#########################################################


# Callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    if rc != 0:
        print("Unable to connect (resultcode " + rc+ ")")
        quit()

    print("Connected, waiting for messages...")
    client.subscribe('v3/' + TTN_APP_NAME + '/devices/+/up')

def save_payload(payload, time, path):
    with open(path, 'a') as dump_file:
        dump_file.write("------------------\n")
        dump_file.write(time + "\n")
        dump_file.write("------------------\n")
        dump_file.write(payload+"\n")
    dump_file.close()

# Callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    try:
        time_str = datetime.now().strftime("%d-%m-%y_%Hh%Mm%Ss")
        print("[received] at " + time_str)
        json_payload = json.loads(msg.payload.decode('utf-8'))
        formatted_payload = json.dumps(json_payload, indent=4)
        print(formatted_payload)
        save_payload(formatted_payload, time_str, PATH)
        print("[saved]")
        app_decoded_payload = json_payload["uplink_message"]["decoded_payload"]
        print("sys_time: " + str(app_decoded_payload["sys_time"]))
        print("temperature: " + str(app_decoded_payload["temperature"]) + " °C")
        print("humidity: " + str(app_decoded_payload["humidity"]) + " %")
        print("[decoded]")
    except Exception as exception:
        print("[malformed]")
        print("---")
        print(exception)
        print("---")
        pass

now = datetime.now()
current_time = now.strftime("%d-%m-%y_%Hh%M")
PATH = current_time + ".dump"
print("TTN application devices uplink dumper started!")
print("Saving in " + PATH)
with open(PATH, 'w') as fp:
    pass

client = mqtt.Client(MQTT_CLIENT_NAME);
client.username_pw_set(TTN_USER, TTN_PWD)
client.on_connect = on_connect
client.on_message = on_message

client.connect(MQTT_SERVER, MQTT_PORT, 60)

# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()

Below shows how standard output looks like when executing the script:

TTN application devices uplink dumper started!
Saving in 17-02-23_14h16.dump
Connected, waiting for messages...
[received] at 17-02-23_14h16m43s
{
    "end_device_ids": {
        "device_id": "eui-0000000000000001",
        "application_ids": {
            "application_id": "iut-valence-ubpe-hao"
        },
        "dev_eui": "0000000000000001",
        "dev_addr": "260B1D3D"
    },
    "correlation_ids": [
        "as:up:01GSFRA5X257A7XPCM47SSJ2V3",
        "gs:conn:01GSAQRDR94P67SK26DBQ5PREA",
        "gs:up:host:01GSAQRDRCAF355XJT6ZK6M3Z3",
        "gs:uplink:01GSFRA5PHMSBH8BSEYGHN967Q",
        "ns:uplink:01GSFRA5PJJ5RM1TMNTSBPA5H5",
        "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01GSFRA5PJ97N01AJZ71M23MYR",
        "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01GSFRA5X120A5YCFQ56Q4DGDN"
    ],
    "received_at": "2023-02-17T13:16:43.298133030Z",
    "uplink_message": {
        "f_port": 1,
        "f_cnt": 97,
        "frm_payload": "AgC4HoVBAAAoQg==",
        "decoded_payload": {
            "humidity": 42,
            "sys_time": 2,
            "temperature": 16.6
        },
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "iut-valence-ttn-gw-02",
                    "eui": "B827EBFFFEBB2805"
                },
                "time": "2023-02-17T13:16:43.066329449Z",
                "timestamp": 1184048548,
                "rssi": -101,
                "channel_rssi": -101,
                "snr": 5,
                "uplink_token": "CiMKIQoVaXV0LXZhbGVuY2UtdHRuLWd3LTAyEgi4J+v//rsoBRCky8y0BBoLCLv8vZ8GEN3y1CogoJGs9rqtJg==",
                "channel_index": 1,
                "received_at": "2023-02-17T13:16:43.073901749Z"
            }
        ],
        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 9,
                    "coding_rate": "4/5"
                }
            },
            "frequency": "868300000",
            "timestamp": 1184048548,
            "time": "2023-02-17T13:16:43.066329449Z"
        },
        "received_at": "2023-02-17T13:16:43.090406034Z",
        "consumed_airtime": "0.205824s",
        "network_ids": {
            "net_id": "000013",
            "tenant_id": "ttn",
            "cluster_id": "eu1",
            "cluster_address": "eu1.cloud.thethings.network"
        }
    }
}
[saved]
sys_time: 2
temperature: 16.6 °C
humidity: 42 %
[decoded]
Clone this wiki locally