How does the BLE low-level Bluetooth map to the modbluetooth.c code?

[brianreinhold]

I want to be able to call the btstack void gap_delete_bonding(bd_addr_type_t address_type, bd_addr_t address) method among other things but I do not understand how the python codes maps to the C code.
For example, I see there is this BLE.gap_advertise() method but the modbluetooth.c file has mp_bluetooth_gap_advertise_start() and mp_bluetooth_gap_advertise_stop(). I dont even see a stop method on the python side.

If I wanted to add a python function BLE.gap_delete_bonding(params) how would I do it? In the btstack C code in the sm.c file there is such a method.

[ned-pcs]

mp_bluetooth_gap_advertise_stop() is called from BLE.gap_advertise() when the interval_us argument is None.

[brianreinhold]

@ned-pcs How would I create a BLE.delete_bonding() method? The method exists in the btstack. What I am looking for is some documentation on how to create such interfaces. Does a python method BLE.gap_advertise() even exist as python source?

I guess what I am asking does a python BLE class actually exist as python source or do the static methods I place in modbluetooth.c with a name like
STATIC mp_obj_t bluetooth_ble_delete_bond() automatically become part of the BLE class such that I could call ble.delete_bond() from my python application? Clearly I have to finish the work to the different bt stacks but that seems much more straight forward.

[ned-pcs]

Ah. Look at https://docs.micropython.org/en/latest/develop/extendingmicropython.html for an introduction to how to write C extensions for Micropython, and https://docs.micropython.org/en/latest/develop/library.html for more detail about how stuff works internally.

BLE.gap_advertise() does not exist as Python source; rather it's in modbluetooth.c. Looking at that file starting at line 452 we have:

STATIC mp_obj_t bluetooth_ble_gap_advertise(size_t n_args, const mp_obj_t *pos_args, mp_map_t *kw_args) {
    enum { ARG_interval_us, ARG_adv_data, ARG_resp_data, ARG_connectable };
    static const mp_arg_t allowed_args[] = {
        { MP_QSTR_interval_us, MP_ARG_OBJ, {.u_obj = MP_OBJ_NEW_SMALL_INT(500000)} },
        { MP_QSTR_adv_data, MP_ARG_OBJ, {.u_rom_obj = MP_ROM_NONE} },
        { MP_QSTR_resp_data, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_NONE} },
        { MP_QSTR_connectable, MP_ARG_OBJ | MP_ARG_KW_ONLY, {.u_rom_obj = MP_ROM_TRUE} },
    };
    mp_arg_val_t args[MP_ARRAY_SIZE(allowed_args)];
    mp_arg_parse_all(n_args - 1, pos_args + 1, kw_args, MP_ARRAY_SIZE(allowed_args), allowed_args, args);

    if (args[ARG_interval_us].u_obj == mp_const_none) {
        mp_bluetooth_gap_advertise_stop();
        return mp_const_none;
    }
# later...
STATIC MP_DEFINE_CONST_FUN_OBJ_KW(bluetooth_ble_gap_advertise_obj, 1, bluetooth_ble_gap_advertise);

This is the typical form for Micropython methods callable from Python but written in C (or C++).
Later on in the file at line 934 we have the bluetooth.BLE locals dictionary definition:

STATIC const mp_rom_map_elem_t bluetooth_ble_locals_dict_table[] = {
    // General
    { MP_ROM_QSTR(MP_QSTR_active), MP_ROM_PTR(&bluetooth_ble_active_obj) },
    { MP_ROM_QSTR(MP_QSTR_config), MP_ROM_PTR(&bluetooth_ble_config_obj) },
    { MP_ROM_QSTR(MP_QSTR_irq), MP_ROM_PTR(&bluetooth_ble_irq_obj) },
    // GAP
    { MP_ROM_QSTR(MP_QSTR_gap_advertise), MP_ROM_PTR(&bluetooth_ble_gap_advertise_obj) }
# ...
};
STATIC MP_DEFINE_CONST_DICT(bluetooth_ble_locals_dict, bluetooth_ble_locals_dict_table);

And the globals dictionary:

STATIC MP_DEFINE_CONST_OBJ_TYPE(
    mp_type_bluetooth_ble,
    MP_QSTR_BLE,
    MP_TYPE_FLAG_NONE,
    make_new, bluetooth_ble_make_new,
    locals_dict, &bluetooth_ble_locals_dict
    );

STATIC const mp_rom_map_elem_t mp_module_bluetooth_globals_table[] = {
    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_bluetooth) },
    { MP_ROM_QSTR(MP_QSTR_BLE), MP_ROM_PTR(&mp_type_bluetooth_ble) },
    { MP_ROM_QSTR(MP_QSTR_UUID), MP_ROM_PTR(&mp_type_bluetooth_uuid) },

    // TODO: Deprecate these flags (recommend copying the constants from modbluetooth.h instead).
    { MP_ROM_QSTR(MP_QSTR_FLAG_READ), MP_ROM_INT(MP_BLUETOOTH_CHARACTERISTIC_FLAG_READ) },
    { MP_ROM_QSTR(MP_QSTR_FLAG_WRITE), MP_ROM_INT(MP_BLUETOOTH_CHARACTERISTIC_FLAG_WRITE) },
    { MP_ROM_QSTR(MP_QSTR_FLAG_NOTIFY), MP_ROM_INT(MP_BLUETOOTH_CHARACTERISTIC_FLAG_NOTIFY) },
    { MP_ROM_QSTR(MP_QSTR_FLAG_INDICATE), MP_ROM_INT(MP_BLUETOOTH_CHARACTERISTIC_FLAG_INDICATE) },
    { MP_ROM_QSTR(MP_QSTR_FLAG_WRITE_NO_RESPONSE), MP_ROM_INT(MP_BLUETOOTH_CHARACTERISTIC_FLAG_WRITE_NO_RESPONSE) },
};

STATIC MP_DEFINE_CONST_DICT(mp_module_bluetooth_globals, mp_module_bluetooth_globals_table);

Then we define the module itself:

const mp_obj_module_t mp_module_bluetooth = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t *)&mp_module_bluetooth_globals,
};

// This module should not be extensible (as it is not a CPython standard
// library nor is it necessary to override from the filesystem), however it
// has previously been known as `ubluetooth`, so by making it extensible the
// `ubluetooth` alias will continue to work.
MP_REGISTER_EXTENSIBLE_MODULE(MP_QSTR_bluetooth, mp_module_bluetooth)

[andrewleech]

At this stage I'd suggest running the multitests that do all this stuff on a board running nimble and following how it works there, to then replicate the same pathways for btstack.

I don't think btstack is ever given control of storing bond information itself (aioble handles json file read/write etc) so I wouldn't think btstack would need to be told to "delete bond"?

Any new python API would need to also make sense on nimble (and zephyr) unless the existing one is incompatible with btstack, at which point a larger discussion would be needed to find a new common API that can work across all stacks.

[andrewleech]

The application does not know that the pairing occurs which is not good as it needs to redo the action that invoked the error response to begin with.

Theoretically aioble could repeat that action too, I'd consider that the advantage of writing in asyncio, this sort of thing is easy to do once the automatic pairing is finished.

[brianreinhold]

Theoretically aioble could repeat that action too, I'd consider that the advantage of writing in asyncio, this sort of thing is easy to do once the automatic pairing is finished.

Doing so would certainly lighten the load on the application. However, one still has the issue on reconnects and first time connects of the application not knowing if pairing or encryption has been done.

In the case of a first time connect and a peripheral security request, the application needs to know when bonding is completed so that it does not attempt to invoke an action which requires bonding. Usually the process of service discovery (which by spec cannot require security) takes long enough that bonding is done before the application can perform some further action.

In the case of a bonded reconnect, most high level APIs automatically invoke encryption, security request or not, aioble does not, so the application needs to handle an insufficient encryption error on its first non-service discovery action. If there is a peripheral security request, the btstack does handle encryption but the application does not know when it is complete, so it can perform some action. In addition, there is no means for the application to invoke encryption given it is bonded.

If aioble did not return on its Device.connect method until encryption was established in a bonded reconnect (regardless of a security request or not), that would solve all the insufficient encryption issues.

[andrewleech]

When bonding is stored and used with aioble on existing nimble based ports encryption is definitely automatically reestablished on reconnection, nimble itself does this and aioble just gets a couple of events saying connection established and encryption update.

I'd expect that btstack should also do the same, as a bonded devices is reconnected the encryption should be automatically handled.

Your other points are really useful, once the basic pairing is working I'd be keen to do some side by side tests with nimble boards I've got and see any discrepancies in workflow, it'll be worth adding new multitests to cover these use cases and automatically handled everything sensible while throwing events to the application / user for things they should know / respond to!

[brianreinhold]

The btstack does do encryption if the bonded device sends a security request. But if I pair via aioble in response to an insufficient authentication error, it does not do encryption on a bonded reconnect. We don't know why. We are going to be looking into this, and perhaps there are people at btstack who can answer that. We do have a good starting point in that everything works correctly if the peripheral does a security request.

Right now what happens is that the peripheral will invoke an insufficient encryption error on a bonded reconnect as soon as the client tries to perform a secured operation. One way around that is to introduce a method in aioble that does encryption and waits until encryption is established -- that is a good thing!

[brianreinhold]

@andrewleech I invoked a few hacks to get around the various pairing/encryption issues.

• I made a new method in aioble that mirrors the security.pair() method which I call security_request_check() which does all the things the pairing method does EXCEPT pair. This method is called right after the connect. It has a timeout and essentially checks to see if there is a security request and if pairing/encryption is invoked under the hood this passes up the encryption changed to aioble.
• I had to make an extra case to handle the SM_EVENT_REENCRYPTION_COMPLETE in the modbluetooth_btstack and pass that up to aioble's encryption changed,

I had to mess around with a few other things, don't think I did anything in the btstack code, however.
However, with these changes, the application can get notified of pairing and encryption due security requests on both the initial connection and bonded reconnect. aioble is also becomes aware of the event and thus reports correct states of DeviceConnection.bond and DeviceConnection.encrypted states. The downside is the application has to 'await' a fixed amount of time and hope the operation completes in that time. I do it right after connection for 2 seconds. If there is no security request nothing happens for that 2 seconds.

If there is no security request I can handle either insufficient authentication or encryption errors by calling the pair method. In the encryption case btstack is smart enough to know that it is bonded and just encrypts. By adding the SM_EVENT_REENCRYPTION_COMPLETE event to the events that get passed up to aioble the application can redo the operation.

Now all cases 'work', connect and bonded reconnect with and without security requests and as well as insufficient authentication and encryption errors. My application also records the bonded state of known devices and that way knows whether to invoke encryption on a reconnect if a security request is not done.

Its messy as I don't know what all the details are and I have not handled all the error cases. Mainstream first!