Removing Objects created from external library and attached to root-pointers

[birdistheword96]

I have a micropython list object, which is set as a root-pointer and therefore excempt from garbage collection. The user can then call an api which returns another type of micropython object, which is stored in that list to avoid garbage collection. What I now need to do is handle when the user deletes that object.

I know I need to call mp_obj_list_remove to remove the object from the list, but what happens if the user calls del to remove the object? Or if they create it within a local function, then when they leave that function the object needs to be removed. How can I safely remove the object and free the underlying memory? Is there a way I can define a function as the destructor? I am guessing it is partly through the use of m_tracked_calloc and m_tracked_free but I can't find any good examples of how to use that.

Any pointers appreciated!

---

EDIT MORE INFO: using uPy v1.10, unable to upgrade yet so please don't suggest that as an answer

I have a Micropython object (lets call it a DataStore object) that is created from a C module, that allows user to read a write data into. The Datastore object contains a C-allocated buffer for storing this data, and then another RTOS task can read and write into the same buffer externally from micropython.

The list object is purely used as a root pointer to store the Datastore objects, it is not accessible from python. The user can then create any number of these Data store objects, and use them within their code. What I want to know is the best way to handle these objects being deleted. I can create a function called 'remove', which removes the object from the root list pointer like so:

mp_obj_t mp_obj_new_datastore(mp_obj_t _name)
{
    /* Create the uPy object. */
    mp_obj_datastore_t *o = m_new_obj(mp_obj_datastore_t);

    o->base.type = &mp_type_datastore;
    o->name = _name;
    memset(o->buffer, 0x00, sizeof(float32_t) * MAX_DEPTH);

    int index = datastore_root_pointer_list->len;
    mp_obj_list_append(datastore_root_pointer_list, o);

    return upy_channels_store->items[index];
}

STATIC mp_obj_t datastore_channel_remove(mp_obj_t self_in) {
    mp_obj_list_remove(datastore_root_pointer_list, self_in);    
    return mp_const_none;
}

This leads me to I suppose two separate questions:

How do I handle this object getting deleted with the del keyword from python, or going out of scope when created as a local variable and exiting the function, etc? In cpython I belive the del function is called when a function is deleted, is that true of Micropython too? How do I assign something to that?

Once the object is removed from the list, how best should I free the memory allocated? Can I just free the memory like I would a normal struct in C, or do I have to do something special in micropython (like set the reference count to zero or something) because of the garbage collector and then let it free the memory?

[jimmo]

I think you might need to explain in more detail what's going on.

Is the list accessible from Python? What is the user applying the del operator to? What are these objects and why do you need to make them reachable (when it sounds like they are being returned to Python).

(I guess what I'm asking is, why do you need to manually make them reachable by the GC when they're being returned to Python which will do that for you by virtue of being referenced either by local variables or some other object. If you want some sort of cleanup for them when they do become unreachable, then you can use a finaliser)

m_tracked_calloc (and free) are kind of like an "auto root pointer" malloc -- i.e. it's a malloc that automatically makes what it returns reachable from a root pointer.

[birdistheword96]

I will try to explain better what is going on (and update the main question with this additional info too):

I have a Micropython object (lets call it a DataStore object) that is created from a C module, that allows user to read a write data into. The Datastore object contains a C-allocated buffer for storing this data, and then another RTOS task can read and write into the same buffer externally from micropython.

The list object is purely used as a root pointer to store the Datastore objects, it is not accessible from python. The user can then create any number of these Data store objects, and use them within their code. What I want to know is the best way to handle these objects being deleted. I can create a function called 'remove', which removes the object from the root list pointer like so:

mp_obj_t mp_obj_new_datastore(mp_obj_t _name)
{
    /* Create the uPy object. */
    mp_obj_datastore_t *o = m_new_obj(mp_obj_datastore_t);

    o->base.type = &mp_type_datastore;
    o->name = _name;
    memset(o->buffer, 0x00, sizeof(float32_t) * MAX_DEPTH);

    int index = datastore_root_pointer_list->len;
    mp_obj_list_append(datastore_root_pointer_list, o);

    return upy_channels_store->items[index];
}

STATIC mp_obj_t datastore_channel_remove(mp_obj_t self_in) {
    mp_obj_list_remove(datastore_root_pointer_list, self_in);    
    return mp_const_none;
}

This leads me to I suppose two separate questions:

1. How do I handle this object getting deleted with the del keyword from python, or going out of scope when created as a local variable and exiting the function, etc? In cpython I belive the __del__ function is called when a function is deleted, is that true of Micropython too? How do I assign something to that?

2. Once the object is removed from the list, how best should I free the memory allocated? Can I just free the memory like I would a normal struct in C, or do I have to do something special in micropython (like set the reference count to zero or something) because of the garbage collector and then let it free the memory?

[stinos]

1. Search the MicroPython source code for __del__ for examples: wrap your datastore_channel_remove like STATIC MP_DEFINE_CONST_FUN_OBJ_1(datastore_channel_remove_obj, datastore_channel_remove); then add that datastore_channel_remove_obj to your locals dict with the name MP_QSTR___del__. It's similar to CPython.

2. You don't do anything, and because your object isn't in the root pointers list anymore, gc will take care of it. Not that that is not a guarantee when or even if that will happen, but that is just how gc works.

[birdistheword96]

You also need to use the 'with finaliser' variant of the new-object funtion:

mp_obj_datastore_t *o = m_new_obj_with_finaliser(mp_obj_datastore_t);

[stinos]

Yes you do need MICROPY_ENABLE_FINALISER, see gc.c around line 374 and my previous remark: 'that is not a guarantee when or even if that will happen, but that is just how gc works.': the finaliser is only called once the object gets garbage collected.

However I just realise there might be a chicken-or-egg problem in this case: if your upy_channels_store is also a root pointer, then your __del__ is never going to be called because the GC will always see your object as root pointer hence will never garbagce collect it. So that's a design problem. If you need a fnialiser, the object you return should itself not be in the root pointer list. Or you need a separate function to tell 'remove yourself from root pointer list'. Or you could use __exit__ for that if it's the type of object for which it makes sense to use it with the with statement.

[jimmo]

However I just realise there might be a chicken-or-egg problem in this case

Yep, @birdistheword96 this is kind of what I was getting at -- why do you need this list?

The main example of when things need to be root pointers is when a Python object is passed to C code, where the only reference to it is a global C variable (i.e. in .data or .bss). An irq handler defined in Python is a good idea -- you pass it to the pin object, but the pin object doesn't live on the heap (they're pre-created in RAM), so we need to ensure that the collector can find that irq object, because possibly nothing in Python refers to it any more.

In your case, if you want an object to be collectable when the user stops referencing it in Python, then it doesn't sound like you have to anything extra.

And just to clarify a couple of things:

• There is no reference counting in MicroPython. The only way we know when an object is "dead" is when the collector runs and discovers that nothing points to it (via mark&sweep). So there's no way to hook del x -- all del x does is remove x from scope (we have no idea at this point what else might be referencing it).
• As a result, the only guarantee is that the finaliser will run "later" when the collector runs (but possibly not at all in some cases like if the board is reset of course).

[birdistheword96]

I initially didn't bother with using a root pointer, but I kept finding that garbage collection was removing them the objects, even though they were globally attached to a class. I also tried delcaring them globally on their own, but found the same behaviour. Because they are created to act as a shared buffer between the uPy task and another task, this was causing a hardFault that brought the rest of the system down.

So I attached them to a root_pointer (the list described above) and the problem immediately went away.

I have a function for removing them from the root_pointer list .remove(), but I don't know how to handle removing them from the list if the user calls 'del' or there are no more references to it. I agree, this is a design issue, and also it sounds like I want garbage collection to do exactly what it is meant to, avoiding root-pointers all together, but I know for a fact this was causing memory fault issues so it has to stay attached to the root pointer

[jimmo]

they were globally attached to a class

What does this mean? If there's a (live) Python object that has a reference to your object (i.e. as an attribute), and the object is being collected, then something is going wrong (or that Python object isn't truly live).

What you're describing has a lot in common with things like file objects or sockets. In MicroPython both of these have finalisers, but they are really intended to be explicitly close()'ed by the user (either directly or with a with statement). The finaliser is just a fallback.

So it's quite common to have a type that has it's __del__ entry in locals dict point to the same function as close (and implement an __exit__ that forwards to close). See for example in a file object:

STATIC mp_obj_t file_obj___exit__(size_t n_args, const mp_obj_t *args) {
    (void)n_args;
    return mp_stream_close(args[0]);
}
STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(file_obj___exit___obj, 4, 4, file_obj___exit__);

STATIC const mp_rom_map_elem_t vfs_fat_rawfile_locals_dict_table[] = {
...
    { MP_ROM_QSTR(MP_QSTR_close), MP_ROM_PTR(&mp_stream_close_obj) },
    { MP_ROM_QSTR(MP_QSTR___del__), MP_ROM_PTR(&mp_stream_close_obj) },
    { MP_ROM_QSTR(MP_QSTR___enter__), MP_ROM_PTR(&mp_identity_obj) },
    { MP_ROM_QSTR(MP_QSTR___exit__), MP_ROM_PTR(&file_obj___exit___obj) },
...
};

So your implementation of close should unregister the object from anything that might cause the other FreeRTOS task to access them, as well as removing them from the list that is itself declared as a root pointer. In your case m_tracked_alloc doesn't make sense because you need the object to still exist after close() (i.e. it can't self-free in close()) it just needs to become unregistered in close() and eligible for collection.