Getting OSError: exception: access violation writing #72
Comments
|
Hello, thanks for the very detailed report. At a glance I do not see anything wrong with the code structure. The only less-usual thing I see is the creation of a second context in On to the error: this looks like a double-free, somehow, which points at a double- Could you add the following line as first line of print('<USBDevice %x>.close called, self.device_p=%r (%s)' % (id(self), self.device_p, bool(self.device_p)) |
|
What is really strange is that when I call the |
Interpreter shutdown is doing very unusual things when trying to call every destructor. Like replacing all module globals with This is why I went with context-manager-based instance lifetime control: this provides unambiguous teardown ordering, as it happens before reaching interpreter shutdown but instead
Thanks, and these outputs all look fine to me: there is no obvious reason why a null pointer appears here. |
|
Actually I don't have a problem with that but cause my unit tests to fail. |
|
I'm sorry but I do not understand what you mean in your latest comment. |
|
Suppose I don't call |
|
If I stick to my gut feeling that this is an interpreter-shutdown-specific issue, one thing I can think of is that maybe the presence/absence of When I wrote the first few version of python-libusb1 I thought I would be able to get away with freeing memory in So I am not too surprised that relying on |
|
Interestingly I do not have any issues under my Mac Mini M1 running macOS Big Sur 11.5 ARM64 version with the test code mentioned in the first post. Recently I got the Cypress FX3 board and it is a wonderful board to play with USB under different OS (Windows, macOS and Linux for me). |
|
But I can reproduce the issue under Windows with the official Python 3.9.6 x64 version. |
|
And if I use MSYS2 MinGW64 Python, it actually segfaults. |
|
With Windows Python, the full debug log (LIBUSB_DEBUG=4). Not so sure if this helps. click to expand |
|
The following debug log seems to be a bit suspicious. Is it possible that only the default context was detroyed and not the second second context? Normally the debug log should be something like the following, without the leak warning. |
Not true. Even if I comment out the following line, it still has the same problem. |
|
This looks like a specific issue with Windows, it may be a libusb issue or a python-libusb1 issue, but more like a python-libusb1 issue to me. I will try out under Linux to see if the issue is there or not. |
|
No issues on libusb under my Linux machine (Raspberry Pi 400 running 64bit Ubuntu Linux ARM). And no leak from the debug log either. |
|
@vpelletier Since the fault seems to happen because libusb_unref_device() is called after libusb_exit() and that seems to cause problem under Windows (but no problems under Linux), the following dirty hack seems to sort out the issue. But of course this does not seem to be the right fix as seen in the warnings of the debug log. |
|
I have raised libusb/libusb#974 to see if libusb can do better, at least for Windows. |
|
Thanks for the extensive testing and debugging.
One particularity of python-libusb1 is that my code never uses the default context, because this seemed the sane thing to do, and it was mapping very well in the object world. I suspect this may differ from most uses, and could cause unusual codepaths to be exercised. Could the windows backend accidentally rely on this ? The default context destruction triggering warnings about |
|
Let me try out on the libusb windows side. It is possible that there is an issue there. Thanks. |
|
So far I have not been able to reproduce the issues with libusb yet. Ref: stress test codes have some simple tests with two contexts. I have also adapted a few examples with non-default context (but single context) but again I am not able to reproduce the issue. But probably they are not good enough. |
|
If we look at the debug log, one strange thing is like the following. There seems to be two times of libusb_get_device_list() and libusb_close() being called. The following line is commented out here. There is only single libusb context here so context handling does not seem to be the issue. click to expand |
|
Adding some debug print. |
There is one special thing about EDIT: This is apparently not actually the case, the code somehow jumps to any I do not see yet why One way to check this could be to call |
|
...and I just noticed that your code calls |
|
Could you give a try to the following patch ? The context refcount wrapper (being patched here) can return before diff --git a/usb1/__init__.py b/usb1/__init__.py
index 1648124..6cd33cb 100644
--- a/usb1/__init__.py
+++ b/usb1/__init__.py
@@ -2085,9 +2085,13 @@ class USBContext(object):
with refcount(self):
if self.__context_p:
# pylint: disable=not-callable
- for value in func(self, *args, **kw):
- # pylint: enable=not-callable
- yield value
+ generator = func(self, *args, **kw)
+ # pylint: enable=not-callable
+ try:
+ for value in generator:
+ yield value
+ finally:
+ generator.close()
else:
def wrapper(self, *args, **kw):
with refcount(self): |
Thanks. I think this is the root cause of the issue. If I delete the read function, then there is no issue. |
This does not seem to help. |
|
Here is the quick hack but working codes with only single __get_handle call. Take note I remove the exception handlings to make it easy for me. |
mcuee
added a commit
to mcuee/python_usb
that referenced
this issue
Sep 1, 2021
From: vpelletier/python-libusb1#72 Quick hacky but working codes with only single __get_handle call. Take note I remove the exception handlings to make it easy for me.
[...] Excellent. And device 1.0 appears too, which I was not sure about. Here I feel we have touched the bottom of this issue. I think I have found a cleaner way to fix this issue than explicit gc calls. I pushed the fix in a temporary branch, could you try this (at least without the |
|
The wip branch seems to help but there are a lot of warnings. And I think it still has some issues. Main issue is that the bahavior is not so consistent and sometimes the app will hang. And often there is this "usb1.USBErrorAccess: LIBUSB_ERROR_ACCESS [-3]" error message. Code: Before the change (using release 1.9.3 version): After: |
|
In the case of hang: |
|
BTW, I am not sure if this has anything to do with the topic here: rene-aguirre/pywinusb#56 . The issue is not related to libusb (pywinusb is only for HID device despite the name. It is Windows only). Interestingly, the pywinusb issue goes away with Python 3.9. So the issue is not exactly the same. Nevertheless it seems to me Windows Python is a bit different from other platforms. |
Woops, I did not notice it would receive the weakref and not the referent. I fixed this and pushed to wip branch. But I also found something else in the weakref documentation which may make this solution impossible. If you get similar errors but with |
I am wary about making conclusions when there is another issue already going on (the weakref callback raising). Garbage collection is a very ticklish beast, small changes can make a large impact. Adding or removing |
Unfortunately this seems to be the case. |
|
I think I have a better solution, just force-pushed in |
|
This does not seem to work. |
Woops, I had fixed one such occurrence, and forgot about the 2 others, sorry. I pushed a fix. |
|
Now it hangs without any errors. debug log |
|
This error is a bit weird: visibly this happens in Checking this, I did found a bug with my change, but it should be unrelated to this hang (it was not possible to open the same device more than once). I fixed it, and tested a minimally modified version of you test code (only changing vendor id and device id to match those used in python-functionfs/examples/usbcat/device.py) and it is working on both win10 (with WinUSB driver attached to my pizero running usbcat/device.py) and linux. Note that usbcat is not a loopback test but a bidirectional pipe (like netcat). As I did not provide any input to it, codeimport usb1
class USB:
"""USB class that handles IO operation with USB device
ENDPOINT_IN: device-to-host, ENDPOINT_OUT: host-to-device"""
# read size chunk
READ_CHUNK = 1024
def __init__(self, pid: hex, vid: hex, endpoint_in: hex, endpoint_out: hex) -> None:
self.endpoint_in = endpoint_in
self.endpoint_out = endpoint_out
self.pid = pid
self.vid = vid
self.context = usb1.USBContext()
def is_connected(self) -> bool:
"""Check if specified device is connected"""
with usb1.USBContext() as context:
for device in context.getDeviceIterator(skip_on_error=True):
if device.getVendorID() == self.vid and device.getProductID() == self.pid:
return True
def open(self) -> bool:
"""Open USB and initialize interface"""
try:
self.context.open()
return True
except Exception as err:
print("open device error:", err)
return False
def __get_handle(self):
"""return handle object"""
return self.context.openByVendorIDAndProductID(self.vid, self.pid)
def close(self) -> bool:
"""Close USB"""
print("Closing USB")
try:
self.context.close()
print("Close handle successfully")
return True
except Exception as err:
print("Close handle error:", err)
return False
def write(self, msg: bytearray, timeout: int = 0) -> bool:
"""write an specific msg to device"""
handle = self.__get_handle()
if not handle:
return False
try:
with handle.claimInterface(0):
bytes_written = handle.bulkWrite(
self.endpoint_out, msg, timeout)
bytes_written == len(msg)
print("Number of bytes written: ", bytes_written)
return True
except Exception as err:
print("write error", err)
#handle.close()
return False
def read(self, timeout: int = 10) -> bytearray:
"""read data from the device"""
data = bytearray()
handle = self.__get_handle()
if not handle:
return False
try:
with handle.claimInterface(0):
while True:
try:
data += handle.bulkRead(self.endpoint_in,
self.READ_CHUNK, timeout)
except usb1.USBErrorTimeout:
break
except Exception as err:
print("read error", err)
return None
handle.close()
return data
def __del__(self):
print("Calling closing method to delete self")
self.close()
if __name__ == '__main__':
VENDOR_ID = 0x1d6b
PRODUCT_ID = 0x0104
ENDPOINT_IN = 0x81
ENDPOINT_OUT = 0x01
usb = USB(PRODUCT_ID, VENDOR_ID, ENDPOINT_IN, ENDPOINT_OUT)
#print(usb.is_connected())
msg = 100 * bytearray(b"B")
if usb.open():
print("write:", usb.write(msg))
print("read:", usb.read())
# error after adding this line
usb.close()linux output (LIBUSB_DEBUG=4)win10 output (LIBUSB_DEBUG=4)EDIT: the linux output includes non-committed debug prints in finalizer {un,}register functions. |
|
Yes libusb sync transfer is internally based on async plus a blocking handle_events call. I believe the device is okay as I tested with pyusb based code as well before using python-libusb1. I will try another device, probably Linux Gadegt Zero. |
|
Ah, I did not finish explaining my train of thought: As you do not create any async transfer and do not call So while I trust that your device is working, I was suspecting that for whatever reason (my previous bug having put it and/or windows in a weird state ?) it was not responding anymore. Maybe unplug/replug could have been enough (more if it was windows that was confused). |
|
I think you are right. I tried it again (but with another Windows 10 laptop at home) and it works fine (tested with libusb-1.0.23, 1.0.24 and latest git 1.0.24.11650). So I think the fixes are good. There is a warning in the end of the debug log though. But I am not sure if it is an issue with libusb Windows or python-libusb1. Full debug log |
|
And your example works fine as well under my OrangePi Zero with old Python 3.5 and ARMBIAN 5.60 stable Debian GNU/Linux 9 (stretch) 4.14.70-sunxi. OrangePi Zero Linux USB Gadget Windows 10 host |
This is indeed a bit surprising. In a previous try (with explicit I now consider my changes to be in the right direction, so I pushed them to master and dropped the
Awesome. So far I personally have only run it on an intel edison (udc=DWC3 but without companion chip for USB3 support and some quirks - some endpoints do not work) and on a raspberry pi zero (udc=DWC2). Good to know that it is working on more gadget-able devices. |
If you look at your Windows 10 debug log, you will notice the following about 'some libusb_devices were leaked'. You are using libusb-1.0.24 release which gives this debug log. libusb-1.0 git now gives more detailed information about which devices were leaked. And then there is an error in the debug log as well.
Great, |
Going back to my original Windows 10 laptop and it also works fine, tested with latest git head now that you have integrated the patches. |
|
This The |
|
I will create a separate issue with the device leak issue. |
Nice catch, I missed this in all the debug output. This should be fixed by 976866d (checked with my win10 vm, the error message does not appear anymore). |
|
Yes the latest git fixed the warning, Thanks. |
|
Thanks for the contribution. |
|
I'm closing this, but please do reopen if you find that it is not fixed. |
This was referenced Sep 9, 2021
mcuee
added a commit
to mcuee/python_usb
that referenced
this issue
Sep 13, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
I've a created a sample class for libusb but when I call the
closemethod I get access violation writing error. When theclosemethod calls with destructor it works just fine. I'm not sure if I'm missing something in here.error:
The text was updated successfully, but these errors were encountered: