[4.6.2] KeyError: '/' in fake_filesystem.FakeDirectory.get_entry() #692

jcfp · 2022-07-19T14:53:03Z

Recent changes in pyfakefs resulted in test failures in sabnzbd, discussed at sabnzbd/sabnzbd#2242. The failures go away when limiting pyfakefs to < 4.6.0 in the test requirements so this appears to be a regression, possibly related to #676.

Example of a unittest that triggers the issue: https://github.com/sabnzbd/sabnzbd/blob/3acfe194996aa8e2409644a9135ccb37b76dc0c9/tests/test_filesystem.py#L623

Environment: github CI, Linux and Mac; Python 3.7 through 3.10; pyfakefs 4.6.2

Traceback:

2022-07-18T20:28:42.2077001Z ___________________ TestListdirFullWin.test_nonexistent_dir ____________________
2022-07-18T20:28:42.2077197Z 
2022-07-18T20:28:42.2077394Z self = <tests.test_filesystem.TestListdirFullWin testMethod=test_nonexistent_dir>
2022-07-18T20:28:42.2077623Z 
2022-07-18T20:28:42.2077733Z     def test_nonexistent_dir(self):
2022-07-18T20:28:42.2078027Z >       assert filesystem.listdir_full(r"F:\foo\bar") == []
2022-07-18T20:28:42.2078205Z 
2022-07-18T20:28:42.2078300Z tests/test_filesystem.py:624: 
2022-07-18T20:28:42.2078562Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2022-07-18T20:28:42.2078818Z sabnzbd/decorators.py:36: in call_func
2022-07-18T20:28:42.2079052Z     return f(*args, **kw)
2022-07-18T20:28:42.2079302Z sabnzbd/filesystem.py:767: in listdir_full
2022-07-18T20:28:42.2079583Z     for root, dirs, files in os.walk(input_dir):
2022-07-18T20:28:42.2080045Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_scandir.py:233: in do_walk
2022-07-18T20:28:42.2080409Z     top_contents = _classify_directory_contents(filesystem, top_dir)
2022-07-18T20:28:42.2080940Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_scandir.py:202: in _classify_directory_contents
2022-07-18T20:28:42.2081308Z     for entry in filesystem.listdir(root):
2022-07-18T20:28:42.2081762Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:3381: in listdir
2022-07-18T20:28:42.2082151Z     target_directory = self.resolve_path(target_directory, allow_fd=True)
2022-07-18T20:28:42.2082664Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:2055: in resolve_path
2022-07-18T20:28:42.2083040Z     path = self.absnormpath(self._original_path(path))
2022-07-18T20:28:42.2083522Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1641: in _original_path
2022-07-18T20:28:42.2083936Z     path = self.replace_windows_root(path)
2022-07-18T20:28:42.2084435Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1901: in replace_windows_root
2022-07-18T20:28:42.2084797Z     if path and self.is_windows_fs and self.root_dir:
2022-07-18T20:28:42.2085268Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:990: in root_dir
2022-07-18T20:28:42.2085608Z     return self._mount_point_dir_for_cwd()
2022-07-18T20:28:42.2086099Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1230: in _mount_point_dir_for_cwd
2022-07-18T20:28:42.2086437Z     return object_from_path(mount_path)
2022-07-18T20:28:42.2086911Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1224: in object_from_path
2022-07-18T20:28:42.2087316Z     target = target.get_entry(component)
2022-07-18T20:28:42.2087574Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2022-07-18T20:28:42.2087731Z 
2022-07-18T20:28:42.2087912Z self = <pyfakefs.fake_filesystem.FakeDirectory object at 0x7f084b6d4250>
2022-07-18T20:28:42.2088247Z pathname_name = '/'
2022-07-18T20:28:42.2088382Z 
2022-07-18T20:28:42.2088592Z     def get_entry(self, pathname_name: str) -> AnyFile:
2022-07-18T20:28:42.2088895Z         """Retrieves the specified child file or directory entry.
2022-07-18T20:28:42.2089144Z     
2022-07-18T20:28:42.2089323Z         Args:
2022-07-18T20:28:42.2089578Z             pathname_name: The basename of the child object to retrieve.
2022-07-18T20:28:42.2089827Z     
2022-07-18T20:28:42.2090006Z         Returns:
2022-07-18T20:28:42.2090230Z             The fake file or directory object.
2022-07-18T20:28:42.2090447Z     
2022-07-18T20:28:42.2090628Z         Raises:
2022-07-18T20:28:42.2090871Z             KeyError: if no child exists by the specified name.
2022-07-18T20:28:42.2091119Z         """
2022-07-18T20:28:42.2094995Z         pathname_name = self._normalized_entryname(pathname_name)
2022-07-18T20:28:42.2095332Z >       return self.entries[to_string(pathname_name)]
2022-07-18T20:28:42.2095636Z E       KeyError: '/'
2022-07-18T20:28:42.2095770Z 
2022-07-18T20:28:42.2096193Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:733: KeyError

And the rather similar:

2022-07-18T20:28:42.1921244Z ____________________ TestSanitizeFiles.test_sanitize_files _____________________
2022-07-18T20:28:42.1921451Z 
2022-07-18T20:28:42.1921657Z args = (<tests.test_filesystem.TestSanitizeFiles testMethod=test_sanitize_files>,)
2022-07-18T20:28:42.1922022Z kwargs = {}, is_windows = False, is_macos = False
2022-07-18T20:28:42.1922191Z 
2022-07-18T20:28:42.1922314Z     def wrapper_func(*args, **kwargs):
2022-07-18T20:28:42.1922548Z         # Save original values
2022-07-18T20:28:42.1922784Z         is_windows = sabnzbd.WIN32
2022-07-18T20:28:42.1923025Z         is_macos = sabnzbd.MACOS
2022-07-18T20:28:42.1923219Z     
2022-07-18T20:28:42.1923641Z         # Set current platform
2022-07-18T20:28:42.1923872Z         if platform == "win32":
2022-07-18T20:28:42.1924092Z             sabnzbd.WIN32 = True
2022-07-18T20:28:42.1924326Z             sabnzbd.MACOS = False
2022-07-18T20:28:42.1924570Z         elif platform == "macos":
2022-07-18T20:28:42.1924788Z             sabnzbd.WIN32 = False
2022-07-18T20:28:42.1925021Z             sabnzbd.MACOS = True
2022-07-18T20:28:42.1925256Z         elif platform == "linux":
2022-07-18T20:28:42.1925485Z             sabnzbd.WIN32 = False
2022-07-18T20:28:42.1925699Z             sabnzbd.MACOS = False
2022-07-18T20:28:42.1925966Z     
2022-07-18T20:28:42.1926230Z         # Perform test
2022-07-18T20:28:42.1926458Z >       value = func(*args, **kwargs)
2022-07-18T20:28:42.1926606Z 
2022-07-18T20:28:42.1926707Z tests/testhelper.py:112: 
2022-07-18T20:28:42.1926947Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2022-07-18T20:28:42.1927214Z tests/testhelper.py:79: in wrapper_func
2022-07-18T20:28:42.1927474Z     value = func(*args, **kwargs)
2022-07-18T20:28:42.1927730Z tests/test_filesystem.py:281: in test_sanitize_files
2022-07-18T20:28:42.1928003Z     self.fs.create_file(file)
2022-07-18T20:28:42.1928691Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:2646: in create_file
2022-07-18T20:28:42.1929027Z     return self.create_file_internally(
2022-07-18T20:28:42.1929533Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:2861: in create_file_internally
2022-07-18T20:28:42.1929891Z     path = self.absnormpath(path)
2022-07-18T20:28:42.1930357Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1685: in absnormpath
2022-07-18T20:28:42.1930694Z     path = self.replace_windows_root(path)
2022-07-18T20:28:42.1931184Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1901: in replace_windows_root
2022-07-18T20:28:42.1931565Z     if path and self.is_windows_fs and self.root_dir:
2022-07-18T20:28:42.1932034Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:990: in root_dir
2022-07-18T20:28:42.1932378Z     return self._mount_point_dir_for_cwd()
2022-07-18T20:28:42.1932874Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1230: in _mount_point_dir_for_cwd
2022-07-18T20:28:42.1933231Z     return object_from_path(mount_path)
2022-07-18T20:28:42.1933696Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1224: in object_from_path
2022-07-18T20:28:42.1934049Z     target = target.get_entry(component)
2022-07-18T20:28:42.1934318Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2022-07-18T20:28:42.1934477Z 
2022-07-18T20:28:42.1934646Z self = <pyfakefs.fake_filesystem.FakeDirectory object at 0x7f084bd449a0>
2022-07-18T20:28:42.1934978Z pathname_name = '/'
2022-07-18T20:28:42.1935114Z 
2022-07-18T20:28:42.1935324Z     def get_entry(self, pathname_name: str) -> AnyFile:
2022-07-18T20:28:42.1935648Z         """Retrieves the specified child file or directory entry.
2022-07-18T20:28:42.1935884Z     
2022-07-18T20:28:42.1936062Z         Args:
2022-07-18T20:28:42.1936445Z             pathname_name: The basename of the child object to retrieve.
2022-07-18T20:28:42.1936686Z     
2022-07-18T20:28:42.1936866Z         Returns:
2022-07-18T20:28:42.1937111Z             The fake file or directory object.
2022-07-18T20:28:42.1937318Z     
2022-07-18T20:28:42.1937498Z         Raises:
2022-07-18T20:28:42.1937758Z             KeyError: if no child exists by the specified name.
2022-07-18T20:28:42.1937989Z         """
2022-07-18T20:28:42.1938256Z         pathname_name = self._normalized_entryname(pathname_name)
2022-07-18T20:28:42.1938575Z >       return self.entries[to_string(pathname_name)]
2022-07-18T20:28:42.1938864Z E       KeyError: '/'
2022-07-18T20:28:42.1938996Z 
2022-07-18T20:28:42.1939305Z ../../../.virtualenvs/.venv/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:733: KeyError

The text was updated successfully, but these errors were encountered:

mrbean-bremen · 2022-07-19T15:40:53Z

Thanks for the report! Looks like that release was premature, next time we should consider a pre-release for changes that may create this kind of regressions.

At a first glance, this looks like the problem is in changing the os enviroment in the test. Due to some internal changes, it might not be suffcient to just change the values as is done in the code:

    def setUp(self):
        self.setUpPyfakefs()
        self.fs.is_windows_fs = True
        self.fs.path_separator = "\\"
        self.fs.is_case_sensitive = False

This now probably needs an additional self.fs.reset(), or better use the standard method to change the os that does that automatically:

    def setUp(self):
        self.setUpPyfakefs()
        self.fs.os = OSType.WINDOWS

I will check if this is the only problem here with your repo (if you have the time, you can try it yourself). If it is, I shall at least update the documentation and the release notes, and try to make this upwards-compatible, for example by automatically resetting the fs if setting fs.is_windows_fs or fs.is_macos.

mrbean-bremen · 2022-07-19T16:30:54Z

Ok, this only fixes some of your tests (10 of 30 failing). I will check the rest - this may take some time...

mrbean-bremen · 2022-07-19T17:55:47Z

The rest of the tests failed due to a regression, that let to os.path.exists return True for any Windows drive root path. Obviously have been missing some tests... If using the branch where this is fixed (together with the previous change), the tests seem to pass.

mrbean-bremen · 2022-07-19T19:48:16Z

Hm, I couldn't get it to work completely without changing the test code - some tests (4 of them) are still failing, and I'm not sure why yet. I may have another look tomorrow evening...

jcfp · 2022-07-19T19:56:29Z

Thanks for the quick response, much appreciated!

- was always detected as existing - caused part of pytest-dev#692

- was always detected as existing - caused part of #692

…xisting - caused part of #692

mrbean-bremen · 2022-07-20T18:29:49Z

Ok, I'm not sure what the problem was yesterday - not sure if the build used the latest version of pyfakefs. It seems now to work, so I will merge this to master and probably make a new patch release shortly afterwards.
This version should then work without changing the test code for your tests. I saw that you already changed the code that changes the OS, which is cleaner anyway, but your code helped me to fix the problem for others, so thanks for that! You still need the fix for the regression, though.

mrbean-bremen · 2022-07-20T18:55:14Z

Release 4.6.3 is out - setting this to fixed. Feel free to reopen if you still have problems.

Safihre · 2022-07-21T09:45:19Z

Confirmed all ✔!
https://github.com/sabnzbd/sabnzbd/runs/7446508804

Thanks for the quick fix!

mrbean-bremen added the regression label Jul 19, 2022

mrbean-bremen added the bug label Jul 19, 2022

jcfp mentioned this issue Jul 19, 2022

set OSType in pyfakefs instead of separate properties sabnzbd/sabnzbd#2243

Merged

mrbean-bremen added a commit to mrbean-bremen/pyfakefs that referenced this issue Jul 20, 2022

Fix handling of Windows drive as root path

34829c0

- was always detected as existing - caused part of pytest-dev#692

mrbean-bremen mentioned this issue Jul 20, 2022

Fix handling of Windows drive as root path #694

Merged

mrbean-bremen added a commit that referenced this issue Jul 20, 2022

Fix handling of Windows drive as root path

c640ddc

- was always detected as existing - caused part of #692

github-actions bot pushed a commit that referenced this issue Jul 20, 2022

Fix handling of Windows drive as root path - was always detected as e…

906e7af

…xisting - caused part of #692

mrbean-bremen closed this as completed Jul 20, 2022

mrbean-bremen mentioned this issue Jul 20, 2022

Infinite loop in test with version 4.6.2 #691

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[4.6.2] KeyError: '/' in fake_filesystem.FakeDirectory.get_entry() #692

[4.6.2] KeyError: '/' in fake_filesystem.FakeDirectory.get_entry() #692

jcfp commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

jcfp commented Jul 19, 2022

mrbean-bremen commented Jul 20, 2022

mrbean-bremen commented Jul 20, 2022

Safihre commented Jul 21, 2022

[4.6.2] KeyError: '/' in fake_filesystem.FakeDirectory.get_entry() #692

[4.6.2] KeyError: '/' in fake_filesystem.FakeDirectory.get_entry() #692

Comments

jcfp commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

mrbean-bremen commented Jul 19, 2022

jcfp commented Jul 19, 2022

mrbean-bremen commented Jul 20, 2022

mrbean-bremen commented Jul 20, 2022

Safihre commented Jul 21, 2022