Added methods to access real files #171

mrbean-bremen · 2017-04-01T17:08:29Z

allow to add really existing files and directory trees to the fake
file system, with the contents read on demand
see [wanted] Easy way to 'embed' certain directories in the virtual-file system #170

mrbean-bremen · 2017-04-02T11:13:56Z

@jmcgeheeiv - please review!

jmcgeheeiv

Beyond the comments I added in-line, I am concerned that this will not work in the context of fake_filesystem_unittest. This is based on my experience with CopyRealFile().

I originally put CopyRealFile() in class FakeFilesystem. Even the unit tests (of the same style you are using) worked. However, when it was in class FakeFilesystem, I had no control over whether it was using the real or fake file methods because fake_filesystem_unittest patches all the file method.

After discovering this, In PR #157, commit a890807, I moved it to fake_filesystem_unittest.TestCase.copyRealFile().

Please add a usage example to example.py and example_test.py and see what happens.

jmcgeheeiv · 2017-04-02T18:36:54Z

pyfakefs/fake_filesystem.py

@@ -886,7 +897,7 @@ def CollapsePath(self, path):
                continue
            if component == '..':
                if collapsed_path_components and (
-                        collapsed_path_components[-1] != '..'):
+                            collapsed_path_components[-1] != '..'):


jmcgeheeiv · 2017-04-02T18:43:46Z

CHANGES.md

@@ -4,6 +4,7 @@ The release versions are PyPi releases.
 ## Version 3.2 (as yet unreleased)

 #### New Features
+ * Added possibility to add lazily read real files to fake filesystem


Perhaps:
Added new methods that make real files and directories appear within the fake file system:

fake_filesystem.FakeFilesystem.AddRealFile()

fake_filesystem.FakeFilesystem.AddRealDirectory()

fake_filesystem.FakeFilesystem.AddRealPaths()

File contents are read from the real file system only when needed.

jmcgeheeiv · 2017-04-02T18:55:45Z

pyfakefs/fake_filesystem.py

-        self.byte_contents = self._EncodeContents(contents, encoding)
-        self.st_size = len(self.byte_contents) if self.byte_contents else 0
+        self._byte_contents = self._EncodeContents(contents, encoding)
+        self.st_size = len(self._byte_contents) if self._byte_contents else 0


if self.byte_contents is not None

jmcgeheeiv · 2017-04-02T19:05:13Z

pyfakefs/fake_filesystem.py

@@ -823,7 +834,7 @@ def GetOpenFile(self, file_des):
        if not isinstance(file_des, int):
            raise TypeError('an integer is required')
        if (file_des >= len(self.open_files) or
-                self.open_files[file_des] is None):
+                    self.open_files[file_des] is None):


jmcgeheeiv · 2017-04-02T19:07:06Z

pyfakefs/fake_filesystem.py

@@ -918,7 +929,7 @@ def NormalizeCase(self, path):
        for component in path_components:
            dir_name, current_dir = self._DirectoryContent(current_dir, component)
            if current_dir is None or (
-                    current_dir.byte_contents is None and current_dir.st_size == 0):
+                            current_dir._byte_contents is None and current_dir.st_size == 0):


jmcgeheeiv · 2017-04-02T19:07:30Z

pyfakefs/fake_filesystem.py

@@ -1003,7 +1014,7 @@ def SplitDrive(self, path):
                # UNC path handling is here since Python 2.7.8, back-ported from Python 3
                if sys.version_info >= (2, 7, 8):
                    if (path[0:2] == self.path_separator * 2) and (
-                            path[2:3] != self.path_separator):
+                                path[2:3] != self.path_separator):


Didn't notice these changes - PyCharm has sometimes its own ideas about hanging indents, obviously I did an auto-format accidentally. I will revert this.

mrbean-bremen · 2017-04-03T17:51:49Z

I added an example as requested. I had to use REAL_OPEN to compare the content with the real file, but you won't need this in a real test, so I think the API can be used in unit tests without problems.

mrbean-bremen · 2017-04-04T16:46:08Z

Also - as CopyRealFile has at least one additional feature (copy to another location), I think it has its own value, but I don't think we need another addition to TestCase - the methods in fake_filesystem should be sufficient.

mrbean-bremen · 2017-04-13T04:24:14Z

@dwt had similar problems as you mentioned, and after reproducing them (or something related ) I found that the call to os.path.exists that is made from the original os.listdir, was the problem in this case. I'm not really sure if this is a good fix for that.

mrbean-bremen · 2017-04-13T04:50:18Z

I may have another go at this later and try to reproduce the problem in a unit test.

mrbean-bremen · 2017-04-13T19:07:33Z

@jmcgeheeiv - I added a test that reproduces the problem. please review!
Note that I rebased against master.

mrbean-bremen · 2017-04-14T18:11:17Z

As to the explanation why the problem happened in the first place:
Other than the other patched modules, path is actually implemented in two modules: ntpath/posixpath (one of which is imported as path by os) and genericpath, which implements stuff common for both systems. If calling os.path.exists from inside the unit test, the real os module is used as it should be, but it calls the implementation in genericpath, which calls os.stat on its part. If genericpath is not added to SKIPNAMES, this last os will be stubbed out, so stat is called on the fake file system.

jmcgeheeiv · 2017-04-14T18:35:15Z

Oh, dear. Could you resolve this conflict and merge it?

Well done, @mrbean-bremen.

- allow to add really existing files and directory trees to the fake file system, with the contents read on demand - see pytest-dev#170

- added example test - fixed documentation and formatting

- this seems to fix the problem that using real os.path functions implemented in this module are stubbed out

- regression test for exception happening if 'genericpath' is not added to SKIPNAMES - do not add 'genericpath' to SKIPNAMES if path is not patched

mrbean-bremen · 2017-04-14T18:57:41Z

Done - caused that conflict myself. Thanks!

mrbean-bremen requested a review from jmcgeheeiv April 1, 2017 17:17

mrbean-bremen self-assigned this Apr 1, 2017

mrbean-bremen force-pushed the real_file_access branch from a52ac94 to e1ca294 Compare April 2, 2017 11:44

jmcgeheeiv suggested changes Apr 2, 2017

View reviewed changes

mrbean-bremen force-pushed the real_file_access branch from 68626dc to 36524bb Compare April 5, 2017 18:33

mrbean-bremen force-pushed the real_file_access branch from e686343 to 6748425 Compare April 13, 2017 19:03

mrbean-bremen force-pushed the real_file_access branch from 6748425 to a5ff2f8 Compare April 14, 2017 18:16

jmcgeheeiv approved these changes Apr 14, 2017

View reviewed changes

mrbean-bremen added 5 commits April 14, 2017 20:45

Added methods to access real files

faa2668

- allow to add really existing files and directory trees to the fake file system, with the contents read on demand - see pytest-dev#170

Changes after review by jmcgeheeiv

a84a9e6

- added example test - fixed documentation and formatting

Changed method names to be PEP 8 conform

951b432

Added genericpath to SKIPNAMES

409f86d

- this seems to fix the problem that using real os.path functions implemented in this module are stubbed out

Added tests to check behavior inside a unittest

a3bf03a

- regression test for exception happening if 'genericpath' is not added to SKIPNAMES - do not add 'genericpath' to SKIPNAMES if path is not patched

mrbean-bremen force-pushed the real_file_access branch from a5ff2f8 to a3bf03a Compare April 14, 2017 18:49

mrbean-bremen merged commit e2f9424 into pytest-dev:master Apr 14, 2017

mrbean-bremen deleted the real_file_access branch April 17, 2017 15:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added methods to access real files #171

Added methods to access real files #171

mrbean-bremen commented Apr 1, 2017

mrbean-bremen commented Apr 2, 2017

jmcgeheeiv left a comment •

edited

Loading

jmcgeheeiv Apr 2, 2017

jmcgeheeiv Apr 2, 2017

jmcgeheeiv Apr 2, 2017

jmcgeheeiv Apr 2, 2017

jmcgeheeiv Apr 2, 2017

jmcgeheeiv Apr 2, 2017

mrbean-bremen Apr 3, 2017

mrbean-bremen commented Apr 3, 2017

mrbean-bremen commented Apr 4, 2017

mrbean-bremen commented Apr 13, 2017

mrbean-bremen commented Apr 13, 2017

mrbean-bremen commented Apr 13, 2017

mrbean-bremen commented Apr 14, 2017

jmcgeheeiv commented Apr 14, 2017

mrbean-bremen commented Apr 14, 2017

Added methods to access real files #171

Added methods to access real files #171

Conversation

mrbean-bremen commented Apr 1, 2017

mrbean-bremen commented Apr 2, 2017

jmcgeheeiv left a comment • edited Loading

Choose a reason for hiding this comment

jmcgeheeiv Apr 2, 2017

Choose a reason for hiding this comment

jmcgeheeiv Apr 2, 2017

Choose a reason for hiding this comment

jmcgeheeiv Apr 2, 2017

Choose a reason for hiding this comment

jmcgeheeiv Apr 2, 2017

Choose a reason for hiding this comment

jmcgeheeiv Apr 2, 2017

Choose a reason for hiding this comment

jmcgeheeiv Apr 2, 2017

Choose a reason for hiding this comment

mrbean-bremen Apr 3, 2017

Choose a reason for hiding this comment

mrbean-bremen commented Apr 3, 2017

mrbean-bremen commented Apr 4, 2017

mrbean-bremen commented Apr 13, 2017

mrbean-bremen commented Apr 13, 2017

mrbean-bremen commented Apr 13, 2017

mrbean-bremen commented Apr 14, 2017

jmcgeheeiv commented Apr 14, 2017

mrbean-bremen commented Apr 14, 2017

jmcgeheeiv left a comment •

edited

Loading