HDFStore.walk() to iterate on groups #21339

CharlesB2 · 2018-06-06T09:27:22Z

This PR adds a walk() method to HDFStore in order to iterate on groups.

This is a revival of the initial PR #10932, rebased on upstream/master and updated tests.

closes ENH: Iterate over HDF store hierarchically #10143
tests added / passed: test_pytables.py:testwalk()
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

jreback

some comments

jreback · 2018-06-06T11:11:45Z

pandas/io/pytables.py

+        group is used.
+
+        The where argument can be a path string
+        or a Group instance (see :ref:`GroupClassDescr`).


what does this refer to? best to directly link to the pytables docs

Actually it doesn't make sense for where to be anything else than str, removing this mention

jreback · 2018-06-06T11:12:04Z

pandas/io/pytables.py

@@ -1106,6 +1106,46 @@ def groups(self):
                 g._v_name != u('table'))))
        ]

+    def walk(self, where="/"):
+        """ Walk the pytables group hierarchy yielding the group name and


should be a single sumary line, then a multi-line extended summary

jreback · 2018-06-06T11:12:20Z

pandas/io/pytables.py

+    def walk(self, where="/"):
+        """ Walk the pytables group hierarchy yielding the group name and
+        pandas object names for each group.  Any non-pandas PyTables objects
+        that are not a group will be ignored.


need a Parameters section

jreback · 2018-06-06T11:12:58Z

pandas/tests/io/test_pytables.py

+        }
+
+        with ensure_clean_store('walk_groups.hdf', mode='w') as store:
+            store.put('/first_group/df1', objs['df1'])


add a table or 2

jreback · 2018-06-06T11:13:35Z

pandas/tests/io/test_pytables.py

@@ -4999,6 +4999,62 @@ def test_read_nokey_empty(self):
            store.close()
            pytest.raises(ValueError, read_hdf, path)

+    # GH10143
+    def test_walk(self):


put the comment inside the test function

jreback · 2018-06-06T11:14:24Z

pandas/tests/io/test_pytables.py

@@ -4999,6 +4999,62 @@ def test_read_nokey_empty(self):
            store.close()
            pytest.raises(ValueError, read_hdf, path)

+    # GH10143


put this near tests for get_node

Sorry, but I couldn't spot any test for get_node in the repo? The nearest I could find is TestHDFStore.test_get()

jreback · 2018-06-06T11:14:46Z

pandas/tests/io/test_pytables.py

+
+            expect1 = {
+                '': ({'first_group', 'second_group'}, set()),
+                '/first_group': (set(), {'df1', 'df2'}),


can you parameterize this test rather than making a long test like this

jreback · 2018-06-06T11:15:08Z

doc/source/io.rst

@@ -3554,6 +3554,22 @@ everything in the sub-store and **below**, so be *careful*.
   store.remove('food')
   store

+
+You can walk through the group hierarchy using the ``walk`` method which


add a versionadded

jreback · 2018-06-06T11:15:20Z

pandas/io/pytables.py

@@ -1106,6 +1106,46 @@ def groups(self):
                 g._v_name != u('table'))))
        ]

+    def walk(self, where="/"):
+        """ Walk the pytables group hierarchy yielding the group name and


add a versionadded

…ile. This implementation is inspired by os.walk and follows the interface as much as possible.

Including small fix to remove redundant '/' from group names.

walk() can be called with where argument to specify the root node. Tests updated with the enhancement

codecov · 2018-06-12T22:30:13Z

Codecov Report

Merging #21339 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #21339      +/-   ##
==========================================
+ Coverage    91.9%    91.9%   +<.01%     
==========================================
  Files         154      154              
  Lines       49562    49577      +15     
==========================================
+ Hits        45549    45564      +15     
  Misses       4013     4013

Flag	Coverage Δ
#multiple	`90.27% <6.66%> (-0.03%)`	⬇️
#single	`41.87% <100%> (+0.01%)`	⬆️

Impacted Files	Coverage Δ
pandas/io/pytables.py	`92.48% <100%> (+0.04%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36422a8...9e7d7a0. Read the comment docs.

CharlesB2 · 2018-06-12T22:36:42Z

I think I addressed your review's comments

jreback

minor comment. can you elaborate on why you think the return values are the right thing here? e.g. is this from the PyTables api directly? e.g. what is the point of having the groups and subkeys? wouldn't you just care about the subkeys?

jreback · 2018-06-19T00:33:13Z

doc/source/whatsnew/v0.24.0.txt

@@ -16,7 +16,8 @@ Other Enhancements
 - :func:`Series.mode` and :func:`DataFrame.mode` now support the ``dropna`` parameter which can be used to specify whether NaN/NaT values should be considered (:issue:`17534`)
 - :func:`to_csv` now supports ``compression`` keyword when a file handle is passed. (:issue:`21227`)
 - :meth:`Index.droplevel` is now implemented also for flat indexes, for compatibility with MultiIndex (:issue:`21115`)
-
+- New method :meth:`HDFStore.walk` will recursively walk the group hierarchy of a HDF5 file (:issue:`10932`)


can you add this to api.rst as well

an HDF5 file

CharlesB2 · 2018-06-19T21:39:34Z

minor comment. can you elaborate on why you think the return values are the right thing here? e.g. is this from the PyTables api directly? e.g. what is the point of having the groups and subkeys? wouldn't you just care about the subkeys?

It was made this way in the original PR, I didn't see any reason to change it. It's a refinement from the PyTables api, I guess the original author @stephenpascoe found it convenient.

I made the docs change in the last forced push.

jreback

minor comment. looks good. ping on green.

jreback · 2018-06-26T10:35:00Z

doc/source/whatsnew/v0.24.0.txt

@@ -17,8 +17,9 @@ Other Enhancements
 - :func:`to_datetime` now supports the ``%Z`` and ``%z`` directive when passed into ``format`` (:issue:`13486`)
 - :func:`Series.mode` and :func:`DataFrame.mode` now support the ``dropna`` parameter which can be used to specify whether NaN/NaT values should be considered (:issue:`17534`)
 - :func:`to_csv` now supports ``compression`` keyword when a file handle is passed. (:issue:`21227`)
- :meth:`Index.droplevel` is now implemented also for flat indexes, for compatibility with :class:`MultiIndex` (:issue:`21115`)
-
+- :meth:`Index.droplevel` is now implemented also for flat indexes, for compatibility with MultiIndex (:issue:`21115`)


this was updated and the diff is taking the original, can you restore (the droplevel line)

Update droplevel change line

jreback · 2018-06-26T22:53:32Z

merged via 45e55af

for some reason couldn't push to your remote, so manually merged.

thanks!

CharlesB2 changed the title ~~Issue 10143~~ HDFStore.walk() to iterate on groups Jun 6, 2018

jreback requested changes Jun 6, 2018

View reviewed changes

jreback added the IO HDF5 read_hdf, HDFStore label Jun 6, 2018

Stephen Pascoe and others added 4 commits June 12, 2018 23:57

ENH: #10143 Function to walk the group hierarchy of a PyTables HDF5 f…

d12ce55

…ile. This implementation is inspired by os.walk and follows the interface as much as possible.

walk will yield tuples containing any pandas type (not just dataframes).

296406a

Documentation and whats-new.

b8d6d7e

Including small fix to remove redundant '/' from group names.

HDFStore.walk enhancement and test update

ea11a34

walk() can be called with where argument to specify the root node. Tests updated with the enhancement

jreback requested changes Jun 19, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Jun 19, 2018

jreback added the Enhancement label Jun 19, 2018

Charles Brossollet added 2 commits June 19, 2018 23:33

Parametrize test_walk, misc documentation enhancements

91452aa

Move test_walk next to test_get

d180c63

Merge branch 'master' into issue-10143

66caa48

jreback requested changes Jun 26, 2018

View reviewed changes

Charles Brossollet and others added 2 commits June 26, 2018 12:40

Update v0.24.0.txt

a9c55ca

Update droplevel change line

Merge branch 'master' into issue-10143

9e7d7a0

jreback closed this Jun 26, 2018

CharlesB2 deleted the issue-10143 branch June 27, 2018 08:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFStore.walk() to iterate on groups #21339

HDFStore.walk() to iterate on groups #21339

CharlesB2 commented Jun 6, 2018

jreback left a comment

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018 •

edited

Loading

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

CharlesB2 Jun 12, 2018

jreback Jun 6, 2018

codecov bot commented Jun 12, 2018 •

edited

Loading

CharlesB2 commented Jun 12, 2018

jreback left a comment

jreback Jun 19, 2018

jreback Jun 19, 2018

CharlesB2 Jun 19, 2018

CharlesB2 commented Jun 19, 2018

jreback left a comment

jreback Jun 26, 2018

CharlesB2 Jun 26, 2018

jreback commented Jun 26, 2018

HDFStore.walk() to iterate on groups #21339

HDFStore.walk() to iterate on groups #21339

Conversation

CharlesB2 commented Jun 6, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CharlesB2 Jun 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 12, 2018 • edited Loading

Codecov Report

CharlesB2 commented Jun 12, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CharlesB2 commented Jun 19, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 26, 2018

CharlesB2 Jun 12, 2018 •

edited

Loading

codecov bot commented Jun 12, 2018 •

edited

Loading