ENH: Iterate over HDF store hierarchically

[nileracecrew]

I am relatively new to pandas, but have been using raw HDF5 files for a long itme, so apologies if this feature or something equivalent already exists.

As stated in the documentation, it is possible to create hierarchies of data stores:

store.put('/first_group/df1', df1)
store.put('/first_group/df2', df2)
store.put('/second_group/df3', df3)

However, it seems that store.keys() can only provide a flattened list of all available groups. It would be nice to be able to walk the HDF5 file hierarchically. A couple examples (this is borrowing a bit from the h5py API):

# do stuff on each subgroup/substore in /first_group
for s in store['first_group'].values() :
    # do stuff on df1 and df2

# visit each group at the root level
for s in store.values() :
    if 'df1' in s :
        # do stuff on any store named 'df1'

The main use case I would have for something like this is that I have many dataframes with different schema but would like to group some of them together and then iterate through those groups.

Best way I can think to emulate this is to parse the strings in store.keys(), but that is pretty ugly.

[jreback]

wouldn't be opposed to adding a .walk() method, similar to how PyTables.Table.walk and os.walk works. You just iterate and yield each node, up to the user to do something with them.

[nileracecrew]

That would provide at least a basic hook for walking through the HDFStore.

Thinking more about my particular workflow, it might make sense instead to maintain a separate DataFrame with the paths to each DataFrame store, and some organizing metadata to query from.

[stephenpascoe]

I'd like to work on this during the EuroSciPy 2015 sprints #10877

[nileracecrew]

In case it's useful, I ended up using this construction to get all the
nodes at a particular location instead of walking the whole tree.

# simplified example from working code but not specifically tested so there may be typos

# added this method to pd.HDFStore
def iter_nodes(self, loc):
   return (n._v_name for n in self._handle.iter_nodes(loc))

# ...

with pd.HDFStore('mystore.h5') as store:
    for g in store.iter_nodes('/'):
        # do something to each node (group, dataset, etc) at the root level

[nileracecrew]

Related #6833

[stephenpascoe]

Created PR #10932

[NumesSanguis]

This issue has been open since 2015 and has the label "Effort Low", but it seems it has not been decided yet how to solve it? What is currently the recommended way to traverse the HDF5 store? I would like to get a list of only the top-level keys/groups.

[jreback]

merged via 45e55af