Is it possible to "open a directory" to remain connected to a server while processing files?

[jmcharter]

Apologies if any of the terminology I use is incorrect. My goal is to connect to an SFTP server, pointed to a directory and then iterate through a series of zip files within that directory and process the inner csv. It's something like this:

.
├── zip1.zip
│   └── file1.csv
├── zip2.zip
│   └── file2.csv
└── zip3.zip
    └── file3.csv

Currently, I'm doing something like this:

fs = fsspec.filesystem("sftp", path, <kwargs>)
files = fs.listdir(file_path, detail=False)
    for file_name in files:
        with fs.open(file_name) as file:
            with zipfile.ZipFile(file) as zf:
                for name in zf.filelist:
                    with zf.open(name, "r") as f:
                        df = polars.read_csv(f)
                        ...

But that, of course, keeps terminating and reconnecting to the sftp server.
Is there a way for me to specific a directory and open that, maintaining the connection whilst I process the files?

[martindurant]

The zip files are closing their respective remote files at the end of each with block. I'm not sure why that would disconnect from the server so long as the original fs stays live, though.

Something you could try, is using like the following. I'm not sure if this is simpler or keeps connections better.

with fsspec.open_files("sftp://...", ...) as ofs:
    for fo in ofs:
        with fsspec.open_files("zip://*.csv", fo=fo) as inner_ofs:
            for f in inner_ofs:
                df = polars.read_csv(f)

[jmcharter]

It's because with fs.open is inside a for loop, so it disconnects and reconnects on each loop. A solution would be to invert it so the for loop with within fs.open, but I can't do that as I have to provide a file to open rather than a directory.

I had not seen open_files before, that looks interesting. My only concern with that is that it might open up every file into memory, which would be... Fun :)

[martindurant]

it might open up every file into memory

It will indeed "open" the files, but this doesn't mean loading them into memory. For ZIP, it probably grabs the archive directory only.