feat: allow to exclude empty file simple directory reader

[antoineKorbit]

Description

Add a new optional argument to the SimpleDirectoryReader class to exclude empty files from the file list. This is implemented to prevent the file limit for extraction from being consumed by empty files, which does not align with my intended use case.

Type of Change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[logan-markewich]

This is a sensible change, but unit tests are failing

>           assert [doc.metadata["file_name"] for doc in documents_include] == [
                "test1.txt",
                "test2.txt",
                "empty1.txt",
                "empty2.txt",
            ]
E           AssertionError: assert ['empty1.txt'..., 'test2.txt'] == ['test1.txt',... 'empty2.txt']
E             At index 0 diff: 'empty1.txt' != 'test1.txt'
E             Use -v to get the full diff

[antoineKorbit]

Sorry about the test file, it's fixed now.