Added support for reading .pyi files.

[alliefitter]

Subject: Added support for reading .pyi files

Feature or Bugfix

• Feature

Purpose

I created a class that dynamically builds its methods at runtime and wanted to create some docs for it.
So, I made a .pyi file with the dynamic methods defined and with docstrings. PyCharm was able to read this file and provide code completion and documentation, but sphinx required a little changing. The change in this commit is more of a proof of concept than anything. I REALLY don't know this codebase or how best to incorporate this--as a change to sphinx or as an extension. I'd really appreciate any thoughts or guidance on what I can do to expose this functionality.

Detail

• When importing a module, checks if a .pyi file exists by using os.path.isfile('{}i'.format(module.__file__)).
• Imports the .pyi file using importlib.
• Replaces the original module with the one imported from the .pyi file.
• From testing, sphinx doesn't seem to have any issues using the object imported from .pyi file.

[tk0miya]

What is .pyi file? Is it standardly used?

[alliefitter]

It's a stub file introduced in PEP 484. Meant so you can use typing to add annotations without breaking in pre-python 3.5. See also typeshed which is included with PyCharm.

[alliefitter]

You guys have an open issue that references .pyi files in #2755.

[tk0miya]

Thank you for description. I understand. I know stub files for mypy (typeshed). But I've never know PyCharm using it as external type hinting.
https://www.jetbrains.com/help/pycharm/type-hinting-in-product.html#pep484

+0 for this for following reasons:

• I think the external type hinting is not needed for py3.
• On the other hand, at present, it is useful to annotate types for py2 code.
• Until the EOF of py2, the proposed feature seems enough useful to me.
• I don't know why external docstrings is needed. So I can't say it is needed or not.

I think this is temporarily needed feature. So it would be nice if we can implement this as an extension. I'll think about it later.

[alliefitter]

Thanks for the consideration. I was leaning towards an extension myself. I'll have to get back to it when I have some time to figure how best to implement it. Also just one more use case for this (though it is pretty narrow) is an object whose methods defined until run time. As I said this is why I needed to experiment with this. Basically, the methods are decided using the strategy pattern when the object is instantiated. The stub files allowed me to define the signature of the methods and add docstrings for the default strategy.

[tk0miya]

I feel your case is not usual. So I can't determine to add this feature to sphinx-core.

As a workaround, you can override import_module() function like following:

# in your extension
from sphinx.ext.autodoc import importer

# keep original import_module()
original_import_module = importer.import_module

def import_module(modname, warningiserror=False):
    module = original_import_module(modname, warningsiserror)
    if hasattr(module, '__file__') and os.path.isfile('{}i'.format(module.__file__)):
        ...  # merge external spec into the imported module

# override import_module() by own
importer.import_module = import_module

I know this is very hacky. But I believe this will work without changing sphinx-core.

[tk0miya]

@shimizukawa any comments?

[shimizukawa]

• Apparently, this code does not seem to work with Py2.
• As @tk0miya said, in order to incorporate this feature into sphinx-core, we need a general case of reading the pyi file in the same directory and doing autodoc.

[shimizukawa]

We need a general case of reading the pyi file in the same directory and doing autodoc.

[tk0miya]

@shimizukawa Thank you for comment.

@alliefitter I markded this as "wontfix". I think there are no proper way. But we can hack it with workaround. So please use it for your extension.
Thanks,

[JosXa]

I just stumbled across this issue by googling and have exactly the same problem as OP. I also created some methods dynamically and documented them using .pyi files. I was very much assuming that Sphinx would use them to generate documentation, as many typing-related things are also supported.
Are you sure this is out of scope and a special case? I'd love to see this included in one way or another, as there really is a valid use case even in Python3.5+.

@tk0miya

I think the external type hinting is not needed for py3.
I don't know why external docstrings is needed. So I can't say it is needed or not.

For dynamically generated types, functions and methods, you really don't have any other way of annotating them properly.
Could someone hint me to some docs or examples on how you could override the internal behavior of the importer by using an extension?

[tk0miya]

@JosXa

Could you let me know about "many typing-related things"? I'd like to know this is commonly used way or not. Is there any specification?

Could someone hint me to some docs or examples on how you could override the internal behavior of the importer by using an extension?

Read my comment above please: #4824 (comment)

[alliefitter]

I apologize for not responding sooner. I get a flood of Github notifications from work everyday and missed these notifications. @JosXa glad that some else has a need for this besides me. @tk0miya I agree that this doesn't belong in sphinx-core. @shimizukawa inherently this isn't compatible with 2.7 because the typing module is 3.5+. Which is another reason why this should be an extension.

With @tk0miya 's workaround, I think I can get this done over the weekend. Thanks, guys!

[alliefitter]

Oh, and @JosXa, just a heads up. If you're using the Google Docstring extension, you have to edit that too in order for it work properly.

[JosXa]

many typing-related things

Sorry, I'm not very experienced with Sphinx as this is the first project I'm making public docs for, so I'm not exactly sure. But all the normal argument type hinting with foo: Type seems to work (as supported by google docstrings?), so I thought .pyi stubs should too.

@tk0miya Ya so I got a custom extension rolling with your guideline, thanks for that. However, it has issues with the non-standard syntax of .pyi stubs, for instance class attributes defined as

class Foo:
    bar: Type

instead of

class Foo:
    bar = Type

I suspect that there might be more instances where the syntax is not quite parseable by a normal python interpreter. Do you not type hint your attributes and properties the same way, @alliefitter? Or have you found a solution to that? I guess it should be fairly simple to script just this one part, but it makes me think that the support for this should come from a more holistic solution than just a workaround/hack.

Is there any specification?

I think the idea stems from the mypi project in order to be able to write and share type hints for third-party libraries that you do not have access to or in order to separate type hints from the source code (PYTHONPATH or shared/typehints/python3.5). Of course there is the issue of code locality, meaning that type hints should usually be right next to the symbols and not in another file, but on the other hand you can also get a lot cleaner projects while still maintaining the autocompletion hints by IDE support (PyCharm). Looking for docs right now...

[JosXa]

Interesting docs:

• typeshed (which is how PyCharm gets its types)
• Very relevant SO post
• The mypy documentation, see chapter 2.5 for .pyi stubs

Note:
Apparently "stub files are written in normal Python 3 syntax, but generally leaving out runtime logic like variable initializers, function bodies, and default arguments, replacing them with ellipses"

[alliefitter]

@JosXa are you using the autodoc typehints extension? I haven't had any issues with the autodoc reading my type annotations.

[JosXa]

Trying it now, and thanks for the link - I included it. Update:

The issue remains:

    bot_under_test: Union[int, str]
                  ^
SyntaxError: invalid syntax

and

    logger: Logger
          ^
SyntaxError: invalid syntax

@alliefitter this is what I'm trying to do: https://github.com/JosXa/tgintegration/blob/freeze/tgintegration/botintegrationclient.pyi#L15

Do you do it differently?

I'm pretty sure it's valid syntax, so something is wrong. Perhaps a too old Python version (edit: no I thinnk 3.5 should be good, right?) or actually an issue with the Sphinx parsing...

[alliefitter]

Yes, it should be fine with 3.5+. I just checked and I'm able to type attributes using that syntax. It could be that you aren't initializing them? Maybe try giving them a default value.

[tk0miya]

The mypy documentation, see chapter 2.5 for .pyi stubs

Thanks! This is what I'd wanted.
This says we can place .pyi file at same directory as the .py script:

Write a stub file for the library and store it as a .pyi file in the same directory as the library module.
http://mypy.readthedocs.io/en/stable/basics.html#library-stubs-and-the-typeshed-repo

It means this is not PyCharm specific behavior. +1 to add this to sphinx-core.

@shimizukawa what do you think?

[Vanuan]

Found an alternative: https://github.com/readthedocs/sphinx-autoapi
Somehow, it reads and understands *.pyi files

[HyukjinKwon]

It'd be great if we have a built-in support :-)

[cmpute]

Any progress here please? this feature is really useful for packages that are heavily written in Cython

[brenthuisman]

I'd like to add to the chorus. It would help add type info for Pybind11 projects.

[arthurp]

If I were to provide a new PR that addresses the issues raised on this PR (and updates it). Would it be something y'all devs would willing to merge?

(@tk0miya)

[AA-Turner]

@arthurp please can you provide a high level (bullets fine) overview of your updated proposal? I wouldn't want you to put in work needlessly.

Personally I'd want to see a switch for .pyi precedence -- the proposal at the top of the PR noted they would always be favoured over .py sources, which sometimes isn't ideal.

A

[Eutropios]

Has there been any update on the inclusion of an option to pull types from stub files?

[picnixz]

Not yet but I plan to work on that next year. I can't promise anything yet but this is indeed an important thing to support imo.

[amirebrahimi]

I'll add that we've run into the same issue for many of our projects, too, where the .py file dynamically loads code, so support of .pyi files would be great. Currently this shows up in sphinx as:

WARNING: error while formatting arguments for <path_to_our_func>: list index out of range

This comes from update_annotations_using_type_comments that calls get_type_comment, which throws an exception because getsource(obj) has no body.

We treat warnings as errors, so, we have to have these files excluded for now.

[amirebrahimi]

For @tk0miya's workaround, I wasn't able to get it to work. I tried a few things for the ... that you suggested, but it isn't clear to me how one merges an external spec into the imported module. Here's the last little bit of what I tried:

        pyi_path = Path(module.__file__).with_suffix(".pyi")

        sys.path_hooks.insert(
            0, importlib.machinery.FileFinder.path_hook((importlib.machinery.SourceFileLoader, [".pyi"]))
        )
        sys.path.insert(0, str(pyi_path))
        spec = importlib.util.spec_from_file_location(modname, pyi_path)
        mod = importlib.util.module_from_spec(spec)
        module = spec.loader.exec_module(mod)
        sys.path.pop(0)
        sys.path_hooks.pop(0)

It seems that spec_from_file_location will ignore any .pyi file extension.

[AA-Turner]

Closing in favour of #13253.

A