NTH: docstring not converted, import statement

[pappasam]

I'm going to open issues for specific items I find in my day-to-day. I'll begin the non-crucial issues with "NTH" so you know it's a "nice to have" and not a blocker. Feel free to ignore these issues, I'm just documenting what I find in case you find it helpful.

The following docstring isn't converted from rst to markdown (raises "UnknownFormatError"):

The "import" statement
**********************

   import_stmt     ::= "import" module ["as" identifier] ("," module ["as" identifier])*
                   | "from" relative_module "import" identifier ["as" identifier]
                   ("," identifier ["as" identifier])*
                   | "from" relative_module "import" "(" identifier ["as" identifier]
                   ("," identifier ["as" identifier])* [","] ")"
                   | "from" module "import" "*"
   module          ::= (identifier ".")* identifier
   relative_module ::= "."* module | "."+

The basic import statement (no "from" clause) is executed in two
steps:

1. find a module, loading and initializing it if necessary

2. define a name or names in the local namespace for the scope
   where the "import" statement occurs.

When the statement contains multiple clauses (separated by commas) the
two steps are carried out separately for each clause, just as though
the clauses had been separated out into individual import statements.

The details of the first step, finding and loading modules are
described in greater detail in the section on the import system, which
also describes the various types of packages and modules that can be
imported, as well as all the hooks that can be used to customize the
import system. Note that failures in this step may indicate either
that the module could not be located, *or* that an error occurred
while initializing the module, which includes execution of the
module’s code.

If the requested module is retrieved successfully, it will be made
available in the local namespace in one of three ways:

* If the module name is followed by "as", then the name following
  "as" is bound directly to the imported module.

* If no other name is specified, and the module being imported is a
  top level module, the module’s name is bound in the local namespace
  as a reference to the imported module

* If the module being imported is *not* a top level module, then the
  name of the top level package that contains the module is bound in
  the local namespace as a reference to the top level package. The
  imported module must be accessed using its full qualified name
  rather than directly

The "from" form uses a slightly more complex process:

1. find the module specified in the "from" clause, loading and
   initializing it if necessary;

2. for each of the identifiers specified in the "import" clauses:

   1. check if the imported module has an attribute by that name

   2. if not, attempt to import a submodule with that name and then
      check the imported module again for that attribute

   3. if the attribute is not found, "ImportError" is raised.

   4. otherwise, a reference to that value is stored in the local
      namespace, using the name in the "as" clause if it is present,
      otherwise using the attribute name

Examples:

   import foo                 # foo imported and bound locally
   import foo.bar.baz         # foo.bar.baz imported, foo bound locally
   import foo.bar.baz as fbb  # foo.bar.baz imported and bound as fbb
   from foo.bar import baz    # foo.bar.baz imported and bound as baz
   from foo import attr       # foo imported and foo.attr bound as attr

If the list of identifiers is replaced by a star ("'*'"), all public
names defined in the module are bound in the local namespace for the
scope where the "import" statement occurs.

The *public names* defined by a module are determined by checking the
module’s namespace for a variable named "__all__"; if defined, it must
be a sequence of strings which are names defined or imported by that
module.  The names given in "__all__" are all considered public and
are required to exist.  If "__all__" is not defined, the set of public
names includes all names found in the module’s namespace which do not
begin with an underscore character ("'_'").  "__all__" should contain
the entire public API. It is intended to avoid accidentally exporting
items that are not part of the API (such as library modules which were
imported and used within the module).

The wild card form of import — "from module import *" — is only
allowed at the module level.  Attempting to use it in class or
function definitions will raise a "SyntaxError".

When specifying what module to import you do not have to specify the
absolute name of the module. When a module or package is contained
within another package it is possible to make a relative import within
the same top package without having to mention the package name. By
using leading dots in the specified module or package after "from" you
can specify how high to traverse up the current package hierarchy
without specifying exact names. One leading dot means the current
package where the module making the import exists. Two dots means up
one package level. Three dots is up two levels, etc. So if you execute
"from . import mod" from a module in the "pkg" package then you will
end up importing "pkg.mod". If you execute "from ..subpkg2 import mod"
from within "pkg.subpkg1" you will import "pkg.subpkg2.mod". The
specification for relative imports is contained in the Package
Relative Imports section.

"importlib.import_module()" is provided to support applications that
determine dynamically the modules to be loaded.

Raises an auditing event "import" with arguments "module", "filename",
"sys.path", "sys.meta_path", "sys.path_hooks".

Future statements
=================

A *future statement* is a directive to the compiler that a particular
module should be compiled using syntax or semantics that will be
available in a specified future release of Python where the feature
becomes standard.

The future statement is intended to ease migration to future versions
of Python that introduce incompatible changes to the language.  It
allows use of the new features on a per-module basis before the
release in which the feature becomes standard.

   future_stmt ::= "from" "__future__" "import" feature ["as" identifier]
                   ("," feature ["as" identifier])*
                   | "from" "__future__" "import" "(" feature ["as" identifier]
                   ("," feature ["as" identifier])* [","] ")"
   feature     ::= identifier

A future statement must appear near the top of the module.  The only
lines that can appear before a future statement are:

* the module docstring (if any),

* comments,

* blank lines, and

* other future statements.

The only feature in Python 3.7 that requires using the future
statement is "annotations".

All historical features enabled by the future statement are still
recognized by Python 3.  The list includes "absolute_import",
"division", "generators", "generator_stop", "unicode_literals",
"print_function", "nested_scopes" and "with_statement".  They are all
redundant because they are always enabled, and only kept for backwards
compatibility.

A future statement is recognized and treated specially at compile
time: Changes to the semantics of core constructs are often
implemented by generating different code.  It may even be the case
that a new feature introduces new incompatible syntax (such as a new
reserved word), in which case the compiler may need to parse the
module differently.  Such decisions cannot be pushed off until
runtime.

For any given release, the compiler knows which feature names have
been defined, and raises a compile-time error if a future statement
contains a feature not known to it.

The direct runtime semantics are the same as for any import statement:
there is a standard module "__future__", described later, and it will
be imported in the usual way at the time the future statement is
executed.

The interesting runtime semantics depend on the specific feature
enabled by the future statement.

Note that there is nothing special about the statement:

   import __future__ [as name]

That is not a future statement; it’s an ordinary import statement with
no special semantics or syntax restrictions.

Code compiled by calls to the built-in functions "exec()" and
"compile()" that occur in a module "M" containing a future statement
will, by default, use the new syntax or semantics associated with the
future statement.  This can be controlled by optional arguments to
"compile()" — see the documentation of that function for details.

A future statement typed at an interactive interpreter prompt will
take effect for the rest of the interpreter session.  If an
interpreter is started with the "-i" option, is passed a script name
to execute, and the script includes a future statement, it will be in
effect in the interactive session started after the script is
executed.

See also:

  **PEP 236** - Back to the __future__
     The original proposal for the __future__ mechanism.

[krassowski]

Thanks, I will greatly appreciate it! As for this specific docstring, this is because it is not RST. I spend quite some time confused here before because it looks like RST, but actually it is stripped of RST qualifiers. Compare with the cPython source: https://github.com/python/cpython/blob/master/Doc/reference/simple_stmts.rst#the-keywordimport-statement

The :keyword:`!import` statement
================================

.. index::
   ! statement: import
   single: module; importing
   pair: name; binding
   keyword: from
   keyword: as
   exception: ImportError
   single: , (comma); import statement

.. productionlist:: python-grammar
   import_stmt: "import" `module` ["as" `identifier`] ("," `module` ["as" `identifier`])*
              : | "from" `relative_module` "import" `identifier` ["as" `identifier`]
              : ("," `identifier` ["as" `identifier`])*
              : | "from" `relative_module` "import" "(" `identifier` ["as" `identifier`]
              : ("," `identifier` ["as" `identifier`])* [","] ")"
              : | "from" `module` "import" "*"
   module: (`identifier` ".")* `identifier`
   relative_module: "."* `module` | "."+

The basic import statement (no :keyword:`from` clause) is executed in two
steps:

#. find a module, loading and initializing it if necessary
#. define a name or names in the local namespace for the scope where
   the :keyword:`import` statement occurs.

When the statement contains multiple clauses (separated by
commas) the two steps are carried out separately for each clause, just
as though the clauses had been separated out into individual import
statements.

The details of the first step, finding and loading modules are described in
greater detail in the section on the :ref:`import system <importsystem>`,
which also describes the various types of packages and modules that can
be imported, as well as all the hooks that can be used to customize
the import system. Note that failures in this step may indicate either
that the module could not be located, *or* that an error occurred while
initializing the module, which includes execution of the module's code.

If the requested module is retrieved successfully, it will be made
available in the local namespace in one of three ways:

.. index:: single: as; import statement

* If the module name is followed by :keyword:`!as`, then the name
  following :keyword:`!as` is bound directly to the imported module.
* If no other name is specified, and the module being imported is a top
  level module, the module's name is bound in the local namespace as a
  reference to the imported module
* If the module being imported is *not* a top level module, then the name
  of the top level package that contains the module is bound in the local
  namespace as a reference to the top level package. The imported module
  must be accessed using its full qualified name rather than directly

[krassowski]

It would be good to determine where (Jedi? cPython?) the conversion happens. Certainly converting it as-is (e.g. without the .. productionlist:: syntax) would lead to formatting issues :(

[pappasam]

Seems like jedi just uses the output from pydoc for keywords. Try running pydoc import from your terminal and you should see jedi's hover text.

[krassowski]

And that gets auto-generated from Sphinx: https://github.com/python/cpython/blob/3.9/Lib/pydoc_data/topics.py (which strips formatting information). It is implemented here: PydocTopicsBuilder.

There is a finite number of keywords so this could potentially be worked around...