Merge pull request #2104 from Kodiologist/result-macros

Replace special forms with result macros

NEWS.rst

@@ -52,13 +52,15 @@ Other Breaking Changes
 * made `calling-module-name` private and `calling-module` internal to Hy
 * `macro-error` has been removed, raise typical errors instead
 * `if` now only accepts one test, if branch, and else branch
-* `defn`, `defn/a` have been made a special forms
 * `annotate*` has been made public and renamed `annotate`
 * return annotation for `defn` has been moved to before the function name
 * Python reserved words can no longer be parameter names or
   function call keywords
 * hy models are no longer equal to their associated Python values. `(= 1 '1)  ; => False`
 * `defclass` no longer automagically adds `None` to end of `__init__` forms
+* All special forms have been replaced with macros. This won't affect
+  most preexisting code, but it does mean that user-defined macros can
+  now shadow names like `setv`.
 
 New Features
 ------------------------------

docs/api.rst

@@ -2,10 +2,11 @@ API
 ===
 
 
-.. _special-forms:
+Core Macros
+-----------
 
-Special Forms
--------------
+The following macros are auto imported into all Hy modules as their
+base names, such that ``hy.core.macros.foo`` can be called as just ``foo``.
 
 .. hy:data:: ^
 
@@ -1496,6 +1497,11 @@ Special Forms
    coroutine, say, if using something fancy like
    `asyncio <https://docs.python.org/3.4/library/asyncio.html>`_.
 
+.. hy:automodule:: hy.core.macros
+   :members:
+   :macros:
+   :tags:
+
 Hy
 ---
 
@@ -1528,10 +1534,10 @@ the following methods
 
 .. _Core:
 
-Core
-----
+Core Functions
+--------------
 
-The following methods and macros are auto imported into all Hy modules as their
+The following functions are auto imported into all Hy modules as their
 base names, such that ``hy.core.language.butlast`` can be called with just ``butlast``.
 
 
@@ -1544,12 +1550,6 @@ base names, such that ``hy.core.language.butlast`` can be called with just ``but
 .. hy:automodule:: hy.core.shadow
    :members:
 
-.. hy:automodule:: hy.core.macros
-   :members:
-   :macros:
-   :tags:
-
-
 Additional Modules
 ------------------
 

docs/language/internals.rst

@@ -83,8 +83,8 @@ Expression
 ``hy.models.Expression`` inherits :ref:`Sequence` for
 parenthesized ``()`` expressions. The compilation result of those
 expressions depends on the first element of the list: the compiler
-dispatches expressions between compiler special-forms, user-defined
-macros, and regular Python function calls.
+dispatches expressions between macros and and regular Python
+function calls.
 
 .. _hydict:
 
@@ -184,176 +184,6 @@ For example::
 
 If needed, you can get the mangled name by calling :hy:func:`mangle <hy.mangle>`.
 
-Hy Internal Theory
-==================
-
-.. _overview:
-
-Overview
---------
-
-The Hy internals work by acting as a front-end to Python bytecode, so
-that Hy itself compiles down to Python Bytecode, allowing an unmodified
-Python runtime to run Hy code, without even noticing it.
-
-The way we do this is by translating Hy into an internal Python AST
-datastructure, and building that AST down into Python bytecode using
-modules from the Python standard library, so that we don't have to
-duplicate all the work of the Python internals for every single Python
-release.
-
-Hy works in four stages. The following sections will cover each step of Hy
-from source to runtime.
-
-.. _lexing:
-
-Steps 1 and 2: Tokenizing and Parsing
--------------------------------------
-
-The first stage of compiling Hy is to lex the source into tokens that we can
-deal with. We use a project called rply, which is a really nice (and fast)
-parser, written in a subset of Python called rpython.
-
-The lexing code is all defined in ``hy.lex.lexer``. This code is mostly just
-defining the Hy grammar, and all the actual hard parts are taken care of by
-rply -- we just define "callbacks" for rply in ``hy.lex.parser``, which takes
-the tokens generated, and returns the Hy models.
-
-You can think of the Hy models as the "AST" for Hy, it's what Macros operate
-on (directly), and it's what the compiler uses when it compiles Hy down.
-
-.. seealso::
-
-   Section :ref:`models` for more information on Hy models and what they mean.
-
-.. _compiling:
-
-Step 3: Hy Compilation to Python AST
-------------------------------------
-
-This is where most of the magic in Hy happens. This is where we take Hy AST
-(the models), and compile them into Python AST. A couple of funky things happen
-here to work past a few problems in AST, and working in the compiler is some
-of the most important work we do have.
-
-The compiler is a bit complex, so don't feel bad if you don't grok it on the
-first shot, it may take a bit of time to get right.
-
-The main entry-point to the Compiler is ``HyASTCompiler.compile``. This method
-is invoked, and the only real "public" method on the class (that is to say,
-we don't really promise the API beyond that method).
-
-In fact, even internally, we don't recurse directly hardly ever, we almost
-always force the Hy tree through ``compile``, and will often do this with
-sub-elements of an expression that we have. It's up to the Type-based dispatcher
-to properly dispatch sub-elements.
-
-All methods that preform a compilation are marked with the ``@builds()``
-decorator. You can either pass the class of the Hy model that it compiles,
-or you can use a string for expressions. I'll clear this up in a second.
-
-First Stage Type-Dispatch
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Let's start in the ``compile`` method. The first thing we do is check the
-Type of the thing we're building. We look up to see if we have a method that
-can build the ``type()`` that we have, and dispatch to the method that can
-handle it. If we don't have any methods that can build that type, we raise
-an internal ``Exception``.
-
-For instance, if we have a ``String``, we have an almost 1-to-1 mapping of
-Hy AST to Python AST. The ``compile_string`` method takes the ``String``, and
-returns an ``ast.Str()`` that's populated with the correct line-numbers and
-content.
-
-Macro-Expand
-~~~~~~~~~~~~
-
-If we get a ``Expression``, we'll attempt to see if this is a known
-Macro, and push to have it expanded by invoking ``hy.macros.macroexpand``, then
-push the result back into ``HyASTCompiler.compile``.
-
-Second Stage Expression-Dispatch
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The only special case is the ``Expression``, since we need to create different
-AST depending on the special form in question. For instance, when we hit an
-``(if True True False)``, we need to generate a ``ast.If``, and properly
-compile the sub-nodes. This is where the ``@builds()`` with a String as an
-argument comes in.
-
-For the ``compile_expression`` (which is defined with an
-``@builds(Expression)``) will dispatch based on the string of the first
-argument. If, for some reason, the first argument is not a string, it will
-properly handle that case as well (most likely by raising an ``Exception``).
-
-If the String isn't known to Hy, it will default to create an ``ast.Call``,
-which will try to do a runtime call (in Python, something like ``foo()``).
-
-Issues Hit with Python AST
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Python AST is great; it's what's enabled us to write such a powerful project
-on top of Python without having to fight Python too hard. Like anything, we've
-had our fair share of issues, and here's a short list of the common ones you
-might run into.
-
-*Python differentiates between Statements and Expressions*.
-
-This might not sound like a big deal -- in fact, to most Python programmers,
-this will shortly become a "Well, yeah" moment.
-
-In Python, doing something like:
-
-``print for x in range(10): pass``, because ``print`` prints expressions, and
-``for`` isn't an expression, it's a control flow statement. Things like
-``1 + 1`` are Expressions, as is ``lambda x: 1 + x``, but other language
-features, such as ``if``, ``for``, or ``while`` are statements.
-
-Since they have no "value" to Python, this makes working in Hy hard, since
-doing something like ``(print (if True True False))`` is not just common, it's
-expected.
-
-As a result, we reconfigure things using a ``Result`` object, where we offer
-up any ``ast.stmt`` that need to get run, and a single ``ast.expr`` that can
-be used to get the value of whatever was just run. Hy does this by forcing
-assignment to things while running.
-
-As example, the Hy::
-
-    (print (if True True False))
-
-Will turn into:
-
-.. code-block:: python
-
-    if True:
-        _temp_name_here = True
-    else:
-        _temp_name_here = False
-
-    print(_temp_name_here)
-
-
-OK, that was a bit of a lie, since we actually turn that statement
-into:
-
-.. code-block:: python
-
-    print(True if True else False)
-
-By forcing things into an ``ast.expr`` if we can, but the general idea holds.
-
-
-Step 4: Python Bytecode Output and Runtime
-------------------------------------------
-
-After we have a Python AST tree that's complete, we can try and compile it to
-Python bytecode by pushing it through ``eval``. From here on out, we're no
-longer in control, and Python is taking care of everything. This is why things
-like Python tracebacks, pdb and django apps work.
-
-
 Hy Macros
 =========
 

docs/tutorial.rst

@@ -65,11 +65,9 @@ separated from each other by whitespace, but some forms, such as string
 literals (``"Hy, world!"``), can contain whitespace themselves. An
 **expression** is a form enclosed in parentheses; its first child form, called
 the **head**, determines what the expression does, and should generally be a
-function, macro, or special operator. Functions are the most ordinary sort of
-head, whereas macros (described in more detail below) are functions executed at
-compile-time instead and return code to be executed at run-time. Special
-operators are one of :ref:`a fixed set of names <special-forms>` that are
-hard-coded into the compiler, and used to implement everything else.
+function or macro. Functions are the most ordinary sort of head, whereas macros
+(described in more detail below) are functions executed at compile-time instead
+and return code to be executed at run-time.
 
 Comments start with a ``;`` character and continue till the end of the line. A
 comment is functionally equivalent to whitespace. ::
@@ -156,9 +154,9 @@ executes and returns the form ``THEN`` if ``CONDITION`` is true (according to
 in its place.
 
 What if you want to use more than form in place of the ``THEN`` or ``ELSE``
-clauses, or in place of ``CONDITION``, for that matter? Use the special
-operator :hy:func:`do` (known more traditionally in Lisp as ``progn``), which
-combines several forms into one, returning the last::
+clauses, or in place of ``CONDITION``, for that matter? Use the macro
+:hy:func:`do` (known more traditionally in Lisp as ``progn``), which combines
+several forms into one, returning the last::
 
    (if (do (print "Let's check.") (= 1 1))
      (do