ExtendingRequire

This page is logging PnkFelix's journey into attempting to extend our require to do something intelligent when you require a library that exports macro definitions. See also YouWantItWhen.

The big issue is handling compilation of such units. PnkFelix is not currently worrying about compiling units that define macros; we can work around that for now. We cannot work around compiling units that import macros, other than ensuring that we've explicitly loaded such macros into the compiler prior to compilation, which is not a very robust approach.

PnkFelix thought this could be a simple matter of redefining require as a transformer macro that expands into a low level procedure require/load (which has semantics equivalent to our current require, but in addition it also does a visit of the target module. This visit should just consist of expanding all of the expressions in the module; due to the way define-syntax is defined, this expansion would mutate the syntactic environment with the necessary macro definitions.

• Update as of June 13, 2006: PnkFelix is abandoning this approach, because there is a difficulty when you promote require to a special form, in that macros that expand into uses of require (such as SRFI-0's cond-expand) cannot be easily handled, because then you end up processing forms like (.require|1 (.quote|1 .srfi-8|1)).
• This is an interesting case of it being difficult to write a transformer macro that tries to inspect its arguments directy as data. It would seem that I need some way for a transformer macro to map a renamed-symbol back to its base symbol (in the manner that m-strip does). That, or use the given equivalence predicate and compare against every possible library name (yuck!).
• Update as of September 1, 2007: You know, if someone used a string instead of a symbol to refer to the desired library, then it would not get renamed, and this approach might actually work...
• Yes, a quick hack I just did seems to work just fine! See Ticket #56 (look at the comments and the attached files).

---

(The below documents PnkFelix's journey towards implementing require as of June 13, 2006, which is when he abandoned the idea.)

The first big problem that PnkFelix encountered was even getting require to be a special form. Something in the core like require needs to be registered in Lib/Common/toplevel.sch, but all of the special forms defined for the top level there are part of the usual macros defined in Compiler/usual.sch. However, you can't put a transformer macro into Compiler/usual.sch. (This might be a bug. See WhyOrderMatters.) PnkFelix finally worked around this by using a trick that RyanCulpepper put into Lib/MzScheme/init.sch, of putting the require syntax into a special variable that gets eval'd very late in the game. So that yielded a simple require special form that is a transformer macro but just expands into a call to require/load. See changeset:3101 (the idea evolved a bit from its original incarnation, which just called eval; the current one calls twobit-expand).

Next is the problem of doing the visit. To do the visit, we need to call twobit-expand recursively with the current syntactic environment. But transformer macros don't actually have a hook on the current syntactic environment. Whoops. (Plus the bootstrap system doesn't expose the-usual-syntactic-environment to the runtime. But that wouldn't be a good solution anyway.) Fixing this seems like it would require rethinking the macro architecture a bit, or at least adding an extension to transformer macros where the procedure gets some additional context to allow it to expand constructed expressions and/or extend the syntactic environment. PnkFelix got this effect by making a new kind of transformer form, called transformer/expand, whcih takes a fouth argument which is an expansion function (with contract SchemeExpr -> CoreSchemeExpr). See changeset:3100 for the implementation of this new transformer.

(Another option would be to just define a "standard" load/visit procedure that does this expansion of a target file without executing the expanded expressions... but one thing that PnkFelix liked about the current solution was that it isolates the added macro definitions to just the syntactic environment of the current expansion (which is sometimes a fresh copy created just for the purposes of expansion (see for example the definitions of compile-file in Compiler/driver-twobit.sch and Compiler/driver-larceny.sch.)

The bulk of the implementation was then developed, which required changes to the codebase in Lib/Common/require.sch. See changeset:3103.

There were a lot of issues with getting the environments set up properly. One problem is with bootstrapping; you want require in the initial environment, but the initial environment needs to be set up before eval, but you need eval before you can define transformer macros, and require is defined as a transformer macro. There's a cycle in that dependence graph. PnkFelix is hacking around this by having two stages to establish the initial environment; one without transformer macros, so that we can define eval sensibly, and then add all the transformer macros (whose definitions presumably cannot be allowed to anything but the usual macros that were added initially; that is, the non transformer macros. All of this will hopefully be replaced by something sane when we adopt something like YouWantItWhen (note that this cycle may be exactly why RyanCulpepper suggested to PnkFelix that we make require a primitive form handled by the compiler directly, rather than a macro layered on top in this wild fashion). See changeset:3102 and changeset:3103.

The other big issue was getting the compiler and the interpreter to handle the new require special form in a consistent fashion. PnkFelix figured out a way to do this eventually, but it involved introducing an extra hook that the two components are expected to call after eval has been prepped accordingly. See changeset:3101 (this is part of the evolution referred to above).