Kernel suggestions about modules

Two suggestions about Kernel modules

Foreword

The first part of this was inspired by my desire to bundle testing submodules with Kernel modules. The second part was inspired by my email exchange with John Shutt after I sent the first part to him.

An issue about modules

Klink is at the point where I'm starting to think about built-in test support. (Till now it has relied entirely on an external tester I wrote for emacs) That brought up an issue about modules.

The dilemma

As I see it, the dilemma (multilemma) is this:

1. IMO as a tester, tests ought to group with modules, for many reasons.
2. I doubt it makes sense for modules to have canonical names or identities, so I can't tell the tests about the module, I have to tell the module about the tests.
   • So a test harness needs to be able to look at an environment and find the tests in it, if any. This probably implies they live somewhere in that environment.
3. Reserving a name to always have a special meaning, such as "tests" or "tests", seems wrong for many reasons.
4. make-keyed-static-variable wants to make a fresh environment.
   • That requires always loading the tests first, which is problematic at best.
   • Even if I can load a module before I load the tests for it, I'd still need to maintain a mapping from module to tests.
   • That makes it impossible to define tests incrementally.
5. I could bind an object that a test-definer would write into. Say, an environment (In fact I will, to name sub-tests). But I'd still have to always place the binder around the entire module.
   • It's an error opportunity, having to always remember to do that.
   • It's structurally noisy.
   • The same would have to be done for every functionality that wants to let individual modules "say something about themselves".
6. I could fake it with gensyms but with all the keyed variable support, it'd be a shame.

Possible solutions

Keyed setters

Have make-keyed-static-variable also make a setter, something with semantics similar to:

($vau (value) env ($set! env KEY value))

where KEY refers to the shared secret. If the "binder" return is analogous to `$let', this would be analogous to `$set!'. This would not make a fresh environment.

• Con: Creates uncertainty and error opportunities about what environment is being defined into.
• Con: Doesn't cooperate with constructs like `$provide'

Let accessors have defaults

Another possibility is for the accessor to optionally, if it finds no binding, make one and record it. Presumably it'd evaluate something to make a default.

• Con: Same cons as above.

Smarter modules

Taking horn #5 of the multilemma as a first draft, provide a construction that surrounds a module with all the binders it should have.

Issues:

• Its identity:
  • Is the entry point `get-module' with a larger mandate?
  • Or is it a separate thing? IMHO no.
  • And should this be available on its own? Meaning "bind all the usual things but don't load anything". IMHO yes.
• So which binders should it use?
  • Interested ones are somehow registered externally.
• What happens for binders that are defined after a module is loaded?
  • Are they missing forever? That seems unfortunate.
  • Alternatively, they could behave as if their binders had been used in the first place, since nothing can have accessed them yet (which must have made an error).
    • Pro: This neatly handles circular dependencies and even self-dependencies.
• How may they be registered?
  • If any combiner of the proper signature can be registered, stray code could be registered and subtly change the behavior of all sorts of modules. That'd be a serious problem.
  • So ISTM registering should be something only make-keyed-static-variable or similar can do. We know it makes a normal, harmless binder.
• What specifically registers a binder?
  • make-keyed-static-variable, on some optional argument?
  • A relative of make-keyed-static-variable that always registers the binder?
    • `make-keyed-module-variable'

Coda

I lean towards the 3rd solution, smarter modules.

What do you think?

Late addendum

This implies that `make-keyed-module-variable' takes code to make the initial object, so I better mention that. Probably it's a combiner that's run with no arguments in the dynamic extent of the call to `get-module'.

Reloading with secrets intact

ISTM surely there are situations where one wants to reload a module but keep its "secrets" (keyed variables, encapsulation types) as they are. The alternatives seem unacceptable:

• Somehow prove that there's never a situation where one wants to reload a module that has secrets.
  • Fatal con: There surely are such situations, eg minor bugfixes.
• Require reloading everything to make sure that every instance of every encapsulation type etc is fully recreated.
  • Con: This makes it very painful to use the interpreter interactively. It would be like having to restart emacs every time you fix an elisp bug.
• Track everything affected and reload just those things.
  • Con: Seems hugely difficult to track it all.
  • Con: Still might require so much reloading that it's effectively a restart.
• Name keyed variables etc in the usual Scheme way
  • Fatal con: defeats their purpose.

A possible solution sketched in terms of the high-level behavior

Let secret-makers (make-keyed-static-variable, make-keyed-dynamic-variable, make-encapsulation-type) optionally be passed a symbol and a version-number (integer). Call those that aren't passed this argument "anonymous secret-makers".

Let each module retain, between loads in the same session, a mapping from (symbol version-number) to private info about the respective secret-maker. Anonymous secret-makers don't participate in the mapping.

When a secret-maker is being created, if its symbol and version-number match an earlier version, then the elements that it returns are to be `eq?' to what the earlier version returned, as if the "two" secret-makers were one and the same.

Anonymous secret-makers never satisfy that test. They behave as if they had a new symbol each time.

It is legal for secret-makers to have the same version-number across source-code changes, but then if changes occur within the same session, proper update behavior is not guaranteed.

Rationale: This allows "secret-makers" to usually carry over automatically, yet allows them to be overridden when desirable, eg when their old definition is wrong.

The version-number is separate to avoid making the user create a new name for each version. In principle it could also let an interpreter react intelligently to "going backwards", or warn on missing redefinitions without also giving false warnings for new versions.

We could have instead required the interpreter to treat source-code changes as new versions, but this seems an unreasonable burden and raises issues of code equivalence, and removes control from the user. But an interpreter is allowed to do this, and since it is legal for secret-makers to keep the same version-number across source-code changes, doing so requires nothing special.

To version a secret-maker, this requires changing source code, because the version-number lives in source code. This is less than ideal because it's really a session property that's being expressed, not a source property. But it is generally reasonable.

We require only that the elements that it returns be `eq?'. Requiring that the whole return value be eq? seems unneccessary, though it probably falls out.

Enabling mechanism: Cross-load memoization

The above all can be accomplished by cross-load memoization, which further makes it possible to make all sorts of objects eq? across multiple loads.

This requires mostly:

• That modules retain an object across repeated loads.
• That that object, relative to the module being loaded, be accessible for this purpose.
• That the object be an environment, because it will map from symbol to object.
• Code that:
  • Takes a (name . version) argument
  • Accesses the above environment relative to current module. It's the only thing that can access it.
  • checks version
  • re-uses old value if appropriate
  • otherwise calls to create new object
  • records current (name . version) and value
• A recipe for using the secret-makers this way. Maybe simply:

  ($module-preserve (my-unique-name 1) (make-keyed-static-variable)) 
  

How should this object be shared?

Code defined in other modules shouldn't normally see this or use it, so these objects are not shared dynamically. They are shared statically. That implies affecting the environment that `get-module' makes, that loading runs in.

Presumably we'd use `make-keyed-static-variable' and share the binder with `get-module' and the accessor with `$module-preserve'.

Sketch of actual requirements on Kernel

• The standard environment would contain `$module-preserve', defined as above.
• `get-module' or similar would take an additional argument, the object, which it would make statically available to `$module-preserve'.
• Any future `require' type mechanism would map module names to these objects.
  • It would create new ones for new modules.