Emsip sexps redux

Emsip sexps redux

A few months ago I blogged about Emsip. I proposed that an Emsip interpreter could neatly handle source code by farming the specialized parts off to special {}-interpreters and handling the core {numbers,symbols,punctuation} with just a few classes and states.

It occurs to me that my design might be made even simpler. Instead of punctuation grouping at the beginning of a sequence of weak constitutents, let it group at the end.

That lets the object-making states be almost entirely regular. There are no longer special conditions for what interrupts them, except for the state Uncommitted, which is interrupted by anything other than a weak constitutent.

That means that punctuation can appear directly before symbols,

'a   +:'b 
;;quoted symbol "a", "+:'" applied to symbol "b".

but not directly after, because the punctuation constitutents will be incorporated in:

a'   b+:' 
;;Two symbols, "a'" and "b+:'"

This is reasonable, since it covers the common use case and because the purpose of punctuation is to modify following sexps, not preceding ones.

The new design

We have these states:

State	Interruptable by
No object	All but whitespace
Uncommitted	All but weak cons
Committed	Only whitespace

The classes of character are the same as before, and now have these transition behaviors:

Class	Object begun	New state
Whitespace	Nothing	No object
Weak constituent	(Possible symbol)	Uncommitted
Strong punctuation	Xforms the tail	No object
Letter	Symbol	Committed
Digit	Number	Committed

We make an object when we enter the No-object state. But that is really a reversed reversal. The real rule is to make a new object when we leave the union of the other two states. So to nail down the details:

• Beginning to read the sexp doesn't make an object, even though we start reading in No-object state.
• Leaving the sexp makes an object unless state was No-object. So it's as if the sexp read is surrounded by No-object state.
• Strong punctuation makes an object even if previous state was No-object, as if we pass thru the Committed state momentarily before re-entering the No-object state.
• Nested reads (which are handled by EMSIP outside of here) enter No-object before they begin, make one object, and leave state as No-object.

suitable SREs

I'll update the SREs as well. And I'll trivially add `nested-object', which I left out last time. It would be supplied by EMSIP framework.

These SREs assume `nested-object' and the classes of character as primitives. They assume longest-match parsing so `uncommitted' will be part of a larger construct when possible (only the symbol second alternative is at issue)

whitespace

(+ whitespace-constituent)

uncommitted

(+ weak-constituent)

number

(seq (? uncommitted) digit (* (~ whitespace-constituent)))

symbol

(or (seq (? uncommitted) letter (* (~ whitespace-constituent))) uncommitted)

punctuation

(seq (? uncommitted) strong-punctuation)

thing

(or whitespace number symbol punctuation nested-object)

(overall)

(* thing)

Notes on numbers

When I say a "number" object is made, I don't mean that "number" must be a primitive type. On the contrary, I expect distinct types including at least integers, exact reals, and inexact reals.

What I mean is that all numbers are of this form. That substring is parsed by another very specialized regular-expression matcher.

Since we can just farm any misfits off to specialized {}-interpreters, we can afford to be very picky about what we treat here. My thoughts on what to support:

Positive vs negative

Of course.

Decimal point

Of course.

Scientific notation

I think so.

Bases

Maybe just decimal and hex.

The infinities and NaN

No. They are semantically numbers but have no digits. They gain nothing by sharing their syntax with this.

Rationals

Maybe. They are convenient.

Inexact vs exact

Probably. A default to exact integers and inexact reals is reasonable. But one would sometimes want exact reals, and it might be a nuisance to have to switch notations just for that. We can't use prefixes #e and #i since they make us read a symbol instead of a number, and trailing "e" is used by scientific notation. So perhaps trailing "+-" could indicate inexactness.

Uncertainty

Maybe. Having "+-", it's tempting to use "+-" followed by a (positive, exact) number to indicate uncertainty.

Precision

Ie, floats vs doubles vs perhaps bignums. No. That's really about object construction. If that sort of control is wanted, real object construction should be used, not syntax tricks.

Digit group separators

As in "1,000,000". Probably, since it almost falls out of this syntax. Since the comma isn't universally accepted in this role, let's accept the underscore too. But IMO going any further would add a great deal of complexity. So no provisions to treat the comma as decimal point in some locales, etc.