

Here are some additional notes on the Charlemagne application language, especially
changes from Orlando.

At the token level -

GetNextToken (the basic tokenizer) has been beefed up in several ways.

1. There are now comment-toggle and end-comment token-types. 
The first time a character of type comment-toggle is encountered 
by GetNextToken, it counts all other characters
(except those of the  comment-toggle and end-comment type) as 
neutral ("white space"). An end-comment character is counted as
neutral (except to turn off comments). Currently, the only characters
of these types are @ (comment-toggle) and <NewLine> (end-comment)

2. There is an exponent-separator token-type which is counted as a
digit if it follows characters of digit-type, and as an ordinary character
otherwise. At the moment, there are no characters of this type,
but when everything gets back in synch, E will be of this type (since
we have decided to return to normal E as the exponent-separator,
right?)

3. digits followed by ordinary characters are broken into separate
tokens: "123XYZ" --> "123" "XYZ", but not the reverse:
"XYZ123" --> "XYZ123".

Token Type Tables -

There are now four token-type tables, instead of the two on Orlando. The
standard one (PalTTT) is used in normal RPN parsing. A variant of it
in which the digits 0-9 are ordinary is used for hex-string and GROB
data parsing. The standard symbolic parser token-type table (solverTTT)
is used when parsing expressions. A variant of it in which space, newline,
carriage-left and carriage-right are all delimiters is used in parsing
unit bodies in the standard RPN parser (so that the end of the 
unit expression can be determined without a trailing delimiter).

Macros and "Program Fragments" -

All of the macros can now be parsed wherever an object is required
(for example, in a list.) When they are parsed this way, the result is
always a secondary, but without the identifying <<..>> pair. These could
be termed "program fragments". 

New Object Types -

There are several new parseable object types. They include GROBs,
directories, tagged objects, and units.

1. GROBS are parsed in the form
   GROB height(real) width(real) data(hex-characters)

The height and width are parsed as real numbers and COERCEd to the
corresponding bint values. The data must exactly match the reqired
number of bytes for the given height and width (including the
necessary padding.) This makes it unlikely that someone will create
a GROB directly through the parser, but it allows them to be transferred
in ASCII mode over the I/O, and to be edited to some degree.

2. Directories (RAM-ROM Pairs) are parsed in the form

   DIR name object...name object END

They O-PAIRS are created in the order which will cause them to
be decompiled in the same relative ordering. No check is made for
duplicate names in the directory, and no ROMPART is attached.

3. Tagged objects are parsed in the form:

   :characters:object

There is no restriction on the characters, except that they cannot include
the colon character itself.

4. Unit objects can be parsed both in RPN and within expressions. The
two forms are quite similar, except for the termination conditions.

In RPN parsing, a unit object has the form
   real_unitbody<delimiter>
where <delimiter> is either space, or newline, or one of the carriage
returns, or any other normal object delimiter, or end-of-line. The
BNF for unitbody is described in symbPA.ers. In short, it is any
products and quotients of allowable units with optional positive real 
coefficients, raised to real powers.

In an expression, the "real" can be any expression, and the <delimiter>
must be some delimiter recognized by the symbolic parser (parentheses,
single-character-infix-or-postfix operator, etc.)


New Macros and Alternative Parsings -

There is really only one new macro, the CASE statement which has
the form
    CASE ...THEN..END ... THEN..END ...END

The outer CASE...END pair delimit the scope of the structure and
each THEN..END pair give the alternatives. The THEN takes a flag
from the stack and if the flag is true, it evaluates the stuff
within the THEN..END pair, and resumes execution after the trailing
END of the CASE...END pair. If the flag is false, it skips the stuff
and resumes execution after the trailing END of the THEN..END pair.

Complex numbers can now be parsed in either rectangular or polar form.
Rectangular form is as always, and polar form looks like:
    (modulus <anglesign> angle)
The angle is assumed to be in radians, degrees, or grads, as per the
current mode.

Vectors can be parsed in rectangular, polar/cylindrical, and spherical
coordinate form. Rectangular is as always, cylindrical looks like
    [XYmodulus <anglesign> angle ..other coordinates..]
Spherical looks like:
    [XYZmodulus <anglesign> XYangle <anglesign> Z-XYangle ..other coord.s]
Angles are assumed in degrees, radians, or grads as per the current mode.

Matrices have an additional parse form, wherein the delimiting marks
between rows, after the first row, may be omitted. In this case, there
must be an integral multiple of the first row's elements total, and the
elements are divvied up among the rows. The form is
   [[number1...numberN]...[number1...numberN] number1...numberK*N]

Finally, there is an alternative method for parsing character strings,
call the counted string. This has one of two forms:
   C$ real characters
or
   $ characters
In the first case, the real is COERCED and the resulting count's worth
of characters (after the character delimiting the real) are gathered
into a string. In the second case, all characters remaining in the edit
line are put into the string.

On-the-fly I/O Parsing -

Although this feature is transparent to the user (except that he can
receive much bigger objects in ASCII mode than he would otherwise.) However,
we should make a note of its quirks so that no one making an extension or
revision to the system gets caught by them. 

On the fly parsing works through revisions to GetNextToken which, when it
either encounters the end of a string in the process of tokenizing, checks
to see if I/O is active (by the value of pre-defined temp. variables). If
it is, it tries to get another packet's worth of data, appends it to the
current token and retokenizes the whole mess, repeating the process if
it again runs into the end. This process will, contrary to Orlando
conventions, change the string currently being parsed in order to 
throw away previously parsed packets. This precludes the possibility
of calling a parser on a single token and then resuming the parse at
the previous location. This was, for example, done in Orlando for
parsing an expression. First, the whole substring between matched '
pairs was broken off and this was sent to the symbolic parser which 
used GetNextToken to break it into tokens (which would screw up
the synchronization between the parser and the I/O if done now.)

While this problem has been completely eliminated from the standard
parsers, there are places (like textbook mode) where it is still done.
Since these are not available (yet) during I/O, there should be no
problem. If someone tries to implement "KEYBOARD IS", however,...

..........Charlie

