de.dante.extex.interpreter
Interface TokenSource

All Known Subinterfaces:
Interpreter
All Known Implementing Classes:
Max, Moritz

public interface TokenSource

This interface describes a class to acquire Tokens from. Beside the pure getter for the next token some higher-level parsing methods are provided here as well.

There are two classes of methods for reading something from a token stream. The methods starting with get perform the raw reading, whereas the methods starting with scan perform expansion as well.

Version:
$Revision: 1.79 $
Author:
Gerd Neugebauer, Michael Niedermair

Method Summary
 void addStream(TokenStream stream)
          Put a given stream on top of the stream stack.
 void closeAllStreams(Context context)
          All input streams are closed and not further Token is available for processing.
 void closeNextFileStream(Context context)
          Close all streams on the stack until a file stream is found.
 void execute(Token token, Context context, Typesetter typesetter)
          Tries to execute a token.
 void executeGroup()
          Scan and execute tokens until the group ends.
 Tokens expand(Tokens tokens, Typesetter typesetter)
          Expand some tokens.
 Box getBox(Flags flags, Context context, Typesetter typesetter)
          Parse the specification of a box.
 CodeToken getControlSequence(Context context)
          Get the next token from the token stream and check that it is a control sequence or active character.
 Font getFont(Context context, java.lang.String primitive)
          Parse the specification of a font.
 boolean getKeyword(Context context, java.lang.String keyword)
          Get tokens from the token stream searching for a sequence of letter tokens.
 Token getLastToken()
          Getter for the token just previously read from the token source.
 Locator getLocator()
          Getter for the locator.
 Token getNonSpace(Context context)
          Get the next token which has not the category code SPACE.
 void getOptionalEquals(Context context)
          Skip spaces and if the next non-space character is an equal sign skip it as well and all spaces afterwards.
 Token getToken(Context context)
          Get the next token form the input streams.
 Tokens getTokens(Context context, TokenSource source, Typesetter typesetter)
          Get the next tokens form the input streams between { and }.
 TokenStreamFactory getTokenStreamFactory()
          Getter for the token stream factory.
 void push(Token token)
          Push back a token onto the input stream for subsequent reading.
 void push(Token[] tokens)
          Push back a list of tokens onto the input stream for subsequent reading.
 void push(Tokens tokens)
          Push back a list of tokens onto the input stream for subsequent reading.
 UnicodeChar scanCharacterCode(Context context, Typesetter typesetter, java.lang.String primitive)
          Scan the input stream for tokens making up a character code, this is a sequence of digits with category code OTHER.
 Token scanNonSpace(Context context)
          Scan the input for the next token which has not the category code SPACE.
 long scanNumber(Context context)
          Deprecated. use Count.scanNumber() instead
 long scanNumber(Context context, Token token)
          Deprecated. use Count.scanNumber() instead
 java.lang.String scanRegisterName(Context context, TokenSource source, Typesetter typesetter, java.lang.String primitive)
          Scan the input streams for an entity to denote a register name.
 Token scanToken(Context context)
          Get the next expanded token form the input streams.
 Tokens scanTokens(Context context, boolean reportUndefined, boolean ignoreUndefined, java.lang.String primitive)
          Get the next expanded token form the input streams between { and }.
 java.lang.String scanTokensAsString(Context context, java.lang.String primitive)
          Get the next expanded token form the input streams between a left brace character (usually {) and a right brace character (usually }) and convert it to a String.
 Tokens scanUnprotectedTokens(Context context, boolean reportUndefined, boolean ignoreUndefined, java.lang.String primitive)
          Get the next expanded token form the input streams between { and }.
 void skipSpace()
          Skip spaces and check whether any tokens are left.
 void update(java.lang.String name, java.lang.String text)
          Send the string to the named observer.
 

Method Detail

addStream

public void addStream(TokenStream stream)
Put a given stream on top of the stream stack. The reading occurs on this new stream before resorting to the previous streams.

Parameters:
stream - the new stream to read from

closeAllStreams

public void closeAllStreams(Context context)
                     throws InterpreterException
All input streams are closed and not further Token is available for processing. This normally means that the interpreter is forced to terminate more or less gracefully.

Parameters:
context - the interpreter context
Throws:
InterpreterException - in case of an error

closeNextFileStream

public void closeNextFileStream(Context context)
                         throws InterpreterException
Close all streams on the stack until a file stream is found. This file stream is closed as last one. The other streams are left unchanged. If no file stream is found the all streams are closed and none is left.

Parameters:
context - the interpreter context
Throws:
InterpreterException - in case of an error

execute

public void execute(Token token,
                    Context context,
                    Typesetter typesetter)
             throws InterpreterException
Tries to execute a token.

Parameters:
token - the Token to execute
context - the interpreter context
typesetter - the typesetter
Throws:
InterpreterException - in case of an error
especially
ErrorLimitException in case that the error limit has been reached

executeGroup

public void executeGroup()
                  throws InterpreterException
Scan and execute tokens until the group ends.

Throws:
InterpreterException - in case of an error
especially
ErrorLimitException in case that the error limit has been reached

expand

public Tokens expand(Tokens tokens,
                     Typesetter typesetter)
              throws GeneralException
Expand some tokens.

Parameters:
tokens - the tokens to expand
typesetter - the typesetter to use
Returns:
the expanded tokens
Throws:
GeneralException - in case of an error

getBox

public Box getBox(Flags flags,
                  Context context,
                  Typesetter typesetter)
           throws InterpreterException
Parse the specification of a box.

This method parses the following syntactic entity:

A Box

   ⟨box⟩ 
During the parsing the flags should be reset to something uncritical and restored at the end.

Parameters:
flags - the flags to be restored
context - the interpreter context
typesetter - the typesetter to use
Returns:
the box gathered
Throws:
InterpreterException - in case of an error

getControlSequence

public CodeToken getControlSequence(Context context)
                             throws InterpreterException
Get the next token from the token stream and check that it is a control sequence or active character. At the end of all input streams the control sequence "inaccessible" is inserted and an exception is thrown. Thus this method will never return null.

This method parses the following syntactic entity:

A Control Sequence

A control sequence is either a active character or an escape sequence. Macros can be assigned to control sequences only.

The definition of a control sequence highly depends on the current configuration of the parser as stored in the category codes. An active character is one with category code 13. A escape sequence starts with a character with category code 0 followed by an arbitrary number of letters – category code 11 – or a single character with any other category code.

Syntax

The formal description of this syntactic entity is the following:
   ⟨control sequence⟩
     → ?13
      |  ?0?
      |  ?0?11*   

Examples

    ~  

    \abc  

Parameters:
context - the interpreter context
Returns:
the token read
Throws:
InterpreterException - in case that the token stream is at its end or that the token read is not a control sequence token

getFont

public Font getFont(Context context,
                    java.lang.String primitive)
             throws InterpreterException
Parse the specification of a font.

This method parses the following syntactic entity:

A Font

   ⟨font⟩ 

Parameters:
context - the interpreter context
primitive - the name of the primitive for error messages
Returns:
a font specification
Throws:
InterpreterException - in case of an error

getKeyword

public boolean getKeyword(Context context,
                          java.lang.String keyword)
                   throws InterpreterException
Get tokens from the token stream searching for a sequence of letter tokens. If all tokens are found then they are removed from the input stream and true is returned. Otherwise all tokens are left in the input stream and false is returned.

Spaces before the keyword are removed from the input stream. Those spaces are not restored, even if the keyword is not found.

Space tokens after the keyword are removed from the input stream.

Parameters:
context - the interpreter context
keyword - the tokens to scan
Returns:
true iff the tokens could have been successfully removed from the input stream
Throws:
InterpreterException - in case of an error

getLastToken

public Token getLastToken()
Getter for the token just previously read from the token source. This is something like a look-back function.

Returns:
the last token or null if not available

getLocator

public Locator getLocator()
Getter for the locator. The locator provides a means to get the information where something is coming from. Usually it points to a line in a file.

Returns:
the current locator

getNonSpace

public Token getNonSpace(Context context)
                  throws InterpreterException
Get the next token which has not the category code SPACE.

Parameters:
context - the interpreter context
Returns:
the next non-space token or null at EOF
Throws:
InterpreterException - in case of an error

getOptionalEquals

public void getOptionalEquals(Context context)
                       throws InterpreterException
Skip spaces and if the next non-space character is an equal sign skip it as well and all spaces afterwards.

This method parses the following syntactic entity:

The Optional Equals

The syntactic entity ⟨equals⟩ skips initial spaces and an equals sign of category 12 if one is found. In the latter case the following spaces are skipped as well.

   ⟨equals⟩
     → ⟨optional spaces⟩
      |  ⟨optional spaces⟩ =12 ⟨optional spaces⟩ 

Parameters:
context - the interpreter context
Throws:
InterpreterException - in case of an error

getToken

public Token getToken(Context context)
               throws InterpreterException
Get the next token form the input streams. If the current input stream is at its end then the next one on the streamStack is used until a token could be read. If all stream are at the end then null is returned.

This method corresponds to the following syntax specification:

A Token

   ⟨token⟩  

A single token depends on the category code of the characters.

Parameters:
context - the interpreter context
Returns:
the next token or null
Throws:
InterpreterException - in case of an error
See Also:
"TeX – The Program [332]"

getTokens

public Tokens getTokens(Context context,
                        TokenSource source,
                        Typesetter typesetter)
                 throws InterpreterException
Get the next tokens form the input streams between { and }. If the current input stream is at its end then the next one on the streamStack is used until a token could be read. If all streams are at the end then an exception is thrown.

This method parses the following syntactic entity:

A Replacement Text

    ⟨replacement text⟩  

Parameters:
context - the interpreter context
source - the source for new tokens
typesetter - the typesetter
Returns:
the next tokens or null
Throws:
InterpreterException - in case of an error

getTokenStreamFactory

public TokenStreamFactory getTokenStreamFactory()
Getter for the token stream factory. The token stream factory can be used to acquire a new token stream.

Returns:
the token stream factory

push

public void push(Token token)
          throws InterpreterException
Push back a token onto the input stream for subsequent reading.

Parameters:
token - the token to push
Throws:
InterpreterException - in case of an error

push

public void push(Token[] tokens)
          throws InterpreterException
Push back a list of tokens onto the input stream for subsequent reading.

Parameters:
tokens - the tokens to push
Throws:
InterpreterException - in case of an error

push

public void push(Tokens tokens)
          throws InterpreterException
Push back a list of tokens onto the input stream for subsequent reading. In case that the argument is null then it is silently ignored.

Parameters:
tokens - the tokens to push
Throws:
InterpreterException - in case of an error

scanCharacterCode

public UnicodeChar scanCharacterCode(Context context,
                                     Typesetter typesetter,
                                     java.lang.String primitive)
                              throws InterpreterException
Scan the input stream for tokens making up a character code, this is a sequence of digits with category code OTHER. The number can be preceded by optional white space. Alternate representations for an character code exist.

This method parses the following syntactic entity:

A Character Code

   ⟨character code⟩ 

A character is a positive number.

Tokens are expanded while gathering the requested values.

Parameters:
context - the interpreter context
typesetter - the typesetter
primitive - the name of the invoking primitive for error handling
Returns:
the value of the integer scanned
Throws:
InterpreterException - in case of an error
especially
InvalidCharacterException in case that no number is found or if it is out of the allowed range
EofException in case that an end of file has been encountered before the character code was complete

scanNonSpace

public Token scanNonSpace(Context context)
                   throws InterpreterException
Scan the input for the next token which has not the category code SPACE.

Parameters:
context - the interpreter context
Returns:
the next non-space token or null at EOF
Throws:
InterpreterException - in case of an error in scanToken()

scanNumber

public long scanNumber(Context context)
                throws InterpreterException
Deprecated. use Count.scanNumber() instead

Scan the input stream for tokens making up a number, this is a sequence of digits with category code OTHER. The number can be preceded by optional white space. Alternate representations for an integer exist.

Parameters:
context - the interpreter context
Returns:
the value of the integer scanned
Throws:
InterpreterException - in case of an error

scanNumber

public long scanNumber(Context context,
                       Token token)
                throws InterpreterException
Deprecated. use Count.scanNumber() instead

Scan the input stream for tokens making up a number, i.e. a sequence of digits with category code OTHER. The number can be preceded by optional white space.

This method implements the generalization of several syntactic definitions from TeX:

A Number

   ⟨number⟩ 

A number consists of a non-empty sequence of digits with category code OTHER.

Parameters:
context - the interpreter context
token - the first token to consider
Returns:
the value of the integer scanned
Throws:
InterpreterException - in case of an error in an observer
especially
MissingNumberException in case that no number could be read

scanRegisterName

public java.lang.String scanRegisterName(Context context,
                                         TokenSource source,
                                         Typesetter typesetter,
                                         java.lang.String primitive)
                                  throws InterpreterException
Scan the input streams for an entity to denote a register name. Upon EOF null is returned.

A Register Name

A register name determines under which key a register can be addressed. In TeX this used to be a positive number only. This has been extended to allow also a token list in braces.

The alternative is controlled by the integer register \maxregister. The following interpretation of the value of this count is used:

  • If the value of this integer register is negative then a arbitrary non-negative number is allowed as register name as well as any list of tokens enclosed in braces.
  • If the value of this integer register is not-negative then a only a non-negative number is allowed as register name which does not exceed the value of the integer register.

The value of the integer register \maxRegister is set differently for various configurations of ExTeX:

  • TeX uses the value 255.
  • eTeX uses the value 32767.
  • Omega uses the value 65536.
  • ExTeX uses the value -1.

Te integer register \maxRegister is defined in the name space system. Thus special actions have to be used to acces it.

Syntax

   ⟨register name⟩
       → ⟨tokens⟩
        | ⟨number⟩  

Examples

  123
  {abc}
 

Parameters:
context - the interpreter context
source - the source for new tokens
typesetter - the typesetter
primitive - the name of the invoking primitive for error handling
Returns:
the register name encountered
Throws:
InterpreterException - in case of an error

scanToken

public Token scanToken(Context context)
                throws InterpreterException
Get the next expanded token form the input streams. If the current input stream is at its end then the next one on the streamStack is used until a token could be read. If all streams are at the end then null is returned.

Parameters:
context - the interpreter context
Returns:
the next token or null
Throws:
InterpreterException - in case of an error

scanTokens

public Tokens scanTokens(Context context,
                         boolean reportUndefined,
                         boolean ignoreUndefined,
                         java.lang.String primitive)
                  throws InterpreterException
Get the next expanded token form the input streams between { and }. If the current input stream is at its end then the next one on the streamStack is used until a token could be read. If all stream are at the end then null is returned.

This method parses the following syntactic entity:

A General text

    ⟨general text⟩  

Parameters:
context - the interpreter context
primitive - the name of the invoking primitive for error handling
reportUndefined - indicator that an undefined control sequence leads to an exception. This parameter is effective only if ignoreUndefined is false
ignoreUndefined - indicator that an undefined control sequence should be treated as \relax
Returns:
the next tokens or null
Throws:
InterpreterException - in case of an error

scanTokensAsString

public java.lang.String scanTokensAsString(Context context,
                                           java.lang.String primitive)
                                    throws InterpreterException
Get the next expanded token form the input streams between a left brace character (usually {) and a right brace character (usually }) and convert it to a String. If the current input stream is at its end then the next one on the streamStack is used until a token could be read. If all stream are at the end then null is returned.

Parameters:
context - the interpreter context
primitive - the name of the invoking primitive for error handling
Returns:
the next tokens as String or null
Throws:
InterpreterException - in case of an error

scanUnprotectedTokens

public Tokens scanUnprotectedTokens(Context context,
                                    boolean reportUndefined,
                                    boolean ignoreUndefined,
                                    java.lang.String primitive)
                             throws InterpreterException
Get the next expanded token form the input streams between { and }. If the current input stream is at its end then the next one on the streamStack is used until a token could be read. If all stream are at the end then null is returned.

Normally all expandable tokens are expanded. This method honors the protected mark and does not try to expand protected code.

Parameters:
context - the interpreter context
primitive - the name of the invoking primitive for error handling
reportUndefined - indicator that an undefined control sequence leads to an exception. This parameter is effective only if ignoreUndefined is false
ignoreUndefined - indicator that an undefined control sequence should be treated as \relax
Returns:
the next tokens or null
Throws:
InterpreterException - in case of an error

skipSpace

public void skipSpace()
Skip spaces and check whether any tokens are left.

This method corresponds to the following specification:

Optional Spaces

This syntactic entity corresponds to an arbitrary number of white-space characters. White-space characters include space, ta and newline characters.

   ⟨optional spaces⟩
     → [ \t\n]*  


update

public void update(java.lang.String name,
                   java.lang.String text)
            throws InterpreterException,
                   NotObservableException
Send the string to the named observer. The observer must be capable to deal with a string argument.

Parameters:
name - name of the observer
text - the text to send to the observer
Throws:
InterpreterException - in case of an error
NotObservableException - in case that the named observer is not accessible