de.dante.extex.scanner.stream.impl
Class TokenStreamImpl

java.lang.Object
  extended byde.dante.extex.scanner.stream.impl.TokenStreamBaseImpl
      extended byde.dante.extex.scanner.stream.impl.TokenStreamImpl
All Implemented Interfaces:
TokenStream
Direct Known Subclasses:
TokenStreamImpl32

public class TokenStreamImpl
extends TokenStreamBaseImpl
implements TokenStream

This class contains an implementation of a token stream which is fed from a Reader.

Version:
$Revision: 1.54 $
Author:
Gerd Neugebauer, Michael Niedermair

Field Summary
protected static int CARET_LIMIT
          The constant CARET_LIMIT contains the threshold for the ^ notation.
protected static de.dante.extex.scanner.stream.impl.TokenStreamImpl.State MID_LINE
          The constant MID_LINE contains the state for the processing in the middle of a line.
protected static de.dante.extex.scanner.stream.impl.TokenStreamImpl.State NEW_LINE
          The constant NEW_LINE contains the state for the processing at the beginning of a new line.
protected  int pointer
          The index in the buffer for the next character to consider.
protected  UnicodeChar saveChar
          The field saveChar contains the saved look-ahead character.
protected static de.dante.extex.scanner.stream.impl.TokenStreamImpl.State SKIP_BLANKS
          The constant SKIP_BLANKS contains the state for the processing when spaces are ignored.
protected  de.dante.extex.scanner.stream.impl.TokenStreamImpl.State state
          The field state contains the current state of operation.
 
Constructor Summary
TokenStreamImpl(Configuration config, TokenStreamOptions options, java.io.InputStream stream, java.lang.String theSource, java.lang.String encoding)
          Creates a new object.
TokenStreamImpl(Configuration config, TokenStreamOptions options, java.io.Reader reader, java.lang.Boolean isFile, java.lang.String theSource)
          Creates a new object.
TokenStreamImpl(Configuration config, TokenStreamOptions options, java.lang.String theLine, java.lang.String theSource)
          Creates a new object.
 
Method Summary
protected  boolean atEndOfLine()
          Checks whether the pointer is at the end of line.
protected  void endLine()
          End the current line.
protected  UnicodeChar getChar(Tokenizer tokenizer)
          Return the next character to process.
 Locator getLocator()
          Getter for the locator.
protected  Token getNext(TokenFactory factory, Tokenizer tokenizer)
          Get the next token when the stack is empty.
protected  UnicodeChar getRawChar()
          Get the next character from the input line.
protected  int hex2int(int c)
          Analyze a character and return its hex value, i.e.
 boolean isEof()
          Check to see if a further token can be acquired from the token stream.
 boolean isEol()
          Check to see if the token stream is currently at the end of line.
protected  boolean refill()
          Get the next line from the input reader to be processed.
 java.lang.String toString()
           
 
Methods inherited from class de.dante.extex.scanner.stream.impl.TokenStreamBaseImpl
closeFileStream, get, isFileStream, put, skipSpaces
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 
Methods inherited from interface de.dante.extex.scanner.stream.TokenStream
closeFileStream, get, isFileStream, put
 

Field Detail

CARET_LIMIT

protected static final int CARET_LIMIT
The constant CARET_LIMIT contains the threshold for the ^ notation.

See Also:
Constant Field Values

MID_LINE

protected static final de.dante.extex.scanner.stream.impl.TokenStreamImpl.State MID_LINE
The constant MID_LINE contains the state for the processing in the middle of a line.


NEW_LINE

protected static final de.dante.extex.scanner.stream.impl.TokenStreamImpl.State NEW_LINE
The constant NEW_LINE contains the state for the processing at the beginning of a new line.


SKIP_BLANKS

protected static final de.dante.extex.scanner.stream.impl.TokenStreamImpl.State SKIP_BLANKS
The constant SKIP_BLANKS contains the state for the processing when spaces are ignored.


pointer

protected int pointer
The index in the buffer for the next character to consider. This is an invariant: after a character is read this pointer has to be advanced.


saveChar

protected UnicodeChar saveChar
The field saveChar contains the saved look-ahead character.


state

protected de.dante.extex.scanner.stream.impl.TokenStreamImpl.State state
The field state contains the current state of operation.

Constructor Detail

TokenStreamImpl

public TokenStreamImpl(Configuration config,
                       TokenStreamOptions options,
                       java.io.InputStream stream,
                       java.lang.String theSource,
                       java.lang.String encoding)
                throws java.io.IOException,
                       ConfigurationException
Creates a new object.

Parameters:
config - the configuration object for this instance; This configuration is ignored in this implementation.
options - ignored here
theSource - the description of the information source; e.g. the file name
encoding - the encoding to use
stream - the input stream to read
Throws:
ConfigurationException - in case of an error in the configuration
java.io.IOException - in case of an IO error

TokenStreamImpl

public TokenStreamImpl(Configuration config,
                       TokenStreamOptions options,
                       java.io.Reader reader,
                       java.lang.Boolean isFile,
                       java.lang.String theSource)
                throws java.io.IOException
Creates a new object.

Parameters:
config - the configuration object for this instance; This configuration is ignored in this implementation.
options - ignored here
reader - the reader
isFile - indicator for file streams
theSource - the description of the input source
Throws:
java.io.IOException - in case of an IO error

TokenStreamImpl

public TokenStreamImpl(Configuration config,
                       TokenStreamOptions options,
                       java.lang.String theLine,
                       java.lang.String theSource)
                throws java.io.IOException
Creates a new object.

Parameters:
config - the configuration object for this instance; This configuration is ignored in this implementation.
options - ignored here
theLine - the string to use as source for characters
theSource - the description of the input source
Throws:
java.io.IOException - in case of an IO error
Method Detail

atEndOfLine

protected boolean atEndOfLine()
Checks whether the pointer is at the end of line.

Returns:
true iff the next reading operation would try to refill the line buffer

endLine

protected void endLine()
End the current line.


getChar

protected UnicodeChar getChar(Tokenizer tokenizer)
                       throws ScannerException
Return the next character to process. The pointer is advanced and points to the character returned.

This operation might involve that an additional bunch of characters is read in (with refill()).

Parameters:
tokenizer - the classifier for characters
Returns:
the character or null if no more character is available
Throws:
ScannerException - in the rare case that an IO Exception has occurred.

getLocator

public Locator getLocator()
Description copied from interface: TokenStream
Getter for the locator. The locator describes the place the tokens have been read from in terms of the user. This information is meant for the end user to track down problems.

Specified by:
getLocator in interface TokenStream
Overrides:
getLocator in class TokenStreamBaseImpl
See Also:
TokenStream.getLocator()

getNext

protected Token getNext(TokenFactory factory,
                        Tokenizer tokenizer)
                 throws ScannerException
Description copied from class: TokenStreamBaseImpl
Get the next token when the stack is empty. This method is meant to be overloaded by derived classes.

Overrides:
getNext in class TokenStreamBaseImpl
Parameters:
factory - the factory for new tokens
tokenizer - the classifies for characters
Returns:
the next Token or null
Throws:
ScannerException - in case of an error
See Also:
TokenStreamBaseImpl.getNext( de.dante.extex.scanner.type.token.TokenFactory, de.dante.extex.interpreter.Tokenizer)

getRawChar

protected UnicodeChar getRawChar()
Get the next character from the input line.

Returns:
the next raw character or null if none is available.

hex2int

protected int hex2int(int c)
Analyze a character and return its hex value, i.e. '0' to '9' are mapped to 0 to 9 and 'a' to 'f' (case sensitive) are mapped to 10 to 15.

Parameters:
c - the character code to analyze
Returns:
the integer value of a hex digit or -1 if no hex digit is given

isEof

public boolean isEof()
              throws ScannerException
Description copied from interface: TokenStream
Check to see if a further token can be acquired from the token stream.

Specified by:
isEof in interface TokenStream
Overrides:
isEof in class TokenStreamBaseImpl
Returns:
true iff the stream is at its end
Throws:
ScannerException - in case of an error
See Also:
TokenStream.isEof()

isEol

public boolean isEol()
Description copied from interface: TokenStream
Check to see if the token stream is currently at the end of line.

Specified by:
isEol in interface TokenStream
Overrides:
isEol in class TokenStreamBaseImpl
See Also:
TokenStream.isEol()

refill

protected boolean refill()
                  throws ScannerException
Get the next line from the input reader to be processed.

Returns:
true iff the next line could be acquired.
Throws:
ScannerException - in case of some kind of IO error

toString

public java.lang.String toString()
See Also:
Object.toString()