de.dante.extex.scanner.stream
Interface TokenStream

All Known Implementing Classes:
TokenStreamBaseImpl, TokenStreamImpl

public interface TokenStream

This interface describes the features of a stream capable of delivering Tokens. In fact it is a push-back stream since Tokens already read can be pushed back onto the stream for further reading.

Nevertheless you should be aware that characters once coined into tokens are not changed – even if the tokenizer might produce another result in the meantime.

Version:
$Revision: 1.22 $
Author:
Gerd Neugebauer

Method Summary
 boolean closeFileStream()
          Close this stream if it is a file stream.
 Token get(TokenFactory factory, Tokenizer tokenizer)
          Get the next token from the token stream.
 Locator getLocator()
          Getter for the locator.
 boolean isEof()
          Check to see if a further token can be acquired from the token stream.
 boolean isEol()
          Check to see if the token stream is currently at the end of line.
 boolean isFileStream()
          Check whether the current stream is associated with a file to read from.
 void put(Token token)
          Push back a token into the stream.
 

Method Detail

closeFileStream

public boolean closeFileStream()
Close this stream if it is a file stream.

Returns:
true if the closing was successful

get

public Token get(TokenFactory factory,
                 Tokenizer tokenizer)
          throws ScannerException
Get the next token from the token stream. If tokens are on the push-back stack then those are delivered otherwise new tokens might be extracted utilizing the token factory and the tokenizer.

Parameters:
factory - the token factory
tokenizer - the tokenizer
Returns:
the next Token or null if no more tokens are available
Throws:
ScannerException - in case of an error

getLocator

public Locator getLocator()
Getter for the locator. The locator describes the place the tokens have been read from in terms of the user. This information is meant for the end user to track down problems.

Returns:
the locator

isEof

public boolean isEof()
              throws ScannerException
Check to see if a further token can be acquired from the token stream.

Returns:
true if the stream is at its end
Throws:
ScannerException - in case that an error has been encountered. Especially if an IO exceptions occurs it is delivered as chained exception in a ScannerException.

isEol

public boolean isEol()
              throws ScannerException
Check to see if the token stream is currently at the end of line.

Returns:
true if the stream is at end of line
Throws:
ScannerException - in case that an error has been encountered. Especially if an IO exceptions occurs it is delivered as chained exception in a ScannerException.

isFileStream

public boolean isFileStream()
Check whether the current stream is associated with a file to read from.

Returns:
true if the stream is a file stream

put

public void put(Token token)
Push back a token into the stream. If the token is null then nothing happens: a null token is not pushed!

Note that it is up to the implementation to accept tokens not produced with the token factory for push back. In general the behaviour in such a case is not defined and should be avoided.

Parameters:
token - the token to push back
See Also:
"TeX – The Program [325]"