TokenizerImpl
public interface Tokenizer
Modifier and Type | Method | Description |
---|---|---|
java.lang.String |
getErrorDescription() |
If hasErrors returns true, returns a description of the error
encountered.
|
Token |
getNextToken() |
Returns the next token.
|
boolean |
hasErrors() |
Returns true if there were errors while reading tokens.
|
boolean |
hasMoreTokens() |
Returns true if there are more tokens, false otherwise.
|
boolean |
isBreak() |
Determines if the current token should start a new sentence.
|
void |
setInputReader(java.io.Reader reader) |
Sets the input reader.
|
void |
setInputText(java.lang.String textToTokenize) |
Sets the text to be tokenized by this tokenizer.
|
void |
setPostpunctuationSymbols(java.lang.String symbols) |
Sets the postpunctuation symbols of this Tokenizer to the given
symbols.
|
void |
setPrepunctuationSymbols(java.lang.String symbols) |
Sets the prepunctuation symbols of this Tokenizer to the given
symbols.
|
void |
setSingleCharSymbols(java.lang.String symbols) |
Sets the single character symbols of this Tokenizer to the given
symbols.
|
void |
setWhitespaceSymbols(java.lang.String symbols) |
Sets the whitespace symbols of this Tokenizer to the given
symbols.
|
void setInputText(java.lang.String textToTokenize)
textToTokenize
- the text to tokenizevoid setInputReader(java.io.Reader reader)
reader
- the input sourceToken getNextToken()
boolean hasMoreTokens()
boolean hasErrors()
java.lang.String getErrorDescription()
void setWhitespaceSymbols(java.lang.String symbols)
symbols
- the whitespace symbolsvoid setSingleCharSymbols(java.lang.String symbols)
symbols
- the single character symbolsvoid setPrepunctuationSymbols(java.lang.String symbols)
symbols
- the prepunctuation symbolsvoid setPostpunctuationSymbols(java.lang.String symbols)
symbols
- the postpunctuation symbolsboolean isBreak()