类 WordTokenizer

所有已实现的接口:
Serializable, Enumeration, OptionHandler, RevisionHandler

public class WordTokenizer extends CharacterDelimitedTokenizer
A simple tokenizer that is using the java.util.StringTokenizer class to tokenize the strings.

Valid options are:

 -delimiters <value>
  The delimiters to use
  (default ' \r\n\t.,;:'"()?!').
版本:
$Revision: 1.4 $
作者:
FracPete (fracpete at waikato dot ac dot nz)
另请参阅:
  • 构造器详细资料

    • WordTokenizer

      public WordTokenizer()
  • 方法详细资料

    • globalInfo

      public String globalInfo()
      Returns a string describing the stemmer
      指定者:
      globalInfo 在类中 Tokenizer
      返回:
      a description suitable for displaying in the explorer/experimenter gui
    • hasMoreElements

      public boolean hasMoreElements()
      Tests if this enumeration contains more elements.
      指定者:
      hasMoreElements 在接口中 Enumeration
      指定者:
      hasMoreElements 在类中 Tokenizer
      返回:
      true if and only if this enumeration object contains at least one more element to provide; false otherwise.
    • nextElement

      public Object nextElement()
      Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
      指定者:
      nextElement 在接口中 Enumeration
      指定者:
      nextElement 在类中 Tokenizer
      返回:
      the next element of this enumeration.
    • tokenize

      public void tokenize(String s)
      Sets the string to tokenize. Tokenization happens immediately.
      指定者:
      tokenize 在类中 Tokenizer
      参数:
      s - the string to tokenize
    • getRevision

      public String getRevision()
      Returns the revision string.
      返回:
      the revision
    • main

      public static void main(String[] args)
      Runs the tokenizer with the given options and strings to tokenize. The tokens are printed to stdout.
      参数:
      args - the commandline options and strings to tokenize