类 StringToWordVector
java.lang.Object
weka.filters.Filter
weka.filters.unsupervised.attribute.StringToWordVector
Converts String attributes into a set of attributes representing word occurrence (depending on the tokenizer) information from the text contained in the strings. The set of words (attributes) is determined by the first batch filtered (typically training data).
Valid options are:
-C Output word counts rather than boolean word presence.
-R <index1,index2-index4,...> Specify list of string attributes to convert to words (as weka Range). (default: select all string attributes)
-V Invert matching sense of column indexes.
-P <attribute name prefix> Specify a prefix for the created attribute names. (default: "")
-W <number of words to keep> Specify approximate number of word fields to create. Surplus words will be discarded.. (default: 1000)
-prune-rate <rate as a percentage of dataset> Specify the rate (e.g., every 10% of the input dataset) at which to periodically prune the dictionary. -W prunes after creating a full dictionary. You may not have enough memory for this approach. (default: no periodic pruning)
-T Transform the word frequencies into log(1+fij) where fij is the frequency of word i in jth document(instance).
-I Transform each word frequency into: fij*log(num of Documents/num of documents containing word i) where fij if frequency of word i in jth document(instance)
-N Whether to 0=not normalize/1=normalize all data/2=normalize test data only to average length of training documents (default 0=don't normalize).
-L Convert all tokens to lowercase before adding to the dictionary.
-S Ignore words that are in the stoplist.
-stemmer <spec> The stemmering algorihtm (classname plus parameters) to use.
-M <int> The minimum term frequency (default = 1).
-O If this is set, the maximum number of words and the minimum term frequency is not enforced on a per-class basis but based on the documents in all the classes (even if a class attribute is set).
-stopwords <file> A file containing stopwords to override the default ones. Using this option automatically sets the flag ('-S') to use the stoplist if the file exists. Format: one stopword per line, lines starting with '#' are interpreted as comments and ignored.
-tokenizer <spec> The tokenizing algorihtm (classname plus parameters) to use. (default: weka.core.tokenizers.WordTokenizer)
-
字段概要
字段修饰符和类型字段说明static final int
normalization: No normalization.static final int
normalization: Normalize all data.static final int
normalization: Normalize test data only.static final Tag[]
Specifies whether document's (instance's) word frequencies are to be normalized. -
构造器概要
构造器构造器说明Default constructor.StringToWordVector
(int wordsToKeep) Constructor that allows specification of the target number of words in the output. -
方法概要
修饰符和类型方法说明Returns the tip text for this property.Returns the tip text for this property.boolean
Signify that this batch of input to the filter is finished.Returns the tip text for this property.Gets the current range selection.Get the attribute name prefix.Returns the Capabilities of this filter.boolean
Get the DoNotOperateOnPerClassBasis value.boolean
Sets whether if the word frequencies in a document should be transformed into:
fij*log(num of Docs/num of Docs with word i)
where fij is the frequency of word i in document(instance) j.boolean
Gets whether the supplied columns are to be processed or skipped.boolean
Gets whether if the tokens are to be downcased or not.int
Get the MinTermFreq value.Gets whether if the word frequencies for a document (instance) should be normalized or not.String[]
Gets the current settings of the filter.boolean
Gets whether output instances contain 0 or 1 indicating word presence, or word counts.double
Gets the rate at which the dictionary is periodically pruned, as a percentage of the dataset size.Returns the revision string.Get the value of m_SelectedRange.Returns the current stemming algorithm, null if none is used.returns the file used for obtaining the stopwords, if the file represents a directory then the default ones are used.boolean
Gets whether if the word frequencies should be transformed into log(1+fij) where fij is the frequency of word i in document(instance) j.Returns the current tokenizer algorithm.boolean
Gets whether if the words on the stoplist are to be ignored (The stoplist is in weka.core.StopWords).int
Gets the number of words (per class if there is a class attribute assigned) to attempt to keep.Returns a string describing this filter.Returns the tip text for this property.boolean
Input an instance for filtering.Returns the tip text for this property.Returns an enumeration describing the available options.Returns the tip text for this property.static void
Main method for testing this class.Returns the tip text for this property.Returns the tip text for this property.Returns the tip text for this property.Returns the tip text for this property.void
setAttributeIndices
(String rangeList) Sets which attributes are to be worked on.void
setAttributeIndicesArray
(int[] attributes) Sets which attributes are to be processed.void
setAttributeNamePrefix
(String newPrefix) Set the attribute name prefix.void
setDoNotOperateOnPerClassBasis
(boolean newDoNotOperateOnPerClassBasis) Set the DoNotOperateOnPerClassBasis value.void
setIDFTransform
(boolean IDFTransform) Sets whether if the word frequencies in a document should be transformed into:
fij*log(num of Docs/num of Docs with word i)
where fij is the frequency of word i in document(instance) j.boolean
setInputFormat
(Instances instanceInfo) Sets the format of the input instances.void
setInvertSelection
(boolean invert) Sets whether selected columns should be processed or skipped.void
setLowerCaseTokens
(boolean downCaseTokens) Sets whether if the tokens are to be downcased or not.void
setMinTermFreq
(int newMinTermFreq) Set the MinTermFreq value.void
setNormalizeDocLength
(SelectedTag newType) Sets whether if the word frequencies for a document (instance) should be normalized or not.void
setOptions
(String[] options) Parses a given list of options.void
setOutputWordCounts
(boolean outputWordCounts) Sets whether output instances contain 0 or 1 indicating word presence, or word counts.void
setPeriodicPruning
(double newPeriodicPruning) Sets the rate at which the dictionary is periodically pruned, as a percentage of the dataset size.void
setSelectedRange
(String newSelectedRange) Set the value of m_SelectedRange.void
setStemmer
(Stemmer value) the stemming algorithm to use, null means no stemming at all (i.e., the NullStemmer is used).void
setStopwords
(File value) sets the file containing the stopwords, null or a directory unset the stopwords.void
setTFTransform
(boolean TFTransform) Sets whether if the word frequencies should be transformed into log(1+fij) where fij is the frequency of word i in document(instance) j.void
setTokenizer
(Tokenizer value) the tokenizer algorithm to use.void
setUseStoplist
(boolean useStoplist) Sets whether if the words that are on a stoplist are to be ignored (The stop list is in weka.core.StopWords).void
setWordsToKeep
(int newWordsToKeep) Sets the number of words (per class if there is a class attribute assigned) to attempt to keep.Returns the tip text for this property.Returns the tip text for this property.Returns the tip text for this property.Returns the tip text for this property.Returns the tip text for this property.Returns the tip text for this property.从类继承的方法 weka.filters.Filter
batchFilterFile, filterFile, getCapabilities, getOutputFormat, isFirstBatchDone, isNewBatch, isOutputFormatDefined, makeCopies, makeCopy, numPendingOutput, output, outputPeek, toString, useFilter, wekaStaticWrapper
-
字段详细资料
-
FILTER_NONE
public static final int FILTER_NONEnormalization: No normalization.- 另请参阅:
-
FILTER_NORMALIZE_ALL
public static final int FILTER_NORMALIZE_ALLnormalization: Normalize all data.- 另请参阅:
-
FILTER_NORMALIZE_TEST_ONLY
public static final int FILTER_NORMALIZE_TEST_ONLYnormalization: Normalize test data only.- 另请参阅:
-
TAGS_FILTER
Specifies whether document's (instance's) word frequencies are to be normalized. The are normalized to average length of documents specified as input format.
-
-
构造器详细资料
-
StringToWordVector
public StringToWordVector()Default constructor. Targets 1000 words in the output. -
StringToWordVector
public StringToWordVector(int wordsToKeep) Constructor that allows specification of the target number of words in the output.- 参数:
wordsToKeep
- the number of words in the output vector (per class if assigned).
-
-
方法详细资料
-
listOptions
Returns an enumeration describing the available options.- 指定者:
listOptions
在接口中OptionHandler
- 返回:
- an enumeration of all the available options
-
setOptions
Parses a given list of options. Valid options are:-C Output word counts rather than boolean word presence.
-R <index1,index2-index4,...> Specify list of string attributes to convert to words (as weka Range). (default: select all string attributes)
-V Invert matching sense of column indexes.
-P <attribute name prefix> Specify a prefix for the created attribute names. (default: "")
-W <number of words to keep> Specify approximate number of word fields to create. Surplus words will be discarded.. (default: 1000)
-prune-rate <rate as a percentage of dataset> Specify the rate (e.g., every 10% of the input dataset) at which to periodically prune the dictionary. -W prunes after creating a full dictionary. You may not have enough memory for this approach. (default: no periodic pruning)
-T Transform the word frequencies into log(1+fij) where fij is the frequency of word i in jth document(instance).
-I Transform each word frequency into: fij*log(num of Documents/num of documents containing word i) where fij if frequency of word i in jth document(instance)
-N Whether to 0=not normalize/1=normalize all data/2=normalize test data only to average length of training documents (default 0=don't normalize).
-L Convert all tokens to lowercase before adding to the dictionary.
-S Ignore words that are in the stoplist.
-stemmer <spec> The stemmering algorihtm (classname plus parameters) to use.
-M <int> The minimum term frequency (default = 1).
-O If this is set, the maximum number of words and the minimum term frequency is not enforced on a per-class basis but based on the documents in all the classes (even if a class attribute is set).
-stopwords <file> A file containing stopwords to override the default ones. Using this option automatically sets the flag ('-S') to use the stoplist if the file exists. Format: one stopword per line, lines starting with '#' are interpreted as comments and ignored.
-tokenizer <spec> The tokenizing algorihtm (classname plus parameters) to use. (default: weka.core.tokenizers.WordTokenizer)
- 指定者:
setOptions
在接口中OptionHandler
- 参数:
options
- the list of options as an array of strings- 抛出:
Exception
- if an option is not supported
-
getOptions
Gets the current settings of the filter.- 指定者:
getOptions
在接口中OptionHandler
- 返回:
- an array of strings suitable for passing to setOptions
-
getCapabilities
Returns the Capabilities of this filter.- 指定者:
getCapabilities
在接口中CapabilitiesHandler
- 覆盖:
getCapabilities
在类中Filter
- 返回:
- the capabilities of this object
- 另请参阅:
-
setInputFormat
Sets the format of the input instances.- 覆盖:
setInputFormat
在类中Filter
- 参数:
instanceInfo
- an Instances object containing the input instance structure (any instances contained in the object are ignored - only the structure is required).- 返回:
- true if the outputFormat may be collected immediately
- 抛出:
Exception
- if the input format can't be set successfully
-
input
Input an instance for filtering. Filter requires all training instances be read before producing output.- 覆盖:
input
在类中Filter
- 参数:
instance
- the input instance.- 返回:
- true if the filtered instance may now be collected with output().
- 抛出:
IllegalStateException
- if no input structure has been defined.NullPointerException
- if the input format has not been defined.Exception
- if the input instance was not of the correct format or if there was a problem with the filtering.
-
batchFinished
Signify that this batch of input to the filter is finished. If the filter requires all instances prior to filtering, output() may now be called to retrieve the filtered instances.- 覆盖:
batchFinished
在类中Filter
- 返回:
- true if there are instances pending output.
- 抛出:
IllegalStateException
- if no input structure has been defined.NullPointerException
- if no input structure has been defined,Exception
- if there was a problem finishing the batch.
-
globalInfo
Returns a string describing this filter.- 返回:
- a description of the filter suitable for displaying in the explorer/experimenter gui
-
getOutputWordCounts
public boolean getOutputWordCounts()Gets whether output instances contain 0 or 1 indicating word presence, or word counts.- 返回:
- true if word counts should be output.
-
setOutputWordCounts
public void setOutputWordCounts(boolean outputWordCounts) Sets whether output instances contain 0 or 1 indicating word presence, or word counts.- 参数:
outputWordCounts
- true if word counts should be output.
-
outputWordCountsTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getSelectedRange
Get the value of m_SelectedRange.- 返回:
- Value of m_SelectedRange.
-
setSelectedRange
Set the value of m_SelectedRange.- 参数:
newSelectedRange
- Value to assign to m_SelectedRange.
-
attributeIndicesTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getAttributeIndices
Gets the current range selection.- 返回:
- a string containing a comma separated list of ranges
-
setAttributeIndices
Sets which attributes are to be worked on.- 参数:
rangeList
- a string representing the list of attributes. Since the string will typically come from a user, attributes are indexed from 1.
eg: first-3,5,6-last- 抛出:
IllegalArgumentException
- if an invalid range list is supplied
-
setAttributeIndicesArray
public void setAttributeIndicesArray(int[] attributes) Sets which attributes are to be processed.- 参数:
attributes
- an array containing indexes of attributes to process. Since the array will typically come from a program, attributes are indexed from 0.- 抛出:
IllegalArgumentException
- if an invalid set of ranges is supplied
-
invertSelectionTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getInvertSelection
public boolean getInvertSelection()Gets whether the supplied columns are to be processed or skipped.- 返回:
- true if the supplied columns will be kept
-
setInvertSelection
public void setInvertSelection(boolean invert) Sets whether selected columns should be processed or skipped.- 参数:
invert
- the new invert setting
-
getAttributeNamePrefix
Get the attribute name prefix.- 返回:
- The current attribute name prefix.
-
setAttributeNamePrefix
Set the attribute name prefix.- 参数:
newPrefix
- String to use as the attribute name prefix.
-
attributeNamePrefixTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getWordsToKeep
public int getWordsToKeep()Gets the number of words (per class if there is a class attribute assigned) to attempt to keep.- 返回:
- the target number of words in the output vector (per class if assigned).
-
setWordsToKeep
public void setWordsToKeep(int newWordsToKeep) Sets the number of words (per class if there is a class attribute assigned) to attempt to keep.- 参数:
newWordsToKeep
- the target number of words in the output vector (per class if assigned).
-
wordsToKeepTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getPeriodicPruning
public double getPeriodicPruning()Gets the rate at which the dictionary is periodically pruned, as a percentage of the dataset size.- 返回:
- the rate at which the dictionary is periodically pruned
-
setPeriodicPruning
public void setPeriodicPruning(double newPeriodicPruning) Sets the rate at which the dictionary is periodically pruned, as a percentage of the dataset size.- 参数:
newPeriodicPruning
- the rate at which the dictionary is periodically pruned
-
periodicPruningTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getTFTransform
public boolean getTFTransform()Gets whether if the word frequencies should be transformed into log(1+fij) where fij is the frequency of word i in document(instance) j.- 返回:
- true if word frequencies are to be transformed.
-
setTFTransform
public void setTFTransform(boolean TFTransform) Sets whether if the word frequencies should be transformed into log(1+fij) where fij is the frequency of word i in document(instance) j.- 参数:
TFTransform
- true if word frequencies are to be transformed.
-
TFTransformTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getIDFTransform
public boolean getIDFTransform()Sets whether if the word frequencies in a document should be transformed into:
fij*log(num of Docs/num of Docs with word i)
where fij is the frequency of word i in document(instance) j.- 返回:
- true if the word frequencies are to be transformed.
-
setIDFTransform
public void setIDFTransform(boolean IDFTransform) Sets whether if the word frequencies in a document should be transformed into:
fij*log(num of Docs/num of Docs with word i)
where fij is the frequency of word i in document(instance) j.- 参数:
IDFTransform
- true if the word frequecies are to be transformed
-
IDFTransformTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getNormalizeDocLength
Gets whether if the word frequencies for a document (instance) should be normalized or not.- 返回:
- true if word frequencies are to be normalized.
-
setNormalizeDocLength
Sets whether if the word frequencies for a document (instance) should be normalized or not.- 参数:
newType
- the new type.
-
normalizeDocLengthTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getLowerCaseTokens
public boolean getLowerCaseTokens()Gets whether if the tokens are to be downcased or not.- 返回:
- true if the tokens are to be downcased.
-
setLowerCaseTokens
public void setLowerCaseTokens(boolean downCaseTokens) Sets whether if the tokens are to be downcased or not. (Doesn't affect non-alphabetic characters in tokens).- 参数:
downCaseTokens
- should be true if only lower case tokens are to be formed.
-
doNotOperateOnPerClassBasisTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getDoNotOperateOnPerClassBasis
public boolean getDoNotOperateOnPerClassBasis()Get the DoNotOperateOnPerClassBasis value.- 返回:
- the DoNotOperateOnPerClassBasis value.
-
setDoNotOperateOnPerClassBasis
public void setDoNotOperateOnPerClassBasis(boolean newDoNotOperateOnPerClassBasis) Set the DoNotOperateOnPerClassBasis value.- 参数:
newDoNotOperateOnPerClassBasis
- The new DoNotOperateOnPerClassBasis value.
-
minTermFreqTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getMinTermFreq
public int getMinTermFreq()Get the MinTermFreq value.- 返回:
- the MinTermFreq value.
-
setMinTermFreq
public void setMinTermFreq(int newMinTermFreq) Set the MinTermFreq value.- 参数:
newMinTermFreq
- The new MinTermFreq value.
-
lowerCaseTokensTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getUseStoplist
public boolean getUseStoplist()Gets whether if the words on the stoplist are to be ignored (The stoplist is in weka.core.StopWords).- 返回:
- true if the words on the stoplist are to be ignored.
-
setUseStoplist
public void setUseStoplist(boolean useStoplist) Sets whether if the words that are on a stoplist are to be ignored (The stop list is in weka.core.StopWords).- 参数:
useStoplist
- true if the tokens that are on a stoplist are to be ignored.
-
useStoplistTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
setStemmer
the stemming algorithm to use, null means no stemming at all (i.e., the NullStemmer is used).- 参数:
value
- the configured stemming algorithm, or null- 另请参阅:
-
getStemmer
Returns the current stemming algorithm, null if none is used.- 返回:
- the current stemming algorithm, null if none set
-
stemmerTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
setStopwords
sets the file containing the stopwords, null or a directory unset the stopwords. If the file exists, it automatically turns on the flag to use the stoplist.- 参数:
value
- the file containing the stopwords
-
getStopwords
returns the file used for obtaining the stopwords, if the file represents a directory then the default ones are used.- 返回:
- the file containing the stopwords
-
stopwordsTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
setTokenizer
the tokenizer algorithm to use.- 参数:
value
- the configured tokenizing algorithm
-
getTokenizer
Returns the current tokenizer algorithm.- 返回:
- the current tokenizer algorithm
-
tokenizerTipText
Returns the tip text for this property.- 返回:
- tip text for this property suitable for displaying in the explorer/experimenter gui
-
getRevision
Returns the revision string.- 指定者:
getRevision
在接口中RevisionHandler
- 覆盖:
getRevision
在类中Filter
- 返回:
- the revision
-
main
Main method for testing this class.- 参数:
argv
- should contain arguments to the filter: use -h for help
-