Package org.apache.spark.ml.clustering

Class LDA

Object
org.apache.spark.ml.PipelineStage
org.apache.spark.ml.Estimator<LDAModel>
org.apache.spark.ml.clustering.LDA
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, LDAParams, Params, HasCheckpointInterval, HasFeaturesCol, HasMaxIter, HasSeed, DefaultParamsWritable, Identifiable, MLWritable
public class LDA extends Estimator<LDAModel> implements LDAParams, DefaultParamsWritable
Latent Dirichlet Allocation (LDA), a topic model designed for text documents.

Terminology: - "term" = "word": an element of the vocabulary - "token": instance of a term appearing in a document - "topic": multinomial distribution over terms representing some concept - "document": one piece of text, corresponding to one row in the input data

Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.

Input data (featuresCol): LDA is given a collection of documents as input data, via the featuresCol parameter. Each document is specified as a Vector of length vocabSize, where each entry is the count for the corresponding term (word) in the document. Feature transformers such as Tokenizer and CountVectorizer can be useful for converting text to word count vectors.

See Also:

  • Constructor Details

    • LDA

      public LDA(String uid)

    • LDA

      public LDA()

  • Method Details

    • read

      public static MLReader<LDA> read()

    • load

      public static LDA load(String path)

    • k

      public final IntParam k()
      Description copied from interface: LDAParams
      Param for the number of topics (clusters) to infer. Must be &gt; 1. Default: 10.

      Specified by:
      k in interface LDAParams
      Returns:
      (undocumented)

    • docConcentration

      public final DoubleArrayParam docConcentration()
      Description copied from interface: LDAParams
      Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").

      This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).

      If not set by the user, then docConcentration is set automatically. If set to singleton vector [alpha], then alpha is replicated to a vector of length k in fitting. Otherwise, the LDAParams.docConcentration() vector must be length k. (default = automatic)

      Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.

      Specified by:
      docConcentration in interface LDAParams
      Returns:
      (undocumented)

    • topicConcentration

      public final DoubleParam topicConcentration()
      Description copied from interface: LDAParams
      Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.

      This is the parameter to a symmetric Dirichlet distribution.

      Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.

      If not set by the user, then topicConcentration is set automatically. (default = automatic)

      Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.

      Specified by:
      topicConcentration in interface LDAParams
      Returns:
      (undocumented)

    • supportedOptimizers

      public final String[] supportedOptimizers()
      Description copied from interface: LDAParams
      Supported values for Param LDAParams.optimizer().
      Specified by:
      supportedOptimizers in interface LDAParams

    • optimizer

      public final Param<String> optimizer()
      Description copied from interface: LDAParams
      Optimizer or inference algorithm used to estimate the LDA model. Currently supported (case-insensitive): - "online": Online Variational Bayes (default) - "em": Expectation-Maximization

      For details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here

      Specified by:
      optimizer in interface LDAParams
      Returns:
      (undocumented)

    • topicDistributionCol

      public final Param<