FASTopic is a neural topic model based on Dual Semantic-relation Reconstruction.
+
+
Turftopic contains an implementation repurposed for our API, but the implementation is mostly from the original FASTopic package.
+
+
This part of the documentation is still under construction
+
References
+
Wu, X., Nguyen, T., Zhang, D. C., Wang, W. Y., & Luu, A. T. (2024). FASTopic: A Fast, Adaptive, Stable, and Transferable Topic Modeling Paradigm. ArXiv Preprint ArXiv:2405.17978.
+
API Reference
+
+
+
+
+
+
+
+turftopic.models.fastopic.FASTopic
+
+
+
+
+
+
+
+ Bases: ContextualModel
+
+
+
Implementation of the FASTopic model with a Turftopic API.
+The implementation is based on the original FASTopic package,
+but is adapted for optimal use in Turftopic (you can pre-compute embeddings for instance).
GMM is a generative probabilistic model over the contextual embeddings.
+The model assumes that contextual embeddings are generated from a mixture of underlying Gaussian components.
+These Gaussian components are assumed to be the topics.
+
+
+ Components of a Gaussian Mixture Model (figure from scikit-learn documentation)
+
+
+
The Model
+
1. Generative Modeling
+
GMM assumes that the embeddings are generated according to the following stochastic process:
+
+
Select global topic weights: \(\Theta\)
+
For each component select mean \(\mu_z\) and covariance matrix \(\Sigma_z\) .
Priors are optionally imposed on the model parameters.
+The model is fitted either using expectation maximization or variational inference.
+
2. Topic Inference over Documents
+
After the model is fitted, soft topic labels are inferred for each document.
+A document-topic-matrix (\(T\)) is built from the likelihoods of each component given the document encodings.
+
Or in other words for document \(i\) and topic \(z\) the matrix entry will be: \(T_{iz} = p(\rho_i|\mu_z, \Sigma_z)\)
+
3. Soft c-TF-IDF
+
Term importances for the discovered Gaussian components are estimated post-hoc using a technique called Soft c-TF-IDF,
+an extension of c-TF-IDF, that can be used with continuous labels.
+
Let \(X\) be the document term matrix where each element (\(X_{ij}\)) corresponds with the number of times word \(j\) occurs in a document \(i\).
+Soft Class-based tf-idf scores for terms in a topic are then calculated in the following manner:
+
+
Estimate weight of term \(j\) for topic \(z\):
+\(tf_{zj} = \frac{t_{zj}}{w_z}\), where
+\(t_{zj} = \sum_i T_{iz} \cdot X_{ij}\) and
+\(w_{z}= \sum_i(|T_{iz}| \cdot \sum_j X_{ij})\)
+
Estimate inverse document/topic frequency for term \(j\):
+\(idf_j = log(\frac{N}{\sum_z |t_{zj}|})\), where
+\(N\) is the total number of documents.
+
Calculate importance of term \(j\) for topic \(z\):
+\(Soft-c-TF-IDF{zj} = tf_{zj} \cdot idf_j\)
+
+
(Optional) 4. Dynamic Modeling
+
GMM is also capable of dynamic topic modeling. This happens by fitting one underlying mixture model over the entire corpus, as we expect that there is only one semantic model generating the documents.
+To gain temporal representations for topics, the corpus is divided into equal, or arbitrarily chosen time slices, and then term importances are estimated using Soft-c-TF-IDF for each of the time slices separately.
+
Similarities with Clustering Models
+
Gaussian Mixtures can in some sense be considered a fuzzy clustering model.
+
Since we assume the existence of a ground truth label for each document, the model technically cannot capture multiple topics in a document,
+only uncertainty around the topic label.
+
This makes GMM better at accounting for documents which are the intersection of two or more semantically close topics.
+
Another important distinction is that clustering topic models are typically transductive, while GMM is inductive.
+This means that in the case of GMM we are inferring some underlying semantic structure, from which the different documents are generated,
+instead of just describing the corpus at hand.
+In practical terms this means that GMM can, by default infer topic labels for documents, while (some) clustering models cannot.
+
Performance Tips
+
GMM can be a bit tedious to run at scale. This is due to the fact, that the dimensionality of parameter space increases drastically with the number of mixture components, and with embedding dimensionality.
+To counteract this issue, you can use dimensionality reduction. We recommend that you use PCA, as it is a linear and interpretable method, and it can function efficiently at scale.
+
+
Through experimentation on the 20Newsgroups dataset I found that with 20 mixture components and embeddings from the all-MiniLM-L6-v2 embedding model
+ reducing the dimensionality of the embeddings to 20 with PCA resulted in no performance decrease, but ran multiple times faster.
+ Needless to say this difference increases with the number of topics, embedding and corpus size.
+
+
fromturftopicimportGMM
+fromsklearn.decompositionimportPCA
+
+model=GMM(20,dimensionality_reduction=PCA(20))
+
+# for very large corpora you can also use Incremental PCA with minibatches
+
+fromsklearn.decompositionimportIncrementalPCA
+
+model=GMM(20,dimensionality_reduction=IncrementalPCA(20))
+
+
Considerations
+
Strengths
+
+
Efficiency, Stability: GMM relies on a rock solid implementation in scikit-learn, you can rest assured that the model will be fast and reliable.
+
Coverage of Ingroup Variance: The model is very efficient at describing the extracted topics in all their detail.
+ This means that the topic descriptions will typically cover most of the documents generated from the topic fairly well.
+
Uncertainty: GMM is capable of expressing and modeling uncertainty around topic labels for documents.
+
Dynamic Modeling: You can model changes in topics over time using GMM.
+
+
Weaknesses
+
+
Curse of Dimensionality: The dimensionality of embeddings can vary wildly from model to model. High-dimensional embeddings might decrease the efficiency and performance of GMM, as it is sensitive to the curse of dimensionality. Dimensionality reduction can help mitigate these issues.
+
Assumption of Gaussianity: The model assumes that topics are Gaussian components, it might very well be that this is not the case.
+ Fortunately enough this rarely effects real-world perceived performance of models, and typically does not present an issue in practical settings.
+
Moderate Scalability: While the model is scalable to a certain extent, it is not nearly as scalable as some of the other options. If you experience issues with computational efficiency or convergence, try another model.
+
Moderate Robustness to Noise: GMM is similarly sensitive to noise and stop words as BERTopic, and can sometimes find noise components. Our experience indicates that GMM is way less volatile, and the quality of the results is more reliable than with clustering models using C-TF-IDF.
+
+
API Reference
+
+
+
+
+
+
+
+turftopic.models.gmm.GMM
+
+
+
+
+
+
+
+ Bases: ContextualModel, DynamicTopicModel
+
+
+
Multivariate Gaussian Mixture Model over document embeddings.
+Models topics as mixture components.
Prior to impose on component weights, if None,
+maximum likelihood is optimized with expectation maximization,
+otherwise variational inference is used.
+
+
+
+ None
+
+
+
+
+ gamma
+
+
+ Optional[float]
+
+
+
+
Concentration parameter of the symmetric prior.
+By default 1/n_components is used.
+Ignored when weight_prior is None.
+
+
+
+ None
+
+
+
+
+ dimensionality_reduction
+
+
+ Optional[TransformerMixin]
+
+
+
+
Optional dimensionality reduction step before GMM is run.
+This is recommended for very large datasets with high dimensionality,
+as the number of parameters grows vast in the model otherwise.
+We recommend using PCA, as it is a linear solution, and will likely
+result in Gaussian components.
+For even larger datasets you can use IncrementalPCA to reduce
+memory load.
+
+
+
+ None
+
+
+
+
+ random_state
+
+
+ Optional[int]
+
+
+
+
Random state to use so that results are exactly reproducible.
KeyNMF is a topic model that relies on contextually sensitive embeddings for keyword retrieval and term importance estimation,
+while taking inspiration from classical matrix-decomposition approaches for extracting topics.
+
+
+ Schematic overview of KeyNMF
+
+
+
Here's an example of how you can fit and interpret a KeyNMF model in the easiest way.
The first step of the process is gaining enhanced representations of documents by using contextual embeddings.
+Both the documents and the vocabulary get encoded with the same sentence encoder.
+Keywords are assigned to each document based on the cosine similarity of the document embedding to the embedded words in the document.
+Only the top K words with positive cosine similarity to the document are kept.
+These keywords are then arranged into a document-term importance matrix where each column represents a keyword that was encountered in at least one document,
+and each row is a document. The entries in the matrix are the cosine similarities of the given keyword to the document in semantic space.
+
+
+
For each document \(d\):
+
+
Let \(x_d\) be the document's embedding produced with the encoder model.
+
+
For each word \(w\) in the document \(d\):
+
+
Let \(v_w\) be the word's embedding produced with the encoder model.
+
Calculate cosine similarity between word and document
You can do this step manually if you want to precompute the keyword matrix.
+Keywords are represented as dictionaries mapping words to keyword importances.
+
model.extract_keywords(["Cars are perhaps the most important invention of the last couple of centuries. They have revolutionized transportation in many ways."])
+
Topics in this matrix are then discovered using Non-negative Matrix Factorization.
+Essentially the model tries to discover underlying dimensions/factors along which most of the variance in term importance
+can be explained.
+
+
+
Decompose \(M\) with non-negative matrix factorization: \(M \approx WH\), where \(W\) is the document-topic matrix, and \(H\) is the topic-term matrix. Non-negative Matrix Factorization is done with the coordinate-descent algorithm, minimizing square loss:
+
\[
+L(W,H) = ||M - WH||^2
+\]
+
+
+
You can fit KeyNMF on the raw corpus, with precomputed embeddings or with precomputed keywords.
+
# Fitting just on the corpus
+model.fit(corpus)
+
+# Fitting with precomputed embeddings
+fromsentence_transformersimportSentenceTransformer
+
+trf=SentenceTransformer("all-MiniLM-L6-v2")
+embeddings=trf.encode(corpus)
+
+model=KeyNMF(10,encoder=trf)
+model.fit(corpus,embeddings=embeddings)
+
+# Fitting with precomputed keyword matrix
+keyword_matrix=model.extract_keywords(corpus)
+model.fit(None,keywords=keyword_matrix)
+
+
Asymmetric and Instruction-tuned Embedding Models
+
Some embedding models can be used together with prompting, or encode queries and passages differently.
+This is important for KeyNMF, as it is explicitly based on keyword retrieval, and its performance can be substantially enhanced by using asymmetric or prompted embeddings.
+Microsoft's E5 models are, for instance, all prompted by default, and it would be detrimental to performance not to do so yourself.
+
In these cases, you're better off NOT passing a string to Turftopic models, but explicitly loading the model using sentence-transformers.
+
Here's an example of using instruct models for keyword retrieval with KeyNMF.
+In this case, documents will serve as the queries and words as the passages:
+
fromturftopicimportKeyNMF
+fromsentence_transformersimportSentenceTransformer
+
+encoder=SentenceTransformer(
+ "intfloat/multilingual-e5-large-instruct",
+ prompts={
+ "query":"Instruct: Retrieve relevant keywords from the given document. Query: "
+ "passage":"Passage: "
+ },
+ # Make sure to set default prompt to query!
+ default_prompt_name="query",
+)
+model=KeyNMF(10,encoder=encoder)
+
+
And a regular, asymmetric example:
+
encoder=SentenceTransformer(
+ "intfloat/e5-large-v2",
+ prompts={
+ "query":"query: "
+ "passage":"passage: "
+ },
+ # Make sure to set default prompt to query!
+ default_prompt_name="query",
+)
+model=KeyNMF(10,encoder=encoder)
+
+
Setting the default prompt to query is especially important, when you are precomputing embeddings, as query should always be your default prompt to embed documents with.
+
Dynamic Topic Modeling
+
KeyNMF is also capable of modeling topics over time.
+This happens by fitting a KeyNMF model first on the entire corpus, then
+fitting individual topic-term matrices using coordinate descent based on the document-topic and document-term matrices in the given time slices.
+
+
Compute keyword matrix \(M\) for the whole corpus.
+
Decompose \(M\) with non-negative matrix factorization: \(M \approx WH\).
+
+
For each time slice \(t\):
+
+
Let \(W_t\) be the document-topic proportions for documents in time slice \(t\), and \(M_t\) be the keyword matrix for words in time slice \(t\).
+
Obtain the topic-term matrix for the time slice, by minimizing square loss using coordinate descent and fixing \(W_t\):
You can also display the topics over time on an interactive HTML figure.
+The most important words for topics get revealed by hovering over them.
+
+
You will need to install Plotly for this to work.
+
+
pipinstallplotly
+
+
model.plot_topics_over_time(top_k=5)
+
+
+
+ Topics over time on a Figure
+
+
+
Online Topic Modeling
+
KeyNMF can also be fitted in an online manner.
+This is done by fitting NMF with batches of data instead of the whole dataset at once.
+
Use Cases:
+
+
You can use online fitting when you have very large corpora at hand, and it would be impractical to fit a model on it at once.
+
You have new data flowing in constantly, and need a model that can morph the topics based on the incoming data. You can also do this in a dynamic fashion.
+
You need to finetune an already fitted topic model to novel data.
+
+
Batch Fitting
+
We will use the batching function from the itertools recipes to produce batches.
+
+
In newer versions of Python (>=3.12) you can just from itertools import batched
+
+
defbatched(iterable,n:int):
+ "Batch data into lists of length n. The last batch may be shorter."
+ ifn<1:
+ raiseValueError("n must be at least one")
+ it=iter(iterable)
+ whilebatch:=tuple(itertools.islice(it,n)):
+ yieldbatch
+
+
You can fit a KeyNMF model to a very large corpus in batches like so:
This is mildly inefficient, however, as the texts need to be encoded on every epoch, and keywords need to be extracted.
+In such scenarios you might want to precompute and maybe even save the extracted keywords to disk using the extract_keywords() method.
+
Keywords are represented as dictionaries mapping words to keyword importances.
+
model.extract_keywords(["Cars are perhaps the most important invention of the last couple of centuries. They have revolutionized transportation in many ways."])
+
You can extract keywords in batches and save them to disk to a file format of your choice.
+In this example I will use NDJSON because of its simplicity.
+
importjson
+frompathlibimportPath
+fromtypingimportIterable
+
+# Here we are saving keywords to a JSONL/NDJSON file
+withPath("keywords.jsonl").open("w")askeyword_file:
+ # Doing this in batches is much more efficient than individual texts because
+ # of the encoding.
+ forbatchinbatched(corpus,200):
+ batch_keywords=model.extract_keywords(batch)
+ # We serialize each
+ forkeywordsinbatch_keywords:
+ keyword_file.write(json.dumps(keywords)+"\n")
+
+defstream_keywords()->Iterable[dict[str,float]]:
+"""This function streams keywords from the file."""
+ withPath("keywords.jsonl").open()askeyword_file:
+ forlineinkeyword_file:
+ yieldjson.loads(line.strip())
+
+forepochinrange(5):
+ keyword_stream=stream_keywords()
+ forkeyword_batchinbatched(keyword_stream,200):
+ model.partial_fit(keywords=keyword_batch)
+
+
Dynamic Online Topic Modeling
+
KeyNMF can be online fitted in a dynamic manner as well.
+This is useful when you have large corpora of text over time, or when you want to fit the model on future information flowing in
+and want to analyze the topics' changes over time.
+
When using dynamic online topic modeling you have to predefine the time bins that you will use, as the model can't infer these from the data.
+
fromdatetimeimportdatetime
+
+# We will bin by years in a period of 2020-2030
+bins=[datetime(year=y,month=1,day=1)foryinrange(2020,2030+2,1)]
+
+
You can then online fit a dynamic topic model with partial_fit_dynamic().
When you suspect that subtopics might be present in the topics you find with the model, KeyNMF can be used to discover topics further down the hierarchy.
+
This is done by utilising a special case of weighted NMF, where documents are weighted by how high they score on the parent topic.
+In other words:
+
+
Decompose keyword matrix \(M \approx WH\)
+
To find subtopics in topic \(j\), define document weights \(w\) as the \(j\)th column of \(W\).
+
Estimate subcomponents with wNMF\(M \approx \mathring{W} \mathring{H}\) with document weight \(w\)
+
Initialise \(\mathring{H}\) and \(\mathring{W}\) randomly.
To sufficiently differentiate the subcomponents from each other a pseudo-c-tf-idf weighting scheme is applied to \(\mathring{H}\):
+
\(\mathring{H} = \mathring{H}_{ij} \odot ln(1 + \frac{A}{1+\sum_k \mathring{H}_{kj}})\), where \(A\) is the average of all elements in \(\mathring{H}\)
+
+
+
+
To create a hierarchical model, you can use the hierarchy property of the model.
+
# This divides each of the topics in the model to 3 subtopics.
+model.hierarchy.divide_children(n_subtopics=3)
+print(model.hierarchy)
+
For a detailed tutorial on hierarchical modeling click here.
+
Considerations
+
Strengths
+
+
Stability, Robustness and Quality: KeyNMF extracts very clean topics even when a lot of noise is present in the corpus, and the model's performance remains relatively stable across domains.
+
Scalability: The model can be fitted in an online fashion, and we recommend that you choose KeyNMF when the number of documents is large (over 100 000).
+
Fail Safe and Adjustable: Since the modelling process consists of multiple easily separable steps it is easy to repeat one if something goes wrong. This also makes it an ideal choice for production usage.
+
Can capture multiple topics in a document.
+
+
Weaknesses
+
+
Lack of Nuance: Since only the top K keywords are considered and used for topic extraction some of the nuances, especially in long texts might get lost. We therefore recommend that you scale K with the average length of the texts you're working with. For tweets it might be worth it to scale it down to 5, while with longer documents, a larger number (let's say 50) might be advisable.
+
Practitioners have to choose the number of topics a priori.
+
+
API Reference
+
+
+
+
+
+
+
+turftopic.models.keynmf.KeyNMF
+
+
+
+
+
+
+
+ Bases: ContextualModel, DynamicTopicModel
+
+
+
Extracts keywords from documents based on semantic similarity of
+term encodings to document encodings.
+Topics are then extracted with non-negative matrix factorization from
+keywords' proximity to documents.
This part of the documentation is still under construction 
+ 
+ 
+