Familia is an open-source project, which implements three popular topic models based on the large-scale industrial data. They are Latent Dirichlet Allocation(LDA)、SentenceLDA and Topical Word Embedding(TWE). In addition, Familia offers several tools including lda-infer and lda-query-doc-sim. Familia could be easily applied to many tasks, such as document classification, document clustering and personalized recommendation. Due to the high cost of model training, we will continue to release well-trained topic models based on the various types of large-scale data.
The details of topic models implemented by Familia can be referred to papers on topic models.
Generally, the applications adopting topic models are categorized into two parts: Semantic Representation and Semantic Matching.
-
Semantic Representation
Topic models are able to mine hidden dimensions (topics) from document collection and generate semantic representations of documents. These generated semantic representations can be used as features for document classification, document content analysis, and CTR prediction.
-
Semantic Matching
We offer two methods to compute semantic similarity between documents:
- Semantic similarity between short-long documents, which can be applied to keyword extraction and computing query-document semantic similarity.
- Semantic similarity between long-long documents, which can be applied to computing semantic similarity between user profile and news article.
More details can be referred to Familia Wiki.
The required third parties include gflags-2.0，glogs-0.3.4，protobuf-2.5.0. The complier should support C++11, g++ >= 4.8 and be compatible with linux and mac. The deps could be obtained and installed automatically by running the following script.
$ sh build.sh
$ cd model
$ sh download_model.sh
More details can be referred to Models.
Familia demo includes the following functions:
-
Semantic Representation utilize topic models to infer the topic distribution of the input document.
-
Semantic Matching compute semantic similarity between short-long or long-long documents.
-
Topic Show demonstrate top words under each topic for users’ better understanding.
More details can be referred to Demos.
-
If libglog.so, libgflags.so and other dynamic libraries could not be found, please add third_party to the environmental parameter
LD_LIBRARY_PATH.export LD_LIBRARY_PATH=./third_party/lib:$LD_LIBRARY_PATH
{familia} at baidu.com
The following article describes the Familia project and industrial cases powered by topic modeling. It bundles and translates the Chinese documentation of the website. We recommend citing this article as default.
Di Jiang, Yuanfeng Song, Rongzhong Lian, Siqi Bao, Jinhua Peng, Huang He and Hua Wu. 2018. Familia: A Configurable Topic Modeling Framework for Industrial Text Engineering. arXiv preprint arXiv:1808.03733.
@article{jiang2018familia,
author = {Di Jiang and Yuanfeng Song and Rongzhong Lian and Siqi Bao and Jinhua Peng and Huang He and Hua Wu},
title = {{Familia: A Configurable Topic Modeling Framework for Industrial Text Engineering}},
journal = {arXiv preprint arXiv:1808.03733},
year = {2018}
}
Familia is provided under the BSD-3-Clause License.