-
Notifications
You must be signed in to change notification settings - Fork 35
/
README.old
185 lines (155 loc) · 7.68 KB
/
README.old
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
README for Termite, a topic model visualization tool.
---------------
INFORMATION
---------------
Termite is a visualization tool for inspecting the output of statistical
topic models based on the techniques described in the following publication:
Termite: Visualization Techniques for Assessing Textual Topic Models
Jason Chuang, Christopher D. Manning, Jeffrey Heer
Computer Science Dept, Stanford University
http://vis.stanford.edu/papers/termite
This tool is developed by:
* Jason Chuang <[email protected]>
* Ashley Jin <[email protected]>
and is distributed under the BSD-3 license.
-----------
LICENSE
-----------
Copyright (c) 2013, Leland Stanford Junior University
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of the <organization> nor the
names of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
----------------------
ADDITIONAL CREDITS
----------------------
Termite requires the use of the following libraries and tools.
We thank their respective authors for developing and distributing the tools.
Mallet: Machine learning for language toolkit
Project website: http://mallet.cs.umass.edu
Developed by Andrew McCallum, et al.
Distributed under a CPL license: lib/LICENSE-mallet
STMT: Stanford topic modeling toolbox
Project website: http://nlp.stanford.edu/software/tmt
Developed by Daniel Ramage, et al.
Distributed under a GNU license: lib/LICENSE-stmt
D3 javascript visualization library
Project website: http://d3js.org
Developed by Mike Bostock, et al.
Distributed under a BSD license: lib/LICENSE-d3
Google closure javascript compiler
Project website: https://developers.google.com/closure/compiler/
Developed by Google engineers
Distributed under an Apache license: lib/LICENSE-closure-compiler
Backbone
Project website: http://backbonejs.org
Developed by Jeremy Ashkenas, DocumentCloud Inc.
Distributed under a MIT license: lib/LICENSE-backbone
Underscore
Project website: http://underscorejs.org
Developed by Jeremy Ashkenas, DocumentCloud Inc.
Distributed under a MIT license: lib/LICENSE-underscore
jQuery
Project website: http://jquery.com
Developed by the jQuery Foundation
Distributed under a MIT license: lib/LICENSE-jquery
html5slider
Project website: https://github.com/fryn/html5slider
Developed by Frank Yan
Distributed under an MIT license: http://opensource.org/licenses/MIT
------------------
ONE-TIME SETUP
------------------
Run the setup script to fetch the following tools, libraries, and prepare all javascript
files needed by Termite. This script only needs to be run once when Termite is first
downloaded onto a new machine.
>> ./setup.sh
Libraries fetched include:
* mallet
* stmt
* closure-compiler.js
* d3.v3.js
* jquery.js
* underscore.js
* backbone.js
* html5slider.js
The script creates a minified version of all javascript files.
-----------------------
BUILD A TOPIC MODEL
-----------------------
Building a topic model in Termite requires running a single python script. The script reads
in an input text corpus, and produces an output folder whose content can be inspected
using a web browser, described in the next section.
Customize configuration file with the following information. A sample configuration
file can be found in 'example.cfg'
[Corpus] path to text corpus
[TopicModel] directory for holding topic model outputs
[TopicModel] number of topics to train
[TopicModel] topic model (either mallet or stmt)
[Termite] number of terms to seriate
[Termite] path to save Termite-internal working files
Process the text corpus, and build a topic model by running the execution script.
Execution time will vary depending on the size of the corpus.
>> ./execute.py <your_config_file.cfg>
The execution script calls in order:
1. pipeline/tokenize.py Tokenize the text corpus
2. pipeline/train_mallet.py Train a topic model using MALLET
pipeline/train_stmt.py or STMT.
3. pipeline/compute_saliency.py Compute term saliency
4. pipeline/compute_similarity.py Compute term similarity
5. pipeline/compute_seriation.py: Seriates terms
6. pipeline/prepare_data_for_client.py: Generates datafiles for client
7. pipeline/prepare_vis_for_client.py: Copies necessary scripts for client
----------------------------
VIEW TOPIC MODEL OUTPUTS
----------------------------
You are now ready to visualize the topic model outputs! Termite's output can be viewed in
a web browser. To view the files locally (on your own computer), you need to set up a local
web server. Alternatively, you may copy the output folder to a web server to publish the results.
Termite outputs are stored in the 'public_html' subfolder within the output directory.
To set up a local webserver:
1. Change into output directory (specified in the configuration file)
>> cd <output_folder>/public_html
2. Start a local server using python
>> ./web.sh
3. Open http://localhost:8888 in a modern web browser (Chrome, Safari, Firefox, or Opera)
to view a visualization of the model outputs.
To publish the results on a webserver:
1. Copy public_html directory to your remote server.
-----------------------------
TOPIC MODEL VISUALIZATION
-----------------------------
The visualization should consist of a matrix view with a frequency bar view to the right.
The top right contains user controls, two slider bars and one input text box. User
controls are explained in more detail in the 'VISUALIZATION USE CASES' section.
When the mouse is placed over terms, topics, circles, or bars in the matrix or frequency
bar view, the associated term, topic, bar, and circles will be highlighted in both views.
---------------------------
VISUALIZATION USE CASES
---------------------------
The Termite visualization tool has a set of user controls in the top right section of the
webpage. The user may specify
* the number of terms with highest affinity to show
* the number of terms with highest saliency to show
* always display specific terms
* always display 20 most frequent terms belonging to selected topics
* click on a topic to select/color the topic
* mouse over a topic/term to highlight the topic/term
(Note: the highest affinity term set and highest saliency term set may contain overlapping words)