Update WEAT and RNSB documentation

doc/about.rst

@@ -190,8 +190,6 @@ query, word2vec embeddings and the WEAT metric.
 To see the implementation of this query using WEFE, refer to 
 the `Quick start <quick_start.html>`_ section.
 
-
-
 Metrics
 =======
 
@@ -200,15 +198,62 @@ The metrics implemented in the package so far are:
 WEAT
 ----
 
-Word Embedding Association Test (WEAT), presented in the paper "*Semantics*
-*derived automatically from language corpora contain human-like biases*".
-This metric receives two sets :math:`T_1` and :math:`T_2` of target words, 
-and two sets :math:`A_1` and :math:`A_2` of attribute words. Its objective is 
-to quantify the strength of association of both pairs of sets through a 
-permutation test. 
-It also contains a variant, WEAT Effect Size. This variant represents a 
-normalized measure that quantifies how far apart the two distributions of 
-association between targets and attributes are.
+Word Embedding Association Test (WEAT) was presented in the paper:
+
+  | Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 
+  | Semantics derived automatically from language corpora contain human-like biases.
+  | Science, 356(6334):183–186, 2017.
+
+The following description of the metric is WEFE's adaptation of what was presented 
+in the original WEAT work.
+
+WEAT receives two sets :math:`T_1` and :math:`T_2` of target words, 
+and two sets :math:`A_1` and :math:`A_2` of attribute words and performs a 
+hypothesis test on the following null hypothesis: 
+There is no difference between the two sets of target words in terms of their
+relative similarity to the similarity with the two sets of attribute words.
+
+In formal terms, let :math:`T_1` and :math:`T_2` be two sets of target words of 
+equal size, and :math:`A_1`, :math:`A_2` the two sets of attribute words.
+Let :math:`\cos(\vec{a},\vec{b})` denote the cosine of the angle between the vectors
+:math:`\vec{a}` and :math:`\vec{b}`. The test statistic is:
+
+.. math::
+
+  \text{WEAT}(T_1,T_2,A_1,A_2) = \sum_{x \in T_1} s(x, A_1, A_2) - \sum_{y \in T_2} s(y, A_1, A_2)
+
+where 
+
+.. math::
+
+  s(w, A, B)=\text{mean}_{a \in A} \cos(\vec{w}, \vec{a}) - \text{mean}
+  _{b \in B} \cos(\vec{w},\vec{b})
+
+:math:`s(w,A,B)` measures the association of :math:`w` with the
+attributes, and :math:`\text{WEAT}(T_1,T_2,A_1,A_2)` measures the differential association 
+of the two sets of target words with the attribute.
+
+This metric also contains a variant: WEAT Effect Size (WEAT-ES). This variant represents a 
+normalized measure that quantifies how far apart the two distributions of association 
+between targets and attributes are. Iin practical terms, WEAT Effect Size makes the 
+metric not dependent on the number of words used in each set.
+
+.. math::
+
+  \text{WEAT-ES}(T_1,T_2,A_1,A_2) = \frac{\text{mean}_{x \in T_1}\, s(x, A_1, A_2) - \text{mean}_{y \in T_2}\, s(y, A_1, A_2) }{\text{std-dev}_{w \in T_1 \cup T_2}\, s(w, A_1, A_2)} 
+
+
+
+The permutation test measures the (un)likelihood of the null hypothesis by
+computing the probability that a random permutation of the attribute words would
+produce the observed (or greater) difference in sample mean.
+
+Let :math:`{(T_{1_i},T_{2_i})}_{i}` denote all the partitions of :math:`T_1 \cup T_2` 
+into two sets of equal size. The one-sided p-value of the permutation test is:
+
+.. math::
+
+  \text{Pr}_{i}[s(T_{1_i}, T_{2_i}, A_1, A_2) > s(T_1, T_2, A_1, A_2)]
 
 RND
 ---
@@ -224,26 +269,59 @@ the more associated are the sets of attributes towards group two (one).
 RNSB
 ----
 
-Relative Negative Sentiment Bias (RNSB), presented in the paper "*A transparent* 
-*framework for evaluating unintended demographic bias in word embeddings*".
+Relative Negative Sentiment Bias (RNSB) was presented in the paper:
+
+  | Chris Sweeney and Maryam Najafian. A transparent framework for evaluating
+  | unintended demographic bias in word embeddings.
+  | In Proceedings of the 57th Annual Meeting of the Association for
+  | Computational Linguistics, pages 1662–1667, 2019.
+
+Originally this metric is based on measuring bias through word sentiment.
+The main idea is that if there were no bias, all words should be equally negative. 
+Therefore, its procedure is based on calculating how negative the words in the target
+sets are.
+
+For this purpose, RNSB trains a classifier that assigns a probability to each
+word of belonging to the negative class (in the original work the classifier is trained
+using `Bing Liu's lexicon <https://wefe.readthedocs.io/en/latest/generated/dataloaders/wefe.load_bingliu.html>`_
+of positive and negative words).
+Then, it generates a probability distribution with the probabilities calculated in
+the previous step and compares them to the uniform distribution
+(case where all words have the same probability of being negative) using KL divergence.
+When the negative probability distribution is equal to the uniform one (i.e., there
+is no bias), the KL divergence is 0.
+
+The following description of the metric is WEFE's adaptation of what was presented 
+in the original RNSB work.
 
 RNSB receives as input queries with two attribute sets :math:`A_1` and 
-:math:`A_2` and two or more target sets, and thus has a template of the 
-form :math:`s=(N,2)` with :math:`N\geq 2`.
-Given a query :math:`Q=(\{T_1,T_2,\ldots,T_n\},\{A_1,A_2\})` and an embedding 
-model :math:`\mathbf{M}`, in order to compute the metric 
-:math:`F_{\text{RNSB}}(\mathbf{M},Q)`  one first constructs a binary classifier 
-:math:`C_{(A_1,A_2)}(\cdot)` using set :math:`A_1` as training examples for the 
-negative class, and :math:`A_2` as training examples for the positive class. 
-After the training process, this classifier gives for every word :math:`w` a 
-probability :math:`C_{(A_1,A_2)}(w)` that can be interpreted as the degree of 
-association of :math:`w` with respect to  :math:`A_2` (value 
-:math:`1-C_{(A_1,A_2)}(w)` is the degree of association with :math:`A_1`).
-Now, we construct a probability distribution :math:`P(\cdot)` over all the words 
-:math:`w` in :math:`T_1\cup \cdots \cup T_n`, by computing :math:`C_{(A_1,A_2)}(w)` 
-and normalizing it to ensure that :math:`\sum_w P(w)=1`.
+:math:`A_2` and two or more target sets. Thus has a template (tuple of numbers that 
+defines the allowed target and attribute sets in the query) 
+of the form :math:`s=(N,2)` with :math:`N\geq 2`.
+
+Given a query :math:`Q=(\{T_1,T_2,\ldots,T_n\},\{A_1,A_2\})` RNSB is calculated under 
+the following steps::
+
+1. First constructs a binary classifier  :math:`C_{(A_1,A_2)}(\cdot)` using 
+   set :math:`A_1` as training examples for the negative class, and :math:`A_2` as 
+   training examples for the positive class. 
+
+2. After the training process, this classifier gives for every word :math:`w` a 
+   probability :math:`C_{(A_1,A_2)}(w)` that can be interpreted as the degree of 
+   association of :math:`w` with respect to  :math:`A_2` (value 
+   :math:`1-C_{(A_1,A_2)}(w)` is the degree of association with :math:`A_1`).
+
+3. Then, the metric construct a probability distribution :math:`P(\cdot)` over all 
+   the words :math:`w` in :math:`T_1\cup \cdots \cup T_n`, by computing 
+   :math:`C_{(A_1,A_2)}(w)` and normalizing it to ensure that :math:`\sum_w P(w)=1`.
+
+4. Finally RNSB is calculated as the distance between :math:`P(\cdot)` and
+   the uniform distribution :math:`Y(\cdot)` using the KL-divergence.
+
 The main idea behind RNSB is that the more that :math:`P(\cdot)` resembles a 
-uniform distribution, the less biased the word embedding model is.
+uniform distribution, the less biased the word embedding model is. 
+Thus, the optimal value is 0.
+
 
 MAC
 ---

wefe/metrics/RNSB.py

@@ -18,6 +18,10 @@
 class RNSB(BaseMetric):
     """Relative Relative Negative Sentiment Bias (RNSB).
 
+    The metric was originally proposed in [1].
+    Visit `RNSB in Metrics Section <https://wefe.readthedocs.io/en/latest/about.html#rnsb>`_
+    for further information.
+
     References
     ----------
     | [1]: Chris Sweeney and Maryam Najafian. A transparent framework for evaluating

wefe/metrics/WEAT.py

@@ -15,9 +15,8 @@ class WEAT(BaseMetric):
     """Word Embedding Association Test (WEAT).
 
     The metric was originally proposed in [1].
-    It measures the degree of association between two sets of target words and
-    two sets of attribute words through a permutation test.
-
+    Visit `WEAT in Metrics Section <https://wefe.readthedocs.io/en/latest/about.html#weat>`_
+    for further information.
 
     References
     ----------