Please email your solution to [email protected] with the subject header Exercise 2, by 15:00, November 16, 2016.
The following tasks come from the book Statistical Machine Translation by Philipp Koehn.
If we flip a coin 10 times, we might get the outcome HTTHTHTHTT (where H is heads, and T is tails).
- Estimate a distribution by maximum likelihood estimation for this event. Estimating a distribution in this case would mean reporting probabilty of two disjoint events; first p(observing a heads) and second p(observing a tails). (3 points)
- We want to test the quality of the estimation. We flip the coin five times and get
HHTTH. What is the probability of this outcome according to - the estimated distribution, and (1 point)
- the uniform distribution or, said another way, assume that the coin is unbiased? (1 point)
- What is the entropy of a coin toss where the coin has a head on each side (fake coin)? (2 points)
- Show that p(y|x) = p(y), if X and Y are independent. Explain each of your steps. (3 points)
- What is the difference between entropy and cross entropy? (2 points)