# HOMEWORK 8

## Question 1

The following table depicts (invented) results from an annotation project. The annotators labeled 10 examples (the rows). There were five annotators (the columns): A, B, C, D, and E. The possible labels for each example were +1 (positive), 0 (neutral), and -1 (negative). The data in CSV format.

| A | B | C | D | E |

ex1 | 0 | 0 | +1 | 0 | 0 |

ex2 | -1 | +1 | -1 | +1 | -1 |

ex3 | +1 | -1 | -1 | +1 | +1 |

ex4 | 0 | +1 | 0 | +1 | +1 |

ex5 | +1 | 0 | +1 | +1 | -1 |

ex6 | -1 | -1 | -1 | -1 | -1 |

ex7 | -1 | 0 | -1 | -1 | -1 |

ex8 | -1 | 0 | -1 | -1 | -1 |

ex9 | +1 | +1 | +1 | +1 | +1 |

ex10 | -1 | +1 | -1 | -1 | +1 |

### Part A [2.5 points]

Some examples are simply more ambiguous than others. Hsueh et al. (2009) propose a method for identifying such cases using the annotation distributions. For this problem, we'll use a modification of their method that seems appropriate for our simpler annotation setting: the entropy of the response distributions. To use this measure for an example ex\(_{i}\), turn its annotation vector into a probability distribution \(P_{i}\) over the category set \(\{-1, 0, +1\}\) and calculate

\[
H(P_{i}) = -\left(\sum_{x \in \{-1, 0, +1\}} P_{i}(x)\log_{2}P_{i}(x)\right)
\]
**Your task**: use this measure to find the most ambiguous example in the data set. Give this example (by its id ex\(_{i}\)) and its associated entropy value.

### Part B [2.5 points]

There is a trouble-maker in our midst: one of our annotators seems to be significantly less reliable than the others. Find the annotator whose column-vector of annotations has the largest mean Euclidean distance from all of the other annotators (excluding self-distances). Provide the name of this annotator and his associated mean Euclidean distance from the others.

(Note: identifying the majority annotation for each example and checking differences from it is a good way to home in on our miscreant/oddball.)

(Note: it is worth considering other methods for making these comparisons, including correlation and measures of inter-annotator agreement. However, we're asking only that you use Euclidean distance.)

## Question 2

Domingos (2012) offers the memorable line “strong false assumptions can be better than weak true ones”.

### Part A [2.5 points]

What problem is Domingos addressing when he says this? Give (i) the name of the problem and (ii) a brief, informal definition of it.

### Part B [2.5 points]

Describe (1-2 sentences) two methods for addressing the problem. For each, give one reason why it is not a complete solution.

**Note**: In both parts A and B, we're looking for a few concise sentences in response to each question. No need to write a treatise.