On the use of an intermediate class in boolean crowdsourced relevance annotations for learning to rank comments

In many Information Retrieval tasks the boundary between classes is not well defined and assigning a document to a specific class may be complicated, even for humans. For instance, a document which is not directly related to the user's query may still contain relevant information. In this scenario, an option is to define an intermediate class collecting ambiguous instances. Yet some natural questions arise. Is this annotation strategy convenient? How should the intermediate class be treated? To answer these questions, we explored two community question answering datasets whose commentswere originally annotated with three classes and re-Annotated a subset of instances considering a binary good vs bad setting. Our main contribution is to show empirically that the inclusion of an intermediate class to assess Boolean relevance is not useful. Moreover, in case the data is already annotated with a 3-class strategy, the instances from the intermediate class can be safely removed at training time.