Why co-occurrence information alone is not sufficient to answer subcognitive questions

Turney (2001) claims that a simple program, PMI-IR, that searches the World Wide Web for co-occurrences of words in 350 million Web pages can be used to find human-like answers to the type of ‘subcognitive’ questions French (1990) claimed would invariably unmask computers (that had not lived life as we humans had) in a Turing Test. This paper shows that there are serious problems with Turney’s claim. We show that PMI-IR does not work for even simple subcognitive questions. PMI-IR’s failure is attributed to its inability to understand the relational and contextual attributes of the words/concepts in the queries. Finally, it is shown that, even if PMI-IR were able to answer many subcognitive questions, a clever interrogator in the Turing Test would still be able to unmask the computer.