Reviewing the Reviewers

Abstract

Listen

Translate article

I just returned from the Association for Computational Linguistics’ 43rd Annual Meeting (ACL-2005). The acceptance rate was 18%. Is this a good thing or a bad thing? When the acceptance rate is low, precision tends to be high. The audience can judge precision for itself. If the presentations are good, everyone knows it. And if they aren’t, they know that as well. ACL-2005 had great precision. Recall is more subtle. When there is an issue with recall, it isn’t immediately obvious to everyone. If you listen closely, you’ll hear some grumbling in the halls. And then the rejects start to appear elsewhere. ACL’s low recall has been great for other conferences. The best of the rejects are very good, better than most of the accepted papers, and often strong contenders for the best paper award at EMNLP. I used to be surprised by the quality of these rejects, but after seeing so many great rejects over so many years, I am no longer surprised by anything. The practice of setting EMNLP’s submission date immediately after ACL’s notification date is a not-so-subtle hint: Please do something about the low recall. When you read some of the ACL reviews for these top EMNLP papers, you realize what is happening. ACL reviewing is paying too much attention to abstentions (and objections from people outside the area). If a reviewer isn’t qualified to say anything on a particular topic, that’s okay. An abstention shouldn’t kill a paper. Controversial papers are great; boring unobjectionable incremental papers are not. The only bad paper is a paper without an advocate. A paper with a single advocate should trump a paper with lots of seconds, but no advocates. Don’t average votes. The key votes are the advocates. Negative votes matter only if they convince the advocates to change their votes. Recall is a problem for many conferences, not just ACL; SIGIR, for example, rejected the classic paper on page rank, a hugely successful paper in terms of citations, perhaps more successful than anything SIGIR ever published.