Sunday, October 5, 2014

On reading/reviewing papers

After reading a paper two times, I consider it well-structured and well-written if I can clearly answer the following:

  • What is the problem/research question being addressed?
  • Why is the problem/research question significant or interesting?
  • What are existing solutions to the problem?
  • What is the proposed solution?
  • Is the proposed solution correct (and supported by proofs)?
  • What is the main contribution to the body of knowledge?
  • Is the contribution "new" and "significant"?
 I accept a paper based on the following: 
  1. The paper is "relevant" to the conference.
  2. The contribution is "significant".
  3. The work is not just a "repeat-experiment" or an application of existing techniques on a different data set. If it is a "repeat-experiment", the insight to be gained from the result is "substantial" and is extensively supported by proofs.
  4. Important claims are supported by citations or proofs.
  5. The context of the work is properly placed in relation to other work. Relevant papers are properly cited. I should be able to identify the one (or two) reference(s) which motivated the work. 
  6. The paper uses simple words. Too much jargon  makes a paper difficult to read and thus defeats the purpose of publication: effective communication of research results.
  7. The paper cites scholarly articles (journal articles, proceedings, technical reports), not just web sites. 
  8. I believe I can duplicate the experiments conducted and get the same results using the same data set, environment, and parameters as the authors' (if I have the resources and given enough time).
  9. The paper is in the proper format for the conference.
  10. Tables and figures can stand on their own and have useful captions.

The above criteria may seem too strict and rigid. However, I believe colleagues who review my work have even stricter criteria.

As a final note, I always review with the mindset of accepting the submission. In case I reject a paper, I see to it that my written review describes the things that will make me accept the paper. Also, my review is based on the paper "as submitted".