After reading a paper two times, I consider it well-structured and well-written if I can clearly answer the following:
- What is the problem/research question being addressed?
- Why is the problem/research question significant or interesting?
- What are existing solutions to the problem?
- What is the proposed solution?
- Is the proposed solution correct (and supported by proofs)?
- What is the main contribution to the body of knowledge?
- Is the contribution "new" and "significant"?
- The paper is "relevant" to the conference.
- The contribution is "significant".
- The work is not just a "repeat-experiment" or an application of existing techniques on a different data set. If it is a "repeat-experiment", the insight to be gained from the result is "substantial" and is extensively supported by proofs.
- Important claims are supported by citations or proofs.
- The context of the work is properly placed in relation to other work. Relevant papers are properly cited. I should be able to identify the one (or two) reference(s) which motivated the work.
- The paper uses simple words. Too much jargon makes a paper difficult to read and thus defeats the purpose of publication: effective communication of research results.
- The paper cites scholarly articles (journal articles, proceedings, technical reports), not just web sites.
- I believe I can duplicate the experiments conducted and get the same results using the same data set, environment, and parameters as the authors' (if I have the resources and given enough time).
- The paper is in the proper format for the conference.
- Tables and figures can stand on their own and have useful captions.
The above criteria may seem too strict and rigid. However, I believe colleagues who review my work have even stricter criteria.
As a final note, I always review with the mindset of accepting the submission. In case I reject a paper, I see to it that my written review describes the things that will make me accept the paper. Also, my review is based on the paper "as submitted".