After many months of conversations with members of our community regarding the health of the reviewing process in security and privacy conferences we think that it would be a good idea to repeat the famous NIPS consistency experiment.
For those who are not familiar, the NIPS experiment was carried out in 2014 at the NIPS conference with the goal of quantifying randomness in the review process. They split the program committee into two independent program committees. Then, 90% of the papers were assigned to either one of the PCs, and 10% were reviewed by both. This enabled them to observe how consistent the PCs were. The results were (not?) surprising: of the 166 papers, the PCs disagreed on the decision in ~25%.
We feel that PoPETs is in a great position to repeat this experiment and get insights about the randomness within reviews in the security domain.
We will organize the experiment as follows:
- We will split the PC in two: PC-A and PC-B, making sure (manually) that both of PCs contain representative expertise in the topics relevant to PoPETS.
- When the papers arrive we will select 20 papers that will be reviewed by both PCs, and the remainder will be assigned to one of the two committees uniformly at random.
- We will duplicate the HotCRP instance, and distribute the papers accordingly. The members of PC-A will be assigned to PoPETS2019 and the members of PC-B will be assigned to PoPETS2019-B.
- Members of each PC will not know whether a paper they review is also being reviewed by the other PC or not.
- Both PCs will run as usual, with the same phases and deadlines.
- We trust you to not transmit information from one PC to the other. (The only case when information transfer might be desired is when one PC discovers a fundamental technical flaw/attack on a paper; such cases will be handled by the Chairs.)
- At the end of the decision phase, we will take the best of the two decisions for the paper (i.e. always benefit the authors). This is important to avoid making the authors feeling uneasy with the experiment.
- In case of Major Revision:
- If only one PC decided MR, when the paper returns it will be assigned to single review by the same set of reviewers that issued the MR decision.
- If both PCs decided MR, the Chairs will merge the meta-review in agreement with the reviewers. When the paper returns it will be assigned to single review by one of the two sets of reviewers, chosen at random.
To ensure that the duplication of papers does not impose a high load on the PC, we have composed a larger PC than in previous years. Also, if the number of submissions grows too much we will suspend the experiment or reduce the number of papers reviewed by both committees.
Authors will be informed of the experiment upon submission so that they can withdraw if desired. They will be aware of whether their paper will get two sets of reviews or not (as they will have to do the rebuttal twice anyway).
We hope that you find this experiment as exciting as we do. Please do tell us your feedback and questions. We want to make this a great experience for everyone.
Looking forward to a successful PoPETs 2019!
Kostas and Carmela