Personal Views on Programme Committees

Having served on many programme committees, been chair of two conferences and listening to other peoples experiences I have come up with the following notes.

Please ignore these, if you want, they are just my opinions.

Selecting a Committee

Keep the balance as wide geographically as possible. Different parts of the world often emphasize different aspects of a subject, hence keeping a geographical balance helps.

For example in Cryptography it is more common for stream and block cipher design to be done in Europe, whilst work on modes-of-operation etc is more likely to be done in the US. Cryptanalysis is less likely to come from the North America, where as theoretical work is more likely to come from North America.

Age should also be balanced. In my view younger PC members have more time and do a more thorough job than older PC members. However, they are also more aggressive about rejecting papers which are not quite up to scratch. They also don't take the long-view as much as older members, either accepting papers which are technically brilliant but which will have no impact, or rejecting papers which will have huge impact but which lack the technical depth of others.

There is a surprising number of PC members who take the mickey out of the entire process, either handing in non-existent or ridiculously short reviews. Some hand in scores, with no justifications. One would get the sack if one did this for students, but apparently it is acceptable for reviewing papers! These are unlikely to be younger reviewers. So I suggest before inviting someone on a committee who you have not seen operate on a committee, that you ask a PC chair who has asked them before to comment on whether they contributed.

Conflicts of Interest

Clearly PC members should not have access to the reviews or comments related to their own papers. Many programme committee's also insist that they also do not have access to the papers from their own institution, which is also perfectly sensible.

A novel, and to my mind very good innovation, made by Antoine Joux at EuroCrypt 2009, was the following. No PC member should have access to the reviews and comments of any paper by an author with which the PC member has published multiple papers. This is quite easy to create such a conflict list for a PC member, for instance using to the table of co-authors on DBLP.

Why is this a good idea? There is a tendency to support papers from your friends. In some committees this is done quite blatantly by certain cliques, bringing the whole procedure into disrepute. Since one is more likely in the Internet age to have a virtual lab with people in other countries, as opposed to a real lab with people in your own building, excluding common co-authors from reviewing each others papers makes a lot of sense.

Blind or Double-Blind Reviewing?

By blind reviewing I mean the paper is anonymous, by double-blind I mean that the PC does not know who is reviewing different papers. Usually in Crypto we adopt blind reviewing, only a few conferences have been double-blind and this was a disaster. Double-blind did not work as one could not judge whether a fellow PC's members comments were due to expertise, incompetance, or known-bias.

There are many arguments against blind reviewing which essentially go along the following lines

However, I feel this is a perfect argument for blind reviewing. If knowledge of an author inclines one to accept or reject with no basis on the actual quality of the work, then clearly marking the paper anonymous is a must. A PC is judging the current papers, and not the papers from twenty years ago after all.

Blind reviewing is now policy for all IACR conferences, but other conferences adopt either a non-blind policy or a mixed policy (where you can choose to be anonymous).

Having a PC Meeting

I had a PC meeting when I was chair of EuroCrypt 2008. It was really nice having a day discussing papers and meeting people with different interests and opinions. It was great fun. But I would not do it again!

Why? Firstly, it is hard as a chair to make sure that a balanced programme is produced when you are simultaneously chairing a meeting. Secondly, some people are more eloquent than others at supporting or not-supporting papers. Thirdly, it is never the case that everyone can attend, so some papers may be treated unfairly by not having their champion present.

Again Antoine Joux's EuroCrypt 2009 is a good case study. There was no PC meeting, and the result was a relatively well balanced programme. A fact which was commented upon by almost all attendees which I spoke to in Cologne.

Nigel Smart
August 2009.