Running a program committee meeting

by Michael Ernst

January, 2006

Contents:

(Also consider reading my advice on organizing a workshop.)

I have been a member of a number of program committees for technical conferences — some successful and some less so. Running a program committee is a heavy responsibility with important consequences. This webpage notes just a few things that I have seen program committee chairs (including myself) doing well or poorly; it is not comprehensive. Additions or links to additional resources are welcome.

Thanks to Andreas Zeller and Sebastian Elbaum for their comments and suggestions.

Choosing committee members

You may need to invite to the PC someone you don't know (someone you have not seen in action at a PC). Get others' opinions of such people before issuing an invitation. It's best to do this by phone, not by email: many people prefer not to communicate negative opinions in writing, and you can learn a lot from tone of voice. If you do ask opinions by email, perhaps ask an open-ended question instead of specific ones (such as “will this person submit reviews on time?”).

When inviting PC members, start early. Otherwise, potential PC members will already have committed to other responsibilities.

Naturally, you will want to achieve coverage of all areas of your field by inviting experts from a variety of sub-fields. Consider making rules for invitation to the PC, such as:

Some conferences use a model of “large PC, light load per member”. This model does not produce the best outcomes for our community. For example, it reduces the likelihood of well-calibrated, fair results (and it thereby reduces PC member engagement). This is one reason that you see it most at second-tier (and worse!) venues. People who care more about improving our field than about padding their cv would rather have a higher load per PC, and be on fewer PCs.

Meeting: physical, phone, or electronic

A face-to-face meeting is the most effective way for the program committee to make decisions. It is also a good way for top researchers to meet one another (and to socialize). However, travel to a common location is costly in time, money, and environmental impact. It can be hard to schedule a time and place that everyone can attend. One PC chair told me that his meeting produced only two surprises — that he could have predicted all the other outcomes from the scores — and others think that even if the physical meeting produces a different program, it isn't necessarily better or worse. To encourage attendance, many conferences remove from the PC anyone who does not attend the physical meeting. A number of conferences schedule a workshop the day before or after the PC meeting, at which PC members can present their work to an influential and highly-qualified audience.

A phone conference call works well for workshops, where there are not so many papers to discuss and the stakes (and the standards) are relatively low. Even a 2-day workshop with 40 papers can be decided in a 2-hour phone meeting.

I am not a fan of electronic committee meetings conducted via bulletin-board-like “chat” servers. The bandwidth is much less than for phone (and phone bandwidth is much less than for face-to-face). In my experience, they cause people to become more tuned out of the discussion, and they tempt uncooperative committee members to prepare long paragraphs to paste in, when those members should be carefully considering what others have to say. They permit members to leave in the middle. On the other hand, electronic meetings (possibly combined with phone) save a lot of travel time and money (easily tens of thousands of dollars). Their proponents note that the effects of any PC meeting are relative minor: most decisions could be predicted from the original review scores, so the PC meeting mostly serves to sort out just a few borderline papers, making a physical meeting not cost effective. Electronic meetings might be effective if the PC chair works very hard at fostering productive discussion.

Especially when the electronic meeting is broken into several segments over multiple days or weeks, the chair must be careful not to let the work pile up on members who keep being asked for additional reviews, and not to let the meeting go longer than scheduled, when some people must go elsewhere due to pre-scheduled commitments.

Perhaps electronic meetings can be run effectively, but I have not yet seen a good example.

Reviews

Identify papers that are too long, and possibly pre-reject them. This is not just those with too many pages, but also those who played too many games with margins and formatting. It's not fair to let some people effectively have more space than others.

Identify which reviewers (on or off the PC) are also an author of another submitted paper. Many such reviewers are objective, but some may be unnecessarily harsh.

Identify which individuals in your community are known not to get along with one another, and avoid making them reviewers of the same paper.

Having more than 4 categories can be helpful in making distinctions among papers, and may reduce the all-to-frequent refrain, “my score of B is actually a low B”. It's true that the final decision is a single bit, but more information can help get there more efficiently. Having an even number of categories can help to discourage fence-sitting; a perfectly neutral score is of little value. Thus, I favor 6 categories of score: 3, 2, 1, -1, -2, -3.

Different reviewers use different criteria. I strongly advocate running a statistical regression to isolate the per-reviewer and per-paper effect. You can normalize out the per-reviewer effect — that is, use the per-paper effect as an estimate of the true relative strength of the paper. In the future, conference management software should offer this functionality, but until then you will have to compute it yourself.

(An alternate strategy is to ask reviewers to effectively normalize, by thinking of A, B, C, and D as quartile rankings and giving approximately equal numbers of them. Such ratings convey more, in an information-theoretic sense, than lopsided ones, and are thus more helpful to the committee, especially when the acceptance rate will be high, as for a workshop. One minor plus to asking reviewers to normalize their scores in this way is that it can get them out of a negative mindset in which they are tempted to reject all the papers; it's not good for a reviewer to rate only one paper “A” and all others lower. A negative of this strategy is that reviewers may resent being asked to normalize, and during the meeting will say “my rating isn't really an A.” Therefore, I believe that normalizing the scores is more effective.)

If you get an external reviewer for each paper, that can provide valuable information, but it can also obligate you to a lot of people! Also, those external reviewers are not well-calibrated compared to other papers, and they may have a (positive or negative) bias. Don't bother to get external reviewers from the same institution as the PC member assigned to a paper (who can already go to those people if they like).

During the review period, it's helpful to send a reminder/summary email every week, letting people know the status. This keeps them involved and reminds them of their responsibility, without nagging them.

As reviews come in, try to read them. Give feedback to reviewers whose reviews are too short or lack detail, whose tone is poor, who have low confidence and have not found an expert co-reviewer, or who make mistakes like comparing two submitted papers to one another (which is a bad idea because of conflicts of interest on the committee). Giving this feedback early can make those people's subsequent reviews much better.

Make sure that each program committee member has read each paper that will come up for discussion. Even if reviews have been farmed out, the PC members are responsible for making the decision, and if they must depend only on reading reviews, the right result may not occur. And in any event, people were invited on the PC for their own expertise, not their ability to ask others to do work for them.

Pre-meeting decisions

Discussions during the review period (or after the review period but before the meeting proper) should resolve many difficult cases ahead of time, reducing the amount of discussion that is necessary at the meeting itself. For any papers that have both positive and negative reviews, you should send mail to each reviewer, asking them to read the other reviews and discuss by email whether it is a worthwhile contribution to the meeting. They should avoid being overly swayed by the scores on the reviews, but should instead focus on the content. Especially for a workshop, remind them that the PC is seeking fruitful discussions and interesting ideas as much as polished work. A completed piece of research that isn't very novel, interesting, or relevant is less appropriate for a workshop than a promising initial foray into a new area, even if there are some potential problems with the approach.

You must decide whether a paper is worth discussing at the meeting. Even if you use the suggestion regarding statistical regression above, factor in the highest score a paper received. Don't just use the average score to make the decision, because a paper about which someone is excited is more likely a good paper than one that no one is excited about, but that no one can find a damning flaw in.

You can remove from consideration papers that are unlikely to be accepted. Communicate such pre-decisions to the PC before the meeting, so that PC members can request that specific papers be discussed or (if the reviewing is broken into stages) that more reviews be obtained. (It's much better to communicate this information explicitly, rather than making committee members adjust their review scores, which can lead to misleading scores and to confusion during the meeting, and lots of comments like “my B is not really a B, but only indicates that the paper should be discussed”.) Likewise, pre-accept papers that have only As and Bs; these papers should at least be presented briefly at the PC meeting, to help with calibration and program balance.

Discussion ordering

I think it's a bad idea to review all the papers in order of their rankings. It tends to bias people regarding their decision, and it gives a very misleading indication of how far along the process is. I think it is better to thematically group them, permitting better comparisons (and possibly better conflicts-of-interest grouping). However, it is useful to start with a slight overdose of positive papers in the beginning, such that the PC comes into “acceptance” mode, and to preserve the energy at the start of the meeting. Do include some negative examples early on to help calibrate the committee. (A different point of view is to put the papers in a random order, in order to increase acceptance rates. The theory is similar to the above, that too many strong papers early sets too high a bar in reviewers' minds.)

Conflicts of interest require people to walk into and out of the room. It is easy to waste a lot of time with this. Try to order papers to leave people out of the room for long periods (e.g., with reviews on multiple papers). Consider bringing a pair of walkie-talkies, so that you can press a button to send a ring signal to the people waiting outside. This can save about 30 seconds per switch — which really adds up! Make sure that each PC member knows when his or her conflicts are coming up. Some conference support software does this for you. If you are using slides to indicate who has a conflict on the next paper, consider having the slides also say who enters/exits the room. This is particularly useful in determining who should be called in from the hallway.

It is useful to delay conflict papers until everyone in the PC has had a chance to speak up at least once. This helps normalizing standards in the PC.

To keep the PC positive, consider putting a strong paper first after each break. Also put a strong paper periodically, to keep people calibrated. But remember that, because of conflicts of interest, the discussion should not make comparisons to previously-discussed papers.

Whatever the ordering, it is certain that the papers at the end of the discussion ordering will get less attention than those at the beginning. This is also true within each day and between each pair of breaks.

Physical logistics

Have a name placard for each person; this is really nice to have, even if everyone “ought” to know everyone else's name.

Some PC chairs assign seats so as to prevent person A from sitting next to person B who is reviewing a paper with which person A has a conflict. (Person A might inadvertently notice person B looking at the review, which reveals who the reviewer was.)

To provide visual feedback, consider putting a sticky note with each paper's number on the wall of the meeting room. It can be moved into the “accepted” or “rejected” column. Be sure to write big so everyone in the room can read it! Some conference support software also supports this, instead.

You might as well provide Internet access to the PC members. Some people will zone out and read email, but mostly at times they aren't needed, and it also does serve a good purpose letting people access reviews and other external resources — or to avoid wasting their time when they have no expertise (though when they have any, they should keep an ear open to provide feedback and ask relevant questions).

Discussion

Be unremittingly positive. It's easy for the whole PC to go negative. But don't annoy the PC by pushing through papers for which there is no buy-in.

Decide how much time you want to spend on each paper. Make sure you spend much less than that on non-controversial papers, and provide a warning about the length of the discussion when one runs over (but obviously it's fine to run longer on some papers).

It's helpful for the champion to first give a brief description of the paper's ideas and contributions.

Consider having a token without which no one is allowed to talk. For instance, it can be a fun distraction to throw around a soft cloth (nerf-like) football.

There should be a strong bias against postponing decisions, including classifying papers as “likely accept” or “likely reject”. Sometimes postponement is necessary, in order to calibrate the committee and understand comparisons to other submissions. But when possible, it is good to resolve each paper as it comes up. If you leave decisions for later, then the hard work just gets deferred, or else bad decisions are made in the last 30 minutes of the PC meeting when you return to all those papers. If you permit “likely accept/reject” outcomes,

Consider every possibly contentious paper, and those with no or few high-confidence reviews (though you should have addressed any such issues before the meeting started), before the break (e.g., on the first day or before lunch of a one-day meeting), to permit additional information-gathering.

If any committee member sends in a “Z” (low-confidence) review, ask that person to find an expert. It's the reviewers' responsibility to find a better reviewer, not the PC chair's. If there are no “X” (high-confidence) reviews, or perhaps if there is only one, ask all the reviewers to look for better co-reviewers. But if the PC members can't find an expert, then the PC chair should assign more reviewers well before the meeting. It is very bad to lack high-confidence reviews for a paper.

The PC chair is often very busy during discussions, and might not have the presence of mind to keep the high-level picture in mind. It can be useful to privately appoint one of the committee members as a "rathole alert", who will send a private instant message to the PC chair if the discussion starts to go down a rathole.

After the meeting

In the proceedings, indicate lesser publications such as posters. For example put “poster” or “short paper” in the title, or put them in a separate volume of the proceedings. Otherwise, unscrupulous authors will pretend to have a regular conference paper — regrettably, I have seen this in many CVs that I have reviewed.

To excite interest in your conference, consider putting abstracts (not just titles) on the webpage. The main page should have just titles, but link to an appropriate anchor in a page that contains all abstracts.

Other sources of advice

Alex Aiken's "Advice for Program chairs"


Back to Advice compiled by Michael Ernst.

Michael Ernst