अनिल एकलव्य ⇔ Anil Eklavya

April 13, 2008

Two Laws of Reviewing

After a few years in research, I have discovered two laws which the process of reviewing (of research papers) follows. Not very original, but here they are:

  1. You can always find some reasons for accepting any paper.
  2. You can always find some reasons for rejecting any paper.

February 29, 2008

English is Language Independent

It’s the Global Language, right? So how can it be language dependent? You propose a theory based on English. It has to apply to all languages. You propose a Natural Language Processing (NLP) or Computational Linguistics (CL) technique for a particular problem. For English. It applies to all languages. You build a software for some purpose. For English. It has to be useful for all languages. You build a dictionary…

Never mind.

But the vice versa is not true. You propose a theory based on Hindi. It is language specific. It doesn’t count for much. You propose an NLP technique for a particular problem. For Hindi. It is language specific. It doesn’t count for much. You build a software for some purpose. For Hindi. It is language specific. It doesn’t count for much.

That’s how it works in practice, if not theory. Or may be even in theory, with some help from the (very valid) idea of Universal Grammar (except that the UG may be the UG of English).

Even today I have got a review of a paper on a problem which is like one of the holy grails of NLP or CL. One of the comments is that the approach has been evaluated on Hindi so it can’t be compared to other techniques that already exist. True. But what is the number of papers published in the ‘first class’ NLP/CL conferences and journals in which the approach has been tried only on English? Doesn’t matter, because English is language independent. If you only evaluate your technique on English, that’s OK. But if you evaluate on only Hindi, that’s not acceptable. Because Hindi is language specific.

We know this very well in India. The Elite talks about (Indian) literature. And sometimes the Elite magnanimously (or dismissively) talks about (Indian) literature in languages. The first, of course, refers to literature in English. The second refers to literature in other languages. Indian languages.

The Elite talks of media. And the Elite (rarely and mostly negatively) talks of language media.

Hindi is a language. English is not a language.

Pardon me.

Hindi is a language. English is the language.

English is above being merely a language.

That’s why all the work done in English is language independent. Not just research. Not just in NLP/CL. Anything. Movies, literature, music.

I am guilty of the sin of indulging too much in mere languages. I should be working mostly on English. Not just writing blog posts in English. Sometimes, of course, I can bestow a bit of my attention on languages. Like Hindi.

But I won’t do that. I will do the opposite. I am incurable.

Can Confrontation be Turned into Collaboration?

In the last post I tried to put across some points against blind reviewing. The intention was to show the limitations of blind reviewing. As I said in that post, I don’t have the correct solution. But I do know now that blind reviewing is not as good an idea as it seems on the surface.

I will have more to say about this, but in this post I will digress from the topic a bit and consider a hypothetical situation.

At present, when the author(s) of a paper get the reviews, they (in most cases) have no venue to reply to the comments made in the reviews. Now, suppose we had completely open reviewing. Not double blind. Not single blind. Zero blind.

Zero blind means that the reviewers know who the author(s) are and the author(s) also know who the reviewers are.

Hard to imagine? But we already have such a reviewing process in real world. In fact, this is the most used process. This is how books are reviewed in the media. Or movies. Or anything artistic or scientific.

Yes, I know, the situation with research conferences and journals is a bit different and there are many practical difficulties. But let’s ignore them for the time being and just assume that a zero blind process is in place and a paper is sent to a conference.

Now, the reviewers do their job and based on their comments and scores (if any) it is decided to reject the paper. This is again not a universal thing, because there is also something called ‘killing a paper’ where a single reviewer can virtually ensure that the paper is, well, killed. Regardless of what other reviewers might have said. This actually happens.

But we will ignore this too for our hypothetical case and just assume that the paper was rejected after considering all the reviews and discussion among the PC members.

Now, when the authors get the reviews, they believe that the reviewers have made mistakes in understanding the paper. Also, that the reviewers have made statements which are not justified. The authors, because they know who the reviewers are, write a detailed reply and counter all the comments and statements made by the reviewers which they think are not justified.

The reviewers are also interested in the problem that the paper is about and they probably are also working on the same problem. Which is why the paper was assigned to them in the first place.

Once again, this is an assumption for the sake of idealization. In reality, the reviewers often are not working on the same problem and may not even be interested in it.

What is going to happen now? The reviewers also reply to the author(s)’ comments. This could start a confrontation which goes on for a long time and arguments are traded back and forth.

Note that the decision about the paper has already been taken because the only reform that has happened in this hypothetical situation is that we have zero blind reviewing instead of single or double blind. So the confrontation was started by the reviewing process, but it is no more a part the process now.

‘Conventional wisdom’ says that confrontations (with words and verbosity) among researchers (or, in general, members of a group) are bad. This is the same wisdom which says that confrontations (with weapons and violence) may be good among nations or communities because there are things called national interests or communal interests.

So how about collaboration? Can this exchange of arguments lead to collaboration among the author(s) and the reviewers?

Apart from practical constraints, is there any reason why this can’t happen? I mean, even if collaboration is ruled out in many cases due to practical constraints, it may still happen is some cases. Wouldn’t that be a good thing for research?

Of course, I am talking about Computational Linguistics and Natural Language Processing, where most of the action takes place on computers. Which are likely to be connected to the Net. So, collaboration, i.e., long distance collaboration, is not a Utopian dream.

And I also know it’s not just that collaboration may not happen, there may be an ugly confrontation. This might lead to worsening of social relations among the researchers. It won’t be a good situation.

But wait a minute!

Aren’t researchers supposed to be mature people who, for the most part, think rationally? Aren’t they supposed to be objective, or at least rigorously and honestly subjective? Aren’t they supposed to be good enough to take the responsibility of deciding which research paper should be accepted and which should be killed. Or not allowed to be born, to put it more politely? In fact, aren’t they assumed to have a lot of qualities which we don’t so easily assume in ordinary mortals? Such as the fact that when they recommend the rejection of a paper and make sure it’s never published, they won’t take some ideas from those rejected papers and use them for their own work as their own contribution? Unconsciously, if not consciously. To suggest otherwise (i.e., that they can be unknowingly plagiarist) would actually be considered a blasphemy. And if someone gets caught, it would be considered a great scandal. There is some social psychology involved here which I would rather not talk about right now.

If the researchers who review papers are already assumed to have all these wonderful qualities, can we also assume that in most cases they won’t get into an ugly confrontation? That they would, if possible, convert a possible confrontation into a collaboration?

It all depends on what our preferred model of relations among researchers is. If there is a four colour (color) spectrum, which one is your preferred color:

  1. Between Newton and Leibniz
  2. Between Einstein and Bose
  3. Between Bertrand Russell and Wittgenstein
  4. Between Hardy and Ramanujan

In case there is anyone reading this, and assuming that they don’t think this is all pure BS…

What do you say? Which model do you think can work? Which is your preferred model? Which model would you recommend for application in the real world?

In today’s world? But more importantly, in tomorrow’s world?

February 26, 2008

On Blind Reviewing (2)

In the first part, I presented a case in favour (favor) of blind reviewing as a bulleted list of ten reasons to have blind reviewing as the most preferable implementation of the concept of peer reviewing.

Excuse again the legal sounding language.

I also indicated that I am not sure that this is indeed correct, i.e., blind reviewing is the best that we can do to ensure quality research. In this post I will try to present a case against blind reviewing, or at least a case against the idea that blind reviewing is only or the best way to go about the business of research.

First, let’s take each of the ten points I had listed:

  1. Human beings can be biased. So, if a reviewer knows that a research paper is written by a person she doesn’t like or has strong disagreement with, she can get biased against the paper and will not be able to review the paper fairly.

    Human beings can be biased and their biases can really come to the fore if they know that their identities won’t be revealed. There may be enough in the paper to trigger their biases.

  2. Apart from the above kind of biases, there can be the bias in terms of the weights associated with the names of the authors, their institutions, their countries, their group, even their academic background. Most of the people who have been working in NLP/CL[1] for some time know about the linguistics vs. statistics or machine learning bias. This kind of bias increases the chance of your paper being rejected or accepted depending on whether you seem to be in favour (or favor) of a linguistics heavy approach to NLP/CL or of a statistics (or machine learning) heavy approach. There are variants of this bias in other fields too. For the closest example, we can consider Linguistics. Where your paper is perceived to be situated along the Chomskyan or Empiricist or Cognitive or Computational axes with respect to the chosen position of the reviewer can have a large impact on the decision about your paper, irrespective of what else your paper says. And the chances of such a perception can be increased if the identities are known.

    Such biases are especially the ones that can be triggered mostly by the content of the paper itself, because enough is revealed about the authors by just the content of the paper. More on this later.

  3. Human beings can be unduly confrontational and they can also be unduly wary of confrontation. So, if the identity of the reviewer is not withheld, the author(s) may be offended by the reviewer and they may also become confrontational and carry on this confrontation with the reviewer, thus making the process of reviewing difficult and something which a lot of people would like to run away from. Also, the reviewer may avoid making adverse comments, especially if the reviewer doesn’t want to offend the author(s).

    This is perhaps the strongest point in favor of blind reviewing. However, the problem is that, like all the other points, it assumes that identities are not revealed at all in the blind reviewing process. The truth is that the content of the paper often gives enough information about the possible identities of the author(s). And this happens more in the cases where there are much higher chances of biases and where a proactively fair reviewing process is needed the most. For example, if someone is working on languages of the Third World, those languages will be mentioned in the paper. From this, and from the language and presentation of the paper, it will be easy to guess that the paper is from some Third World researcher. In many cases, the reviewer might even be able to guess the author(s), or at least the groups of which the author(s) are part. It is these cases where the need for fairness is the highest because it is so easy for a reviewer to become biased against the paper. So much so that she doesn’t even care to read the paper carefully. If you get two papers to review and one of them seems to be from a major research group from a major university from a First World country, while the other is possibly from a graduate student from a second class university in a Third World country, would there be any difference in the way you review the papers? Would there be any prejudgment? Won’t you be more careful in reviewing the second paper if you knew your identity will be revealed?

  4. If the author(s) don’t know who the reviewer is and vice versa, the whole reviewing process may be more fair for the above specified reasons and because of the general association between anonymity and fairness. If you don’t know who is criticizing and the person criticizing also doesn’t know who is being criticized, then you can expect more fairness.

    Seems to be a valid point on the surface. And it is: To a certain extent. But there is an even more valid counter-point. If the reviewer knows her identity is not going to be revealed, she can be as biased as wants. Even as biased as she doesn’t want. There is not much ‘incentive’ to read the papers carefully. You can get away with anything, especially if the authors are not ‘prominent’ ones so that even the chair(s) won’t probably notice. Unknown authors, paper rejected, extremely negative comments and ridiculously low scores. So what? Common occurrence. No need to take note.

  5. If the Program Committee (PC) chair(s) also don’t know who the authors are and who the reviewers are, then they can assign equal weight to all the reviews for making the final decision about a paper.

    This is actually a hypothetical point. It doesn’t apply even to double blind reviewing. The chair(s) always know who the authors are. The reviewing process is not actually completely blind. In almost all cases, the chair(s) assign the papers to the reviewers. Now, the reality is that the fate of a paper depends on which reviewers are assigned to review it. Moreover, in many cases, the chair(s) or the area chair(s) actually overrule the assessment of the reviewers. They are supposed to do this to ensure fairness, but why should the chair(s) be assumed to be free from bias? They are as much human as other reviewers. Reviewers in one conference or journal are often the chair(s) or the editor(s) in some other conference or journal.

  6. If the author(s) don’t know who the reviewer is, then they won’t have any reason to attribute bias or prejudice to the comments made or ratings given by the reviewer.

    Only if they assume that the reviewer didn’t guess anything about the possible identities of the authors. That might happen with their first few papers, but later on they might catch on to the fact that this is not really so. In fact, just as the reviewers can guess the possible identities of the author(s), the vice versa can also happen. One small remark might give away the identity of the reviewer. And it often does.

  7. Peer reviewing of research papers, like the administration of justice, should not just be fair, but seen to be fair. And this can only happen with blind reviewing.

    Again a superficially valid point, but there is a more valid counter-point. If everything is ‘blind’ but not really blind, how can the process be seen to be fair? Perhaps there can be some other ‘non-blind’ way which actually seems more fair.

  8. Blind reviewing, through the use of the device of anonymity, gives a true meaning to the idea of ‘peer reviewing’, because if the identities are not known, all the people involved can be treated as peers, even if some of them are senior most pioneering researchers or Directors of first class institutions in first world countries, while some others are graduate students in second class institutions in third world countries.

    As discussed earlier, anonymity doesn’t really happen in the cases where it is supposed to matter the most, i.e., in cases where there can be biases. Could it be that ‘blind reviewing’ actually gives an impression of fairness where it doesn’t really exist? This could be a very undesirable situation if we are really sincere about ensuring fairness.

  9. If the identities are not known, both the reviewer and the author can focus on the content of the paper and the review, respectively.

    Another strong point in favor of blind reviewing, but again it depends on the assumption of anonymity. Also, could it be that (consciously or unconsciously) the reviewer starts guessing who the paper is possibly from so that she can review accordingly, without spending too much time in understanding the paper? And when the author(s) get the reviews, could they be guessing who the reviewers are (from the remarks they have made). Both of them might reach right or wrong conclusions, and whichever they reach, such a situation will not be an ideal one for the purpose of fair peer reviewing. This is, of course, more likely to happen with papers which get rejected, fairly or unfairly.

  10. Finally, the very practical reason that blind reviewing provides a reasonably fair mechanism to ensure the selection of the best research papers such that everyone can be more or less satisfied with the outcome and no one will have valid reasons to complain.

    Not all conferences or journals have blind reviewing. Yes, some studies have been done which show that blind reviewing reduces biases, but these studies have not considered the cases where the biases are most likely (as mentioned earlier). What these studies show that biases are somewhat less in cases where biases are likely to be less anyway.

So the case it not as clear as it might seem at first sight.

Now I will mention the really bleak reality of blind reviewing which made me think about the reviewing process and, over the years, has provoked me enough to write this.

Let me emphatically state first that when I started research and was thinking of publishing my first paper, I was really happy to know that most NLP/CL conferences and journals use blind reviewing process to select papers. My reasoning was exactly as I have listed above as points in favor of blind reviewing. So this post is not being written just out of a whim.

What I have found is that blind reviewing, though it does work to some extent, actually becomes a cover for reviewers to be as irresponsible as they want because they are anonymous. As an analogy, there are some good anonymous commentators on blogs, but many use anonymity as cover for their mean and nasty (completely unjustified) comments with perfect unaccountability. Something similar happens with the blind reviewing process. Many reviewers, including those who are most probably very senior researchers, use the cover of anonymity to let all their biases flow freely into their reviews because they know they are safe. And, unlike on the blogs, the author(s) can’t even reply because most conferences don’t have a rebuttal phase. Even if there is one, the reviewers simply don’t care. They have reviewed and their comments are final and unchangeable, whatever the author may have to say. They don’t change their reviews in response to author(s)’ rebuttals or clarifications. They don’t because they are safe in anonymity.

What makes this kind of situation even worse is the fact that a lot of reviewers review for many conferences and journals and, therefore, your paper if unfairly rejected from one place, is quite likely to run into the same set of reviewers at another place.

It can be quite depressing. I am sure many researchers have thought of or have actually quit from the research arena because of what I have described above.

I think it’s time we gave another look to the reviewing process. I don’t have a solution, but I will try to make some suggestions later. Perhaps more experienced researchers and organizers can say something better.

I know there are many great people out there who put their best in writing a review and actually go beyond what’s expected of them, but I think such people would do the same even if their identities were revealed because they know they are doing what’s right to the best of their abilities.

January 20, 2008

On Blind Reviewing

This is something about which I have wanted to write for a long time. Since, like many other things about which I want to write, it is quite an important matter, I didn’t want to write in a hurry. Which meant that I had to wait for a time when I could write at enough leisure to be able to write at enough length with enough time for making it rigorous enough. Now, since it is very difficult (for me at least) to get enough of all these, this effectively meant that writing about this topic was postponed indefinitely.

But I don’t want this to be postponed indefinitely. I want to write about this now. So, I would just write and try to be as rigorous as it is possible to be in a blog post written in one or two short sittings. This applies to many other posts, whether written already or to be written in future. You can take it as an apology or you can take it as a disclaimer.

What is the problem? Well, the problem, or rather the question, is whether what is called ‘blind reviewing’ is a good thing or not. And, of course, this is in the context of peer reviewing of scientific (or claimed to be scientific) research papers or articles for the purpose of selection for inclusion in the proceedings of a conference or workshop or for inclusion in a journal.

Excuse the legal sounding language.

First of all, let me list all the reasons in favour (‘favor’ for the dominant party) of the so-called ‘blind reviewing’ process, so that no one can jump and dismiss the whole affair as trivialization by saying you don’t know what you are talking about:

  1. Human beings can be biased. So, if a reviewer knows that a research paper is written by a person she doesn’t like or has strong disagreement with, she can get biased against the paper and will not be able to review the paper fairly.
  2. Apart from the above kind of biases, there can be the bias in terms of the weights associated with the names of the authors, their institutions, their countries, their group, even their academic background. Most of the people who have been working in NLP/CL[1] for some time know about the linguistics vs. statistics or machine learning bias. This kind of bias increases the chance of your paper being rejected or accepted depending on whether you seem to be in favour (or favor) of a linguistics heavy approach to NLP/CL or of a statistics (or machine learning) heavy approach. There are variants of this bias in other fields too. For the closest example, we can consider Linguistics. Where your paper is perceived to be situated along the Chomskyan or Empiricist or Cognitive or Computational axes with respect to the chosen position of the reviewer can have a large impact on the decision about your paper, irrespective of what else your paper says. And the chances of such a perception can be increased if the identities are known.
  3. Human beings can be unduly confrontational and they can also be unduly wary of confrontation. So, if the identity of the reviewer is not withheld, the author(s) may be offended by the reviewer and they may also become confrontational and carry on this confrontation with the reviewer, thus making the process of reviewing difficult and something which a lot of people would like to run away from. Also, the reviewer may avoid making adverse comments, especially if the reviewer doesn’t want to offend the author(s).
  4. If the author(s) don’t know who the reviewer is and vice versa, the whole reviewing process may be more fair for the above specified reasons and because of the general association between anonymity and fairness. If you don’t know who is criticizing and the person criticizing also doesn’t know who is being criticized, then you can expect more fairness.
  5. If the Program Committee (PC) chair(s) also don’t know who the authors are and who the reviewers are, then they can assign equal weight to all the reviews for making the final decision about a paper.
  6. If the author(s) don’t know who the reviewer is, then they won’t have any reason to attribute bias or prejudice to the comments made or ratings given by the reviewer.
  7. Peer reviewing of research papers, like the administration of justice, should not just be fair, but seen to be fair. And this can only happen with blind reviewing.
  8. Blind reviewing, through the use of the device of anonymity, gives a true meaning to the idea of ‘peer reviewing’, because if the identities are not known, all the people involved can be treated as peers, even if some of them are senior most pioneering researchers or Directors of first class institutions in first world countries, while some others are graduate students in second class institutions in third world countries.
  9. If the identities are not known, both the reviewer and the author can focus on the content of the paper and the review, respectively.
  10. Finally, the very practical reason that blind reviewing provides a reasonably fair mechanism to ensure the selection of the best research papers such that everyone can be more or less satisfied with the outcome and no one will have valid reasons to complain.

I think the above list makes as strong a case for blind reviewing as can be made. I mean in a blog post, not in a book.

Now, in the next post (that means in some future post) I will discuss what is or can be wrong with blind reviewing and will try to draw some conclusions. You must have guessed that the reason I am writing all this is that I am not sure whether blind reviewing is the best thing possible. But by writing all this, I am also trying to get things straight in my own mind.

[1]: With apologies to Martin Kay and others, I am using NLP and CL as interchangeable terms because I think my arguments in this matter are not affected by the distinction between the two, a distinction which may be important in many but not all contexts (i.e., in my opinion).

Blog at WordPress.com.