On Blind Reviewing

This is something about which I have wanted to write for a long time. Since, like many other things about which I want to write, it is quite an important matter, I didn’t want to write in a hurry. Which meant that I had to wait for a time when I could write at enough leisure to be able to write at enough length with enough time for making it rigorous enough. Now, since it is very difficult (for me at least) to get enough of all these, this effectively meant that writing about this topic was postponed indefinitely.

But I don’t want this to be postponed indefinitely. I want to write about this now. So, I would just write and try to be as rigorous as it is possible to be in a blog post written in one or two short sittings. This applies to many other posts, whether written already or to be written in future. You can take it as an apology or you can take it as a disclaimer.

What is the problem? Well, the problem, or rather the question, is whether what is called ‘blind reviewing’ is a good thing or not. And, of course, this is in the context of peer reviewing of scientific (or claimed to be scientific) research papers or articles for the purpose of selection for inclusion in the proceedings of a conference or workshop or for inclusion in a journal.

Excuse the legal sounding language.

First of all, let me list all the reasons in favour (’favor’ for the dominant party) of the so-called ‘blind reviewing’ process, so that no one can jump and dismiss the whole affair as trivialization by saying you don’t know what you are talking about:

1. Human beings can be biased. So, if a reviewer knows that a research paper is written by a person she doesn’t like or has strong disagreement with, she can get biased against the paper and will not be able to review the paper fairly.
2. Apart from the above kind of biases, there can be the bias in terms of the weights associated with the names of the authors, their institutions, their countries, their group, even their academic background. Most of the people who have been working in NLP/CL^[1] for some time know about the linguistics vs. statistics or machine learning bias. This kind of bias increases the chance of your paper being rejected or accepted depending on whether you seem to be in favour (or favor) of a linguistics heavy approach to NLP/CL or of a statistics (or machine learning) heavy approach. There are variants of this bias in other fields too. For the closest example, we can consider Linguistics. Where your paper is perceived to be situated along the Chomskyan or Empiricist or Cognitive or Computational axes with respect to the chosen position of the reviewer can have a large impact on the decision about your paper, irrespective of what else your paper says. And the chances of such a perception can be increased if the identities are known.
3. Human beings can be unduly confrontational and they can also be unduly wary of confrontation. So, if the identity of the reviewer is not withheld, the author(s) may be offended by the reviewer and they may also become confrontational and carry on this confrontation with the reviewer, thus making the process of reviewing difficult and something which a lot of people would like to run away from. Also, the reviewer may avoid making adverse comments, especially if the reviewer doesn’t want to offend the author(s).
4. If the author(s) don’t know who the reviewer is and vice versa, the whole reviewing process may be more fair for the above specified reasons and because of the general association between anonymity and fairness. If you don’t know who is criticizing and the person criticizing also doesn’t know who is being criticized, then you can expect more fairness.
5. If the Program Committee (PC) chair(s) also don’t know who the authors are and who the reviewers are, then they can assign equal weight to all the reviews for making the final decision about a paper.
6. If the author(s) don’t know who the reviewer is, then they won’t have any reason to attribute bias or prejudice to the comments made or ratings given by the reviewer.
7. Peer reviewing of research papers, like the administration of justice, should not just be fair, but seen to be fair. And this can only happen with blind reviewing.
8. Blind reviewing, through the use of the device of anonymity, gives a true meaning to the idea of ‘peer reviewing’, because if the identities are not known, all the people involved can be treated as peers, even if some of them are senior most pioneering researchers or Directors of first class institutions in first world countries, while some others are graduate students in second class institutions in third world countries.
9. If the identities are not known, both the reviewer and the author can focus on the content of the paper and the review, respectively.
10. Finally, the very practical reason that blind reviewing provides a reasonably fair mechanism to ensure the selection of the best research papers such that everyone can be more or less satisfied with the outcome and no one will have valid reasons to complain.

I think the above list makes as strong a case for blind reviewing as can be made. I mean in a blog post, not in a book.

Now, in the next post (that means in some future post) I will discuss what is or can be wrong with blind reviewing and will try to draw some conclusions. You must have guessed that the reason I am writing all this is that I am not sure whether blind reviewing is the best thing possible. But by writing all this, I am also trying to get things straight in my own mind.

[1]: With apologies to Martin Kay and others, I am using NLP and CL as interchangeable terms because I think my arguments in this matter are not affected by the distinction between the two, a distinction which may be important in many but not all contexts (i.e., in my opinion).

Too Good to be True

So there were a lot of things to be done in a short time and, apart from one or two minor hiccups, everything went quite well: The paper presentation, the demo, submission to the ACL main conference, and the two workshops. In fact, one of the workshops (NLP for Less Privileged Languages) was an unexpected success, both in absolute and relative terms. The funny thing is that while this workshop was the one my heart was set on, I had to do much much more work for the other workshop (NER for South and South East Asian Languages). Believe me, the ‘much much more’ part is more than justified. Conducting a workshop (when you have to do most of the work) is not very easy. Conducting a workshop that includes a shared task is tough. And a shared task which requires new training and testing data is tougher. And it’s even more tough when you don’t have any funding etc. You get the point.

So, everything went quite well.

Not quite.

After the workshop work was finished I just happened to look at the final version of the paper which was submitted to ACL. And I found to my shock that a figure which I had added and on which I had spent around two hours on the night before the workshop (my heart was set on) was missing. The caption was there and so were the references to the figure, but the figure itself was gone.

What happened was that while I was in the middle of the workshop, one of the co-authors noticed a minor typo. But he is now doing a job and is not here. He was unable to compile the Latex files on his system. So he asked another co-author to correct the mistake. The other co-author did as he was told, but he didn’t check before submitting the revised version. Apparently some library was missing or incompatible on his system and so the figure simply disappeared from the paper. Now, I had worked all night before the workshop (my heart was set on) to complete that paper and I thought I had done a good enough job. Moreover, that paper has already taken two rejections, in spite of the fact that it has taken a lot of work from more than one person. Even as an unbiased reader (to the extent it is possible for me to be), I think that paper was worth selection in a major (’first class’) conference.

So many things to do in so short a time and everything going almost perfectly?

It was too good to be true. Something had to go wrong. Murphy’s law.

That jinxed paper seems set for another rejection. Leading to more dejection.

Just because a library was missing (or incompatible) on a system and the version submitted had one figure missing.

I don’t know whether I should say this, but had this happened to some other author and I was the organizer, I would allow the author to submit a revised version with the figure included, provided the authors had not added anything more after the deadline.

Does that sound like a crazy idea?

Hectic Days are Here

There was a lull before the storm (in a tea cup?). The easy days are over and the hectic days are here. Today I have to submit a paper for the ACL conference. Yesterday I had to present a demo at IJCNLP conference. The day before that we had a paper presentation at the same conference. And the day before that I had to attend two tutorials, again at the same conference.

Tomorrow there is an IJCNLP research workshop of which I am the organizer. The day after tomorrow there is another workshop of which I am the co-organizer. And I have to make introductory presentations at both the workshops.

After that starts the ‘real’ work which I am supposed to do. Some professional, some formally academic and some informally academic.

These must be among the most hectic days of my life. Certainly the most hectic involving public presentations.

I hope I survive.

A Silly Undreamt-of Adventure

Let me first remind you (whoever you are) that my secondary Gmail account is still disabled and there has been no response from Google, even though I had filled in their contact form days ago.

So, just to make myself a bit secure, I created a Yahoo! account as a backup. Some of you will understand what I mean. Now, after creating this account, there was a link on the left side of the window asking ‘will you marry in 2008?’. I don’t know why I clicked on it. But there is more. I created a profile on the page that came up. Yes, a Shaadi.com page. So, I actually did something as silly as creating a profile on Shaadi.com! Even if I didn’t put much information (or misinformation) there.

What came over me? Nothing, it’s just that my sense of humor is gaining breadth.

Anyway, since the deed is done, I thought may be I will let the profile remain there for some time and see what happens.

After having a reasonably good time at the research conference (IJCNLP), may be it’s time to have a silly undreamt-of adventure. Why not?

As always, I am thinking that things can’t get any worse.

I know. They can.

Back to Notwork as Usual

The wireless network had been working well for sometime. Now there is major conference at the institute in which I am involved in several things (very unusual). Tomorrow I have to give a demo. Today I had to setup the system for that demo.

I should have known. The network is back to being the notwork with a perfect timing. Period. Either no one else is affected, or no one else cares.

Life is beautiful.
Hope springs eternal.
Human being are social animals.
All of us are good people.
Great are the wonders of civilization.
What a wonderful world.

Deja vu.
Which means snafu.

By the way, the Gmail account is still disabled.

Summary execution Google style.

Gmail Account Disabled (2)

It’s official now. Actually it has been for some time, I just didn’t know about it. A large number of genuine Gmail accounts being used without any abuse are being disabled. Some accounts seem to be up again, but only some. Mine isn’t either, although it’s much more than 24 hours (the expected response time). And, yes, I had filled the required contact form quite early.

As the Economic Times (India) article says, “The Internet giant Google Inc reportedly disabled a few Gmail accounts in an overzealous [attempt?] to combat spammers”. Except that ‘few’ here refers to a very large number, judging from the lists of complaints I have seen while searching with the query ‘gmail account disabled’.

It is being suggested that only the new accounts are affected by this, but my account is not really a new one. If I am not mistaken, it’s more than an year old. That’s old enough for Gmail.

The funny thing is that the spate of spam mails (I was the recipient, not the sender, in case someone is mistaken) after which my account was disabled asked the recipients to furnish information about their Gmail accounts and threatened that if such information was not sent within the specified period, the account will be disabled. And the account was actually disabled, even before that deadline. Most of the people who saw that mail (it was sent multiple times to my account and many people commented about it) immediately thought that it was another of those fake mails which are a regular feature of the email world. But given this funny situation, I can’t help wondering whether the mail was actually an official Google mail. Of course, it was not, but an absurd situation anyway.

I am still waiting for my account to be enabled again. And I hope the primary account is not disabled in the ‘overzealous attempt to combat spammers’. As I said earlier, I oppose spam but I don’t support anti-spam fanaticism.

I had once mentioned that my homepage had suddenly disappeared from the Google index. It has since come back on the index. I hope Google doesn’t become comparable with the official censors (of anything anywhere).

What if WordPress suddenly, ‘with or without any notice’ (as the Gmail Terms of Use say) disables my blog? Sounds sinister and utterly unethical to me. Even if I am using the free version of WordPress. Google may be providing the email service for free, but they are putting ads on the side, aren’t they? That’s the Internet business model. It’s not the same as the old world economic model. If the user accepts ads being put on their pages, they are actually paying in an online sort of way. So, you get what you pay for argument is not really valid. The (’free’) service may be theirs, but the content is ours.

The least acceptable could be this: give me back my content and I will look for another ‘free’ service provider, or I will get a ‘non-free’ provider if I can afford it. I want access to my content, and in the case of emails, I want access to them immediately, as soon as the account is blocked. You may have the right to disable my account, but you don’t have the right to take away my content. This, of course, applies to all service providers, not just to Google.

Gmail Account Disabled

I just tried to login to my secondary Gmail account found that my account has been disabled. What happened? Well, during the last two days, there were many spam (actually phishing) mails sent to my account. These mails asked the recipients to submit Gmail account information ‘otherwise the account will be disabled’. Several people commented about this to the others on the To: list. I responded to one of those mails (just a small retort to a stupid racist remark, which had a denial attached in advance, by one of the people who responded).

I thought I was the victim of a phishing attack, though I realized it was a case of phishing so I didn’t do what was suggested in the fake mail. I mean I was not the perpetrator of the attack or anything similar.

Does Google punish the perpetrators of such attacks or the victims? Nice New Year’s gift.

A few days ago when I tried to login to my primary account, I got the message that my account has been locked as ’some unusual activity has been detected’. Unusual? Yes, I do a lot of unusual things, but certainly not on my Gmail account. There I am as usual as you can get. Fortunately, I succeeded in logging in on my third attempt.

While I am on this topic, I might as well write about one suggestion for improvement in all email systems. Whenever you log into your account, you should be shown if there is some other session open for the same account. This will help in ensuring that one has not accidentally left one’s account open somewhere. And, in a worst case scenario, one would also be able to check whether someone else has somehow got access to one’s account, by stealing password or by hacking.

One lesson learnt from this story: It is not so good to rely on one email provider alone, even if it’s Google. I will have to do something about this.

Small comfort to find that I am not alone: [1], [2], [3], [4], [5], [6], [7].

There is one more thing I should say here. I am against spam like most people, but I don’t think heavens would fall if one gets a few useless mails on some particular day. I don’t support anti-spam fanaticism either.