Progressive Geographies

The comments following the post on referee etiquette on the Leiter Report blog are worth reading.

One theme in particular is worth reflecting on. Would the process of double blind review – referees don’t know the author; authors don’t know their referees – be improved by editors not knowing whose papers had been submitted? Either by manuscript handling software, or by people acting as journal managers or administrators, it would be possible to make that stage of the process anonymous.

The positive reason that it would take away any bias editors might have in the process – presumably things like picking perceived soft or hard referees, how they handle conflicting reports, what decision they make, what they ask of the authors, etc.

But there are downsides as well.

– In selecting referees we avoid author’s colleagues, research collaborators, known friends, etc. This is much harder if the author is not known and these names are entered into a machine. I wonder how many times, working blind, we would pick the author of a paper to review their own work? A person doing the role of administrator could catch some – such as colleagues – but by no means all of the referees we might chose not to use if we knew author identity. There is an lot of editor knowledge that would be lost this way.

– Authors would then not be able to have any contact directly with the editor. Initial queries about potential papers, suitability, encouraging people at conferences to send their work to the journal, etc. – all would need to be prohibited.

– Just as referees can sometimes guess author identies, so too would editors. This can happen simply by knowing the field, attending conferences, etc., but the process of finding referees would exacerbate it. I might remember hearing a paper at a conference and think that person would be a good referee. A bit of online searching later I might put a name to that memory. But wouldn’t I likely stumble upon the author? If authors genuinely made their papers anonymous – a large number still don’t – what chance I would realise their identity simply because their own work was not cited? All these are true of referees too, but I think the average editor has a much better sense of these things than the average referee.

– With people early career I tend to show what might be called a generosity of process, not outcome. In other words, papers need to meet the same standard to get accepted, regardless of career stage, but I might be more forgiving of some things up until decision with someone who is learning the system. So I might suggest that authors do a few things to the paper before review, or suggest that they rework it more throughly before second-round review after a revise and resubmit decision. That would be very difficult without knowing the author identity. Similarly with issues over language competency.

There may be other problems too. But the idea seems to rest on the assumption that editors carry biases. I’m sure we do, but at Society and Space we work quite hard to try to minimise their impact. There are four co-editors and an editor – we work collectively and so I’d hope that would highlight and/or stop any unfairness or unreasonable generosity. We each have a policy of keeping out of decisions on colleagues, close friends or students (current or ex-) and leaving it to others in the team.

I also worry that it would take away from the role of editor. With the volume of submissions it can be hard to give each paper a personalised touch as it goes through review, but I think we try. Take away the author identity and it becomes just another piece of paper. Editors should edit, not just manage, a journal and I think this idea would reduce that ideal still further.

The Leiter Blog posts some suggested guidelines from Thom Brooks on referee behaviour, along with a request for comments.

I was hoping that you could help publicize some comments for feedback from your readers concerning referee guidelines. I am part of an APA committee looking into journal practices and sit on a subcommittee on guidelines for referees. We have three different areas where I would be keen to solicit comments from your readers: 

A.    Appropriate response times to an invitation and for an evaluation

My view is that potential referees should have at least two weeks to respond to invitations for reviews. This is because they should have an extra few days to assess whether they have the time to commit to providing a review. Once a referee accepts, I believe the maximum evaluation time should be 6-8 weeks and ideally one month. Colleagues should have time to review given their many existing commitments, but no more than two months. Otherwise, it is best to find someone else.

B.    Guidelines for comments with regard to civility and the promotion of quality research.

Referees should be expected to treat authors with courtesy and even where authors have been uncharitable towards their targets. Reports should convince editors to follow their recommendations and uncivil reports may have the opposite effect.

C.    Conflicts of interest

While it is necessary that reviewers inform editors where they may know the author’s identity, I am not sure this need always be a problem or that another reviewer is needed in every instance.

The second and third seem straight-forward: although especially with the second there are still plenty of uncivil reports. As an editor I don’t ordinarily edit reports, but I might remove an unnecessarily nasty sentence or two. On the other hand, I do find it difficult when the report for authors is softened, perhaps suggesting substantial revisions, while the confidential comments to editors are much harder. The editor can then be in the difficult position of knowing the paper should be rejected but without the justifications to do so. So I’d encourage civility but directness. (For this reason I discourage comments just to editors, unless they are to inform instances in the third category, or other more technical issues.)

The first is perhaps uncontroversial too: who could argue with giving people enough time to decide if they can commit to do something, and then giving them a sufficient period of time to do it? The time given is balanced by editor needs to keep things moving, and authors want to hear about their papers in reasonable amounts of time. But, in my experience as editor, this is what can introduce delays into the review system. Referees often take a long time to agree to do a report, and many do not reply at all, and then many miss deadlines, even if these were extended to accomodate their needs. Once a referee has failed to reply, or to deliver what was agreed, another referee is asked, and the process begins again. I’d happily give all referees two weeks to decide, and up to eight weeks to referee the paper, if we always heard within two weeks and then always received the report within eight. Ten weeks maximum time in review if all the first group of referees agree; twelve if you need to ask a second group to replace those who decline. But I’m cautious about expanding the time if there is no guarantee of delivery. So this is the question about the ‘no more than two months’ maximum. Yes, if you know at the outset that it will appear in two months, no if that’s when you are forced to give up on this particular referee and start again with another referee. In other words, this proposal only improves things if all referees reply within that time, and do what they agree to within that time.

Elsevier are launching a new system of publication they are calling ‘article-based publishing’. Details here.

I’m sure most people have seen ‘online-first’, forthcoming papers, or similar systems. Society and Space has a page of just such papers here.

But the Elsevier system is somewhat different. They are not simply making an earlier version of the paper available, with temporary page numbers starting at 1. They are building the issue up as they go – assigning volume and issue number, and run-on pagination. The advantage of this is that they can get things up online early, as before, but with reference details that remain static for citation purposes.

But there are disadvantages. The key one is for the journal editor – any discretion over lineup of issues seems to be removed. The papers are ordered as they come off the production line, and so working out an effective running order for an issue is much more difficult, if not impossible. For a theme issue, the last thing finalised is often the introduction or guest editorial. But that’s the first thing that appears in the issue. Working on that sometimes suggests a different running order than originally planned. And even non-themed issues are often ordered in particular ways that are not simply first in, first out. Commentaries or editorials, sometimes on a very timely topic, come out quicker and appear first in an issue. Papers don’t always come out as the number of pages first expected, which might mean that the overall pagination of the issue is exceeded or light, and other things need to be juggled; some papers take longer through proof stage and have to slip down an issue, meaning something else needs to be moved up, but that may not work best as the last item, etc. Review essays and reviews are often used to make issues work to length, but this system would seem to make them more likely to be used in that way, rather than allowing flexibility.

It is clear that most people no longer read a journal as a print copy in a library or by subscription, but access papers online. They increasingly do this not through the journal contents page but through links, google scholar, etc. For this reason each piece in the journal needs to work as a standalone item, even if some pieces gain extra value and context when seen alongside others. Things like feeds, contents alerting, forthcoming sections, etc. all help with the visibility of individual pieces. But equally it still seems worthwhile to work on the overall shape of a single issue, and that requires much more flexibility than this new system would seem to allow. I spend a lot of time with a planning spreadsheet for future issues, and am continually moving things around and working with the publishers to try to make things work as well as possible.

In other words, don’t expect Society and Space to move to this model. But do keep checking the forthcoming page, or better yet subscribe to rss or email updates.