September 13, 2013

My paper's journey through the review process

Plus some reflections

In a previous blog post I mentioned the perils of big data. I can also use the paper from that post to talk about the publication process.  Today I received an email that this paper has just been published at International Studies Quarterly.  This is a very good journal and I am very excited to have people read this paper. 

I had a little twitter exchange about this paper so I very quickly wrote up a little post.  Sorry if this is a bit half assed, but I am on leave with my 3 month old son.  I have very little time to do work…and even less to blog. And it is a Friday night.  Shooting from the hip.

But here is a quick rundown of my paper.

My paper won an award for the best paper presented in the American Political Science Association Political Economy section 2008 and was first sent to the American Political Science Review on January 10, 2008.  It was finally published in ISQ on September 13, 2013.

What took so long?  It was rejected by multiple journals, often for very fair criticisms.  But this paper, more than many other papers, had a bunch of near misses.  

Here is my recollection (with the aid of emails) of the publication process:

Submitted APSR 1/20/08.  Rejected 5/13/08

Submitted IO 10/14/2008, Rejected 1/06/09

Submitted JOP 7/11/09, Rejected 11/23/09

Submitted AJPS  6/19/10, R&R 12/?/10, Rejected 8/19/2011

Submitted ISQ 9/30/11, R&R on 2/24/12, resubmitted 6/6/12

Accepted: 6/22/12

I swear I had another R&R in there, but I might have this mixed up with another paper that shared a similar fate. I can think of at least five papers that have gone through more than five journals.

In my previous post I mentioned that this was a very labor-intensive project.  It required me to apply for a special access to the data data, obtain a security clearance, and then fly to Washington DC to do the analysis, and then to do more analysis for each reviewer.  So in between each one of these rejects are a series of flights to Washington DC, analysis, rewriting and finally resubmitting. 

I finally quit on future projects with this data when we had my first child two years ago.

It was a pretty painful experience.

I’m a tenured professor at a great institution (WashU) and I’m moving as a tenured professor to another great institution (GWU).  If anything, I think I have caught a lot of breaks in my career and have encountered lots of reviewers that gave me more constructive feedback than I could have ever expected.  I have few complaints about how things have worked out for me.

But I think the point is that you can often look at the CV of a person and you only see the published papers and don’t see any of the failures along the way.  This paper is definitely on the long side from writing to publication, but the process rarely is quick and clean.  Here are a few quick facts.

  • My most cited paper (a 2003 International Organization paper) went through 3 rounds of Revise and Resubmit.
  • I have submitted three career NSF grants and one cross-division grant that were rejects.  I think I’m at seven or eight other rejected regular grants. NSF 12 Nate 0
  • In the last two weeks I received two rejections from the American Political Science Review.  I have also received two Reject and Resubmits from the APSR in the last three years.  Both of these were rejected on the second go round as well.  Not sure of my total APSR rejects, but I know the accepts are zero.
  • I have publish over 25 papers and I can only think of three that hit on the first submission.  I would guess that the modal number of journals before publication is three.  But that is a guess.

These are stories from mostly my own papers.  I’ve had some other crazy experiences with my co-authored papers.  This includes what seemed to be something like six rounds of R&R with a journal, a three line review after 9 month of waiting (basically saying that they didn’t like this kind of work), and a few projects that simply died in the publication process.

What are my own person conclusions from this process?

1. Editors have a lot of power, but they tend to be fair.

In my many, many rejected publications, I have only complained to the editor once.  This was a case where a reviewer basically vetoed the paper saying that they didn’t like it, but without any real explanation.  This was the pivotal review and my paper was rejected.  I complained to the editor  and the editor agreed to go out for a new review.

But other than that, I have found that most editors and reviewers are fair.  I often disagree with some of the points in a review, but I can’t think of a case where an editor has unfairly read the reviews and rejected my manuscript.  It is often a judgment call, and sometimes it has gone my way.  Probably the majority of times it went the other direction. But I can't call it unfair.

I actually have a lot of sympathy for editors given the volume of submissions, the difficulty of getting good reviewers, and the complexity of making a decision of accept, reject, or R&R on some many difficult cases.

2. Sloppiness kills

As I noted above, I think most reviewers do their job.  But you can really, really piss off a reviewer in the first few pages.  This can be a typo, an incorrect reference, or a flashy intro that oversells your contribution.  There are lots of ways to set off a reviewer, and I can’t say for certain if this is why a review turns nasty, but I can say it is strongly correlated.  Ironically, this post probably has a bunch of typos.  Not peer reviewed.

3. Work on multiple projects…

I would often ask senior job candidates and speaker series visitors about their most famous projects.  Did they think it was their best work?  In general, can they gauge where a piece will land?  Most of these people were surprised what stuck and what didn’t. 

This isn’t to say that quality isn’t correlated with where a paper lands, but there is a lot of uncertainty even with the strongest paper.  If it gets published, does it get cited?  I think this is even more of a mystery.

My personal response to this uncertainty is to diversify.  This also fits my own personality and interests.  I love working on and finishing projects, often with co-authors.  Obviously this isn’t a strategy that works for all types of research.

4.But…

There is some tradeoff between quantity and quality.  This seems so obvious that it isn't worth mentioning. But, I’m not sure how much better my work would be if I cut the number of papers I write in half.  There are tradeoffs you have to make, but remember that even the most famous people have trouble telling ex ante what is their best work.  Make sure you have a real diamond that has been vetted by others if you’re going to polish and polish and polish. This is especially important for grad students who want to have a great project.  A great project doesn't emerge from sitting in a room working and working and working.  It comes from writing, feedback, failure and trying again.  

5. And…

It can be hard to manage workflow when you have a lot of papers floating around. This gets even more complicated with co-authors that are also juggling multiple projects.  It is very easy to ignore a project and let it sit for a year while you work on other projects.  For some people it might be better to work on sequential projects finishing one and then moving on to the next.

I work on multiple projects at the same time.  Mostly because I enjoy it, but I also have found some ways to really manage my workflow.  The most obvious of these is to make sure to get projects done and out the door.  The other is to make sure to make the best use of “downtime”.  When I have some free work hours, I write. 

6. Most importantly…

Take reviewing manuscripts seriously.  Like most senior scholars, I get 20-40 manuscripts to review a year.  Add to this tenure reviews, papers by grad students, conference discussant duties and I’m sure I end up commenting on at least one research paper per week of the year.  Maybe two.

But these reviews really matter.  In a few cases I have taken to complaining to editors if I think another one of the reviewers has been unfair to a paper that I reviewed.  

I’m not sure if these complaints matter and I am as guilty as anyone of some sloppy reviews.  But reviews matter.

This is a reminder to myself.  Reviews matter.

DISCLAIMER:  This is a blog post I whipped up in an hour or so.  It is rough and probably make some pretty stupid points.  I am tempted to delete it and not post it until I have a more polished version and some clearer thoughts.  But I have done this a few other times, only to never get back to the post.  So I am taking my own advice.  I'm going to throw a few ideas out there and see what sticks.  See what people say about them, and use this feedback to revise my own thoughts.  The only advice I didn't take is the one on sloppiness.  Sorry for all of the typos Mom.

 

 

Comments

This is depressing. I teach at an LAC and regularly have a 4/4 and sometimes 5/4 teaching load along with being chair of a multi-disciplinary department. If this is what it takes, I don't have a chance of getting published in a decent journal. There just isn't enough time in the day to fulfill all of my responsibilities and then produce something of publishable quality.

Thanks for sharing your experience. Some times a blog post is more effective than a reviewed paper transmitting ideas. Saludos

It sticks, all of it! An excellent post, some great insights on success (must work butt off), and very useful advice. I really enjoyed reading it. Many thanks for sharing thoughts and advice. Best, Soo Yeon

Very sensible post, Nate, and very honest. Thanks so much. Your conclusions should be required reading for graduate students and tenure-track assistant professors.
If it's any consolation, I had a paper published last October in JCR on which we started to work in June 2006. All the more important to several irons in the fire at any given time.

Thanks for this - a breath of fresh air. I mean that in terms of underscoring the reality of publication and the behavior of editors (I've been one in the past). I'll pass this along to our graduate students.

A note on doing reviews: The paper whose history you detail took up (depending on whether you count readers on an R&R) between 12 and 15 referees. If we were to adopt double entry bookkeeping that means a reasonably high deficit to be covered by your own willingness to referee. And that is on a single paper.

I am just about to submit my first single author paper for publication and your blog post has braced me for the eventuality. I have one professor that loves to tell the story of a paper of his that was held for review for a year. He sent it a birthday card saying something to the effect of "you have been out in the world for a year now and I just wanted to check in and see how you were fairing." He received notice 3 weeks later that his paper had been selected for publication.

It seems that publishing is kind of like fishing. You have to cast a lot of times, sometimes over and over in the same spot, but hopefully you will get a bite eventually. Thank you for your post.

Really nice post (and judging how many people posted it on Facebook, seems like lots of people agree). I forwarded it on to my graduate scope and methods course. It's good for people to see the long path some papers take (in that class I always talk about how my public opinion and war paper got rejected three times -- often quite harshly -- before it got accepted. and even when it got accepted it just barely made it through. and its on track to be the most cited thing I'll ever write). And the advice at the end is spot on. thanks so much for sharing.

I do too and it is a hard thing to get things out the door. I've had several publications, but only one in a top-flight journal. Here's what I do:

1. I write on topics that others don't. One thing that can distinguish your work is that it's on something that's important, but that not too many people work on. That sounds impossible, but it isn't. For instance, I've done a lot of work on judicial administration. Important? Yes. Lots of work being done on it by political scientists (in contrast to pub ad types)? No.

2. I do lots of targeting research when I'm ready to submit. I look around for journals that publish the kinds of work I do and send it to them. With planned backups. This limits the impact of the research to more specialized groups of scholars, but it amplifies your chances and saves time.

3. Do research on teaching and learning. My one publication in a widely recognized journal was on a new teaching method. You're going to try new things anyhow; may as well write them up.

4. Try the occasional book. I had to wait around for the US to start torturing people to find a topic that moved me enough to want to write a book. It takes something you really care about to get you to put aside the time. NB also that this is related to point 1.

5. Don't volunteer to be head of a multi-disciplinary department! I know you probably didn't have much choice, but ditch that responsibility asap!

Hope this helps.

Good article that, along with the comments above, sums up neatly the sheer insanity of academic publication and tenure. Fortunately, progress does not depend on academic journals, more conference presentations and informal communications. Unfortunately, academic systems that rate academics based on publication scores don't know this yet.

It's amazing that it takes 5 years for an award winning article to get published in a good journal. I wonder if you can skip the 'rejects' stages. What would happen if you submitted the paper the first time to the one you finally got the paper published? I think there are a few factors that play roles in the publication process:

1. Selection of a journal.
2. Reviewers' credentials, attitudes, likes and dislikes
3. Whehter reviewers' comments are fair and most importantly whether you can implement their comments in the revised version. Sometimes I find the revision based on the reviewers' comments are a mission impossible within a short period of time.
4. You've got to have faith in your paper and try it again and again till the paper finds a home.

I've been struggling to get two papers published for two years. Some of the criticisms have been reasonable, but others have been absolutely baseless. The peer view process is what separates actual science and just rhetoric or opinion, but it has become more volatile and it is strongly dependent on the reviewer. We can only do the work and just keep trying. While impact factors are becoming overly important, keep in mind that even a low impact pub is better than an unpublished computer file!

Thanks for some of the insights.

As an author, reviewer and editor (in that order) I want to thank you for this frank discussion. The review process is needed but sometimes I feel very cynical about the whole process - it seems almost like the luck of the draw.
Also, I submitted an article last night to a third journal (long story) and I think this article is better than other articles I have written that went through the process without much fuss (not that the latter is a regular occurrence).

This is a fair summary and I appreciate your frankness (and of course could provide very similar stories of my own, though I'm not as patient). Still, I'd flip this around and conclude that this is yet more evidence that the journal model, *as a means of actual scholarly communication* is seriously broken (as a means of allocating ever-diminishing positions in a pecking order, it works fine).

So, we've got five years from the initial research to publication. Let's pretend that such journals were the only means of communication (obviously they aren't) then the first paper that had gone through a similar process which *cited* this work would be available a full ten years after the original research. Ten years.

In contrast, I'm at the Conflict Research Society meetings and just gave a 90-minute workshop on a dataset (gdelt.utdallas.edu) that has been available to the public only since March 2013 and has already accumulated such a large user community that I wasn't able to cover everything in that time. But GDELT is moving at "internet time" due to open collaboration across the web, and things happen in days (or at times, hours), not years. There are downsides as well -- more than one visualization has gone viral in both content and misinterpretation -- but shouldn't there be some mode of professional communication that wastes less time and effort?

Really good post. I didn't know how much you had made efforts to publish this paper, but you should know that your paper inspired readers like me to think of corporate tax issues seriously.

I'm with you Robert. I'm at a teaching university and with my teaching and service load, finding time to write is difficult. I have one article out for review and I'm working on a couple of others. Honestly, this blog is frightening.

Super cool and interesting post! You really got my brain goin on that one! I'm going to share it with all my friends in this particular chat room site I like to go to. The site is adult 4 sex chat if you're wondering. It's super duper fun and cool just like your magnificient post.

Sorry I am so slow to respond to this. On leave with the baby.  But your point on the referee deficit is an excellent one!

Your example on GDELT is a great one.  I wish there was an easier way to "weight" this sort of contribution for junior faculty.

Add new comment