Tag: academia

Braindumping on Rejection

I’m on a PhD life advice roll. So, here we go.

Handling rejection is quite possibly the most horrible part of life in academia, whether as a grad student or a faculty member (I assume :D).

But, actually, is it?

I realized recently, that it’s really one of the most awesome parts of academia, even though it doesn’t always seem that way, when it happens.

I’ll explain why . . . with an anecdote!

I wrote a paper about a year ago, that was something I put my heart and soul into (and a notable amount of personal finances, but I digress). I submitted it — rejected.

I then tried to turn this same paper into my MS thesis. I presented it — edits requested.

It hurts, it really does, to have something that you worked hard on, that you feel proud of, and that you’ve spent a significant amount of time on, potentially at the expense of other things in your life.

And, oh, it did hurt. But, I realized I was thinking about the whole process of peer review the wrong way.

When we, or at least I, get rejected, the go-to response is often a feeling of worthlessness, frustration, or maybe even anger. But, you know what? In reality, rejection means some of the smartest people on the planet, with the most expertise in the exact area of my research, took the time to read my work and criticize the living $#!^ out of me. That might not sound like a privilege, but it is.

We have the privilege of playing in the big leagues, especially in when we swing for the fence. And sometimes, aiming for the fence means we miss. But, it also means that we have a whole panel of the most esteemed and qualified “coaches” (the program committee) telling us why we missed.

Submitting a publication, and ultimately getting a PhD, are not “gatekeeper” experiences. When we submit a publication, or we pursue a PhD, we basically get to sit in an astronomy class, taught by Neil deGrasse Tyson and Bill Nye, and ask if our ideas make sense. Even if they don’t, you receive feedback from two preeminent thinkers in the field — which is usually more like five preeminent thinkers, on a program committee. And with their feedback, you get to ask them, even if only briefly, to participate in your research, and improve it.

Similarly, getting a PhD is not about having three papers published as quickly as possible, stapling them, and riding into the sunset. It’s about improvement, and becoming the best researcher you can become, which involves a lot of struggle, criticism, and rejection. (At least, so far, it has for me. If not for you, share your secret. :D)

Back to my anecdote: I’m glad my paper was rejected. I’m glad I was asked to edit it. What it is now is much better than what it was, or what it could have been, without the feedback of the program committee and my thesis committee. And who I am now, as a researcher, is much better than who I was, or who I could have been, without the feedback of these same people. For instance, I now know WAY more about statistics, mainly because my anger drove me to dig deeper, and prove my reviewers wrong (which I did not: they’re experts, after all).

So, the next time you get rejected — which you will, and so will I — don’t be tempted to put down the bat. Accept that even negative feedback is an honor, take the steps to improve your swing, and walk back up to the plate. (Thanks, Jacob Sorber, for the baseball analogy.)

Braindumping on Literature Reviews

I’ve had a few questions about lit reviews in the past few days, and mostly I find myself repeating the same advice. As I’m currently (rather, I’m almost always) in the ongoing process of literature reviewing, I wanted to write a brief overview of how I personally go about systematic literature reviews, with some tips, to hopefully help anyone who is now, or will be, conducting a lit review. I want to note, most importantly, that this is how I do it. It’s not the only way, and perhaps not even the best way (though, I’m biased to think it is :D). The best advice is to find what works for you.

The direct answer for where to start, is that the most important place to look is prior years’ proceedings of the venue you plan to publish to.

OK, that’s the 5 second answer. But, I want to make sure we’re all aware of the lit review resources available to us, as computing researchers. Google Scholar is AWESOME, and hopefully we’re all familiar with it. But, if you haven’t checked out the ACM digital library, you should. It’s here: http://dl.acm.org/

The ACM digital library improves on Scholar in a few ways. One, it’s centered on ACM publications, meaning the results returned for your search terms are more likely to be applicable to our research — which is likely going to be an ACM publication. Two, it is easier to find the papers that are related within the venue where you’ll be publishing, meaning you’re less likely to leave a paper uncited that is well-known to (and potentially even written by) your reviewers. Put bluntly, there is no way to ever be sure you’ve cited every piece of related work, from every possible venue, big or small — it’s just too vast. But, the ACM DL will give you an excellent way of forming a good foundtion, and also being reasonably sure you touched on all the related work from the most relevant venues.

So, once I’ve collected papers that are clearly related, my next stop is the papers cited by my related papers (confusing sentence). What I mean is, if paper A is related to my work, what papers did A cite? Are those papers related? You can then branch continuously from this point, and collect more related work from those citations, too. If paper A cites B, C, and D, and B and C seem relevant, then I can check B and C’s citations for more related work. This is especially useful for identifying canonical papers (aka “must cites”). If A, B, and C, all cite the same paper E, it’s probably safe to assume E may be relevant, too.

Once I’ve searched the ACM DL, then I generally turn to Scholar. Scholar is useful, but if you’re working in an area that is even remotely interdisciplinary, it can return a high volume of papers from other disciplines, which may be less relevant than ACM publications. Moreover, it’s often difficult to determine the credibility of the venues of the papers returned in Scholar, whereas we have a general conception of the prestige/rigor of ACM venues. But, nevertheless, Scholar is a necessary stop on the literature review train. Choo choo.

The next stop, for me, is Scimago Journal Rankings, here: http://www.scimagojr.com/journalrank.php

The SJR lets you browse journals by area, and ranks them for you (which is not to imply the most relevant are the best ranked). Regardless of ranking, SJR gives an itemized list of computing-related journals. The titles are pretty informative, so it should be trivial to determine which journals are most likely to contain citeable papers. Warning: reviewing journals can be exhausting. Elsevier, a publisher of a huge number of journals, has a searchable database here: https://www.elsevier.com/catalog

It’s not quite as useful as some of the other search tools, but it’s better than nothing. There have been times, though, that searching the database was not useful. In which case, my recommendation is to find the most relevant journals, and review the articles within them, for at least this year and a few past years (5-10). Nobody expects you to cite a journal article from 1975, unless it’s a fundamental building block of all related work — in which case you should turn it up from your other review strategies.

Also, I’ve had recommended to me, that if you should find yourself working in an unusual area, with very limited related work, you can include your search metrics, and number of returned results as evidence of this limitation. Eg. “Our search on the ACM Digital Library for ‘Communist Cat Brain-Computer Interfaces’ returned no results. We cite this lack of related work as a potential limitation of our study.” HOWEVER, this is an extreme edge case, and has never happened to me. So, don’t mistake this for a recommendation, so much as a bit of knowledge I acquired in a systematic lit review class that I’m passing on, in case you are in this strange scenario. Note: if you find yourself researching communist cats using brain-computer interfaces, you may need to reconsider your project, anyway.

So, to summarize, the places I look for related work are, in order of importance (for me):
1. Past years’ proceedings of the target venue
2. ACM Digital Library
3. Google Scholar
4. Related journals in the past few years (5-10 years)
5. Papers cited by papers in 1-4

Happy hunting!