Braindumping on Literature Reviews

I’ve had a few questions about lit reviews in the past few days, and mostly I find myself repeating the same advice. As I’m currently (rather, I’m almost always) in the ongoing process of literature reviewing, I wanted to write a brief overview of how I personally go about systematic literature reviews, with some tips, to hopefully help anyone who is now, or will be, conducting a lit review. I want to note, most importantly, that this is how I do it. It’s not the only way, and perhaps not even the best way (though, I’m biased to think it is :D). The best advice is to find what works for you.

The direct answer for where to start, is that the most important place to look is prior years’ proceedings of the venue you plan to publish to.

OK, that’s the 5 second answer. But, I want to make sure we’re all aware of the lit review resources available to us, as computing researchers. Google Scholar is AWESOME, and hopefully we’re all familiar with it. But, if you haven’t checked out the ACM digital library, you should. It’s here:

The ACM digital library improves on Scholar in a few ways. One, it’s centered on ACM publications, meaning the results returned for your search terms are more likely to be applicable to our research — which is likely going to be an ACM publication. Two, it is easier to find the papers that are related within the venue where you’ll be publishing, meaning you’re less likely to leave a paper uncited that is well-known to (and potentially even written by) your reviewers. Put bluntly, there is no way to ever be sure you’ve cited every piece of related work, from every possible venue, big or small — it’s just too vast. But, the ACM DL will give you an excellent way of forming a good foundtion, and also being reasonably sure you touched on all the related work from the most relevant venues.

So, once I’ve collected papers that are clearly related, my next stop is the papers cited by my related papers (confusing sentence). What I mean is, if paper A is related to my work, what papers did A cite? Are those papers related? You can then branch continuously from this point, and collect more related work from those citations, too. If paper A cites B, C, and D, and B and C seem relevant, then I can check B and C’s citations for more related work. This is especially useful for identifying canonical papers (aka “must cites”). If A, B, and C, all cite the same paper E, it’s probably safe to assume E may be relevant, too.

Once I’ve searched the ACM DL, then I generally turn to Scholar. Scholar is useful, but if you’re working in an area that is even remotely interdisciplinary, it can return a high volume of papers from other disciplines, which may be less relevant than ACM publications. Moreover, it’s often difficult to determine the credibility of the venues of the papers returned in Scholar, whereas we have a general conception of the prestige/rigor of ACM venues. But, nevertheless, Scholar is a necessary stop on the literature review train. Choo choo.

The next stop, for me, is Scimago Journal Rankings, here:

The SJR lets you browse journals by area, and ranks them for you (which is not to imply the most relevant are the best ranked). Regardless of ranking, SJR gives an itemized list of computing-related journals. The titles are pretty informative, so it should be trivial to determine which journals are most likely to contain citeable papers. Warning: reviewing journals can be exhausting. Elsevier, a publisher of a huge number of journals, has a searchable database here:

It’s not quite as useful as some of the other search tools, but it’s better than nothing. There have been times, though, that searching the database was not useful. In which case, my recommendation is to find the most relevant journals, and review the articles within them, for at least this year and a few past years (5-10). Nobody expects you to cite a journal article from 1975, unless it’s a fundamental building block of all related work — in which case you should turn it up from your other review strategies.

Also, I’ve had recommended to me, that if you should find yourself working in an unusual area, with very limited related work, you can include your search metrics, and number of returned results as evidence of this limitation. Eg. “Our search on the ACM Digital Library for ‘Communist Cat Brain-Computer Interfaces’ returned no results. We cite this lack of related work as a potential limitation of our study.” HOWEVER, this is an extreme edge case, and has never happened to me. So, don’t mistake this for a recommendation, so much as a bit of knowledge I acquired in a systematic lit review class that I’m passing on, in case you are in this strange scenario. Note: if you find yourself researching communist cats using brain-computer interfaces, you may need to reconsider your project, anyway.

So, to summarize, the places I look for related work are, in order of importance (for me):
1. Past years’ proceedings of the target venue
2. ACM Digital Library
3. Google Scholar
4. Related journals in the past few years (5-10 years)
5. Papers cited by papers in 1-4

Happy hunting!