Published February 2, 2021
In this post I write down some of my thoughts on how we currently organize scientific knowledge (with a focus on anatomy and physiology) and how this might be improved. I don’t have all the answers but I wanted to get my thoughts down “on paper”.
I apologize in advance for the length; this is a topic I’m very passionate about. I truly hope this article resonates with at least one or two of my one or two readers!
When I first started as a postdoc I was new to the wonderful world of urology. My first set of experiments involved testing electrical stimulation in the peripheral nerves that innervated the lower urinary tract (bladder & urethra). Being new to the anatomy I figured I would read some papers that explained the nerves involved, how they coursed from the spinal cord to the periphery (so I could manipulate them), and other important details that might hint at how activating these nerves artificially (via electrical stimulation) would influence bladder and urethral function.
So I read some papers; I read a lot of papers. I found some decent papers and some really interesting tidbits, but overall I was confused and found it difficult to synthesize and understand the information I had read. It was 2014 and I was surprised that there wasn’t a paper that summarized the information I was looking for. For a while I considered writing my own review paper, but there were three big issues in my way.
First, good review papers are criminally undervalued. Despite the obvious need for clear and precise reviews of what we know – and by extension, what we don’t know – review papers have been delegated to a lower class of papers that are not viewed as being as valuable as original research. Many reviews are also of poor quality. Whether reviews are poor because they are undervalued, or whether generally poor reviews leads to them being undervalued is unclear. Regardless, there was no way I could justify investing the amount of effort it would take to write a good review paper on the topic. I needed to do original research!!
Second, review papers are subject to the limits of current publishing models. Want to write a paper that is 2-3 paragraphs on a very specific topic? Good luck getting that published. Instead you’ll need to incorporate it into a larger group of topics. Unfortunately this requires more work than just writing a few paragraphs! Want to include a lot of detail? Too bad, word limits. No one wants to read endless rambling, but if you are covering a lot of topics, then word limits seriously start to impact your ability to write thoroughly about each sub-topic. And then there is one of my biggest pet peeves, limits to the number of citations a paper may include. I’ve never understood this limit. I can understand having recommended limits, but hard limits just seem to encourage plagiarism and making up information that is not backed up with a citation to actual evidence. If I had a nickel for every time I read some amazing claim with no reference …. Come on reviewers!!
Finally, reviews have a fundamental flaw that I don’t think can be addressed with our current publishing model – our knowledge is constantly being updated and revised, but (most?) reviews are not. In our field, the king of basic science research is Dr. Chet de Groat. At the time of this writing Dr. de Groat has just recently retired. However during his career it seemed like every year he would publish another big review on the lower urinary tract (it was probably more like every 2-3 years). These reviews were generally pretty good but they weren’t perfect (sorry Chet). Additionally, despite the presence of newer reviews by de Groat himself, old versions of his reviews still get cited. Thus even if our scientific understanding is updated and manages to make its way into a newer review, people are still being pointed to outdated versions, which again, weren’t perfect even when they were written, much less so now.
After starting to write a review paper I realized I didn’t want to write a review paper, I wanted to create something else.
At this point it may be helpful to clarify what exactly I was trying to write. I wanted something that described how to isolate and dissect the peripheral nerves of the lower urinary tract in a rat. Doing this however required also knowing and being able to describe the rat vasculature system, the bones of the rat, and the muscles.
Identifying the nerve is just the start. Nerves may change over their length as branches split off. There are structural considerations too for interfacing with the nerves. Certain places are easier to instrument along the nerve due to surrounding support structures or crossing of blood vessels.
And that is just nerve access and interfacing. But what about the composition of the nerve? How large is the nerve? How does the size vary as a function of age? What types of neurons does it contain? How large are those neurons? Where do these neurons start and where do they end? What do we already know about stimulating these nerves? How does that change if we transect the nerves first or administer a drug systemically? Answers to some of these questions exist, scattered in tens or hundreds of manuscripts, not collated nicely but further scattered amongst the millions of existing biomedical papers.
The fundamental problem I’ve run into with anatomy and physiology is that so much of it matters. You can try and draw your moat around what you think matters, but inevitably you’ll find yourself wishing to expand. You may hope that there will be a relevant review that answers all of your questions, but often that’s not the case. Thus your stuck searching the tens of millions of biomedical papers that currently exist for knowledge.
Our approach to gathering knowledge from literature reviews is outdated and inefficient. I have many thoughts on how to improve the situation, but that’s a topic for another time. Instead, I want to focus on how we might create good “reviews.”
Essentially, I want a “Wikipedia” for the scientific literature. Wikipedia itself would almost work, but there are a few issues I see. First, my instinct is that editing such content should require slightly more verification than Wikipedia provides. I tend to be in favor of trying to let as many people participate in science as possible, but I think in this case there may be a need to distinguish “researchers” (loosely constrained) from completely open access. Second, Wikipedia itself is generally shunned in the scientific community. Despite I suspect many scientists using Wikipedia to brush up on or learn areas that they are not knowledgable in, I’ve yet to see anyone use Wikipedia in a journal citation. Third, although Wikipedia tracks edits and has versions, it is not clear to me if these versions are sufficiently distinct as to be worth citing (but perhaps here I am getting ahead of myself, more on this later).
So, what do I want?
I want a place where small focused articles can be created, even though they are super specific and may only interest a handful of people. For example, there is evidence that the pudendal nerve sends neurons towards pelvic ganglia in many species. As far as I know the function of these neurons is unknown. A small article might summarize the evidence of these connections and identify that the function of these connections is unknown.
Articles should contain links to other topics with more information on keywords. For example, if I am describing the pudendal nerve in the above example article, you should be able to follow a link to more in depth explanation of the pudendal nerve. Again, this is very Wikipedia like (or like the internet, as opposed to a book where these links are more difficult to execute).
As new information is collected, articles can be updated. This was one of the primary motivators for this approach. Let’s say Dr. de Groat has written a marvelous review but it misses out on one key bit of information that a new lab has just discovered. Rather than writing a completely new review, the lab (or anyone) should be able to add the information to the existing review. I see this as a move towards team science, with everyone contributing their bits and pieces of the knowledge towards the general collective.
These articles should not be the sole source of scientific observations. Instead, all text with evidence should cite published articles. Writing these “reviews” may lead to some scientific understanding or clarification, but ideally any new insights generated would get published as standard journal articles. This would also extend to digital assets (images, videos) which would ideally either be published in papers or archived elsewhere and then duplicated in these articles. Bonus: You should be able to enter a published paper into a search interface for the review web site and get information on all the times that paper is cited in this “review universe.” If it isn’t, and that paper says something of interest/relevance (hopefully true, but maybe not always??), than one or more reviews are missing information.
Hopefully at this point you understand roughly what I’m going for and why I don’t think reviews are sufficient to meet my goals. Additionally, this may sound a lot like Wikipedia … but not Wikipedia.
With that in mind here are some questions that need answers:
Incentives How could we incentivize contribution to such a resource? As I mentioned earlier, reviews don’t count for much. There have been efforts to move beyond standard metrics, such as tracking of peer-review effort, tweets of papers, blog posts (either about a paper or as a unique contribution in and of itself), etc., but my impression is that these alternative metrics (altmetrics) currently count for very little and that there is little sign of that changing.
One potential approach is just to start doing it. I think in the long term it would be better to recognize that synthesizing information is valuable. Without that however, sometimes you just need to do things for yourself. If others follow, that’s great. If not, at least you have something you like and appreciate. I may eventually do that myself but there are also potentially non-trivial infrastructure issues to work out.
Also, it is unclear how to give credit for group efforts. Number of words written? Number of articles created? Again, I’m not really a fan of trying to create metrics but I don’t have an alternative either. I could see paying people to do this job, but with biomedical funding being relatively tight, is there the will to do so?
Editors How do we ensure that the information that gets added is accurate? For normal papers there is a peer review system that attempts to help ensure accuracy and quality. It’s not clear to me if such a system would work here. Perhaps we could have information that gets added but that isn’t verified, and eventually it gets marked as verified if a pre-selected group of people look over it and say it is OK. I’m not sure of the right answer here and my guess is that other people have thought about this problem more than I have and have solutions. Similarly, some knowledge points are unclear and there isn’t necessarily a single correct answer as much as many potential answers. Having a good way of identifying this ambiguity up front may be useful.
Comments It seems like the framework should support comments so that people can make public notes without needing to actually modify the text itself. Wikipedia has a separate comments page for each content page. It is unclear to me if that is sufficient or not. Something like “tracked changes” in Word would also be useful. However, I think it may be better if the comments could somehow be detached from the words they were originally attached to so that they aren’t accidentally deleted if some minor editing is done. It may also be useful to be able to silently mark other peoples’ comments as no longer relevant, or provide other private feedback, so that the original commenter (or a moderator?) can remove the stale comment.
Organization Organizing information is always tricky. It is not clear to me if there are organization (naming) principles that should be utilized for individual articles. Here my instinct is to start organically, focusing first on getting content and then later on organizing if necessary. It may be beneficial to create articles that are solely focused on creating navigation links (again, I think this is a Wikipedia thing). The big issues I am concerned about here are duplicative efforts and content discovery.
Partial Information Writing a thoughtful and complete article is a lot of work. If instead people are able to write only a few sentences for an article or even just start with bullet points, that should make it easier to get content added. I don’t know if there needs to be any delineation between pages that are essentially notes and those that have been flushed out more. Once a page has been flushed out, can additional information be added as loose notes to the page? Perhaps this is where similar to comments, a secondary page is needed that is only for content notes.
Citing and Versioning Finally, I’m not sure whether these “articles” should support being cited. My guess here is yes, but that version numbers need to be included with every edit. Presumably being able to restore any particular version is critical.
While writing this I was reminded of Scholarpedia. Scholarpedia seems to be close to something I would like, although not exactly. A few thoughts:
I wrote everything up to this paragraph many months ago (minus some edits) but I was just not able to find the time or effort to finish. Part of this was due to a job transition. But also, I just don’t really know a way forward, so my ending here sort of just … ends. I’m convinced that what I call “knowledge management” is critical, and yet, like software development (in an academic setting), there seems to be little support for something so important. I can’t count how many times I’ve heard things said at conferences that were just wrong or out of date. I don’t blame the people saying these things, I don’t know how anyone is supposed to actually read the vast amount of scientific literature out there. And with everyone and their mom needing to publish papers for graduation or promotion, the problem is only getting worse. Importantly, I think this also means we are currently doing research with incomplete information. We conduct experiments that fail to properly build on, or take advantage of, what others have done. Subsequently, new discoveries are slowed simply because we’re not aware of what we already know.
As I alluded to earlier, for many years now I’ve been thinking of ways to make it easier for people to understand and manage the vast amount of scientific literature that we are now generating. Efforts in that area are coming along slowly but I hope in the near future to not only start sharing some of those “nights and weekends” efforts but to start getting paid to work in that area. In this related area of scientific consensus building and community reviews, I have nothing. If you, dear reader, have some suggestions, I’m all ears.