Cherry-picking things will lead you off the cliff
What to expect?
I have taken a great interest in reviewing research papers lately. I’ve even set reviewing them as one of my goals for this spring and summer so I have a good motivation for them. But truly reviewing articles and research papers is not an easy job. There is a lot of papers out there and most importantly, many of them give heterogenetic answers. One paper says that wool pants cure your back pain and one says that’s only because of the effect of strong placebo. So the question is, if research shows that a lot of things MIGHT work, what should I include as part of my own actions?
I will be graduating from this school with a bachelor’s degree in physiotheraphy and there has NOT been a single course of how to critically appraise an article. We end up doing our bachelors thesis without a single advice on how to ACTUALLY BE CRITICAL and what to actually look in a research paper. I’m making an anecdotal guess here, but most of the bachelor’s thesis that get published are going to be filled with biased looks on their own research questions. People go to Pubmed and take the first possible papers that looks good for their own thesis agenda and present it as the “absolute truth”. More often than not these also offer no bigger picture look on the subject. In their papers there is no counter-papers of how their research may actually be false.
But I have a feeling. That this is not because they don’t find these kind of papers. I feel like it’s more about fear. Fear of your thesis not answering your research questions the way you wanted it to. I feel like people don’t realize that bachelor’s thesis is not supposed to answer your questions the way you thought it was going to, but to show your own work of the subject, whether or not the results were what you hypothesized. It’s about doing a good research and thesis. To do that, you should always try not to be biased on your own agenda.
I myself also find this truly hard. As far as I have gathered information about the subject, it takes time to get good at it and not having a single lecture about doing critical appraise, we are already behind. So here is where I hope my essay will help you.
I have gathered few key points from researchers that push forward the behavior of being critical. Here are few good advices I took from blog of Adam Meakins, who does considerable amount of critical appraise of research papers as part of his work.
- Use papers to guide your actions, but make sure you have actually read a lot of papers.
- Use papers as illumination, not as support
- Making good researches is hard, so be critical about what you have just read
- Just because it’s on a paper doesn’t mean that’s the absolute truth, it’s just more probable
So what do they mean?
1 Use papers to guide your actions, but make sure you have actually read a lot of papers.
The first advice is straight from the intro I wrote. If you only cherry pick papers that agree on your own vision and agenda, you’ll end up being blind to the big picture. I have done this in the past as well and I see the problem in it now. If you or I base “what we know” on only few papers then we actually know only few papers worth about the subject. We don’t know the whole scale of possibilities that there are and we might end up even doing harm for our project and actions. When you do research for a project or for your own papers, thesis or master’s thesis, read a lot of papers so you can see the big picture.
2 Use papers as illumination, not as support. Like I earlier said, if you only cherry pick papers that agree on your own vision and agenda, you’ll end up being blind to the pick picture. More often than not people use papers to support their own views and when someone tries to argue them about their beliefs, people get defensive. I have heard many times “research shows that…” as a counter-argument and many times I have the exact same counter-counter-argument “actually research shows that…”. And we end up in a stalemate. Both of us have cherry picked our own views and unless both are up for good dialog, there will be no conclusion. What I want to bring you with this, is the idea that papers can help guide you towards something bigger, but then again, they may lead you towards the end of a cliff. Use papers as illumination, to guide your own views of the bigger picture and to see what out there may be.
3 Making good research is hard, so be critical about what you have just read. “Research is a tool, and like any tool, it’s only as good as the person using it” (A. Meakins. 2019). For general public, that I consider myself part of as well, don’t really have good knowledge of scientific literacy and understanding. If we don’t practice how to critically appraise an article, we will never get out of the pitfall of not thinking critically. There may be many factors in a paper that critically lower its validity, but do we actually notice it? Without a doubt we don’t, especially if we chose that paper because it looked good for my agenda.
When people do research there are a lot of variables that need to be addressed. Researchers don’t always address all the variables and may also end up showing up results that may not be true. If you are not aware of this and be critical, you may end up also be spreading this nonsense. I recently saw a post in a Facebook group, that consists of doctors and physiotherapist and the poster asked if “Thing A is relevant after exercise”. This caught my eye, because a friend of mine answered at the Facebook group post, that because of their work in their bachelor’s thesis, “A is not relevant”. Well first thing came to my mind was, well how can you be sure about it? Because I knew their research consist of only six candidates. What didn’t work for these six candidates may actually end up working for the rest of the population.
Well what is the point in this? Take pride for what you have accomplished, but be critical of your own work too. Don’t show it up as absolute truth.
4 Just because it’s on a paper doesn’t mean that’s the absolute truth, it’s just more probable. Evidence and results from research won’t tell you what’s absolutely true or wrong. It doesn’t tell you what to do or not to do. It tells you what’s more probable, more less or likely true. Just because this one paper told you that doing pushups helps your shoulder pain, doesn’t mean that it absolutely will. Just because some paper told us that “this is the best way to brand your company” doesn’t mean that it actually is. These are guidelines that should help you, but if you don’t think with your own brain, you may end up missing a great chance, just because it wasn’t in the paper.
If doing pushups make your shoulder hurt even more, you should do something else. Don’t try to force your own agenda to someone just because the research told you that it’s true. Telling someone they should use a car for travelling because it’s the fastest way may be true, but if the other person doesn’t have a car it’s not true in their situation.
I have really only talked about appraising papers, but all of these advices should also be utilized when reading books and also in general conversations. People are genuinely adamant about their opinions and changing those opinions are even harder. The more of an expert they are of a certain area the more challenging their opinion will be. But be open minded when trying to tell someone their opinion is wrong, because it actually might be you, who is in the wrong here. Only way to know what the other person means, is by listening to them, understanding them and when you understand them. That’s where you will find what is wrong in their perception and that’s where you want to go for with your arguments.
Conclusion of sorts
So what do I really want you to get from these four advices? Well I hope you managed to grab the main point and the advice number 3 really crystallizes it, which is always to be critical and question what you have read or heard. Don’t believe things just because someone told you “that is how it is”. Do your own research and when you do, be critical. Try to find things in the papers that may make it less valid so you don’t walk into pitfall. Read papers when you can, especially if you are “evidence-based” kind of worker. You need to be critical to actually get better at being critical.
A lot of the things I have told you are truism, but they still need to be said. To help you little further here’s an article that gives nice guidelines into critically appraising articles; if you want to travel deeper into the rabbits hole.
Young JM. Solomon MJ. 2009. How to critically appraise an article.
Meakins A. 2019. Three dangerous words. Blog.