This is such good information, and should be bound in hardback and then used to smack in the head every online publisher that publishes junk science under the guise of news. I learned a bunch of this when I was studying library science, and I STILL get caught out every so often.
This was great Josh. I have so much to say about this one (anaesthetist, dabbling in academia ie I have research interests but no formal higher degree apart from Masters of Public Health). Interpreting scientific literature is so fucking complex - which basically goes to show how unqualified/ anti-qualified/ negatively-qualified all those “experts” on the Covid vaccine (or whatever) were. A massive bug bear of mine is that scientific journals currently operate on a shitty business model - fundamentally they make money... which is always a conflict of interest whether you like it or not. So much RUBBISH gets published. It is almost impossible to tell what is published rubbish unless you have very good knowledge of research methodology. The stats part is huge of this discussion is huge. Even if you look at massive clinical trials (thousands) so many results might be statistically significant (reliable) but are they clinically significant? The study abstract might say “New meditation technique significant increases your time asleep.” Ok... great! But in the actual article you find out that it statistically significantly increases it by 3 minutes. Even statistical significance (p-value, traditionally defined as <0.05) is a whole discussion in itself. It’s a made up value! Journals will publish an article with a p-value of 0.049 but not 0.051. The ACTUAL difference between these studies is... nothing.
Anyway I’m RAMBLING. My summary is that it is really complicated.
Oh PS - I occasionally review articles for journals. If something gets published and there is a TYPO in the heading... you can safely assume the peer review process has been shite. Another red flag.
That did invoke a cold sweat recall of university days, but truth be told I enjoyed the hunt if not the citations. The Listener cover seems to have become a rolling maul of "wellness" topics in recent times. I shall be reading in the latest edition how breathing techniques "will change my life". I find that in and out is a pretty reliable one for sustaining it? It becomes overwhelming as the list of things that need improving continues to grow ^^
Re the surveys point: some data can ONLY be gathered through surveys.
The reason I'm pro surveys: gender should always be self reported as the gold standard. The best way to know someone's gender is to *ask them*. However, a lot of studies do things like automate gender ID based on first name (culturally insensitive, also good luck being called Alex), get an interviewer to guess (this is in some WHO guidelines!), get a random stranger to guess (the "who asks questions at conferences" papers do this), or use medical records without considering the wildly different standards for recording what someone's gender is across jurisdictions.
Kia ora Josh, this is a great overview but skews a bit too much towards the hard sciences and academia IMO - something that could be caveated. I believe that you can very much do *proper* science outside of academia, whilst being industry-funded, and focusing on qualitative insights with a small number of targeted audiences. That also means it's likely there won't be statistics, or limited ones (if you try to apply stats at all cost to qual social science, you can get into the "lies, damn lies and statistics" territory very quickly). My research, for example, is based on almost 20 years professional experience post-PhD (which was on a completely different topic and discipline, more akin to the hard science approach you talk about here), in a variety of sectors, with a variety of stakeholders, and on a variety of target audiences. Because I research human behaviour and energy (in)justice, a lot of the aspects you ascribe to "pure science" really don't apply - it's a highly complex area of research that requires socio-ecological approaches, which in themselves often fall outside of quantitative inquiry and stupid "gold standards" such as randomised control trials. One thing I do definitely agree with though, is to run away from economists - and that includes a lot of behavioural economists who have taken over my field. To me, they are usually represented by the typical patriarchic, Eurocentric, neoliberal, overprivileged fools that give STEM (which economics definitely isn't part of) a bad name - and they have WAY too much influence over CEOs and politicians. Instead, I'd like to see *real* scientists of all colours get more sway with evidence- and fact-based analysis, something I believe saw us have such a strong COVID-19 response. Though I believe that triangulating that with more qualitative science on e.g. human behaviour, culture and the power of misinformation would have helped predict and better manage some of the unintended outcomes like the intensity of the blow-back to public health measures that evidently saved 1000s of lives...
Another major issue for me when it comes to research papers is so many are just horribly written, like there is some award for using the biggest words possible.
And I really wish journalists would ignore press releases about studies unless they are willing to carry out the due diligence outlined in this newsletter (which of course they are not because they don't have the time).
The article in footnote two is one of the scariest things I've ever read. I'm sure I've seen the chocolate diet reports many times in recent years, and while not making any lifestyle changes based on them, thought it had some real scientific background.
I consider(ed) myself pretty clued in about junk science, but this post has been a real wake-up.
So eating kilos of chocolate each week isn’t the magical path to weight loss? I’ll just tell myself it’s all about the antioxidants as I wash it down with red wine.
Great article- thank you both for breaking it down into laypersons language
2) Having flirted with going into academia a bit too much, I don't have enough appreciation for the fact that these tips are not common knowledge. Thanks for working to get them to more people. :)
Great approach and points. As a lazy lay person, I try said approach and find it time consuming and I also don't feel qualified to verify and validate. That said I try. Smiling at the LinkedIn post you were subjected to & can't help but feel I may have contributed to that 🙃 royal jelly. Good news on the sleep.
An Actual Neuroscientist's Guide for Adults Who Can't Science Good
This is such good information, and should be bound in hardback and then used to smack in the head every online publisher that publishes junk science under the guise of news. I learned a bunch of this when I was studying library science, and I STILL get caught out every so often.
This was great Josh. I have so much to say about this one (anaesthetist, dabbling in academia ie I have research interests but no formal higher degree apart from Masters of Public Health). Interpreting scientific literature is so fucking complex - which basically goes to show how unqualified/ anti-qualified/ negatively-qualified all those “experts” on the Covid vaccine (or whatever) were. A massive bug bear of mine is that scientific journals currently operate on a shitty business model - fundamentally they make money... which is always a conflict of interest whether you like it or not. So much RUBBISH gets published. It is almost impossible to tell what is published rubbish unless you have very good knowledge of research methodology. The stats part is huge of this discussion is huge. Even if you look at massive clinical trials (thousands) so many results might be statistically significant (reliable) but are they clinically significant? The study abstract might say “New meditation technique significant increases your time asleep.” Ok... great! But in the actual article you find out that it statistically significantly increases it by 3 minutes. Even statistical significance (p-value, traditionally defined as <0.05) is a whole discussion in itself. It’s a made up value! Journals will publish an article with a p-value of 0.049 but not 0.051. The ACTUAL difference between these studies is... nothing.
Anyway I’m RAMBLING. My summary is that it is really complicated.
Oh PS - I occasionally review articles for journals. If something gets published and there is a TYPO in the heading... you can safely assume the peer review process has been shite. Another red flag.
That did invoke a cold sweat recall of university days, but truth be told I enjoyed the hunt if not the citations. The Listener cover seems to have become a rolling maul of "wellness" topics in recent times. I shall be reading in the latest edition how breathing techniques "will change my life". I find that in and out is a pretty reliable one for sustaining it? It becomes overwhelming as the list of things that need improving continues to grow ^^
Also, you can't tell me what to do! Gonna eat so many bees today
Re the surveys point: some data can ONLY be gathered through surveys.
The reason I'm pro surveys: gender should always be self reported as the gold standard. The best way to know someone's gender is to *ask them*. However, a lot of studies do things like automate gender ID based on first name (culturally insensitive, also good luck being called Alex), get an interviewer to guess (this is in some WHO guidelines!), get a random stranger to guess (the "who asks questions at conferences" papers do this), or use medical records without considering the wildly different standards for recording what someone's gender is across jurisdictions.
Kia ora Josh, this is a great overview but skews a bit too much towards the hard sciences and academia IMO - something that could be caveated. I believe that you can very much do *proper* science outside of academia, whilst being industry-funded, and focusing on qualitative insights with a small number of targeted audiences. That also means it's likely there won't be statistics, or limited ones (if you try to apply stats at all cost to qual social science, you can get into the "lies, damn lies and statistics" territory very quickly). My research, for example, is based on almost 20 years professional experience post-PhD (which was on a completely different topic and discipline, more akin to the hard science approach you talk about here), in a variety of sectors, with a variety of stakeholders, and on a variety of target audiences. Because I research human behaviour and energy (in)justice, a lot of the aspects you ascribe to "pure science" really don't apply - it's a highly complex area of research that requires socio-ecological approaches, which in themselves often fall outside of quantitative inquiry and stupid "gold standards" such as randomised control trials. One thing I do definitely agree with though, is to run away from economists - and that includes a lot of behavioural economists who have taken over my field. To me, they are usually represented by the typical patriarchic, Eurocentric, neoliberal, overprivileged fools that give STEM (which economics definitely isn't part of) a bad name - and they have WAY too much influence over CEOs and politicians. Instead, I'd like to see *real* scientists of all colours get more sway with evidence- and fact-based analysis, something I believe saw us have such a strong COVID-19 response. Though I believe that triangulating that with more qualitative science on e.g. human behaviour, culture and the power of misinformation would have helped predict and better manage some of the unintended outcomes like the intensity of the blow-back to public health measures that evidently saved 1000s of lives...
Another major issue for me when it comes to research papers is so many are just horribly written, like there is some award for using the biggest words possible.
And I really wish journalists would ignore press releases about studies unless they are willing to carry out the due diligence outlined in this newsletter (which of course they are not because they don't have the time).
In conclusion, boo science? Idk fuck
The article in footnote two is one of the scariest things I've ever read. I'm sure I've seen the chocolate diet reports many times in recent years, and while not making any lifestyle changes based on them, thought it had some real scientific background.
I consider(ed) myself pretty clued in about junk science, but this post has been a real wake-up.
So eating kilos of chocolate each week isn’t the magical path to weight loss? I’ll just tell myself it’s all about the antioxidants as I wash it down with red wine.
Great article- thank you both for breaking it down into laypersons language
1) Giggled at the Zoolander reference.
2) Having flirted with going into academia a bit too much, I don't have enough appreciation for the fact that these tips are not common knowledge. Thanks for working to get them to more people. :)
Great approach and points. As a lazy lay person, I try said approach and find it time consuming and I also don't feel qualified to verify and validate. That said I try. Smiling at the LinkedIn post you were subjected to & can't help but feel I may have contributed to that 🙃 royal jelly. Good news on the sleep.