genophoria

Genomics euphoria: ramblings of a scientist on genetics, genomics and the meaning of life

Tag Archives: scientific method

Reviewing the “peers”

A while back, with some not too complex detective work on the part of USA Today, we got our hands on the reviewer comments from the infamous #arseniclife paper. Looking at the reviews, I agreed with the experts, including Leonid Kruglyak, who “reviewed the reviews”, that they generally looked normal… yes, they were quite positive at times but let’s face it, some scientists are nice and some aren’t and by chance alone, you might have a group of three that use encouraging words even if there are criticisms. But since then, I’ve seen opinions written out there that are not too kind to these reviewers… good thing they’re anonymous, or we were looking for our pitchforks now! Even Ash Jogalekar over at “The Curious Wavefunction“, whose posts I often find refreshing, called the reviewers out on their subpar job.

But what do “I” think about this whole fiasco? I think hindsight is 20/20… It all comes down to taking the authors at their word that there was no phosphorus in the media they used… everything else follows… But as it turns out, that is very difficult to achieve. But I didn’t know that and if I was a reviewer, I wouldn’t have brought it up either. I probably would’ve asked for specific experiments but they would be based on my background (and could be claimed to be “outside the scope”). And that is all it comes down to, isn’t it? Our backgrounds… and the reviewers are in fact chosen from different areas. For a paper like this, you probably will get a bacteriologist, may be someone working on Archea and maybe a chemist would not even be on the short list (let alone one that would have the relevant knowledge). In the end of the day, I don’t think that as researchers, we want reviewers to be too ambitious… we have all got bad reviews that we think are nonsensical. I actually think the reviewers should just make sure, given the facts and their knowledge, the results are not seriously flawed. And we all know that a published paper is not exactly the word of God and it can be easily refuted/corrected/expanded upon.

So, if we all know these things, then what is the source of the outrage? I think there are a number of points:

  1. This is Science we are talking about here… we like to think that the impact factor of the journal has a strong correlation to the strength of its underlying science. We all toil away for years hoping to get a paper in Science and it breaks our heart to see a flawed paper like this appear on its pages. I agree that this paper, in retrospect, is outrageous but this is not the first time a bad paper is published in Science or Nature or any other journal.
  2. This process shows every thing that’s wrong in research nowadays… holding press conferences like a true salesman, relying on seemingly random reviews to accept a paper and the disproportionate value associated to papers in big name journals.

Are there solutions to these problems? Of course there are… pretty old ones actually. Switching to Arxive model instead of the journal model makes a lot of sense… in that model, studies are presented on equal footing and it is on their own merit that they gain attraction. Can we ever do that? Not in the near future… biology is expensive and researchers need decent publications for every grant cycle. We simply don’t have the luxury of time to wait for our papers to climb the citation ladder (that is assuming citation is even a good measure of merit). The current journal structure enables us to “assign” a “value” to a paper based on a couple of samplings, even before the paper is published. Is it a flawed process? Of course. Is it at times unfair? Absolutely. But let’s realize what the problem is: just too many people, not enough money. Let’s not heap all that on the shoulders of three reviewers…

The rise of “entertainment-science”

I wanted to write about the very interesting back to back Cell papers of c-myc (this and this). But at the last second, I changed my mind. I want to write, instead, about what people have been writing about all week… that is the paper by French scientist attacking the safety of genetically modified food. Genetically modified organisms are crops and animals that have been genetically engineered to improve their efficiency and productivity, from pest-resistance to higher milk productions. They have been around for sometime now and despite countless studies, there is very little evidence threatening the safety of the whole category…

The controversy, however, is alive and kicking and the Seralini et al paper drops the hammer on the safety of GM maize. I don’t want to talk about the short-comings of this paper, you can get that from other sources (e.g. here and here). Instead, I want to talk about a broader phenomenon, the rise of entertainment value in controversial science:

  1. I am sure there are many world-class scientists who focus on GM-food safety. But I don’t know any of them, do I? Instead, I and many like me (who read science news) know Seralini, mainly because his papers are always controversial and go against the consensus of the field. There is nothing inherently wrong with that… but I think amazing claims require amazing accompanying evidence. It is OK when papers show up that go against the norm… either the whole field is wrong (which sometimes happens) or there is something wrong with that study (which happens quite often, case in point: faster than light neutrinos). But what I think is weird, is that these studies get WAY more publicity than they should. And in the end of the day, it makes us scientists look bad. People hear about all these extraordinary findings that are then debunked in a couple of months… no wonder we have problems with the perception of science in public.
  2. From linking disease to vaccination to discovering a new Arsenic-based form of life, we have let science down again and again… not by making mistakes. Being wrong is fine, it is the first step towards getting it right, but rather through pushing our results into the spotlight. Holding press conferences, enforcing gag orders on collaborators and pulling off all the PR stops. I assume, soon we’ll see trailers of upcoming publications on TV (“…starring POLR2A as RNA polymerase II”) with entertaining twists of course.

The majority of science is still working the way it should, but that is not the part we read about in science news sections. I want to think scientists are better than this PR stuff… We were supposed to be skeptics, we were supposed to be above this… we were supposed to be living in our ivory towers. We shouldn’t care about the results of our experiments (positive or negative), we are supposed to be searching for truth… or so I thought.

Living an “organic” life: debunking the supremacy of the organic produce

A couple of years ago, I took a course titled “The use of science in public policy” taught by Prof. Lee Silver at Princeton University. The goal of the course was to bring basic science and policy students together and expose them to the challenges generally faced by each group. The take home message for me, as a scientist, was the paucity of black and white issues and how complex even mundane policies become when they’re applied to the entirety of a society. However, what shocked me the most was how difficult it is to bring the scientific world-view, which is some times counter-intuitive, into policy making. And this is nowhere more obvious than issues like “genetically modified food”, “homeopathic medicine” or even “organic food”. Organic produce, which have been supposedly grown chemical-free, is branded as “natural food” and has formed one of the most successful industries in the US with a growth of about 10,000% over 10 years. What distinguishes the organic food from conventional products however is for the most part in its sticker price. But the consumers are buying organic products in droves, assuming that its health benefits far outweighs the cost. There have been studies looking at this claim but a recently published study in Annals of Internal Medicine does a good job of bringing together all the available data over many years to perform an effective meta-analysis of this subject.

Challenges

I think the main challenge in discussing organic produce is the laxity of standards in its definition. From what I understand, every farm has its own way of defining organic food. And without a national standard, it is more difficult (although not impossible) to study the health benefits of this category. Nevertheless, organic farms have been very successful in convincing the consumers about this matter, so much so that other industries are taking notes. Case in point, the rise of the “organic laundry” and “organic detergents” that (spoiler alert) has nothing to do with “organic” in the sense we use in our everyday lives.

In general, the industries seem to be very good at persuading the public about health benefits of food (e.g. brand names like Vitamin water). Also, as humans, it seems it is difficult for us to take into account non-linear relationships; for example, if a little bit Vitamin E is essential for your health, adding as much as you can to your diet should be even more beneficial. Teaching the public that there isn’t a linear relationship between health impact and intake seems to be a daunting task. Labels that connotate “nature” are very potent, which for me is very counter-intuitive. Nature is full of dangerous toxins… not every natural product is beneficial.

Another important challenge has to do with how well we can disentangle “organic food” consumption from other aspects of life. We can assume that people who care enough to pay 2-3 times more for organic produce just because they think it’s healthier, are also more likely to go for their annual checkups, exercise and in general be concerned about their health. Statistically speaking, it would be very difficult to correct for these covariates without doing a true double-blind experiment. Double-blind experiments need sponsors, which leads me to ask whether the government may in fact have a role to play in this matter. At this point, FDA doesn’t regulate anything that is “natural” which puts organic food outside of its jurisdiction.

Given these challenges, it is rather obvious why we needed several decades worth of data to be able to perform a decent meta-analysis. And to be honest, I still think this study could be much better and more distant from academic hype.

Findings

Despite these challenges, these researchers, I think, have done a decent job of analyzing the data. They find very little evidence in support of organic produce. Sure, they see marginally higher pesticide levels in conventional produce, but the levels are far lower than the risky threshold. Also, organic farms do use pesticides, but use natural ones instead of chemical ones and we don’t know if natural pesticides are in any way safer than chemical ones. More importantly, because natural pesticides are less potent, higher quantities needs to be used (adding more stuff to the soil and environment).

There isn’t enough data to fully debunk the perceived value of organic food, but at this point, I’m pretty sure it’s not worth the significantly higher sticker price.

The best book of all time

What is the best book of all time? That seems like a stupid question. It’s vague, subjective and quite meaningless. However, before I get replies like “Harry Potter”, “Twilight” or “Fifty shades of grey”, let me provide some context for this question. As this is a scientific blog, first and foremost, the answer should be a scientific book… and no, I am not talking about “On the origin of species”. I am looking for something grander, something without which Darwin’s publication was not even possible. The answer to this question, I think resoundingly, is Novum Organum by Francis Bacon, the 17th century philosopher scientist. Why is that? Well, I don’t know… may be because it formulated scientific method in its most basic form. For centuries, thinkers, authors and philosophers were expanding upon the ideas put forth by Aristotle and Plato. However, Francis Bacon made the case that, for the most part, these were mostly just ideas (what we now call hypotheses). He advocated a new setting in which philosophers could rise above these ideas and formulate new ones. However, he also clearly indicated that the validity of these ideas need to be tested through a rigorous “method” involving data collection, its interpretation, and even designing new experiments.

Novum Organum

Now why do I think Novum Organum is the best scientific book of all time? Because it made a case for scientific method, before it was cool. And most important of all, he didn’t only talk the talk… he actually walked the walk. He contracted pneumonia while studying the effects of freezing on preservation. While I don’t condone working oneself to literal death, we should realize that the scientific process owes an undeniable debt of gratitude to giants who devoted their lives to science. Novum Organum is available on Amazon for $1 (the Kindle edition that is), I don’t have to tell you that it is worth every penny. I don’t believe in the “Great man theory“. I am not saying that had he not published this body of work, we would still follow an Aristotelian method. If Edmund Hillary hadn’t climbed Everest, someone else would have. But this fact does not make his trial more trivial and his adventure less dangerous. The same holds for Francis Bacon and his seminal work “Novum Organum”.

Francis Bacon

Francis Bacon