False Research Studies?

What you read is not always true; even if you read it in the otherwise sacrosanct academic journals. Is that really true? "In an article in the Public Library of Science Medicine, John P. A. Ioannidis, an epidemiologist, argues that more than 50 percent of the conclusions drawn in papers published in scientific journals are false." [via]

This reminds me of an exchange that I heard on radio not too long back. The RJs were talking about a research study that made similar claims, saying that most of the findings that research studies make are simply manipulated and not entirely rooted in reality. This can be quite shaking and almost make you disbelieve or at least suspect everything that you have read that started with “According to a research study; ” But the RJs offered a ray of hope. They summarized that if the researchers of the above-mentioned study found out that almost 50% of research findings are false; which half does their research study fall under?

On a serious note, being in an academic field, I admit that some manipulation does occur (after all it is publish or perish) although I am still feeling my way around in the research field and have not come across blatant misrepresentation of reality or screwing around with data. I hope I never will or have to. Other senior researchers that read this blog care to comment?

Tweet This“False Research Studies?” [Short URL: http://www.ipatrix.com/?p=893] was published Wednesday, August 31st, 2005 at 3:57 pm by Patrix, filed under Science, and tagged as , , , . Both comments and linkbacks are closed.

«
»

Related Posts

  1. Eliminating NSF Funding for Political Science?
  2. Research Park
  3. Google Algorithm for Finding Scientific and Research Papers

  • http://ashujo.blogspot.com Ashutosh

    Patrix, rather than call it ‘manipulation’, I would call it setting up an experiment to see what you want to see. Then you can get good results, dictated by the restraints that you have imposed by your biased wants. The truly creative research is frequently the unexpected, and observed even in the work of methodical researchers; Enrico Fermi’s suddenly using a paraffin block for his experiments ‘just like that’ (as he himself said to S Chandrasekhar; unusual for a supremely organized man such as Fermi) and making the discovery of slow neutrons which are the foundation of nuclear reactors, is a good example. In biological sciences, Ionnidis rightly notes that because of inherent complications, there may be ‘false’ results. But rather than false, I would say it is the result of an overcontrolled experiment, one in which and unrealistic number of variables are kept constant.

  • http://ipatrix.com Patrix

    Ashutosh, of course what you say is absolutely correct. Most of the big discoveries in science were accidental.But I was referring to some researchers especially social scientists who search for and fit data to their predetermined conclusion. Selective data retention can be dangerous because it makes you believe in relationships that may not necessarily exist. Tsk tsk…that almost reminds me of my previous personal relationships :)

  • http://balancinglife.blogspot.com sunil

    Which is why any result is expected to be proven independently (by a research group not connected to the group which first reported the findings) before it even starts gaining widespread acceptance. Only after a number of different groups can reproduce the results are they taken to be absolute. In any research field, controversies about a theory abound, untill proven otherwise.

    There are a number of findings which remain controversial because they cannot be proven independently. Every now and then you’ll see articles in Science, Nature, Cell (or other high profile scientific journals) that are retracted….because some times the same group (but other individuals in the group) cannot reproduce the same results.

    Which also raises the importance of a vibrant, independent research culture. The more the researchers there are working independently on a problem, the less the chances are of reports that aren’t true, or are only selectively true.

    But yes…..overinterpretting results, or “sexing up data” are both common problems in research.

  • http://chocolateandgoldcoins.blogspot.com/ Michael H.

    Hi Patrix
    The first thing I thought when I read that research had discovered that 50% of all papers are false was: “What’s the chance that that paper was wrong?” But apparently someone beat me to that punchline.

  • http://nanopolitan.blogspot.com Abi

    Hi all, the paper is a *theoretical* study about the kinds of errors that could creep into an experimental studies in fields such as medicine and many social sciences such as economics and psychology.

    First, since this paper is a ‘theoretical’ one, it is, by definition, ‘true’ (of course, subject to the validity of its underlying assumptions).

    Second, as Ashutosh mentioned, there are many confounding variables that have to be dealt with in an experimental study in such fields. If each of them introduces even a small probabilty of error, I can imagine a situation in which *all* of them put together conspire to produce a false finding. It is surprising that the overall probability works out to more than 50 %.

    It is unfortunate that Patrix has chosen to highlight manipulation of results. Now, data manipulation and personal bias are not ‘errors’; they are fraud. The paper is certainly not about fraud!

    I think a basic feature underlying all of science is that you don’t suspect ‘personal bias’ and ‘sexing up data’ unless you have a strong reason to. I don’t think these are common (sure they are common enough that people retract some papers every once in a while, but come on: in proportion to the total number, the bad ones are a tiny fraction), and I think it is unfair to throw these allegations around so carelessly.

    Full disclosure: I am *not* an experimental researcher; much of my work revolves around computer modelling of strange processes that occur within materials. ;-)

  • http://ipatrix.com Patrix

    Sunil, creating identical conditions and producing the same results by an independent researcher is the key test that all research studies should be subjected to and the results should not be taken at face value. This is often done in social sciences where often specific studies are generalized to a larger scope.

    Michael, sorry dude! maybe next time :)

    Abi, thanks for your comments. I think the ‘errors’ that creep into a paper are a combination of the methods used and nature of the topic. Often the methods used aren’t appropriate or are simply asking the wrong questions. Social sciences, as I am mentioning again and again, is very different from pure science which often has clear cut results. There can be hundreds or even thousands of variables that often shift over time in social studies that researchers often find it difficult to keep track. Choosing those variables for the desired results treads the fine line between manipulation and ‘sexing up’ data.

  • http://nanopolitan.blogspot.com Abi

    Thanks, Patrix, for your response.

    So we agree that this is a problem related to some sciences (including social) that rely on statistical inference while testing hypotheses regarding phenomena involving tons of variables (phew! that was a long sentence …). Alex Tabarrok has an excellent (and, as is expected from him, a highly readable) explanation of the Ioannidis paper, by giving concrete examples (this explanation is followed by a nice analysis as well. So, don’t miss it). He too identifies the large number of variables as the key to understanding the rather provocative claim by Ioannidis.

    If I think someone has chosen a ‘wrong’ variable to study, my first guess would be ‘this person is misguided or, at worst, incompetent’, which is a long way away from ‘this person is a fraud’. People ought to be very careful to rule out other incompetencies (!) before throwing terms like ‘sexing up data’ and ‘manipulation’. The latter cannot be — and should not be — the first line of offence.


Copyright © 2012 Nerve Endings Firing Away. Designed by ckunte. Powered by WordPress.