Day 14: How to tell good research from bad

60/60: 60 short videos for PhD students in 60 days

A few days ago, I talked about filtering the literature. There are 3 main criteria for this; quality of research, relevance to your work and influence on the field. But how can you judge the quality of the work? How can you tell good work from bad? You have to get experience doing your own research. Without this, the only experience you have is what other people have said. Start getting practical experience of research and analysis as early as possible.

6 thoughts on “Day 14: How to tell good research from bad”

  1. Unfortunately not everyone can start practical research early. Indeed, in some subjects there is an awful lot of reading that goes on first.

    The approach I always take is one of ‘the sceptic’. The research I read not only has to inform me, it has to convince me.

    So although I look for the strengths of a particular piece of research I also actively look for the other side too.

    I’m always checking for error. I ask myself whether there are any opportunities for the author of a piece of research to get things wrong or to be biased or simply to be mistaken. Is there any potential for systematic or unsystematic errors? Is the sample size large enough to be convincing? Is the study double-blinded where medical treatments are being considered? Is the method tried and trusted, or new and innovative? How does the author deal with the limitations or weaknesses or disadvantages of their research – are they open about them or do they ignore them? Has this study been superseded (e.g. it was a great study in the 1990s but techniques are so much better now)? What are the researcher’s underlying assumptions? Do all the researchers in the field have the same underlying assumptions? Are those underlying assumptions reasonable? Does the researcher agree or disagree with other researchers in the field?

    This way, rather than simply absorbing the information, I’m actively engaging with it.

    James, do you have a similar approach? Are you a sceptic too?

    • Skepticism is good, but I think it’s easier when you have experience.

      To give one example, I saw a talk at a conference on a variation of one of the techniques I specialised in. It took 2 seconds to see their data were fake (and it was obvious they didn’t know or chose not to use basic techniques for validating their results). This wasn’t something I could have learned from reading because it was a perspective based on skill and experience.

      I do understand that there are restrictions on some types of research, but there are usually ways to start getting at least some practical experience early on. Is it really impossible in your field?

      • Luckily the two things – scepticism and practical experience – aren’t mutually exclusive. Scepticism just an approach I find to be really helpful and I wondered if other people would too. It really makes me engage with other people’s research and hones my arguments. And like you I’ve had the happy experience of catching someone out.

        i think that it is worth noting that in the arts and humanities, people often have to do lots of reading and writing before becoming ‘hands-on’ researchers. It’s different in the sciences and (to some extent) the social sciences.

        • Agreed, they aren’t mutually exclusive!

          I also agree that in many fields people are expected to do a lot of reading before becoming hands-on researchers, but I think it’s often a mistake to spend too long reading and not doing research of your own. I tend to see the ones who end up in trouble, though. I don’t hear from people who breeze through!

  2. Hi James,

    Really appreciating the latest vids. I am requesting a video please. I am struggling to structure my findings/discussion chapters. I am attempting to link the two sections rather than separate them. I am starting to think this is a bad decision! Any advice on how to integrate these sections. I am doing qualitative research in sociology of work.

    • hard to know without seeing your chapters, but it depends on the kind of findings you have.

      Sometimes you have to give a bit of discussion of one finding before the next one makes sense (for example if you found something surprising that led to further work to clarify).

      If you are struggling to integrate them, why not keep them separate?

Comments are closed.