Doing data analysis

'Research is about more than empirical evidence, but evidence is at the heart of finding out more about the social and education world. One way of marshalling evidence on a topic, or to answer a research question, is to use the findings of others as published in the literature. This use of evidence at third-hand is common – in the notorious literature review for a PhD, for example. I say ‘third-hand’ because the analyst does not have access to the primary evidence, nor are they re-presenting an analysis of the data. They are presenting a summary of what a previous author presented about an analysis of data. Done well, with a clear focus, such a review of literature can be useful, at least in establishing what others think, how a topic is usually researched, and why the topic might be important to research further. Some of the inherent weaknesses of using the accounts of others might be overcome by ensuring that all of the relevant literature was used, even accounts of unsuccessful studies and evidence from unpublished studies, and then conducting a full meta-analysis of the results (I recommend using a Bayesian approach, see appendix to Gorard et al. 2004, which allows the relatively simple combination of different kinds of evidence). But such systematic reviews of evidence are rare, very difficult to do properly, and both expensive and time-consuming. And anyway this second approach does not overcome the chief drawbacks of the literature which are that we have no direct access to the evidence of others, and often face a very partial view of the assumptions made and the analyses conducted.'