I am still trying to get a grip on John Hattie’s two books, Visible Learning (2009) and Visible Learning for Teachers (2012). The sheer volume of studies reviewed, students involved and the meta-effect methodology are just three reasons this is so hard to digest. To say I am late to this work is a given but it also allows me to see what others have said about the work. And it does take time for the research community to evaluate claims from studies large and small. Three years ago, Peter DeWitt writing in EdWeek said this:
Educational publisher Corwin Press announced that they are bringing the work of New Zealand born educational researcher John Hattie to North America. Hattie is the Director of the Melbourne (University of Melbourne, Australia) Education Research Institute and is best known for his Visible Learning approach to student achievement (Corwin & Visible Learning).
In their press release, Corwin Press said,
“Hattie’s work is based on his meta-analysis of more than 1,000 research reviews comprising more than 50,000 individual studies–the largest meta-analysis ever conducted in the field of education. Hattie identified the major factors and practices that influenced student achievement, from family background to teacher training to specific instructional practices. He then went a step further and calculated how much of an effect each factor had on students.”
SO what is a meta-analysis? It is the analysis of not one study, but many studies. Please let me try to explain, without losing you in “edspeak” talk, ok? Most individual research reports are judged by statistical significance. This means that the research tried to see if the treatment (e.g., class size, mother’s education-level, or teaching style) made a non-random difference in the student results. The reported results might look like this: “smaller class size resulted in an increase of student achievement by 10%.” When this research is reported, however, you can only judge it’s worth to you, let’s say as a teacher or parent, by the rigor of the research, what population of student it was done with, how many students were involved, etc. So you are left thinking, class size really matters. But does it? The next study, done in a different state with different students, might show that class size really didn’t make a difference in student learning. Now what do you think? One study says yes class size matters and one study says no it does not.
Well, researchers try to compare different studies through meta-analysis. Meta-analysis allows all research on a given topic, such as class size, to be compared. This comparison is framed along one line of a continuum. So picture a number line with negative one on the left, zero in the middle, and positive one on the right.
-1…………………………….-0.5………………………….0……………………….+0.5……………………………+1
Each piece of research, let’s say 8 studies on class size or 33 studies on mother’s level of educational attainment, can now be compared. Through meta-analysis all of these studies can be compared by type of study and also can be compared to each other. So what John Hattie has done is compared 1000 studies and ranked their effectiveness as a number along this line of this continuum. To understand the 138 variables that were studied and their ranked effectiveness in helping students learn, please visit John Hattie’s website: Visible Learning.
Controversy
In the New Zealand Post Primary Teachers Association blog there is mention of a commentary by New Zealand professors (Massey university academics Ivan Snook, John Clark, Richard Harker, Anne-Marie O’Neill and John O’Neill banded together to produce Invisible Learnings? A commentary on John Hattie’s “˜Visible Learning: A synthesis of over 800 meta-analyses relating to achievement‘):
The commentary looks at Hattie’s findings on class size and expresses concern at the validity of these.
“Hattie has been cited as ‘finding’ that class size is not important and this has excited the attention of those concerned about financing of schools, who conclude they can economise on class size,” it reads. These results however fail to look at contributing factors such as professional development and the ability for teachers to spend more time with students.
“Reducing class size may have only a small effect when considered in isolation, but that’s not the issue. What matters is that reducing class size permits the teacher (and children) to do things differently,” it says.
No evidence to support performance pay
The report also voices concern that Hattie’s conclusions on the importance of what teachers do have led some to advocate performance pay.
“We have seen no evidence at all to support the claim that performance pay improves teaching or learning and there is nothing in Hattie’s massive research which even remotely suggests that it does. On the contrary, much of what he says suggests the very opposite… (he says) what is needed for school improvement is a caring supportive staffroom, a tolerance of errors and … a peer culture among teachers of engagement, trust, shared passion and so on’. Such a co-operative, trusting and self-critical school atmosphere is the very kind of atmosphere which regimes of performance pay destroy.”
Determining effect size and cut off points
The commentary looks closely at the methods Hattie used in his research. All the findings in his book derive from a synthesis of 800 “meta-analyses” of more than 50,000 studies of variables affecting the achievement of students.
The main aim of the exercise was to determine effect sizes of certain factors on student achievement and Hattie does this by reducing these issues to decimal points.
“There are debates about where a small effect size ends and a moderate or large effect size begins. Hattie adopts 0.4 as the cut off point, basically ignoring effect sizes lower than 0.4. Thus, for example, class size is interpreted as a small effect size since it is 0.2 (In public debate this tends to turn into ‘class size has no effect at all’.) Selecting a cut-off point is a very hazardous exercise, as it means potentially important effects may be overlooked,” the commentary says.
Classrooms are complex
Professor Snook did not believe the figures Hattie presented amounted to a “holy grail” for education. In fact he was rather dubious about the benefits of reducing a complex area like a classroom to mere decimal points.
“A classroom is a complicated sort of place.” New Zealand Post Primary Teachers Association
Paradigm Shift or Complicated Place?
I think we are on the edge of a new paradigm for how we think about teaching and learning. Yes, classrooms are complicated, and yes, teaching and learning is one very complicated dynamic. But Hattie’s work may help teachers who will have to examine their practice not just from how they were trained or what worked last year, or even how they were taught, but much more from the point of view of what works in the best classrooms based on world-wide research. The reconciliation of a teacher’s current teaching practice with what works in all classrooms may be a new paradigm for improving teaching. Most importantly, this improvement cannot be separated from students, so the new paradigm should always include improving teaching and learning.
