Is some research just too different to robustly average?

This is a common criticism of meta-analyses and has been dealt with in many other sources. Certainly, combining remains critical and this is why care is needed to discover moderators and mediators. Moderators have been continually searched for in the research and there were very few. Where they did exist they were mentioned.   In Visible Learning (2009) Hattie points out the need to think about moderators.

One concern that is considered more important is when two quite divergent effects are combined and then assuming the average is a good measure of the “typical value“.  For example, in “Inductive Teaching” two meta-analyses with effect sizes of d = .06 and d = .59 are combined to a mean effect size of d = .33.  Here, it is important to look less at the numbers and more at the interpretation.  An interpretative approach was used when reviewing Lott’s (1983) meta-analysis where this included a comparison of inductive versus deductive teaching approaches in science education (where it made little difference) and Klauer and Phye (2008) who were more interested in inductive reasoning across all subject areas (this is where they did find higher effects).   The details of the interventions matter crucially which is why the story underlying the data is so critical.