Scientific Meta-Analysis as Organzing System
The controversial recent Stanford study finding that organically grown food is no more nutritious than conventionally grown food is an interesting example of resource description and classification and how the resulting interactions in an organizing system come together to form conclusions. The study was a meta-analysis of previously published studies of conventional and organic food. As detailed in “Parsing of Data Led to Mixed Messages on Organic Food’s Value” (New York Times, October 15, 2012), a similar 2011 meta-analysis by scientists at Newcastle University in England that looked at much of the same data came to the conclusion that organic food is indeed more nutritious than conventionally grown. How can two studies look at the same data and conclude such different things? The answer is in how the resources (the data) are treated in the organizing system (the meta-analysis).
Some examples of the differences in organizing principles that lead to different interpretations of the data are granularity, ranking the importance of different properties, and categorization into equivalence classes. On the matter of granularity, the Newcastle study treated each year for a given crop as a data point, because “variations are expected because of differences in plant species, weather, soil and other conditions.” Stanford researchers, on the other hand, averaged data for each crop over several years, thus reducing the influence of fluctuations in different years on the results.
On the matter of assigning importance to different properties, the Stanford researchers emphasized in their findings the levels of vitamins and minerals in organic and conventionally grown meat, dairy, and produce. Although their report did show that organic food had lower levels of pesticides and antibiotics, they didn’t consider these properties as important to their definition of nutrition as vitamins and minerals. The Newcastle study, however—much like people choosing to buy organic food—considered these properties to be more central to the definition of nutrition and thus weighted them more strongly.
Classification comes into it through a category of plant compounds called flavonoids (and a subset of flavonoids called flavonols). Both studies sought to measure levels of flavonoids, but the Stanford researchers did not measure as flavonoids some specific flavonol compounds named in papers in the data set. If they had thought through their equivalence classes better for the purposes of accurate measurement, this mistake could have been avoided.
All of these elements invite questions about bias, and demonstrate how inevitable biases in organizing can manifest in important ways.
