Study #2 "Full Year Pilot Grades 3 and 4" Mokros, et al, 1994

• The students were picked randomly from classes that were selected to achieve similarity between groups on demographic and other factors.  However, they did not describe the subjects statistically so nothing is known about how the study groups compared.  Baseline data were obtained.
• Again, the comparison curriculum is unknown.
• On the objective tests of computation, no differences were seen between the groups.
• The written test was untimed.  Results may have been different if the test had been timed.  Invented strategies may ultimately provide an accurate answer, but they are often cumbersome and inefficient.
• No information is given about how well the groups did on the objective tests.  We are told they improved over time, but no descriptive data is given. 
• On the interview (subjective) portion of the test, the Investigations group made more gains than the comparison group. This is hardly surprising, since the questions were written by TERC "to reflect the goals stressed by the NCTM Standards and the Investigations curriculum" (their own words). For example, the children were asked to use manipulatives, explain their strategies and show how they were thinking through with the use of drawings and constructions. It is not surprising that the Investigations kids would outperform the comparison group, since that is presumably what they had been doing all year, while these were probably new concepts to the kids in the comparison group. The real question is whether the fact that the Investigations kids could answer these questions means that they are any better at math.

Summary: Without descriptive data about the composition of the groups and objective performance, no conclusions can be drawn.  The fact that children in the Investigations group performed better on the interview portion is not particularly meaningful. 

Study #3 "Construction of Number Sense by Second Graders", Goodrow, 1998

• There is a discrepancy between the description of the study and the actual paper. The description says 46 children participated, while the actual paper gives the number of subjects as 30 (10 in each of the study groups). Either way, this study is too small, but 10 in a group is so small it is unlikely to provide meaningful data.
• Even more importantly, there is not one word in this paper about how the participants were chosen, so we can assume it was not randomly. We don't know a thing about socioeconomic status, a critical variable. This essentially makes this study useless. (This comes from a doctoral dissertation!)
• The tests were untimed.  As above, results may have been different if the test had been timed.  Invented strategies may ultimately provide an accurate answer, but they are often cumbersome and inefficient.  In fact, they note that only 6% of the Constructivist group used the standard algorithm when adding two digit numbers. 
• Similar to study #2 above, the outcome measures were aligned with the NCTM standards and Investigations curriculum. For example, students were asked to choose a number and write as many number sentences as they could that would result in that number. This is a typical problem in constructivist classrooms but would be foreign to children in traditional math classes. While the Investigations children were able to come up with significantly more number sentences, again, the real question is, so what? It is certainly conceivable that if the kids in the traditional classes were given a 5 minute introduction to this concept, they could have performed just as well. I am utterly nonconvinced that thinking up more number sentences implies a child is better at math.

Summary: The fact that we know nothing about how these children were selected and have no information about confounding variables (most importantly socioeconomic status) makes this study essentially useless.

Study #4 "Children's Understanding of Multiplication, Division, and Proportional Reasoning", Flowers, 1998

• Once again, no mention is made of how the subjects were selected, indicating that they were not chosen randomly. And again, we have no information about socioeconomic status. In fact, there are obvious differences between the groups on the baseline test, so one would not be surprised if there were significant differences between them with respect to demographics. This is inherently biased and makes interpretation of the results impossible.
• The two comparison groups were not identified. Although the paper states that one of the comparison groups was called "traditional", it used a 1992 textbook. Since the NCTM standards were written in 1989, it seems unlikely that this textbook would not have been influenced by the standards. A credible study would identify what the comparison group was using.

Summary: Without knowing the socioeconomic status of the groups, and the way the participants were chosen, meaningful conclusions cannot be drawn from this study.

Synthesis of the review: Three of the four studies have critical methodological flaws, making their conclusions invalid. The remaining study, while methodologically sound (if we assume that the groups were reasonably well matched) uses outcome data with questionable validity.

The most important point is that none of these studies compared Investigations to a curriculum that is truly substantively different, such as SRA's Connecting Math Concepts, Saxon Math, or SRA Math: Explorations and Applications. When these types of comparisons are made, we may expect to see some meaningful data about how best to teach children math.

Contact us Parents for Evidence Based Education PO Box 11144 Cedar Rapids, IA 52410-1144 Background courtesy of Webground updated: September 3, 2003