EducationNews.org

1) Marcus, you and Jay Green have recently released a report about what is transpiring in Florida entitled " Building on the Basics: The Impact of High-Stakes Testing on Student Proficiency in Low- Stakes Subjects.  First of all, where can our readers get a copy of this report?

The original report and a more technical version of it are available at the Manhattan Institute's website, www.manhattan-institute.org.

2) You have been given a gargantuan amount of data from Florida. How did you get access to the data and what were you trying to find out?

We are among several researchers who have gained access to Florida's administrative dataset over the last few years. The availability of Florida's rich dataset has thus far produced a wide and growing body of important studies from a variety of researchers. We applied for these data in order to study a wide scope of policies in Florida, which we are continuing to research at this time. In this particular paper, we are using this individual level data to study the impact of incentives under Florida's accountability system on student proficiency in a subject
unrelated to the system -- science.

3) Tell us about the FCAT and why Florida seems to be a "model" for NCLB.

I think that the Florida and Texas policies are both important models followed by NCLB. In Florida, students are administered math and reading exams in grades 3-10 on the state test, the FCAT. The results of these exams are then used to grade schools from A - F, and schools that receive a failing grade multiple times in a four-year period receive sanctions. At the time of our study, students in schools that received two F-grades within four years received vouchers that they could use to attend a private school. The policy is similar to NCLB in its focus on using these scores to identify failing schools and hold them accountable. However, NCLB is a bit less nuanced in that it deems schools either passing or failing to meet a particular AYP benchmark.

4) Basically many people are concerned that there is an overemphasis on math and reading to the exclusion of science, social studies, and (gasp!) art, music and physical education. What did you find in your analysis?

Accountability policies across the states as well as NCLB tend to focus entirely on measured student proficiency in math and reading. Some research indicates that these programs have improved student proficiency in those subjects. However, one important criticism of these policies is that they provide schools with an incentive to focus entirely on math and reading and ignore the other subjects. Some anecdotal and qualitative research finds that schools have responded to these programs by re-allocating time away from science, social studies, and the arts toward math and reading instruction. However, we know much less about the total impact that these programs have had on actual student proficiency in these low-stakes subjects.

Our goal was to measure the extent to which the incentives of an aggressive accountability program focused entirely on math and reading exams impacted student proficiency in a low stakes subject. We could do this in Florida because the state tests students in science as well as math and reading, though the results on the science exam are not used for any accountability purpose.

Our findings were intriguing. We found that students in schools facing an F-sanction actually made substantial test score improvements in science that matched the improvements they were making in math and reading. That is, not only did we fail to find evidence that the incentive to focus on math and reading harmed student proficiency in science, we found evidence that it led to even higher student achievement in the low-stakes subject. These results are particularly surprising given recent evidence in Florida and elsewhere that these policies tend to lead schools to focus less time on these low-stakes subjects.

We think that these puzzling results can be explained in potentially two ways. First, it could be that schools are adopting school-wide reforms that are intended to improve math and reading proficiency but could have an impact in other subjects as well. For instance, a recent paper in Florida found that schools responded to the F-grade sanction by lengthening the school day and adopting block scheduling.

Second, it may be that as students become more literate and numerate they also become better able to learn material in other applied subjects such as science.

It is important to note that while we find convincing evidence that the F-sanction in Florida led to student achievement gains in science in the elementary grades we know less about the impact on other subjects such as social studies and the arts. Future work is necessary evaluating the impact of these testing programs on those other subjects and in later grades.

5) Now, help me out here personally. What is a regression discontinuity design? And why did you use this particular statistical analysis on this data?

Regression discontinuity is a tool that has been growing in popularity among education researchers in order to account for heterogeneity of school productivity. The idea is to take advantage of situations where an institution or individual must meet a particular measurable threshold in order to be subject to a particular treatment. In these cases, individuals falling within a very narrow margin above or below this threshold are likely very similar in their observed and unobserved characteristics.

In this study, schools earn points for a variety of measures of student math and reading proficiency under Florida's somewhat elaborate accountability system. Schools that receive fewer than 280 points receive an F-grade.

We take advantage of the fact that the point system is actually a more complete measure of the school's productivity than is the grade received by the school. For instance, imagine two schools. One school receives 281 points under the policy and thus earns a D grade, while the other earns 279 points under the policy and thus earns an F. These schools are nearly identical in their effectiveness as measured under the point system -- in fact, randomness probably determined whether they fell just above or below the 280 threshold.

However, these schools face very different incentives under the policy because one faces the sanction of the F-grade while the other does not. The result is that when we control for the number of points earned by the school we are accounting for their similarities and differences, and thus whatever remaining differences in the productivity of the two schools can be attributed to earning the F sanction.

6) How can one really know if improved reading and math results in improved science thinking, and performance and lab work?

Our only measure of science proficiency is student achievement on a standardized exam, so we are not in a good position to evaluate lab work, etc. As we do in other evaluations using exams, we are forced to trust that they are true measures of student proficiency in these subjects. Clearly they are imperfect measures.

7) Marcus, this research reminds me of the Dutch boy with his finger in the dyke- the idea that if the state or Feds indicate there is a major problem, the schools rush to address it, but there is no long term re-construction of the dam. Am I off on this?

I suppose only time will tell -- though I think I am more hopeful on this. The idea of these policies is to alter the incentives of schools so that they face consequences for poor performance. It is true that the potential consequences only occur if a school falls behind and that they can avoid them by making quick changes, but the hope is that schools will keep such positive changes in place in order to avoid the potential of sanctions in the future.

8) To use your words, there is a "sparse previous literature quantitatively evaluating whether high-stakes testing policies have "crowded out" learning in a low-stakes subject '. Has there been any long term analysis of the effects of high stakes testing? I am thinking of writing skills and higher order thinking skills and critical thinking skills in particular.

Most papers tend to focus on short term benefits of the accountability systems, and nearly all papers use performance on standardized exams as their measure of student proficiency. Writing skills are particularly difficult to measure because the tests are often quite easy to "teach to" in such a way that students earn high grades but may not have developed real writing skills. This is certainly an issue that researchers need to address in the future.

9 What research still needs to be done in this realm?

I think that this paper helps to move the ball forward toward thinking about researching the potential externalities of accountability systems.

It appears from past research that these policies can improve student proficiency in math and reading. However, we are also interested in student proficiency in other subjects. Further, future research can look at other potential unintended consequences of these laws, such as whether they lead to schools labeling students in ways that make them exempt from the exams, etc. I think that the issue here is not whether these policies can be effective, but how we can best develop the policies in order to maximize their benefits and minimize their potential costs. In this paper we are finding that one potential cost --loss of science proficiency due to its low-stakes status -- may not only be lower than we had thought, but might in fact prove to be a positive externality. Though these results are encouraging for such policies, we need more research in other states to see if these results hold elsewhere and to evaluate other potential unintended effects of the policies.

Published July 24, 2008