A Missing Piece in the Debate on School Performance

by Walter R. Tschinkel

Professor of Biological Science

Florida State University

Our family has long supported public schools, and has followed the public debate about school performance with interest for several years now. Five years ago, the legislature mandated that standardized tests were to serve as measures of school performance, and that they were to be published for all to see. As a scientist, I felt it was essential to analyze these school performance data before rushing to champion "remedies". I want to share the outcome of this analysis because it demonstrates that most of the proposed solutions are political expediencies, and are irrelevant to the real challenge of assuring that all of our children receive a quality education. Briefly, the analysis shows that practically the only factor that explains the differences in performance among our public schools is the proportion of the children that come from poor families. Because we fail to recognize this connection, we ignore what is most important. Instead, we debate remedies that, at best, can have only minor effects, we spend money on ineffective measures and make decisions that are shots in the dark.

Annual measures of the performance of each county’s elementary schools have been published in local newspapers for the last 5 years, and are based on standardized math and reading tests. My data are for Leon County, but the results would not be any different for any other county, and so can serve as a representative analysis. Because the math and reading scores are highly correlated, I used the combined test scores in the analysis. The annual school report also contains information on each school’s characteristics, including the number of teachers, number of students, the racial makeup of students and staff, cost per student and percent of students on supported lunch. The last can be used as an estimate of the percent of the student body from lower income families, because to qualify for this lunch support requires that family income be below a certain level.

A simple inspection of the data shows that there is a striking spread in the performance of the schools --- the top schools have scores four or five times greater than the bottom ones. To find out what might be causing this variation, the data were analyzed using a reliable statistical method called regression, which determines which factors are related to which others and how strongly they are related. The outcome of this analysis was strong and unambiguous, and gives rise to my first point: Point 1. Eighty-five percent of the differences in school performance are associated with differences in the percent of students on supported lunch, that is the percent of the student body from poor families (see graph 1). Turning our finding around, it follows that only 15% of the differences among schools remain to be explained by factors other than socioeconomic composition. Most of the various "causes" and remedies discussed in the public forum are included in this 15%, so none of them can account for more than a small fraction of the differences in school performance.

None of the other factors has a large effect. As the regular cost per student increases, school performance actually decreases, but this relationship is weak. It is the result of low-scoring schools assigning fewer students per teacher in order to create smaller classes, raising the cost per student. The racial composition, which is often cited as affecting performance, has no effect except through its strong association with poverty. Mostly white schools like Woodville and Ft. Braden have similar scores to mostly black schools with similar levels of poverty.

Recognizing that more than three-fourths of the differences in school performance are associated with socioeconomic make-up of the student body, it follows that one of the surest ways, in fact the only sure way, to improve performance is to reduce the proportion of poor students. This is exactly what happens in our exurban schools--- their high performance is almost entirely due to the relative absence of poor students, as I will show in a moment. Conversely, no matter how much low-performing schools are punished, it is not likely to erase the performance gap. It is like punishing a person because he is tall.

Before I continue, it is important to understand that these results deal with the behavior of populations, not individuals. Even though we concluded that schools with many poor children perform more poorly, this does not mean that all poor children perform poorly. Our analysis predicts the performance of populations of children (schools), not individuals.

Point 2. When school performance is adjusted for socioeconomic composition, that is, when the effect of the number of poor children is statistically removed, only one elementary school in the county performed significantly better than average and none worse than average. In other words, if we take away the effect of student poverty, almost all of our schools are actually pretty much the same (see Graph 2), as judged from the published data (of course, better data might reveal differences).

These results press us to answer two questions. First, would it be fairer to compare schools only after adjusting for the effects of socioeconomic make-up? After all, there is little a school can do about the poverty of its students. Some educators argue that adjusting for such extrinsic factors is dangerous because pressure mounts to adjust for other extrinsic factors as well, thus opening a can of worms. Perhaps they are right, but shouldn’t we, the public, at least acknowledge that these relationships exist, and that some have large effects, and that they should be part of the public debate?

The second question is, when we take away the socioeconomic effect, is there still evidence that our schools are failing? If you choose to answer yes, on what evidence do you base your choice? . Do you have data to support your belief? The published data show that most of the differences among our schools are associated not with differences in the efforts or competence of schools or teachers, but with the degree of poverty of their student bodies, a factor that is beyond the control of the schools

So can schools improve test performance? The answer is, yes. Over the last five years, the scores have crept upward about 20 points, finally becoming a significant trend in 1996-7. However, this small improvement was experienced by all schools more or less equally, so that the difference between the best and worst schools changed little, remaining about 110 points. Thus, Point 3 is: Yes, schools can improve performance, but these improvements do not erase or even reduce the effects of socioeconomic composition.

How does all this relate to our efforts toward school improvement? First, it means that the political debate about school performance is focused on features of the educational landscape that either have little or no impact, or whose impact is not different from school to school, and largely ignores the factor which determines most of the differences among schools. We have to ask, is this a reasonable way to frame the argument?

Second, we need to focus on the link between poverty and low performance. What is it about the culture within poor families that is expressed as low performance in public schools? The educational level of the parents, especially the mother, is known to be related to a child’s performance, but parental education is neither under discussion, nor a mission of our public schools. Unless we redefine this mission, we need to accept that most of a school’s characteristics reflect those of the families whose children attend that school, and that our ability to change the achievements of that school will always be relative and limited.

Third, once we see the causes of differences in school performance clearly, we need to take these causes into account when we propose remedies. For example, vouchers are likely to have a negative effect on school performance. The children left behind after voucher-sponsored children have transferred out, are likely to be from the poorest, least motivated, most stressed families, who cannot afford the transportation, or are not involved enough, or have insufficient parental education, or cannot cope in one way or another.

Fourth, this analysis raises a question that is far more important and interesting than the overall performance of each school: do students from similar socioeconomic backgrounds perform differently in schools of different socioeconomic composition? Does a middle-class child with college-educated parents test out worse in a school like Woodville than in a school like Gilchrist? Does a poor child with parents who dropped out of high school test out better at Killearn than at Bond? The school performance data published in the Democrat do not allow such an analysis, and fall seriously short of adequate. However, the Leon County school system collects, but does not publish, the data that would allow this analysis to be made. What would we do if the answer were yes? What if the answer were no? Either way, the answer ought to shape our plans for improving education.

Perhaps most important of all, this analysis points out that ultimately, the performance of schools is irrelevant. The effectiveness of our educational efforts is much more strongly dependent on the cultural background each student brings to school than it is on the overall performance of the school. We need to focus our attention on the performance of each individual student, because only then do we have any hope of overcoming the effects of poverty on educational achievement.