A Missing Piece in the Debate on School Performance

by Walter R. Tschinkel

Professor of Biological Science

Florida State University

e-mail: tschinkel@bio.fsu.edu

Tel.: 850-644-4489; Fax: 850-644-0481

Our family has long supported public schools, and has followed the public debate about school performance with interest for several years now. Five years ago, the legislature mandated that standardized tests were to serve as measures of school performance, and that they were to be published for all to see. As a scientist, I felt it was essential to analyze these school performance data before rushing to champion "remedies". I want to share the outcome of this analysis because it demonstrates that most of the proposed solutions are political expediencies, and are irrelevant to the real challenge of assuring that all of our children receive a quality education. Briefly, the analysis shows that practically the only factor that explains the differences in performance among our public schools is the proportion of the children that come from poor families. Because we fail to recognize this connection, we ignore what is most important. Instead, we debate remedies that, at best, can have only minor effects, we spend money on ineffective measures and make decisions that are shots in the dark.

Annual measures of the performance of each county’s elementary schools have been published in many local newspapers for the last 5 years, and are based on standardized math and reading tests. The Florida Department of Education also makes the data for all school systems available on their website (http://www.firn.edu/doe/bin00068/doestart.html), from which I obtained the 1996-7 data for seven major school systems of Florida (Dade, Broward, Palm Beach, Pinellas-Hillsborough, Orange and Duval Counties). Because the math, reading and Florida Writes scores are highly correlated, I used the combined test scores in the analysis. The annual school report also contains information on the characteristics of each school and its students. School characteristics include the number of teachers, their average years of experience, the frequency of master’s degrees, the number of students, the average class size, the number of staff, the per-student cost for several types of students and how the budget is allocated among instruction, administration and support. Together, we can call these "school factors". The student body is characterized by several "student factors": their mobility, the frequency of long absences, the frequency of gifted students, of students with disabilities, of students with limited English proficiency, and finally the percent of students on supported lunch. The last is important because it can be used as an estimate of the percent of the student body from lower income families, because to qualify for this lunch support requires that family income be below a certain level.

A simple inspection of the data shows that there is a striking spread in the performance of the schools --- in all counties, the top schools have scores three to seven times greater than the bottom ones. To find out what might be causing this variation, the data were analyzed using a reliable statistical method called regression, which determines which factors are related to which others and how strongly they are related. I first analyzed school factors, then student factors, then both together. The outcome of this analysis was strong and unambiguous, and gives rise to my first point: Point 1. Most of the differences in school performance are associated with characteristics of their student bodies, not with characteristics of the schools. In all cases, sixty to eighty percent of the differences in school performance are associated with a single factor--- the percent of students on supported lunch, that is the percent of the student body from poor families (see graph 1). Factors obviously detrimental to student performance, such as mobility and long absences were strongly related to the percent of students on supported lunch, and probably are part (but not all) of the cause of low performance.

What about school factors? Whereas some of these explained some of the differences in school performance, many of these school factors were themselves related to the proportion of poor students. As a result, when both school and student factors were analyzed together, student poverty factors were overwhelmingly stronger predictors of school performance than were any school factors. In most counties, the only school factors having any significant predictive power were teacher experience (or training) and regular cost per student (sometimes class size), none of which was strong. Typically, as the regular cost per student increases, school performance actually decreases, but this relationship is weak. It is usually the result of low-scoring schools assigning fewer students per teacher in order to create smaller classes, raising the cost per student. School performance often increased with average teacher experience (or masters degrees), but the effect was weak. Furthermore, the higher the poverty levels of the schools, the less experienced or trained were the teachers, exacerbating the effects of student poverty. The racial composition (which was available only for Leon County) is often cited as affecting performance, but it has no effect except through its strong association with poverty. Mostly white schools and have similar scores to mostly black schools with similar levels of poverty.

Turning our finding around, it follows that only a small proportion of the differences among schools remain to be explained by factors other than those associated with socioeconomic composition. Most of the various "causes" and remedies discussed in the public forum are included in this small proportion, so none of them can account for more than a small fraction of the differences in school performance.

Recognizing that most of the differences in school performance are associated with socioeconomic make-up of the student body, it follows that one of the surest ways, in fact the only sure way, to improve performance is to reduce the proportion of poor students. This is exactly what happens in our exurban schools--- their high performance is almost entirely due to the relative absence of poor students, as I will show in a moment. Conversely, no matter how much low-performing schools are punished, it is not likely to erase the performance gap. It is like punishing a person because he is tall.

Before I continue, it is important to understand that these results deal with the behavior of populations, not individuals. Even though we concluded that schools with many poor children perform more poorly, this does not mean that all poor children perform poorly. Our analysis predicts the performance of populations of children (schools), not individuals.

Point 2. When school performance is adjusted for the socioeconomic composition, that is, when the effects of the proportion of poor children are statistically removed, the vast majority of schools did not perform significantly better or worse than average, and performance was not related to "school quality". In other words, if we take away the effect of student poverty, we can no longer tell the difference in their performance (see Graph 2). (This is not to say that they may not actually differ, but the published data do not allow us to make this discrimination.)

These results press us to answer two questions. First, would it be fairer to compare schools only after adjusting for the effects of socioeconomic make-up? After all, there is little a school can do about the poverty of its students. Some educators argue that adjusting for such extrinsic factors is dangerous because pressure mounts to adjust for other extrinsic factors as well, thus opening a can of worms. Perhaps they are right, but shouldn’t we, the public, at least acknowledge that these relationships exist, and that some have large effects, and that they should be part of the public debate?

The second question is, when we take away the socioeconomic effect, is there still evidence that our schools are failing? If you choose to answer yes, on what evidence do you base your choice? . Do you have the data to support your belief? The published data show that most of the differences among our schools are associated not with differences in the efforts or competence of schools or teachers, but with the degree of poverty of their student bodies, a factor that is beyond the control of the schools

So can schools improve test performance? The answer is, yes. In Leon County over the last five years, the scores have crept upward about 20 points, finally becoming a significant trend in 1996-7 (I did not have multiple-year data for other counties). However, this small improvement was experienced by all schools more or less equally, so that the difference between the best and worst schools changed little, remaining about 110 points. Thus, Point 3 is: Yes, schools can improve performance, but these improvements do not erase or even reduce the effects of socioeconomic composition.

How does all this relate to our efforts toward school improvement? First, it means that the political debate about school performance is focused on features of the educational landscape that either have little or no impact, or whose impact is not different from school to school, and largely ignores the factor which determines most of the differences among schools. We have to ask, is this a reasonable way to frame the argument?

Second, we need to focus on the link between poverty and low performance. What is it about the culture within poor families that is expressed as low performance in public schools? The educational level of the parents, especially the mother, is known to be related to a child’s performance, but parental education is neither under discussion, nor a mission of our public schools. Unless we redefine this mission, we need to accept that most of a school’s characteristics reflect those of the families whose children attend that school, and that our ability to change the achievements of that school will always be relative and limited. Turning this reasoning around, the high performance of relatively affluent schools must be largely attributed to the influence of the students’ families, not the excellence of the schools.

Third, once we see the causes of differences in school performance clearly, we need to take these causes into account when we propose remedies. For example, vouchers are likely to have a negative effect on school performance. The children left behind after voucher-sponsored children have transferred out, are likely to be from the poorest, least motivated, most stressed families, who cannot afford the transportation, or are not involved enough, or have insufficient parental education, or cannot cope in one way or another.

Fourth, this analysis raises a question that is far more important and interesting than the overall performance of each school: do students from similar socioeconomic backgrounds perform differently in schools of different socioeconomic composition? Does a middle-class child with college-educated parents test out worse in a low-performing school than in a high-performing one? Does a poor child with parents who dropped out of high school test out better at a high-performing school than a low-performing one? The school performance data published by the Florida Department of Education do not allow such an analysis, and fall seriously short of adequate. However, each county school system collects, but does not publish, the data that would allow these analyses to be made. What would we do if the answer were yes? What if the answer were no? Either way, the answer ought to shape our plans for improving education.

Perhaps most important of all, this analysis points out that ultimately, the overall performance of schools, as currently reported, is of little relevance. The effectiveness of our educational efforts is much more strongly dependent on the cultural background each student brings to school than it is on the overall performance of the school. We need to focus our attention on the performance of each individual student, because only then do we have any hope of overcoming the effects of poverty on educational achievement.