Continued Misuse of Test Score Data to Rate and Rank Schools

Zachary Smith, the Plain Dealer‘s current data wonk, just published another article ranking Ohio’s schools, Ranking Ohio Public High Schools from 1 to 823, based on Ohio’s 2022 state school report card’s Performance Index.  A few weeks ago he ranked all of the state’s school districts by the same Performance Index.

Ohio released its annual state school report cards in mid-September and changed its ratings from A-F to a five star system.  At the time, the Plain Dealer‘s Laura Hancock reported that because the state’s schools were dealing with COVID all of last year and the federal government, therefore, eliminated demands for school improvement plans, the state would not calculate an overall summative rating for schools and school districts: “This year the Ohio Department of Education is not offering an overall rating for each school and district, due to the reprieve on sanctions. In future years, there will be an overall star rating.”

This year the stars were awarded in five categories: Achievement, Early Literacy, Graduation Rate, Progress, and Gap Closing.  A new category was added, “College, Career, Workforce and Military Readiness,” but there was were no stars assigned this year in this area due to ongoing COVID recovery.

While the state did not assign any overall summative grade for schools and school districts this year, reporter Zachary Smith discovered—in the state’s description of the category of “Achievement”what he considers an overall way to rank the state’s schools. In its description of measuring Achievement,  the state lists a “Performance Index” number for each school and school district.  The state says it calculates the Performance Index based primarily on aggregate standardized test scores: “The Achievement component represents the number of students who passed the state tests and how well they performed on them. This component includes three additional performance indicators —the Chronic Absenteeism Improvement Indicator, End-of-Course Improvement Indicator and Gifted Indicator.” Based on Performance Index scores, Smith and the Plain Dealer have been ranking the state’s school districts and high schools.

The problem is that test scores are known to reflect a community’s family economics more than they measure the quality of a school or school district. I wish Smith would go back a couple of years to look at Rich Exner’s profound 2019 Plain Dealer article on Ohio’s school report cards. See How Closely Ohio School Report Card Grades Trend with District Income. Here is an example of one of Exner’s bar graphs, which profoundly depict the story.

Two weeks ago, Jack Schneider, a professor at the University of Massachusetts, Lowell and Joel Boyd, the superintendent of the Lowell Public Schools, explained that the correlation of standardized test scores with family income has been an issue from state to state through the past two decades since No Child Left Behind thrust us into school accountability based on standardized test scores: “As research indicates, test scores are highly indicative of the inequalities that afflict our communities, and are not a valid basis for determinations about overall school performance… Scholars have repeatedly shown (that) the leading predictors of student standardized test scores are demographic variables like family income and parental educational attainment.”

Schneider and Boyd show how test scores this year particularly are likely to reflect the disparate economic realities for families during the COVID pandemic: “Imagine that in one school community, students were insulated from the worst effects of the pandemic. Parents were able to work from home, oversee remote schooling, and offer additional support. Young people felt safe, and their families remained intact. Family resources were deployed for educational purposes and enrichment. The pandemic was a challenge, but one that was mitigated to a significant degree.  In the other school community, students felt the effects of the pandemic acutely. Family members became sick, were hospitalized, and may have even died.  Working in so-called essential fields drew caregivers away from home during the day. Internet was often slow and unreliable, and students competed for quiet space with siblings. Young people felt vulnerable, frightened, and isolated.”

The No Child Left Behind Act, formulated in 2001 and signed into law on January 8, 2002, restructured public education by demanding quantitative, standardized-test-based school accountability and by using sanctions to punish the public schools struggling to raise aggregate scores.  Gail Sunderman was one of NCLB’s early critics as the lead author of NCLB Meets School Realities, published for the Harvard Civil Rights Project in 2005, in collaboration with James Kim and Gary Orfield. Sunderman is now a research scientist in the Department of Teaching and Learning, Policy and Research and director of the Maryland Equity Project at the University of Maryland School of Public Policy.

Sunderman reminds readers that No Child Left Behind was replaced by the Every Student Succeeds Act in 2015, but the new law still requires states to rate their schools and put the lowest scoring schools on corrective action plans. Here are Sunderman’s concerns today about the damage wrought by state school rankings: “There’s… evidence that state school rating systems often reflect personal and ideological preferences of state leaders… States with a more liberal orientation… are more likely to incorporate indicators related to school quality and indicators of student success, such as growth measures, while states with a more conservative leaning maintain a focus on student test scores…  While school rating systems may be a practical means to a political end, their educational value is questionable. Despite the proliferation of school rating systems, there is very little peer-reviewed, empirical research on their effects on student performance, and school and teacher practices…  Summative ratings also tend to obscure the well-documented relationship between student achievement scores and demographic variables, most notably race and socioeconomic status. An analysis of the Maryland five-star rating system, for instance, examined why no high-poverty schools earned a five-star rating, but when the researchers adjusted ratings to account for economic disadvantage, the number of five-star schools increased.”

Sunderman concludes: This inability of summative school ratings to distinguish school performance from student demographic variances disproportionately harms schools serving marginalized children and inflates the quality of schools serving wealthy and white students.”

This blog previously covered Ohio’s 2022 school ratings here.


Injustice: The Disparate Impact of EdChoice Vouchers Across Ohio School Districts

Ohio Governor Mike DeWine just announced large, recession-driven cuts in statewide funding for public education at the same time more and more students in 140 of Ohio’s 610 school districts will be carrying EdChoice vouchers out of their local school district budgets to pay for private school tuition. Ohio’s EdChoice voucher program is funded through a local school district deduction, which sucks education program dollars out of local school district budgets.  At this time when Ohio faces a sudden and serious revenue shortfall due to Covid-19 job losses and business closures, the Legislature must immediately protect extremely vulnerable local school districts by merging EdChoice and another Ohio voucher program—EdChoice Expansion, which is fully state funded. The EdChoice voucher program was created by the state legislature. The Legislature ought to pay for the vouchers out of the state budget for which the Legislature is responsible.

In February, the Ohio House passed legislation which would begin collapsing the two programs into a fully state funded Buckeye Opportunity Scholarship voucher program. While the Ohio Senate did not concur with the plan, the pandemic-driven fiscal emergency has now radically reshaped Ohio’s school funding calculus. The growth of EdChoice vouchers next year in the school districts shouldering the burden of the EdChoice program will catastrophically reduce local school budgets. The burden falls unevenly, because the state has designated only 140 of the state’s 610 school districts for the punitive EdChoice program, and 40 of those EdChoice districts will suffer the greatest impact.

Last week Ohio Governor Mike DeWine announced an immediate state budget reduction before the end of the fiscal year (ending June 30, 2020) of $775 million, with $300 million of the cuts to be absorbed across the state’s K-12 public school districts. The Covd-19 pandemic has led to business shutdowns and job losses, which reduced Ohio’s state tax revenue through April by $776.9 million below fiscal year estimates.

The EdChoice voucher program is particularly burdensome for public school districts because the vouchers are funded by a method called “school district deduction.”  Before the current school year, only 255 individual public schools had been designated for the EdChoice voucher program, but for the 2019-2020 school year, the state doubled the size of the program, for a total of 517 EdChoice Designated Schools. Legislators had been scheduled to designate 700 additional EdChoice school buildings for the program for the 2020-2021 school year. However, at the end of March,—under pressure to shut itself down due to Covid-19—the Legislature merely froze in place (at 517) the number of public schools targeted for EdChoice vouchers.

In his daily e-mail on May 5, William Phillis, executive director of the Ohio Coalition for Equity and Adequacy of School Funding, published a white paper by Susan Kaeser, a co-convener of the Heights Coalition for Public Education. In her new paper, Kaeser explains that, although legislators did not enact an expected expansion of the number of EdChoice Designated schools, they did nothing to change the ongoing operation of the program, which sucks money out of the budgets of 140 of the state’s 610 school districts, according to Kaeser’s data.  In these 140 school districts, students in the attendance zones of specific EdChoice Designated schools  will continue to qualify—including students who already have a voucher, their siblings, incoming Kindergartners, and all students in grades 9-12.  These 140 school districts will continue to experience EdChoice voucher growth. “This includes 40 districts that have had at least one designated EdChoice school since FY 2018. The other 100 districts started to fund EdChoice vouchers in the 2019-20 school year.”

Students qualify for EdChoice vouchers when they live in the attendance zone of a school the state has deemed “EdChoice Designated” based on state report card school ratings that rely primarily on standardized test scores. Kaeser explains that, because aggregate test scores correlate highly with family and neighborhood income, EdChoice vouchers in Ohio decimate the local budgets of some of the state’s most vulnerable school districts: “The financial burden of EdChoice vouchers is not shared evenly by school districts across the state, punishing some districts and not others. This disparity is made worse by the reality that the majority of students in the districts most affected by vouchers live in poverty and are not white… This means some districts lose funds and others don’t. Test scores determine if a school is designated Edchoice.”

Kaeser presents background on the correlation of poverty with low test scores: “Research demonstrates that aggregate family income, neighborhood economics, and many other out of school factors drive test performance…. As education researcher David Berliner notes, ‘Research demonstrates that if you know the average income, the average level of parent education, and the percent of single-parent households in a community – just three variables- you can predict with great accuracy the performance on the standardized test scores used by the community to judge its schools.'”

Kaeser also quotes the Plain Dealer‘s Patrick O’Donnell who describes research by Ohio’s Howard Fleeter: “State test scores continue to rise right along with a school district’s affluence, and fall as poverty rates increase…. Ohio may have changed academic standards and its state tests last school year, but the recurring relationship between test scores and poverty remains the same…. Fleeter has reported the relationship between test scores and family income on an annual basis the last several years…. As he does each year, Fleeter compared the percentage of students scoring ‘proficient’ or better on state tests in each school district to the percentage of students considered ‘Economically Disadvantaged’….”

Each EdChoice voucher removes from a school district budget $4,650 for a student in grades K-8 and $6,000 for a high school student. According to Kaeser, “Ohio is the only state that funds vouchers by the deduction method which assigns funding responsibility to the public school district of residence of the voucher student. Prior to FY 2020, the state contributed to voucher costs by providing to the school district for each voucher student the same amount of per pupil state aid as public school students in that district.” But per pupil aid varies widely among Ohio’s districts and in a significant number of school districts, the state’s per pupil contribution was significantly less than the cost of the voucher.

However, the fiscal damage of EdChoice to school district budgets grew explosively in the current, 2019-2020 school year. The Ohio Legislature did three things that exacerbated the severe impact of the vouchers for the 140 school districts with EdChoice Designated schools. First, the Legislature expanded the number of eligible schools. Second, the Legislature expanded the program by allowing all high school students in EdChoice Designated school zones to qualify for a voucher even if they have never been enrolled in the school district whose budget will pay for the voucher. And third, as Kaeser reports: “The Legislature froze state aid to school districts at the FY 2019 level for FY 2020 and FY 2021…  Starting in FY 2020: In the current biennium, the state has frozen its contribution and now school districts must cover the full cost of new vouchers.”

The expansion of the EdChoice program by the Legislature in the current state budget especially targeted the 40 districts hardest hit by EdChoice: “When it comes to EdChoice vouchers, there are three kinds of districts in Ohio: 40 districts that have been EdChoice for multiple years and costs have accumulated over multiple years; 100 districts that have been EdChoice for one year; and 472 districts that have never lost funds to EdChoice vouchers. In FY 2019 there were 255 schools designated as EdChoice in 40 districts…. Because of multiple years of funds lost to Edchoice vouchers, the 40 districts with the longest history of EdChoice designation are the most vulnerable to any new voucher costs going forward. Because voucher use is concentrated in these districts, most new vouchers will also be awarded by them. This cohort of districts is the hardest hit by EdChoice vouchers.”

Here is Kaeser’s conclusion: “The Edchoice voucher program is expensive, impacts individual districts unequally, and fuels education funding inequality. In FY 2020, 92% of all EdChoice vouchers and 71% of all new EdChoice vouchers were awarded to students in 40 Ohio districts that educate 16% of Ohio’s students. This small share of Ohio’s districts…. are most likely to be the districts that will incur most of the cost of new vouchers in FY 2021…  Most of the students enrolled in the hardest hit districts are poor and minority. Students with the greatest needs are the most likely to lose out on public funds because of vouchers. The state is obligated to provide equal and adequate funding for all of its public school students. EdChoice vouchers do the opposite.”

Here we are at the very end of the FY 2020 Fiscal Year, with an additional fiscal blow to Ohio school districts: Governor DeWine is reducing funding to school districts by $300 million before June 30, and significant additional cuts are expected in the fiscal year that will end on June 30. 2021.

And there is breaking information that makes matters even worse for districts already shouldering the heavy burden of EdChoice vouchers. The state is allocating the $300 million budget cut to school districts on a per-pupil basis. And the state counts voucher students as though they are enrolled in the school district from whose budget the voucher is deducted.

For, Thomas Jewell explains the impact for the Cleveland-Heights-University Heights School District.  The CH-UH district will lose $1.4 million due to DeWine’s new budget cut, but $300,000 of that lost money will be because the state counts 1,400 voucher students as though they are enrolled in the district’s public schools.  Jewell quotes the district’s treasurer, Scott Gainer: “Unfortunately, the state chose not to do a straight percentage cut and instead did a ‘per pupil reduction’—including EdChoice students….”   First money flows out of the district budget for each voucher, and then the state applies DeWine’s new budget cut for the district on a per-pupil basis and inflates the school district’s budget loss by counting students who are not attending the public schools.

Jewell also quotes CH-UH Superintendent, Liz Kirby: “Superintendent Liz Kirby contends that the district has already endured the loss of $7.2 million this year by way of the EdChoice tuition voucher program, with the state legislature ‘freezing’ the current formula into next year, as well… ‘Our district is ground zero for losses due to the way EdChoice scholarships are funded, and this cut exacerbates our challenge….’ Kirby added that the district’s 1,416 EdChoice voucher students are part of the state’s deduction calculation, ‘so $300,000 of that $1.4 million deduction is for students who aren’t even enrolled in our schools.'”  It is all very discouraging for school district personnel and for the local taxpayers who will be asked to make up the difference. The state seems to be finding multiple ways to reduce state education funding for a particular set of Ohio school districts.

In the current fiscal climate, the Legislature must move immediately to end voucher funding by school district deduction. Reducing state school funding overall during the recession and at the same time expanding the diversion of state and local dollars out of school districts’ budgets to pay private school tuition will force vulnerable school districts to put large school levies on the November ballot or make deep cuts to programming for students. The combined loss of millions of dollars for the 40 most affected school districts is unsustainable. The disparate impact of the EdChoice program is unjust.

This blog has recently covered the crisis created in Ohio by the legislature’s recent expansion of EdChoice vouchers here, here, hereherehereherehere, and here.