Beware of Lies, Damn Lies, and Statistics

graph-41423_1280I liked this book when I read it years ago.  Still do. I really look at numbers and think about what they do and do not tell us.  When I learned that the Florida Legislature had changed the evaluator for the Florida Tax Credit scholarship (FTC) program, I wondered why.

Before we judge, let’s look at the data from the April 2014 FTC scholarship program for private schools.  These are the scholarships funded by tax rebates to corporations.

How are tax credit voucher programs evaluated in other states?  Let us know.  Here is what was reported in Florida.

David Figlio: Evaluation of Florida Tax Credit Program Participation, Compliance and Test Scores in 2013. April 2014

Professor Figlio reports the data and describes his methods.  Sometimes it is hard to tell if the glass is half full or half empty, but the results are there if you look.

I have excerpted his major findings on the achievement of students who participated in the FTC program.  Eligibility for this program in 2012-13 was limited to low income and minority students whose family incomes were below $44,000.  In 2014, the Florida Legislature expanded the program to include at least partial scholarships for family incomes up to $62,ooo.  Was this warranted?  On the surface one might say, “No harm, no foul”.

Achievement gains for FTC students were slightly below average for mathematics and slightly above average for reading.  The differences were not statistically significant, so you can say that students on average made a year’s worth of gain.  (But there is wide variability from very good to very poor improvement in learning.)

Look underneath the averages:

–A strong relationship in gain scores was reported for schools with more than 180 days (10.8% of schools); weakest gains were for schools with fewer than 180 days (17%).  Public schools are required to have 180 days.  Why would private schools have fewer days?

–Differences exist in achievement gain scores by religious orientation of the schools.  Strongest gains were achieved for non religious schools and Catholic private order schools in mathematics. Other Catholic, non-affiliated Christian schools and Seventh Day Adventist schools had gains in reading but not in mathematics.  All other religious denominations (nearly 50% of scores) reported a negative gain in reading and mathematics compared to national percentile rankings.  Are these differences due to socio-economic or school quality factors?

–Students who struggle the most tend to leave and return to public schools.  These students’ scores (about 5%) were not included in FTC results.  Another two percent, also struggling students, were reported ill on testing days.  Did public schools include returning students in their scores?

–Only about 12% of FTC scholarship participants came from public ‘D or F’ schools.  Thirty five percent came from public ‘A’ schools.  Why are students leaving good schools?

–4 private schools closed from the previous year, and the Florida DOE suspended 129 schools from the program.  Why?

–In the 2013 report, Dr. Figlio stated that it is not advisable to compare FTC students with public school gains any longer; they do not take the same tests.  The legislature changed evaluators in 2014.  Here is a quote from Professor Figlio regarding the new legislation to revise the methodology for the FTC program evaluation:  “My read of the new law (SB850) is that average scores will be reported, rather than gain scores. This will, sadly, mean that it will be harder to have a reasonable read of how children are progressing. In addition, my research (and that of others) suggests that accountability based on average scores is much more likely to lead to cheating and other forms of manipulative behavior on the part of educators than accountability based on gain scores.”

What is the ‘truth’ about the FTC scholarship program?  Dr. Figlio provides the data and explains the limitations of the study.   His conclusion is that FTC students disproportionately represent the lowest performing  students and schools.  We need to be clear on this.  He means more than expected given the percentage of students with low scores in public schools.) Yet, on average, FTC students achieve the same learning gains as the average student nationally.

Averages mask realities.    Students of all achievement levels have FTC scholarships.  There is a 5% difference in the percentage of eligible FTC students and public school students in the lowest scoring group (28.7 % of the FTC students are in  the lowest quartile of achievement compared to 23.5% of eligible public school students).  These are the students, moreover, who are most likely to leave the FTC schools and return to public schools.  Differences in the percentages of students in higher achievement groups of smaller.

What have we learned from these FTC evaluations over the last several years?  They are relatively consistent.  Students who struggle the most tend to leave public schools and then return to them.  Students who remain in private schools learn at about the same rate as the average student nationally.  A longer school year appears to improve achievement.  Changing the evaluator is not likely to change the results.  Is there harm?

Posted in Achievement, Research studies, Tax credit scholarships.

One Comment

Leave a Reply