FYI: Charter School Performance in 16 States

Study Methodology

Stanford University’s Center for Research on Education Outcomes (CREDO) has released new findings about American choice schools in 16 states. The first of three reports scheduled to be released in 2009, “Multiple Choice: charter School Performance in 16 States” searches for measures of charter school “effectiveness.” CREDO chooses student outcomes or standardized test scores as an indicator of effectiveness. Specifically, the approach is a value added approach which takes initial student scores as a base which can be removed from latter scores in order to measure the charter school’s effect on score growth. The data in the report, a project undertaken with the use of a data collection methodology pioneered by RAND corp., is an advance over previous studies as it utilized standardized data from a good number of charter schools nationally, about “1.7 million records from more than 2400 charter schools.” Standard data allows for a reading of charter school characteristics between individual states that are comparable. Also, taken as a whole, the data is representative of all charter schools on the national level. CREDO notes: “the states included in this study enroll more than half the charter school students in the United States, so the consolidated results begin, for the first time, to tell the story of the policy of charter schooling at a macro level.” In order to mitigate the effects of selection bias, the CREDO researchers employ a Virtual Control Record (VCR) methodology by which a virtual twin for each student in the study is created to be sure that both charter students and the traditional public school (TPS) students are comparable. Basically, by taking both charter and TPS students from the same origin, the students are considered characteristically similar.  CREDO explains thus:

We identify all the TPS that have students who transfer to a given charter school; we call each of these schools “feeder schools.” Once a school qualifies as a feeder school, all the students in the school become potential matches for a student in a particular charter school. All the student records from all the feeder schools are pooled – this becomes the source of records for creating the virtual match. Using the records of the students in those schools in the year prior to the test year of interest, CREDO selects all of the available records that match each charter school student.

Study Findings

To begin, CREDO’s major findings do not do much to champion the cause of charter schools. Their reports finds the following: on average, charter students experience a decrease in math and reading when compared to students in traditional public schools. But these aren’t the most interesting findings in the report. The most interesting conclusion, in fact, is probably that the students performing best in choice schools happen to be English Language Learners (ELL) and students in poverty. Also, despite there being a significant demographic overlap between students in poverty and many American minority groups, CREDO finds that Black and Hispanic charter school student performance is “significantly worse” when compared to the performance of demographically comparable students (data twins) enrolled in traditional public schools. The two conclusions together are interesting in as much as charter schools are often viewed, specifically, as a positive alternative to TPS education for low income students which, as mentioned, tend to overlap demographically with minority groups. This is troubling information, though the report mentions that the statistics do not yet reveal any specific causes to account for the data.

According to the report, race and economic status are not the only determinants of educational outcomes. Time and education level are also factors. In regard to time, CREDO researchers find:

Students do better in charter schools over time. First year charter students on average experience a decline in learning, which may reflect a combination of mobility effects and the experience of a charter school in its early years. Second and third years in charter schools see a significant reversal to positive gains.

Regarding education level, charter school students at the elementary and middle school level were found to have learned at “higher rates” than their peers in TPSs and at lower rates on the high school level. From another vantage, a charter that serves only high school level students may be predictably worse. The proper niche for charters, therefore, may be in serving the lower grades, a conclusion that must be considered by policy makers as perhaps now more than 48 percent of all charters, according to Center for Education Reform data in 2005, serve students on the high school level. Further, in regard to CREDO’s prior conclusions, the National Charter School Research Project also concluded in 2005 that “nation¬ally, charter schools serve a larger proportion of minority and low-income students than is found in traditional public schools, a characteristic due largely to the disproportionate number of charter schools located in urban areas.” Often, the mission of charter school education reform is to offer to minority and underserved communities a positive alternative to their local TPSs which are by many reformers considered to be the root cause of unsatisfactory educational outcomes. This thought, presumably, is what has motivated the CREDO researchers to undertake the comparison. However, CREDO’s data may run contrary to the intuitions of many reformers. And, there is a danger that, if policy makers are not informed, certain communities will be harmed by the same act of reform that was intended to serve them and their children better. Indeed, if the data allows us to do so, taking all the report’s conclusions together produces the image of the ideal charter: the ideal is a charter that serves only non-Hispanic and non-Black students in grades K-8. Of what use, we might then ask, is such a school?

These questions aside, once they are reproduced and thus confirmed in other studies these conclusions should have a significant impact on the way policy makers view charter schools in their own states. Knowing how and for whom charter schools work best might cause education officials to pause before closing an underperforming school as, for instance, the number of first year students enrolled at a charter may bear significantly on educational outcomes reported for a certain year. In this way, for officials, the “first year effect” might be actively treated with some new policy, or acknowledged as more an enrollment phenomenon rather than a determinant of charter failure. Rather than contributing to keeping charters open, however, researchers understand that their efforts may have the opposite effect as keeping charters open is not the greatest challenge facing charter school reform, currently in an “authorizing crisis.” CREDO describes that “Evidence of financial insolvency or corrupt governance structure, less easy to dispute or defend, is much more likely to lead to school closures than poor academic performance.” As poor performing schools do more harm than good to the charter reform movement by corrupting the data gleaned from successful charters, from the perspective of statistical research, CREDO suggests that their work should add imperative to the difficult task of closing underperforming schools. Unfortunately, according to their research, many of these underperforming schools may serve the majority of at risk students.

Policy Environment

Perhaps more interesting than the forgoing conclusions, the effect of policy environment on charter school effectiveness will be particularly useful to officials and those who study governance and education administration. Studying the following: “the use of a cap on the supply of charters”; “the availability of multiple authorizers”; and “the availability of an appeals process to review authorizer decisions.” Caps, presumably, limit the total number or charters and thus the overall number of low performing charters by increasing the pressure to close old and to grant entry to new charters. More authorizers may add more knowledge to support judgments about closing or authorization—and it may also permit poor performing schools to pick sympathetic judges. Lastly, the opportunity to appeal the decision of an authorizer may “increase the proportion of marginal schools, dragging down the overall performance of the sector” or it may add a higher level of scrutiny to authorizer judgments, the CREDO researchers suggest. Briefly, CREDO found that the existence of charter caps lowers student performance in a state; multiple authorizers has “a significant negative impact on student academic growth”; the opportunity for appeal seemed to have a growth effect for student academic outcomes, though the conclusions are tentative because none of the states changed their policies during the study period.

In the report, there is also included the breakdown into math and reading performance for several kinds of students and the analysis of outcomes with both charter and TPS understood in a market framework. At large, much of the research really draws attention to how subtle the differences are between charter and TPS environments. The subtlety of the differences almost lends support to less recent studies that conclude no significant difference between charter and TPS. However, though these differences are subtle, they are still important given the size and demography of the student population enrolled in charter schools.

Is it the case that the perfect charter school student looks something like this?: in grades K-8, in poverty, an ELL of non-Hispanic or non-Black heritage. Similarly, is the ideal school one which operates in states that: shun caps, have a single authorizing entity and favors appeals?  Perhaps these characteristics are only determinants of success when they are each by themselves.  By lumping them together we supposes that all these factors are inclusive.  Further, making up ideal schools and districts based on the data is interesting but not very productive.  If this kind of idealization accomplishes anything it begs us to investigate more – into the why and the how of the CREDO’s findings.

New Jersey

Though New Jersey is not part of the CREDO’s analysis, a simple inventory of the identified indicators would be interesting. Unfortunately, no data exists for some of the indicators such as race and at-risk which might be why New Jersey was not included among the other states in the charter school study. We do have some information, however. According to the New Jersey Department of education, over 18,000 New Jersey students attended 62 charter schools as of May 2009. About 13 of those 62 charters serve over 4,413 students, roughly 24.5 percent of all students. Further, regarding state policy, New Jersey, joined by 14 other states, Arizona, Colorado, Delaware, Florida, Georgia, Kansas, Maryland, Minnesota, Nevada, Oregon, Pennsylvania, South Carolina, Virginia, and Wyoming, has no charter school cap. However, the rigorous process of application and accountability to the state’s sole authorizer adds a significant amount of a barrier to charter school entry.

1. Though there is no cap and no leveraged pressure to eliminate low performers, school performance assessment is administered every two years. Upon entry, charters are all granted a four year term, at the end of which they must apply for renewal to the education commissioner.

2. New Jersey is the only state to invest in one person, the commissioner of education, the responsibility of authorization. As most other states charge single or multiple boards with that responsibility, the effects of New Jersey’s unique charter policy and that of other states are not comparable in studies such as the foregoing.

3. New Jersey’s appeal process is open to the affected school district as well as the charter operator as a means of reconsidering an authorizer’s decision. Appeals are directed to State Board of Education. Whether this has an effect on growth in New Jersey charters is uncertain. By 2005, New Jersey had authorized 91 charter school applications and had received 237 in total. Despite the availability of appeal, the number of charters in New Jersey has never grown over 67, the number reached in 2005. No great augmentation of the charter school count has occurred since the beginning of charter schools in New Jersey.

Though the performance of New Jersey’s charters cannot be directly understood in reference to the CREDO findings, it would still be interesting to add to this study the relative effects that New Jersey’s policy environment has on student performance. Policy makers should pay attention to this and the following two reports from CREDO as, even if they do not support them as a solution to the education of underserved student populations, there is much to be learned about the education environment of traditional public schools in the laboratory environment of the charters.