More on Education Reform and Campbell's Law
04/24/12 09:52 AM Filed in: School Reform
Yesterday I posted a bit on Campbell’s Law. To repeat:
"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Wikipedia entry
My evaluator pals relate numerous seemingly innocent examples that might represent the Law at work. A generic (and hypothetical) example: imagine well intentioned reformers develop a program to increase the number of disadvantaged students applying to and matriculating at four year colleges. After one year, the program operators (and the funding source) were disappointed by the low rate of success. Counselors and teachers who worked with the students suggested re-examining admission criteria. They observed that many students admitted during the first year of operation were on paper academically qualified to enter four year colleges, but exhibited only superficial interest in higher education. Admission criteria the second year were refined to include interest level in college as well as include academic readiness–think of it as learning from experience. Outcomes for the second year of operation were dramatically better.
Did the adjustment of admission criteria represent a corruption of standards? If the original goals included increasing the number of students interested in attending college, unless the goals were also adjusted, then the change in admission process might be Campbell’s Law at work. If in the second year, the goals were explicitly revised to focus only on application and matriculation, then perhaps changing admission criteria was not as clear an example of corrupted standards. In practice, busy program personnel are so caught up in the day to day demands of their work that goals sometimes implicit shift without awareness. So perhaps there was in this instance a “slippery slope” kind of corruption in which admissions criteria were changed, but the operators never explicitly addressed the goals issue–and continued to perceive and describe the program as unchanged. But the Law makes no distinction between unintended and intended forms of standards corruption.
A second hypothetical example: a charter school operation within a public district was under close scrutiny by many community interests. For each of the first five years operation, standardized achievement test results in the charter schools were significantly higher than in non-charter schools in the district. After several years, it was discovered the charter operators gradually changed policies, by reducing the number of student in their program with learning disabilities and a history of behavior problems. Charter operators argued they were only trying to maximize the number of students benefiting from an academically challenging instructional program. Critics argue they had stacked the deck and bled desperately needed funds away from the rest of the schools in the district, who could not “screen out” students so easily; and that the charters school were duplicitous in claiming their program accounted for the superior achievement gains. The charter operators said they did not stack the deck deliberately, that they tried very carefully to screen out only a limited number of students ill suited to their program. A professional evaluator was consulted; she suggested that a careful review of the selection process might be yield useful information provided the charter operators had kept detailed records on each case screened or counseled out. The dispute continues.
Then there is out-right fraud. Fraudulently changing achievement scores to protect a program under scrutiny has happened, and likely will happen again. Making up and changing achievement scores to protect a program under scrutiny has happened, and likely will happen again. Fraud cases might be easier to recognize as examples of Campbell’s Law.
But it is more difficult to decide when the issue involves the sometimes complex and difficult job of selecting participants in or out of a high value program. How can we decide whether it is corruption or not when operators seek to improve the functioning of a program by selecting in those mostly likely to benefit?