Ronald GallimoreEveryone's a teacher to someone (John Wooden)

Focus more on processes to achieve valued outcomes

Brad Ermeling alerted me to this blog posting by Daniel Markovitz in the Harvard Business Review.

Markovitz points out several potential dangers of so called “stretch goals” including sapping motivation, fostering unethical behavior, and encouraging excessive risk-taking.
Better, he argues, to set reasonable goals and focus on process improvement. Process improvement refers to what is required to achieve goals, sometimes referred to as opening the “black box” to see how something works in order to make it work better.

He closed the blog with two sentences written for a business and industrial audience that could also be addressed to education, medicine, and behavioral interventions among others.

“The heavy lifting has to be done at the outset — a deep understanding of the current condition is a prerequisite for true improvement. This approach also requires a subtle — but critical — shift in focus from improving outcome metrics to improving the process by which those outcomes are achieved.”

Education reformers and policymakers are now engaged in a great debate about the value of standards and assessment. Once in a while someone alludes to Markovitz’s point that settings standards and developing outcome metrics accomplish little unless the mediating process is examined and improved. In education that mediating process is what transpires in classrooms. Teaching that provides effective learning opportunities for students.

Promising method for assessing teaching effectiveness

An Arizona and California research team (Kersting, Givvin, Thompson, Santagata, & Stigler, 2012) reported a novel and promising new approach to assessing teaching effectiveness. Teachers were asked to analyze thirteen 3 to 5 minute classroom video clips from fraction lessons, and write detailed comments for each. The researchers rated the written comments for how attentive teachers were to mathematical content and student thinking portrayed on the video and the degree to which teachers made suggestions for instructional improvement. They also rated the depth of teachers' analyses, e.g. was the written response purely descriptive or evaluative versus connecting analytic points to form a cause-effect argument. The team defined these 4 dimensions as reflections of a teacher’s usable knowledge for teaching fractions. The fraction clips covered such topics as part-whole relationships, equivalency, operations with fractions, etc.

But does a teacher’s “usable knowledge for teaching” transfer into the classroom? The research team addressed that question as well. Teachers who completed the video analysis were videotaped in their own classrooms teaching a fractions lesson, which was scored for instructional quality. Based on an extensive review of the mathematics teaching research, teaching quality was defined as developing concepts, appropriate use of representations to explain algorithms, and connecting concepts and topics. And the answer? Yes, a teacher’s usable knowledge for teaching is correlated with the quality of instruction they deliver in a classroom lesson.

But do these assessments of teaching knowledge and quality lead to more student learning? Yes. Students of the 36 teachers who participated in the study completed pre and post fractions quizzes. Teachers who did better on the video analysis task had higher scores on classroom teaching quality and their students had larger gains on the post-test fractions quiz. Usable knowledge predicted better classroom teaching and together these two assessments of teaching quality predicted greater student learning. Few studies have attempted to connect these three dots of knowledge, practice, and achievement, and even fewer reported positive correlations.

The approach used by Kersting, et al. (2012) is a promising alternative to the questionable approaches currently pursued at the national level. In the last several years, several major policy efforts have focused on assessing teaching quality as part of the standards and accountability reform. This latest wave of reform acknowledges that improved teaching is critical to improved student achievement. To help teachers, reformers have been developing teaching assessments based on live-observation of classroom instruction. Armed with powerful psychometric development strategies, researchers have been struggling to find a cost-effective way that educators can assess teachers based on a few or even a single classroom observation. This approach is questionable on several accounts. To get a reliable or accurate assessment of an individual teacher’s classroom practices probably requires multiple observations over at least a unit of instruction. The cost is prohibitive, and hardly appealing schools already strapped for resources.

A second limitation of live observation methods is the complexity of behavior to be captured. “Live observations are limited to whatever an observer can record. Checklists can be useful, but it is possible for a live observer to make only a limited number of reliable judgments at the speed required for classroom research. There simply is too much going on. Video, on the other hand, can be paused, rewound, and watched again. Two observers can watch the same video, independently, and go back to re-play and discuss those parts that they saw differently. Videos can be coded multiple times, in passes that require only limited judgments by an observer on any single pass. This makes it easier to train observers and enables reliable coding of complex events.
The most important advantages of video derive from its concrete, vivid, and “raw, un-analyzed” nature (i.e., the categories can be derived from the data rather than vice versa, leaving the data open to a vast array of analyses)
(Stigler, Gallimore, & Hiebert, 2000, p. 90).

It’s premature to argue that video clip analysis is a workable, scalable alternative to live observations. Although this newly published study replicates earlier work by Kersting and colleagues, so far only mathematics instruction has been investigated, and only with secondary school samples. But given the stakes, I hope that national policymakers will not become so wedded to live observations that the nation spends massive resources on a single approach when such a promising alternative is available. It is possible to imagine that rather than an army of classroom observers, knowledge useful for teaching could be assessed using modern technologies at a fraction of the cost of live observations
(Gallimore & Stigler, 2003).

The Kersting, et al. study was published in the June, 2012 issue of the
American Education Research Journal.

Kersting, N.B., Givvin, K. B., Thompson, B.J., Santagata, R., & Stigler, J. W. (2012). Measuring usable knowledge: teachers’ analyses of mathematics classroom videos predict teaching quality and student learning.
American Education Research Journal, 49, 3, 568-589.

No single number can identify a good school.

Every once in a while a comment comes along that bears repeating.
Many times. This is one: “There is no single number that anyone can look at and say this is a good school or this is a bad school.”
(Dr. Joan Herman, Director, UCLA’s CRESST (National Center for Research on Evaluation, Standards, & Student Testing).

For more wise words by Joan, click here

I’d add there is no single number that tells who is and who is not a good teacher. If only it was that simple.


Ermeling tells how PLCs can connect the dots

Connect the Dots is the title of a new article by Brad Emeling, published by Learning Forward (formerly the National Staff Development Council), in the Journal of Staff Development (April, 2012, volume 33, No.2). Connect the Dots summarizes what it takes to turn schools or districts into communities of educators learning together how to sustain improvements in student achievement. Connect the Dots provides concrete guidelines to educators who want to convert PLC aspirations into functioning reality. The key, as Brad explains, is not only establishing settings for teacher teams and leaders, but strengthening the connections between settings so that each role group is focused on assisting learning of the next immediate group they directly lead or support. He means district personnel assist principals who assist school leadership teams that assist teacher teams. Each role–group takes ownership for planning for and providing assistance to the learning and productive work of the next immediate group–one month at a time. The result is a coherent system of settings functioning to improve teaching and learning.

Not only has Brad spent a decade+ building learning teams and PLCs in America, he participated in lesson study for 7 years as a teacher and administrator in Japan. As a fluent Japanese speaker he is one of the few America educators with such extensive experience with lesson study. In 2003, Brad joined a research team at UCLA and Stanford that had been researching teacher collaboration and learning teams for more than three decades.
Connect the Dots is based on Brad’s years of experience helping highly challenged schools set up learning teams that get results. The learning teams program he helps direct is based on ideas and concrete guidelines that are direct applications of 40+ years of research by Claude Goldenberg, Bill Saunders, and myself.
Personal point of privilege: The privilege was serving as Brad’s doctoral committee advisor, and as a member of the UCLA/Stanford research team on which
Connect the Dots is based. Brad’s dissertation extended the team’s work for the first time to secondary settings. After finishing his UCLA doctoral program, Dr. Ermeling helped scale the research to districts and schools across the nation.

For a PDF of Brad’s
Connect the Dots, click here.


Slippery slopes of education reform

“Complex, difficult to implement well” is a description that fits many education reforms and innovations. Not only must practitioners accept new goals, to achieve them they often must learn new practices and figure out how to secure durable implementation. A reform that does not improve students’ opportunities to learn is unlikely to succeed. To improve student learning opportunities nearly always means teachers have to change how they teach as well as what they teach. Implementing changes in instruction is lot harder to figure out than many realize when they launch a reform.

As the first year of implementation unfolds, there is a slope on which many slip that might ultimately wreck a reform effort. Gradually, unwittingly the reformers and practitioners begin to adjust their goals. Instead of better learning opportunities and improved student achievement, focus shifts to implementing some feature of the reform. Success gets gradually re-defined as “how much of the reform are we doing every day” instead of how much more are the students learning. The means have become the ends. A means–ends reversal. One of the most common slips is to unwittingly replace the goal of improved learning and achievement with greater use of one of the reform elements

This is one of my favorite examples: Therese, a UCLA graduate student, and I visited a school which was implementing a form of cooperative learning. Everyday during reading instruction, teachers were asked to pair up students to work cooperatively on an assignment. The day we visited the cooperative learning activity involved text comprehension. Therese was an experienced elementary teacher, a warm, approachable personality who seamlessly roved through the classroom, sitting at tables, chatting with students. She asked students how they liked the cooperative learning segment, and got surprising answers reflected in the response of a 4th grade girl: “working together is ok, but it keeps us from getting our work done.” Therese and I thought the girl’s comment revealing since our observations of the cooperative learning segments suggested students carried out the activity in a mechanical and hurried fashion; they were working together but it hardly seemed like the descriptions of “cooperative learning” that the program operators were pursuing.

In a debriefing meeting after our observations, the program operators talked excitedly about how well the cooperative learning was working. At the launch of the reform, they had struggled to stand up the cooperative learning activity. But after much effort, they were pleased with the results. Their evidence focused on how reliably cooperative learning was being incorporated by teachers into lessons and the participation level of the students. When we probed for evidence of improved learning, they offered more anecdotes about participation. Our visit took place in late Spring, after the reform program had been in place for nearly a year. Had these well intended reformers slipped down the slope of a means–ends reversal? Were they forgetting that the original ends was better learning, not increased cooperative learning participation? Perhaps. But this can easily happen when education reformers underestimate how hard it is to fundamentally change teaching practices.

Another example. A truancy reduction program hired as director a young adult from the neighborhood. Kai was once a notorious truant who had turned his life around. Kai’s charismatic personality was attractive to the adolescent boys who were the main focus of the truancy program. A “club house” was created near the school, where the boys could gather. Kai offered counseling, diversion activities, exhortations to return to school, tutoring, and a place to do homework. Participation in the program increased quickly after Kai began work. There were less complaints about the truants in the neighborhood, less hanging around the schoolyard and making trouble. The program operators were pleased, and so was the funding agency. But a brief evaluation suggested that despite these improved participation and less behavioral complaints, the program seemed to have increased the truancy rate of the boys in the program, and also increased the number of truants over all. More boys than before were skipping classes to spend time at the club house. Kai’s personality and the attractions the program offered had certainly increased participation, but at the cost of the original truancy reduction goal. Slipping into a means-ends reversal can be deceptively easy to do.

More on Education Reform and Campbell's Law

Yesterday I posted a bit on Campbell’s Law. To repeat:
"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Wikipedia entry

My evaluator pals relate numerous seemingly innocent examples that might represent the Law at work. A generic (and hypothetical) example: imagine well intentioned reformers develop a program to increase the number of disadvantaged students applying to and matriculating at four year colleges. After one year, the program operators (and the funding source) were disappointed by the low rate of success. Counselors and teachers who worked with the students suggested re-examining admission criteria. They observed that many students admitted during the first year of operation were on paper academically qualified to enter four year colleges, but exhibited only superficial interest in higher education. Admission criteria the second year were refined to include interest level in college as well as include academic readiness–think of it as learning from experience. Outcomes for the second year of operation were dramatically better.

Did the adjustment of admission criteria represent a corruption of standards? If the original goals included increasing the number of students interested in attending college, unless the goals were also adjusted, then the change in admission process might be Campbell’s Law at work. If in the second year, the goals were explicitly revised to focus only on application and matriculation, then perhaps changing admission criteria was not as clear an example of corrupted standards. In practice, busy program personnel are so caught up in the day to day demands of their work that goals sometimes implicit shift without awareness. So perhaps there was in this instance a “slippery slope” kind of corruption in which admissions criteria were changed, but the operators never explicitly addressed the goals issue–and continued to perceive and describe the program as unchanged. But the Law makes no distinction between unintended and intended forms of standards corruption.

A second hypothetical example: a charter school operation within a public district was under close scrutiny by many community interests. For each of the first five years operation, standardized achievement test results in the charter schools were significantly higher than in non-charter schools in the district. After several years, it was discovered the charter operators gradually changed policies, by reducing the number of student in their program with learning disabilities and a history of behavior problems. Charter operators argued they were only trying to maximize the number of students benefiting from an academically challenging instructional program. Critics argue they had stacked the deck and bled desperately needed funds away from the rest of the schools in the district, who could not “screen out” students so easily; and that the charters school were duplicitous in claiming their program accounted for the superior achievement gains. The charter operators said they did not stack the deck deliberately, that they tried very carefully to screen out only a limited number of students ill suited to their program. A professional evaluator was consulted; she suggested that a careful review of the selection process might be yield useful information provided the charter operators had kept detailed records on each case screened or counseled out. The dispute continues.

Then there is out-right fraud. Fraudulently changing achievement scores to protect a program under scrutiny has happened, and likely will happen again. Making up and changing achievement scores to protect a program under scrutiny has happened, and likely will happen again. Fraud cases might be easier to recognize as examples of Campbell’s Law.

But it is more difficult to decide when the issue involves the sometimes complex and difficult job of selecting participants in or out of a high value program. How can we decide whether it is corruption or not when operators seek to improve the functioning of a program by selecting in those mostly likely to benefit?


Campbell's Law of Criterion Corruption

Campbell's law:
"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Wikipedia entry

Recently, in an Education Week blog, Walt Gardner observed that Campbell’s Law struck the use of standardized achievement tests for high-stakes education decisions. He notes several high profile scandals in school testing programs. Examples of cheating by districts and schools he dates back to at least 1969 in Texas, in addition to more recent cases in Georgia, Indiana, Nevada, Virginia, Massachusetts, South Carolina, and California.

Education is not the only field in which Campbell’s Law applies, according to a comment in 2007 by Nichols and Berliner in the Harvard Education Letter. They claim there are in other fields numerous examples of “corruption, cheating, gaming the system, taking short cuts, and so forth whenever high stakes are attached to performance in athletics, academia, politics, government agencies, and the military.”

The nature and extent to which performance criteria inevitably lead to corruption is a matter of debate, and a tough problem to investigate. But there’s every reason Campbell’s Law operates to some extent in any field, and if nothing else is a warning to be humble about claims that setting quantitative criteria to meet is a easy way to improve performance.

If we don’t use standardized tests to assess student learning, what do we use? It is wise to remember standardized tests were once hailed as a democratic alternative to privilege and patronage that determined access to valued assets such as college entrance and employment opportunities. So what now?
Feb 2018
Oct 2015
Mar 2015
Apr 2014
Aug 2013
Jul 2013
Mar 2013
Jan 2013
Nov 2012
Jul 2012
Jun 2012
May 2012
Apr 2012
Mar 2012