IN ADDITION TO MY GOLF ANALOGY (see other message) ... I WOULD SAY THE FOLLOWING ABOUT STUDENT OUTCOMES...THERE SHOULD BE MINIMAL STANDARDS IF WE ARE TO SERIOUSLY USE STUDENT ACHIEVEMENT AS ANY KIND OF EFFICACY MEASURE OF A PROFESSIONAL DEVELOPMENT EFFORT...

BASICALLY, IF YOU LISTEN TO MIKE AND BEN, WHAT THEY ARE SAYING IS TRUE... THAT IS, WE LIVE IN AN AGE OF OVERSIMPLIFIED ACCOUNTABILITY AND THAT PEOPLE SEEM TO BE DEMANDING EVIDENCE OF EFFICACY ON THE BASIS OF STUDENT ACHIEVEMENT MEASURES...

BUT I HAVE TWO QUESTIONS ABOUT THEIR PROPOSITION THAT THEREFORE THE BEST POLITICAL THING TO DO IS TO TRY AND GET THOSE MEASURES...

1) WHAT IS THE COST OF PROMOTING OUR WORK IN TERMS THAT ARE INAPPROPRIATE AND CONVEYING THE SENSE THAT OUR WORK CAN AND SHOULD BE JUDGED ON THESE MEASURES... ???AND

2) IS THERE A DIFFERENCE BETWEEN LONG TERM AND SHORT TERM POLITICAL GAINS... PERHAPS STUDENT ACHIEVEMENT CAN BE USED TO HELP OUR EFFORTS IN THE SHORT TERM... BUT IS THIS OVERALL EFFORT GOING TO ENHANCE OUR WORK POLITICALLY IN THE LONGT TERM... I SUGGEST IT IS NOT A GOOD IDEA TO SELL OUR EFFORTS BY RAISING INAPPROPRIATE, UNSCIENTIFIC EXPECTATIONS...

HERE IS MORE THINKING ABOUT THE MINIMAL CONDITIONS THAT SHOULD EXIST IF WE ARE TO HAVE ANY KIND OF VALID CONNECTION BETWEEN STUDENT ACHIEVEMENT AND PD EFFORTS...

The term "student outcomes" is based on the metaphor of there being some sort of process in which there are "input" and "outcomes". In this case there is the assumption that the professional development activities are the inputs and the scores of students are the outcomes... And for evaluation purposes there is the assumption that measurement of the "outcomes" is a valid way to assess the value of the "inputs." The proposition, then, is that it is both possible and desirable to assess the value of the NSF investment in porfessional development (LSCs) by examining the scores of students on assessments.

This proposition is fundamentally and fatally flawed -- it is not possible to assess the value of the NSF investments in professional development by looking at the scores of students.

Un fortunately, this is a fact that seems to require a lot of discussion. The idea that it is possible to evaluate professional development this way is driven a by sense of political necessity, not by substantive and logical thinking ... Almost everyone I talk to acknowledges this fact but regrets that we must proceed ahead anyway.... They argue that funders and legislators demand this evidence... I think that it is ultimately very unwise to try to make the case for investments in professional development by asserting that we can prove they cause enhanced student achievement... I think at the very least that it would be good to find rigorous ways to assess appropriate outcomes of professional development (rather than attempt to find pseudo-rigorous ways to assess inappropriate outcomes....) I think it would be better to think again about how we might evaluate the ways in which professional development adds value to a complex system...

Having said all that, let me ask if there are any conditions under which I can imagine trying to connect the NSF funding for an LSC with student outcomes... that is is it ever possible to do a rigorous sound evaluation where LSC work is the input... and some sort of student scores are the "outcome"...

Here are the conditions that would have to be met if, indeed, the LSC investment were to be legitimately evaluated using any kind of student outcome measures.

1) The LSC would have to work with teachers consistently and over a long term, focusing on multiple key dimensions required to improve instructional practice. (If the LSCs do not work with individual teachers consistently and in depth, then it is hard to imagine that the contribution to classrooms to classrooms and students is significant.)

2) The LSC would have to be a major input into the consistent use of high quality curricular materials in the school or district being studied.. That is, the LSC would have to be a major reason that new curricula were now being "implemented" . (Otherwise it may well be that the district on its own has implemented good curricula without NSF help).

3) The LSC would have to be the sole, or certainly most significant, source of professional development for teachers. (Otherwise it might well be the participation of teachers in other Eisenhower or NSF PD that is the major source of teacher learning.)

4) The LSC would clearly have to show that the teachers and classrooms it serves show a significant CHANGE in the nature and quality of instructional practice.... The LSC must be a clear cause of a discernable change in the amount and/or type and/or quality of instruction that is offered. (It may be that these classrooms were good long before the LSC).
This is why the LSCs are currently evaluated by Horizon by examining random samples of classroom instruction and carefully rating its quality and qualities... (If there is no discernable change in classroom instruction as a result of the LSC, can it be argued that the LSC significantly increased student learning?) And it would have to be clear that the LSC was the major CAUSE of improved classroom instruction...

5) It must be clearly demonstrated that the students who are to be tested (who generate the outcomes) must have had a steady consistent diet of instruction that itself was clearly and strongly shaped by the work of the LSC. Probably this is a multi-year requirement... That is, we must be assured that students who represent the work of the LSC have in fact received strong steady LSC influenced instruction for significant periods of time... We must take account of the fact that students may well have received very different amounts of the "treatment"....

6) Similarly we must take into account that teachers have received different amounts of the LSC treatment... Our evaluation must take into account the type and amount of LSC participation by individual teachers....

6) The assessments used to measure student ability must be consistent with the goals and work of the LSCs...

7) Finally, and this is a really tough requirement, we need to show that the LSC is the only logical cause --the only feasible explanation --- for the increase in classroom quality and the increase in student achievement... This requires random assortment of students... or some other valid way of eliminating all other factors. It also requires that we have an accurate measure of the GAINS that students have made compared to the GAINS other students, not influenced by the LSC, have made over a similar period of time.... Note , again, that we msut use comparative gains in students scores on valid and relevant assessments; we can not use absolute scores as a comparative measure (We know that these scores are influenced by many many factors other than instruction....) .

In Summary, we must show unequivocally that 1) the LSC treatment is a major contributor to improved classroom practice; 2) that students have had a significant dose of the LSC treated classrooms and an educational experience that is significantly different than in other classrooms, and 3) that the LSC treatment is the only possible explanation of the cause of the differential gains in"treated" and "not-treated" students on appropriate and relevant assessments.

If teachers have not had significant amounts of learning experiences through the LSC, why would we expect their practice to change?

If teachers have had significant amounts of experience, but the system is non supportive, how much would we expect their practice to change?

If LSC classrooms are, in fact, quite different in the amount and/or type of instruction they offer, but students come and go, and do not get a steady diet of improved instruction, how much impact do we expect to see on their learning?

If LSC classrooms are but one of many inputs to student learning, or if students are not motivated or empowered to learn, how much influence will an improved classroom have on student knowledge and skills?

If assessments that are used to measure student learning are not closely related to the thrust of the work of the LSCs, then to what extent is it possible for the assessments to measure the value added component of the LSC?

There may be situations in which these conditions can be met. Mike Klentschy in El Centro has had the advantage of geographic and educational isolation in being able to assess student outcomes and compare them to the amount of time they spent in LSC classrooms... And Diane Briars in Pittsburgh was able to find five out of 6o schools in which Every Day was thoroughly and consistently implemented. Hence, where these stringent conditions can be met, it may be worthwhile to look for correlations... But in most LSCs I know we are not close to meeting these conditions... Hence the negative or unclear results from tests in these districts will not add to the knowledge of anyone; it will take lots of time and effort; and it will confuse project leaders and decisionmakers about the nature and purpose of professional development.

Finally, I think there is a real and important paradox to understand here. If you are serious about improving student achievement, then you have to focus on the long term development of system capacity; And if you focus too much on student achievement you will actually impair the effectiveness of the LSC in developing that long term capacity...