EducationNews.org

Ronald K. Hambleton, University of Massachusetts Amherst

For the recent meeting of AERA and NCME in New York, Kristen Huff and Linda Cook suggested that we bring together six prominent measurement specialists and have each of them talk about one or two problems that they felt were timely for research in the next few years. All of the speakers were enthusiastic about their choice of topics.

Joanna Gorin from Arizona State University continued a theme the measurement field has been hearing for many years and at least as far back as the early 1980s--cognitive psychology will be the link to developing instructionally-relevant assessment measures, and she encouraged more research. Her feeling is that psychometricians and teachers live in two worlds (she didn't mention in which world the psychometricians were living), and cognitive psychology can bring them together, and increase the impact of assessments on learning. Now that real progress can be seen, more of us need to be paying attention, and begin to participate in the transformation that is happening—cognitive-based testing is upon us.

Continuing the message, Suzanne Lane from the University of Pittsburgh lamented the limited utility of large scale assessment for measuring skills that educators value. Her plea included a request for substantially more research on how students acquire and develop knowledge and processes. For those in our field who are model builders, she wants to see more attention given to the development of psychologically important models linking cognitive variables to performance assessments. She too made a plea for linking cognitive psychology, testing, and instruction.

Increasing the measurement and statistics knowledge of K-12 teachers and administrators was the challenge Rebecca Zwick from the University of California at Santa Barbara presented to the field at the symposium. Of course, this problem is not new to most of us—many educators have a disappointingly low level of knowledge about educational testing, and in part this flows from a lack of training or ineffective training provided to them at the universities. Professor Zwick talked about some recent work she has been doing for the National Science Foundation in both identifying what educators need to know about measurement and statistics, and then she has been researching ways to develop these skills. She views this problem as one that more measurement specialists need to be committed to. It is difficult to justify all the assessment taking place if measurement and statistics literacy remains low among educators.

Jim Popham, from UCLA seemed disappointed that after all these years we still don't seem to be willing and/or able to design tests for accountability that can identify quality instruction when it is present. He argued persuasively, I think, that tests to assess accountability are not very sensitive to instruction and therefore they are of limited value to educators. He challenged the measurement field to construct instructionally sensitive tests and proposed both judgmental and empirical strategies for evaluating test instructional sensitivity.

Whether you agree or disagree with his ideas, Howard Wainer from the National Board of Medical Examiners is one of the more interesting and provocative speakers in our field, and on this occasion he laid out his personal views about what is especially important to investigate in the coming years: Bayesian estimation, learning to handle missing data, picturing data, and learning much more about causal inference and type I and II errors, were five of his favorite topics for more intensive research. He suggested too that we back off our intensive study of IRT, reliability, factor analysis, and DIF, arguing that we know enough to use these methodologies well in practice, while there are other more fundamental problems requiring our attention. He also singled out evidenced centered design and value added models for more attention.

Finally, a great thinker from the north, Bruno Zumbo from the University of British Columbia, was enthusiastic about multi-level variables in educational research, but felt that we have been surprising lax in our concern for reliability, generalizability, and validity issues. He noted that nearly all of our psychometric practices are geared to individual scores, but a good amount of data today are aggregated data, such as data being reported for a school, a neighborhood, or state, and we need to invent reliability and validity approaches for handling these data. Clearly, we are almost no where with respect to his concerns.

A spirited discussion among the presenters, and later, among the presenters and the audience took place with Professor Popham taking on all comers in his defense of the use of tests than can detect effective instruction when it is present. No attempt was made to achieve consensus and any of the suggested topics for future research but it was clear the presenters felt that research attention ought to be somewhat refocused to start addressing timely assessment questions that can make a difference in education. All of the topics suggested by the speakers were more or less agreed upon as important. The take-home messages for me were that (1) measurement training programs around the country need to expand their course offerings to insure some training in cognitive psychology, and linking theories of cognition to assessment development, and extending this work to score reporting (another important topic for future research), (2) tests need to be designed that have more sensitivity to detect good instruction of state curriculum objectives, (3) progress is being made regarding assessment skills for educators, but much more attention is needed, if the vast investment this country is making in assessment is to be worthwhile, and (4) graduate students and even university professors would do well to look around for the important unsolved problems and work on those rather than tweak well established methodologies. I found the session stimulating and expect many of those present did also. At a minimum you may want to write the speakers for their slides, and several provided papers too. I think we need to see more sessions like this one at upcoming meetings of NCME and AERA. Big ideas are important and we need to hear them, and lively debate is refreshing and enlightening, and may spur many of us on to greater research accomplishments.

Published July 1, 2008