September 27, 2006

Multiple Choice Tests: A-Great, B-OK, C-Overused

The correct answer is C

It is very common in nursing programs to evaluate students' learning primarily with multiple-choice exams. The reason for this is often stated that because the NCLEX-RN exam is multiple-choice then we should also. I have concerns about trying too closely to follow the NCLEX-RN exam model. They have the advantage of a huge population to pretest all questions, and a large group of contributors to the test question pool. Our use of multiple choice is more for testing efficiency more than emulation of the Board exam. To truly test learning we should be using different types of evaluation: essays, open book, short answer, concept mapping, etc. I've always thought to continue to use multiple choice because "that what the boards are" does our students a disservice in promoting limited ways of learning, and limited ways of demonstrating learning. I have yet to meet a patient with A, B, C, D on his chest. The Board exam may be multiple choice but my hypothesis is that a student who has good depth of knowledge (previously evaluated through multiple methods) can prepare for the boards with exam practice as he or she nears graduation.


savvydapunch said...
This comment has been removed by a blog administrator.
savibus said...

Brent, you have identified "the problem" and while there is no "the answer", there is plenty in the nursing literature to support and expand your thinking on how to truly know and value what students are learning.

Start with Carper's fundamental patterns of knowing for nursing - all four of those patterns ought to be part of the evaluation program because all four, if one accepts Carper's thesis, which I do, make an essential contribution to the whole of knowing in nursing. If you haven't studied the idea of the patterns of knowing in depth already, I would recommend the Chinn and Kramer book on nursing knowledge development as a second excellent starting place - they really develop out Carper's work in a current theoretical and practical sense. Once one understands the idea of these four fundamental patterns of knowing in nursing (and there are at least two others in the literature including sociopolitical knowing by White and unknowing by Munhall), then it is fruitful to begin to read about how other nurse educators are using these patterns not only for teaching but also for evaluating what students are learning. I co-auhtored a chapter in a book that is now out of print but available in most good nursing libraries, Living a Caring Based Program edited by Boykin; the article is Chapter 8 Valuing, Prizing and Growing in a Caring Based Program, by Schoenhofer and Coffman. In addition the literature on aesthetic knowing in nursing and aesthetic knowing in nursing education teaching and evaluation is growing. One paper that comes to mind is by Barry on quilting as an expression of student understanding of their nursing practice situations. And then there's the excellent book by Oermann and Gaberson, Evaluating and Testing in Nursing Education, that honors what, for want of a more generally understandable term, subjective evaluation. They have many examples, although they really haven't gotten into evaluation strategies in the aesthetic pattern of knowing per se.

Savina Schoenhofer

Brent Thompson, DNSc, RN said...

Thanks for the great resources for more information on patterns of knowing and evaluation of learning.

The problem I see is that while some nursing faculty may try to teach the many ways of knowing, they fall short in the many ways of evaluation.

Many nursing programs use team-taught classes that depend on multiple-choice exams to consolidate evaluation for efficiency's sake. Discussion of using different methods of teaching and evaluation is quickly ended due to fears of "inconsistency" in both teaching and evaluation.

Bear said...

Hi Brent - Hi Savina,

Brent - taking your analogy a bit tongue in cheek - have you ever seen a patient with essays, open book, short answer, concept mapping, etc on his/her chest?

The difficulties with many alternative ways of evaluating learning is not that they are inherently inappropriate. Neither MC, essays, concept maps, nor open-book exams are problematic assessment tools when students exceed any reasonable expectations. But all except MC are likely to prove terrible at distinguishing and ultimately 'out-counseling' people who come nowhere near meeting expectations. The biggest problems I have encountered with non-MC learning assessments is that some students have no idea at all how far below acceptable their work really is. BTW - I have never given a MC test since I started teaching nursing. You get a paper from a senior nursing student and it reads like it was written by an over-tired third grader - but the student thinks it is divinely inspired and brilliant. When you try to discuss the lag between what you were expecting and what they delivered - they argue that you, not they, have the problem. In really unpleasant situations it will involve the classic games described by Eric Berne's and other TA people. If, we were all at Lake Wobegone School of Nursing where all the students are above average,there would be no problem at all with any assessment approach - but as we increase the number of nursing programs and the numbers of nursing students per program, not everyone is intellectually equipped for what is a very, very information intensive, as well as caring, profession. As well, we deviate further and further from the assumptions of the LW-SON.

I often make the point that one of the most caring of my BSN peers flunked out by a few questions out of 300 during a semester - and I think his loss was a loss for nursing - but at the same time, it really wasn't that hard to pass the MC tests. He didn't fail because the assessment instruments were inherently tainted but because he didn't put in the time or simply did not have the intellectual ability to succeed in nursing school. While it is true that I would want his 'caring' for a patient I cared about - I wouldn't want his inability to quickly and accurately process information, make connections between VS and electrolytes and past, current, and future patient status....

As it was, he tried to press the case that the deficit in the MC tests wasn't significant enough to result in his failing out of school. With a reasonably objective standard of below 74% you failed - there was at least some measuring rod. Granted, the number of easy questions was manipulated by the instructor - but so would be any standard based on essays, short answers, or concept maps. If, he had to hand in essays or concept maps he probably would have argued that his essays were meritorious and then, the instructor would have a far less defensible position. Even with a concept map - how many interlocking concepts constitute satisfactory or superlative responses? 2, 5, 25? Who decides? Is it specified up front? What if a student thinks they have 10 concepts but the student only sees three - albeit worded differently? What if the student's concept maps or essays bear little resemblance to material in texts, class lectures, or assigned readings?

With MC tests a student can see in very tangible terms, how they compare with other students. There are very few people at the top and very few people at the bottom - but at least the student knows which group they are in. If the only issue were that we should (and we should) turn out nurses who feel good about themselves, a free-floating evaluation would be reasonable - but if we are also supposed to turn out students that can evaluate their probability of managing patient care, passing the NCLEX, and succeeding in nursing, don't we have some responsibility to help them see when they are well below the standards by which they will ultimately be evaluated?

I would be very interested to know how you would address poor performance, if at all, in cases where students dispute your assessment?

Bear said...

mea culpa...

"What if a student thinks they have 10 concepts but the student only sees three - albeit worded differently?"

should have read

"What if a student thinks they have 10 concepts but the teacher only sees three - albeit worded differently?"

Brent Thompson, DNSc, RN said...

Bear, please keep in mind my thesis is that MC is overused, not that it should be eliminated.

Your story about the friend who failed is very familiar and I think only supports my case. There is an illusion of objectivity with MC tests. The built-in biases, the lack of pretesting to determine item difficulty levels and discrimination abilities mean that the grades earned on MC tests are much cruder measures of learning then some teachers realize. This is especially visible when schools set very different passing levels. I have seen passing as 70, 73, 75, and 80 at different schools.

Of course all evaluations are subjective to some degree. Faculty are hired for their expertise in the subject area and are given lots of room to determine how students will demonstrate learning.

In the end, students are at our mercy on whether or not they will succeed. I have always thought that education is a bit of a scam compared to anything else you spend your money on. Tuition only buys you the right to attempt a class to get credit at the determination of someone else. Get the credit or not, you still pay.

Post a Comment

Comments are welcome. However, if they are of a commercial nature they will be immediately deleted.