Assessing Competence in Medicine and Other Health Professions

Claudio Violato

• Publisher: CRC Press
• Year: 2018
• ISBN:
  9780429761829
• Edition: 1
• Topic: Medizin
• Subtopic: Medizinische Theorie, Praxis & Referenz

This comprehensive, yet accessible, text demystifies the challenging area of competence assessement in medicine and the health sciences, providing a clear framework and the tools for anyone working or studying in this area. Written by a single, highly experienced, author, the content benefits from uniformity of style and is supported and enhanced by a range of pedagogic features including cases, questions and summaries. Essential reading for all students and practitioners of medical education, it will also be an invaluable guide for allied health professionals and psychologists with a general interest in assessment, evaluation and measurement and a useful library reference.

SECTION III

Test construction and evaluation

In Chapter 10, the various formats of testing for cognition, affect, and psychomotor skills are summarized. These include selection type items (multiple-choice questions, constructed response, checklists, and other item formats). There is a discussion of Bloom’s taxonomies of cognitive, affective, and psychomotor skills, Miller’s Pyramid and further work on the use of tables of specifications.

Chapter 11 deals specifically with multiple-choice items especially on how to write items, types of MCQs, number of options, how to construct MCQs and appropriate levels of measurement. MCQs are objective tests because they can be scored routinely according to a predetermined key, eliminating the judgments of the scorers. The multiple-choice item consists of several parts, the stem, the keyed-response, and several distractors. The keyed-response is the right answer or the option indicated on the key. All of the possible alternative answers are called options. The stems should include verbs such as define, describe, identify (knowledge), defend, explain, interpret, explain (comprehension), and predict, operate, compute, discover, apply (application).

In Chapter 12, constructed response items are detailed. These include essays, short answer, matching, and hybrid items such as the Script Concordance Tests. These item types are sometimes also called subjective tests because scoring involves subjective judgments of the scorer. Constructed-response questions are usually intended to assess students’ ability at the higher levels of Bloom’s taxonomy: application, analysis, synthesis, and evaluation. There are at least two types of essay questions, the restricted and extended response forms. The restricted response question limits the character and breadth of the student’s composition.

There are two basic approaches to the scoring of constructed-response test questions: analytic scoring and holistic scoring.

Measuring clinical skills with the objective structured clinical exams (OSCE) is discussed in Chapter 13. The focus is on describing the OSCE and its use in measuring communications, patient management, clinical reasoning, diagnoses, physical examination, case history, etc. Additional issues of case development and scripts, training assessors, and training standardized patients are detailed. Content checklists and global rating scales are used to assess observed performance of specific tasks. Candidates circulate around a number of different stations containing various content areas from various health disciplines.

Most OSCEs utilize standardized patients (SPs), who are typically actors trained to depict the clinical problems and presentations of real issues commonly taken from real patient cases. OSCE validity concerns are (1) content based—defining the content to be assessed within the assessment; (2) concurrent based—this requires evidence that OSCEs correlate with other competency tests that measure important areas of clinical ability; and (3) construct based—one of the ways of establishing construct validity with OSCEs is through differentiating the performance levels between groups of examinees at different points in their education.

Chapter 14 deals with checklists, questionnaires, rating scales, and direct observations. Checklists, rating scales, rubrics, and questionnaires are instruments that state specific criteria to gather information and to make judgments about what students, trainees, and instructors know and can do on outcomes. They offer systematic ways of collecting data about specific behaviors, knowledge, skills, and attitudes and can be used at higher levels of Miller’s Pyramid. A checklist is an instrument for identifying the presence or absence of knowledge, skills, and behaviors. Surgery has made extensive use of checklists to improve patient safety. A rating scale is an instrument used for assessing the performance of tasks, skill levels, procedures, processes, and products. Rating scales allow the degree or frequency of the behaviors, skills, and strategies to be rated. Rating scales state the criteria and provide selections to describe the quality or frequency of the performance. A five-point scale is preferred because it provides the best all-around psychometrics.

Chapter 15 is about evaluating tests and assessments using item analyses: conducting classical item analyses with MCQs (item difficulty, item discrimination, and distractor effectiveness), conducting item analyses with OSCEs, conducting item analyses with constructed response items, and computing the reliability coefficient and errors of measurement.

A complete analysis of a test requires an item analysis together with descriptive statistics and reliability. There are three essential features for an item analysis for multiple-choice questions: (1) difficulty of the item, (2) item discrimination, and (3) distractor effectiveness. All of these criteria apply to every other test or assessment form (e.g., OSCE, restricted essay, extended essay, survey) except for distractor effectiveness since there are no distractors in these test formats.

The difficulty of the item is the percentage or proportion of people who got the item correct. If everyone gets the item correct, it is an easy item; if very few test-takers get the item correct, it is a very difficult item. Item difficulty (P) is usually expressed as a proportion such as P >= 0.72 (72% got it correct).

Item discrimination (D) is the extent to which an item discriminates between high-test scorers and low-test scorers. The point-biserial is commonly used for item discrimination, D. Distractor effectiveness refers to the ability of distractors in attracting responses.

Grading, reporting, and methods of setting cutoff scores (pass/fail) are discussed in Chapter 16. The focus is on norm-referenced versus criterion-referenced approaches, setting a minimum performance level (MPL) utilizing the Angoff method, Ebel method, Nedelsky method.

Grading systems can be based on norm-referenced (grading on the curve) or on criterion-referenced bases. All of these systems have some problems associated with them. The main and historically oldest symbols for grading are the letter grade, usually ranging from A to F. Substitutes have been attempted but have achieved little success because these usually involve reducing the number of categories (e.g., good, satisfactory, unsatisfactory). Numerical grades (percent or 1–10) have also met with limited success as have pass/fail systems and checklists of objectives, as well as empirical methods (borderline regression, cluster analyses, etc.).

The Dean’s Letter or Medical Student Performance Evaluation employs multiple reporting systems dealing with the cognitive, affective, and skills domains. There are provisions for reporting learner achievement, effort, attitudes, interest, social and personal development, and other noteworthy outcomes. The wealth of information on the MSPE, however, should be balanced against the need for brevity and simplicity so that it can be readily understood.

10

Formats of testing for cognition, affect, and psychomotor skills

ADVANCED ORGANIZERS

• It is common practice to think of human behavior in three categories or domains: cognition (thinking), affect (feeling), and psychomotor (doing). Although we tend to discuss human behavior in these separate ways, most behavior involves all three aspects or domains.
• Thurstone’s negative exponential or hyperbolic model learning curve represents a principle of learning in a variety of disciplines and for a variety of learners; learning increases rapidly with practice but attains an upper limit quickly and then flattens out.
• Notwithstanding scientific advances in the study of thinking, perception, memory, learning, much of assessment in medicine and the other health professions is based on lore, intuition, anecdotal evidence, and personal experience. The use of Bloom’s taxonomies is helpful in systematizing assessment.
• Much of the testing in these domains involves selection items (e.g., multiple choice) or constructed response (e.g., essays) items.
• Jean Piaget’s theory of intelligence is that we adapt to the world employing the dual cognitive processes of assimilation and accommodation. The theory both helps to explain key ideas such as deliberate and interleaved practice, retrieval practice, and learning styles and how to assess them.
• The script-concordance test (SCT) is used to assess the ability to interpret medical information under conditions of uncertainty, tapping Piaget’s highest levels of cognitive functioning. There still remain a number of challenges to assess the processes of problem-solving, abstract reasoning, and meta-cognition.
• Students should be continuously tested so as to capitalize on the testing effect. Using retrieval practice with testing—working memory to recall or retrieve facts or knowledge—is more effective than reviewing content or re-reading text.
• Both the experimental work and the correlational-based research suggest the need for purposeful, direct teaching for integration and encapsulation of basic sciences into clinical reasoning and clinical skills. This integration and encapsulation needs to be assessed and tested dynamically to determine the cognitive processes and outcomes of such pedagogy.
• Bloom is credited for developing a taxonomy in the affective (feeling) domain that provides a framework for teaching, training, assessing, and evaluating the effectiveness of training as well as curriculum design and delivery.
• In the affective domain, the structure of professionalism, aptitude, achievement, and personality are accounted by Aptitude for Science (cognitive variables, and Extraversion—a characteristic of working with people (e.g., medicine); Aptitude for Medicine (cognitive variables, and Conscientiousness and, inversely, Neuroticism); Professionalism is composed of the peer assessment and Openness; General Achievement is composed in almost equal parts of GPA and Agreeableness; and Self-Awareness
• In health sciences education, psychomotor skills and technical skills—rarely defined in the health sciences literature—most commonly refer to motor a...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedicaion
• Table of Contents
• Preface
• Author
• Section I Foundations
• Section II Validity and Reliability
• Section III Test Construction and Evaluation
• Glossary of Terms
• Index