VI. Scoring Constructed Response Items
B. Analytic vs. Holistic Rubrics
Rubrics are scoring tools used to evaluate student performance based on specific criteria. There are two types of rubrics: analytic and holistic. Analytic rubrics provide detailed feedback on multiple aspects of performance, while holistic rubrics provide a single score based on an overall impression of a student's performance. The choice between the two types depends on the purpose of assessment, the nature of the task, and scorer reliability.
BASIC Information
Rubrics are scoring tools used to evaluate student performance based on specific criteria. There are primarily two types of rubrics: analytic and holistic.
Analytic Rubrics: These divide an assignment or task into its component parts and provide a detailed description of what constitutes acceptable or unacceptable levels of performance for each part.
Holistic Rubrics: These provide a single score based on an overall impression of a student's performance on a task as a whole.
PA (PA) students are frequently assessed on clinical skills, knowledge, and patient interactions. The choice between analytic and holistic rubrics can impact how feedback is given and how students identify areas of improvement.
INTERMEDIATE Information
Analytic Rubrics:
Advantages: Provide detailed feedback on multiple aspects of performance, allow for more objective grading.
Disadvantages: Can be time-consuming to develop and apply, may overwhelm students with feedback.
Holistic Rubrics:
Advantages: Quicker to develop and score, can be useful when looking at the overall quality of work.
Disadvantages: Less detailed feedback, potential for scorer subjectivity.
PA students are expected to master both the science and art of medicine. While an analytic rubric might be used to assess the specific steps in performing a procedure, a holistic rubric might better capture the nuances of patient interaction or overall clinical decision-making.
Task: Conduct a physical examination on a mock patient.
Holistic Rubric:
5 Points: Student demonstrates mastery in technique, communication, and clinical reasoning.
3 Points: Student exhibits competence but may miss minor details or lack fluidity.
1 Point: Student lacks confidence, misses significant findings, or communicates poorly.
Analytic Rubric:
Technique (5 Points Max):
5 Points: Demonstrates flawless technique.
3 Points: Correct techniques, but lacks smoothness or sequence.
1 Point: Improper techniques or omissions.
Communication (5 Points Max):
5 Points: Engages patient, explains each step, ensures comfort.
3 Points: Generally communicates but may miss explaining some steps.
1 Point: Limited communication, does not ensure patient comfort.
ADVANCED Information
Purpose of Assessment: For formative assessments, where the goal is to provide detailed feedback for improvement, analytic rubrics may be more suitable. For summative assessments, where a general understanding of student competence is needed, holistic rubrics might suffice.
Nature of the Task: Complex tasks with multiple components might benefit from analytic rubrics, while holistic tasks where the overall impression is more critical might be better suited for holistic rubrics.
Scorer Reliability: Analytic rubrics can reduce scorer variability as they provide more explicit criteria. However, with proper training, scorers can achieve reliable scores with holistic rubrics as well.
As PA students progress and encounter more multifaceted clinical scenarios, integrating both holistic and analytic rubrics can provide a balance of detailed feedback and overall competency assessment.
Task: Diagnose and propose a management plan for a complex patient with co-morbidities.
Holistic Rubric:
5 Points: Provides an accurate diagnosis and comprehensive management plan, integrating all aspects of the patient's condition.
3 Points: Accurate diagnosis but misses some nuances in the management plan.
1 Point: Struggles with diagnosis or proposes an incomplete management plan.
Analytic Rubric:
Diagnosis (5 Points Max):
5 Points: Accurate diagnosis with justifications using patient data.
3 Points: Correct diagnosis but lacks full justification.
1 Point: Missed or incorrect diagnosis.
Management Plan (5 Points Max):
5 Points: Comprehensive plan addressing all issues, potential interactions, and patient lifestyle.
3 Points: Adequate plan but overlooks minor details or doesn't fully consider patient lifestyle. - 1 Point: Incomplete plan, misses significant aspects or doesn't address potential complications.
For the advanced PA student, it's crucial to understand that scoring constructed response items, especially complex clinical scenarios, requires rigorous scorer training. This ensures that all examiners have a consistent understanding of the rubric's criteria.
With Analytic Rubrics: Examiners should be well-acquainted with each criterion's nuances. In the given example, understanding what justifies a comprehensive management plan versus an adequate one is crucial. Training might involve practice sessions where examiners score sample responses, then discuss and calibrate their scoring to align with one another.
With Holistic Rubrics: Examiners should have a clear understanding of what constitutes an overall mastery versus basic competence. Given the subjective nature of holistic scoring, it's even more vital for examiners to undergo training sessions to calibrate their scoring for consistency.
In the context of a PA program, instructors can use these rubrics not only for grading purposes but also as formative tools. For instance, the rubric can be shared with students before they undertake the task, helping them understand what's expected and how they can excel. Over time, as students internalize the criteria laid out in the rubrics, they'll develop an intrinsic understanding of clinical excellence, driving both their academic and professional growth.