In the minds of OSCE examiners: uncovering hidden assumptions

Abstract

The Objective Structured Clinical Exam (OSCE) is a widely used method of assessment in medical education. Rater cognition has become an important area of inquiry in the medical education assessment literature generally, and in the OSCE literature specifically, because of concerns about potential compromises of validity. In this study, a novel approach to mixed methods that combined Ordinal Logistic Hierarchical Linear Modeling and cognitive interviews was used to gain insights about what examiners were thinking during an OSCE. This study is based on data from the 2010 to 2014 administrations of the Clinician Assessment for Practice Program OSCE for International Medical Graduates (IMGs) in Nova Scotia. An IMG is a physician trained outside of Canada who was a licensed practitioner in a different country. The quantitative data were examined alongside four follow-up cognitive interviews of examiners conducted after the 2014 administration. The quantitative results show that competencies of (1) Investigation and Management and (2) Counseling were highly predictive of the Overall Global score. These competencies were also described in the cognitive interviews as the most salient parts of OSCE. Examiners also found Communication Skills and Professional Behavior to be relevant but the quantitative results revealed these to be less predictive of the Overall Global score. The interviews also reveal that there is a tacit sequence by which IMGs are expected to proceed in an OSCE, starting with more basic competencies such as History Taking and building up to Investigation Management and Counseling. The combined results confirm that a hidden pattern exists with respect to how examiners rate candidates. This study has potential implications for research into rater cognition, and the design and scoring of practice-ready OSCEs.

Appendix

The Ordinal Logistic Hierarchical Linear Modeling (OLHLM) is slightly more complex as the outcome variable is an ordered categories and not continuous data. OLHLM is very similar to Logistic Hierarchical Linear Modeling, where the outcome is binary, and we estimating likelihood of receiving very good or poor ratings. Since there are four categories (poor, borderline, satisfactory, very good) of probability in our case three functions (1–3) are used and propensity of being in each of the categories is based on the change in log odds. The model below, represents a three level OLHLM with 4 category outcomes, 3 of the 4 categories are estimated, as it assumed that the 4th category is 1—the probability of being in the other three.

Station level model

$$ {\text{Category}}\, 1 :\;\;\log \left[ {\frac{{\phi_{ijm\left( 1 \right)}^{\prime } }}{{1 - \phi_{ijm\left( 1 \right)}^{\prime } }}} \right] = \pi_{ojm} + \pi_{1jm} X_{1jm} + \pi_{2j} mX_{2jm} + \cdots + \pi_{kjm} X_{kjm} $$

(1)

$$ {\text{Category}}\,2:\;\;\log \left[ {\frac{{\phi_{ijm\left( 2 \right)}^{\prime } }}{{1 - \phi_{ijm\left( 2 \right)}^{\prime } }}} \right] = \pi_{ojm} + \pi_{1jm} X_{1jm} + \pi_{2j} mX_{2jm} + \cdots + \pi_{kjm} X_{kjm} + \delta_{\left( 2 \right)} $$

(2)

$$ {\text{Category}}\,3:\;\;\log \left[ {\frac{{\phi_{ijm\left( 3 \right)}^{\prime } }}{{1 - \phi_{ijm\left( 3 \right)}^{\prime } }}} \right] = \pi_{ojm} + \pi_{1jm} X_{1jm} + \pi_{2j} mX_{2jm} + \cdots + \pi_{kjm} X_{kjm} + \delta_{\left( 3 \right)} $$

(3)

Candidate level

$$ \begin{aligned} &\pi_{0jm} = \beta_{00m} + r_{0jm} \hfill \\ &\pi_{1jm} = \beta_{10m} \hfill \\ &\pi_{1jm} = \beta_{10m} \hfill \\&\qquad \vdots \hfill \\ &\pi_{kjm} = \beta_{k0m} \hfill \\ \end{aligned} $$

(4)

Year level

$$ \begin{aligned} &\beta_{00m} = \gamma_{000} + u_{00m} \hfill \\ &\beta_{10m} = \gamma_{100} \hfill \\ &\beta_{20m} = \gamma_{200} \hfill \\ &\qquad \vdots \hfill \\ & \beta_{k0m} = \gamma_{k00} \hfill \\ \end{aligned} $$

(5)

Its important that when reading an HLM model to look at the different levels. In our case since there are no person characteristics at level 2 and no year characteristics at level 3, we are estimating the error variance at each of those levels and the competency coefficients. As such the:

$\phi_{ijm(1)}^{\prime }$ is the probability that person j in year m scores a 1; $\phi_{ijm(2)}^{\prime }$ is the probability that person j in year m scores a 1 or 2; $\phi_{ijm(3)}^{\prime }$ is the probability that person j in year m scores a 1, 2 or 3; $\pi_{ojm}$ is the intercept term; $\pi_{kjm}$ is the coefficient for the kth competency for person j in year m; $X_{kjm}$ is the kth competency score for person j in year m; $\delta_{(2)}$ is the threshold value between category 2 and 1; $\delta_{(3)}$ is the threshold value between category 3 and 2; $r_{0jm}$ is the random component of $\pi_{0jm}$; $\beta_{00m}$ is an effect of the reference competency in year m; $\beta_{k0m}$ is an effect of the kth competency in year m; $u_{00m}$ is the random component of $\beta_{00m}$; $\gamma_{000}$ is the overall effects of competencies; $\gamma_{k00}$ is the coefficient for competency k.

The HLM program uses the above model and provides estimates for each of the values above. However it does not provide the probability estimates of a candidate belonging to the poor, borderline, satisfactory or very good category at each station. These probability estimates need to be calculated. The following formulas are used to calculate the probability of being in each of the categories.

$$ {\text{Category}}\,1:\;\;\phi_{ijm\left( 1 \right)}^{\prime } = \frac{{e^{{(\gamma_{000} + \gamma_{100} X_{1jm} + \gamma_{200} X_{2jm} + \cdots + \gamma_{k00} X_{kjm} )}} }}{{1 + e^{{(\gamma_{000} + \gamma_{100} X_{1jm} + \gamma_{200} X_{2jm} + \cdots + \gamma_{k00} X_{kjm} )}} }} $$

(6)

$$ {\text{Category}}\;2:\;\;\phi_{ijm\left( 2 \right)}^{\prime } = \left( {\frac{{e^{{(\gamma_{000} + \gamma_{100} X_{1jm} + \gamma_{200} X_{2jm} + \cdots + \gamma_{k00} X_{kjm} + \delta_{\left( 2 \right)} )}} }}{{1 + e^{{(\gamma_{000} + \gamma_{100} X_{1jm} + \gamma_{200} X_{2jm} + \cdots + \gamma_{k00} X_{kjm} + \delta_{\left( 2 \right)} )}} }}} \right) - \phi_{ijm(1)}^{\prime } $$

(7)

$$ {\text{Category}}\;3:\;\;\phi_{ijm(3)}^{\prime } = \left( {\frac{{e^{{(\gamma_{000} + \gamma_{100} X_{1jm} + \gamma_{200} X_{2jm} + \cdots + \gamma_{k00} X_{kjm} + \delta_{\left( 3 \right)} )}} }}{{1 + e^{{(\gamma_{000} + \gamma_{100} X_{1jm} + \gamma_{200} X_{2jm} + \cdots + \gamma_{k00} X_{kjm} + \delta_{\left( 3 \right)} )}} }}} \right) - \phi_{ijm(2)}^{\prime } - \phi_{ijm(1)}^{\prime } $$

(8)

$$ {\text{Category}}\;4:\;\;\phi_{ijm(4)}^{\prime } = 1 - \phi_{ijm(3)}^{\prime } - \phi_{ijm(2)}^{\prime } - \phi_{ijm(1)}^{\prime } \;(2) $$

(9)

$\phi '_{ijm\left( 1 \right)}$ denotes the probability of being in the poor category, when the $X_{kjm}$ (i.e. competency scores) are at the average (i.e. set to zero). To calculate $\phi '_{ijm\left( 4 \right)}$ the probability of being in the very good category we subtract 1 from the probability of being in the satisfactory, borderline, or very good categories.