Introduction

Disruptive student behavior is a common stressor for elementary school teachers and can significantly detract from instruction time (Greenberg et al. 2016; Robb et al. 2011). Although classroom management strategies and targeted interventions (e.g., daily report card) are effective in preventing and reducing disruptive behavior (Epstein et al. 2008; Pyle and Fabiano 2017), teachers’ use of some of these strategies is limited (Hart et al. 2017; Owens et al. 2018). One means to support teachers’ use of interventions is via problem-solving consultation with performance feedback (also referred to as coaching) to improve teacher knowledge about the interventions and skills in applying the interventions (Frank and Kratochwill 2014). Yet, even when teachers receive this support, intervention implementation is variable (Fabiano et al. 2010; Owens et al. 2019), suggesting that other factors (in addition to teacher knowledge and skills) may influence implementation (Han and Weiss 2005). One hypothesized variable is teacher beliefs (e.g., about the importance of the intervention or confidence in their own skills). Indeed, because implementation integrity often drops precipitously when consultation is withdrawn (Mortenson and Witt 1998; Noell et al. 1997), other factors such as teacher self-efficacy and motivation may be needed for ongoing strategy use. Given that low and variable implementation can severely compromise student outcomes (Conroy et al. 2015; Owens et al. 2017), innovative consultation methods that address knowledge, skills, and teacher beliefs may be needed to adequately support teachers who are struggling with implementation integrity.

Innovations in Consultation: The Use of Motivational Interviewing

In response to this need, our team and others have developed and evaluated enhanced multi-component consultation packages that are built upon best practices in consultation (Frank and Kratochwill 2014) and informed by cognitive behavioral therapy and motivational interviewing (MI) techniques (e.g., Bradshaw et al. 2018; Owens et al. 2017; Reinke et al. 2008). MI theory purports that specific therapist strategies elicit client change talk, where change talk is defined as statements that express a desire to change, reasons or benefits of using a strategy, and beliefs in one’s ability to implement the plan; and subsequently, client change talk predicts behavior change or favorable outcomes (Magill et al. 2014; Miller and Rollnick 2013). A recent review of the MI literature provides empirical evidence of the predictive relationship between client change talk and subsequent change in behaviors such as alcohol use, medication adherence, and healthy food intake (Romano and Peters 2016).

In the context of our multi-component consultation package, consultants assessed each teacher’s values and intervention-related beliefs and used MI techniques, including motivational rulers, to elicit change talk to address beliefs. For example, after the teacher and the consultant reviewed data from recent classroom observations, they collaboratively developed the teacher’s plan for the next 2 weeks. Then, the consultant asked two motivational ruler questions: (1) Among all your other responsibilities how important is the implementation of this plan? Importance ratings convey the extent to which the plan is a priority to the teacher (Miller and Rollnick 2013); (2) How confident are you that you can carry out the plan? Confidence ratings convey the extent to which teachers believe they can contribute to or make the change happen (Miller and Rollnick 2013). Consistent with MI procedures, if the teacher’s ratings were high (9 or 10), the consultant elicited change talk by asking What makes you a [reported number] instead of a [lower number]? If the teacher reported a lower number (8 or below), the consultant elicited change talk by asking What might it take to move you from a [reported number] to a [higher number]?

The results of the pilot test (Owens et al. 2017) revealed that teachers with lower baseline knowledge, skills, and intervention-supportive beliefs (i.e., more barriers to integrity) demonstrated more improvement in classroom management skills (e.g., use of labeled praise and appropriate response to rule violations) in response to multi-component consultation, as compared with a comparison condition designed to mirror best practices (Cohen’s d ranged from 0.33 to 1.12 across skills). This study and others (Bradshaw et al. 2018; Reinke et al. 2008) are revealing promising results about the use of MI in teacher consultation.

However, we are unaware of any studies that have examined the relationship between teacher motivation and change in teacher behavior. Miller and Rollnick (2013) describe both importance and confidence as critical for motivation to change, as people are unlikely to commit to making a change unless they believe the change is important and possible. Yet, as suggested by MI theory, a better understanding of teachers’ ratings of importance and confidence could help consultants differentiate consultation to better support teacher needs. If a teacher had high importance ratings but low confidence ratings, the consultant could focus on skills practice. If a teacher had high confidence ratings but lower importance ratings, the consultant may focus on trying to understand how implementation could be better linked to other teacher priorities. Tailoring consultation may allow for greater facilitation of change talk, problem-solving, and a greater reduction in barriers, ultimately leading to better teacher and student outcomes.

Motivational Rulers as Assessment Tool

Given the relationship between change talk and behavior change (Magill et al. 2014; Romano and Peters 2016), practitioners are interested in identifying efficient, psychometrically sound assessments that could provide a proxy for motivation for change. Motivational rulers are a common component of MI-based protocols and may serve as such an assessment (Miller and Rollnick 2013). Motivational rulers ask clients to rate the importance of change, readiness for change, and/or confidence in change or a given plan on a 1 to 10 scale. There has been some psychometric evaluation of ruler ratings in clinical contexts. Boudreaux et al. (2012) examined the psychometric properties of three ruler ratings (importance of, readiness to, and confidence in ability to quit smoking in the next month) in patients attending an emergency room visit. This study documented the moderate association between these constructs (rs range from 0.50 to 0.70) and the utility of ruler ratings in predicting smoking behavior 2 weeks later. However, we are unaware of any studies assessing the use of motivational rulers in teacher consultation.

Thus, the goal of this study was to examine psychometric properties of motivational ruler ratings provided by teachers during the early stages of consultation in classroom management, as change in adult behavior is often observable after a brief MI intervention (Dishion et al. 2003). We sought to answer the following questions: What is the relationship between confidence and importance (aim 1)? If ratings on these two variables are highly correlated, they may not represent distinct constructs. If the constructs are distinct, do they show stability (as evidence of credibility) over a brief time period (aim 2)? Lastly, is there preliminary evidence of predictive validity of ratings (aim 3)? Namely, are importance or confidence ratings associated with teacher implementation of classroom management strategies in the 2 weeks following the ratings? Findings will provide insights into the reliability and validity of motivational ruler ratings in this context and generate hypotheses for future research.

Method

Participants

Participants were 29 teachers (13 in Ohio, 16 in Florida) in the multi-component condition of our randomized trial (Owens et al. 2017), as motivational ruler ratings were only collected in this condition. Three teachers in this condition were excluded from this study because the target student in their class moved prior to the second consultation session, resulting in missing data for the analyses. Most (89.70%) teachers were women, who identified as Non-Hispanic White (51.70%) or Hispanic (any race; 48.30%) with an average of 13 years of teaching experience (range 0.5 to 36 years); 55.20% had a master’s degree. There were 9 consultants (see Owens et al. (2017) for more details about teachers and consultants).

Procedures

All general education teachers in the eight participating schools were invited to a 3-h workshop on best practices in classroom management and the daily report card (DRC) intervention. At the workshop, teachers were recruited for the consultation study (across schools, response rate was 10 to 58%). To participate, teachers were required to identify one student with or at risk for attention deficit hyperactivity disorder (see target student identification procedures in Owens et al. (2017)). Consultation focused on general classroom management strategies (i.e., labeled praise, use of rules, effective instructions, and appropriate response to rule violations) and the use of a DRC intervention with the target student. A DRC is an evidence-based classroom intervention for reducing disruptive behavior and improving academic performance. When using a DRC, teachers identify target behaviors (e.g., interruptions, work completion), set daily goals for success (e.g., 7 or fewer interruptions), provide the student with feedback throughout the day, and send the DRC home to parents where privileges can be provided contingent upon achievement of goals. With the consultant, teachers participated in (a) an interview about the teacher’s values and approach to classroom management, (b) a target behavior interview to identify DRC target behaviors, and (c) a DRC development meeting to review baseline data (collected by the teacher) and finalize the DRC, each occurred during separate consultation meetings. Once the DRC was launched, teachers were observed weekly and met biweekly (every other week) with the consultant. During each consultation session (referred to as session 1 through 4), the consultant and the teacher discussed an informational handout, reviewed DRC data and observation data from the past 2 weeks, and created a plan for the next 2 weeks based on the feedback. Consultants were trained to adopt an MI spirit and use MI strategies during all sessions (see Owens et al. (2017) for details). Implementation integrity was conceptualized as teacher use of a strategy (e.g., praise) and data were obtained via observation.

Measures

Motivational Rulers

At the end of each consultation session (session 1 through 4), teachers rated the importance of and their confidence in implementing their plan over the next 2 weeks. Ratings were made on a 1 (not at all) to 10 (very confident/important) scale.

Student Behavior–Teacher Response (SBTR) Observation Rating System

The SBTR is a psychometrically sound class-wide observation system designed to capture discrete student-teacher interactions (Fabiano et al. 2010; Owens et al. 2017). Observers obtained frequency counts of (a) classroom rule violations by the target student, (b) classroom rule violations by all other students, (c) teacher’s appropriate responses to each rule violation (by the target student and other students, separately), and (d) teacher’s praise statements (labeled and unlabeled) toward the target student and other students. Each frequency count variable (e.g., total rule violations, total labeled praise) was summed for a given observation, divided by the duration of the observation in minutes, and multiplied by 60 to produce a rate per hour. The total number of appropriate teacher responses to target student violations for a given observation was divided by the total number of violations by the target student for that observation period. This produced the percent of appropriate response to violations per observation. This was repeated for appropriate response to violations by all other students. Rates and percentages were each averaged, respectively, across observations during baseline and during the 2 weeks between consultation sessions. Inter-observer assessments were conducted for 24% of all observations. Intra-class correlations (ICC) of type 1 for average of k raters ranged from 0.78 to 0.98 across variables.

Results

Aims 1 and 2: Correlations Between Importance and Confidence Ratings

Motivational ruler ratings ranged from 5 to 10; with averages falling between 8 and 9 (see Table 1). Correlations between importance and confidence ratings from the same session (see italics in Table 1) ranged from low (session 1: r = 0.24, p = 0.22) to moderate (session 2 r = 0.54, p < 0.05). Stability between initial importance ratings at session 1 and later sessions (see boldface in Table 1) was moderate, ranging from r = 0.42 (session 3, p < 0.05) to r = 0.54 (session 4, p < 0.01). Similarly, stability between initial confidence ratings (at session 1) and later sessions was moderate, ranging from r = 0.32 (session 4, p = 0.13) to r = 0.53 (session 3, p < 0.01).

Table 1 Descriptive statistics and bivariate correlations for teacher motivational ruler ratings at sessions 1 through 4
Full size table

Aim 3: Association Between Motivational Ruler Ratings and Behavior Change

Using session 1 importance and confidence ratings, we categorized teachers as having lower (ratings of 5 to 8) or higher (ratings of 9 or 10) ratings relative to each other. These cutoff scores were selected because they aligned with consultant interview procedures (e.g., whether they elicited change talk in relation to a higher or lower number) and because the small sample size precluded a three-level or four-level group (e.g., representing the interaction between the two constructs). We examined change in six teacher behaviors between the baseline period and the session 2 observations. Baseline and end of consultation data provided elsewhere (Owens et al. 2017; Owens et al. 2019) document that teachers had room for growth in these behaviors. Cell sizes (see Table 2) vary for each strategy due to missing observation data (e.g., if there were no target student rule violations in an observation, there were no data for percent appropriate response). Due to small cell sizes, we report descriptive rather than inferential outcomes.

Table 2 Rates of teacher strategy use during baseline and subsequent 2 weeks among those with high and low ratings of confidence and importance
Full size table

For importance ratings, teachers with higher ratings showed more improvement than those with lower ratings for rates of praise (labeled and unlabeled) toward target student and other students, and percent appropriate response to other student rule violations (see Table 2). For these behaviors, the improvement of teachers in the higher rating group was nearly twice the rate of that of teachers in the lower rating group. Hedge’s g effect sizes representing group differences at session 2 ranged from 0.11 to 0.78. Rates of change for percent appropriate response to target student rule violations were similar for both groups.

For confidence ratings, the pattern was similar (to importance ratings) for praise (labeled and unlabeled) toward target student and other students and for percent appropriate response to target student rule violations. Hedge’s g effect sizes representing group differences at session 2 ranged from 0.20 to 0.58.

Discussion

This study provides preliminary descriptive evidence of the psychometric properties of motivational ruler ratings with teachers. First, the correlations between importance and confidence rating from the same session were low to moderate. Consistent with Boudreaux et al. (2012), these findings suggest that ratings of importance and confidence are related yet distinct. Thus, when a teacher’s importance and confidence ratings differ, it may indicate that different strategies should be used in consultation to facilitate implementation (e.g., additional skills practice, or exploring the connection between teacher values, priorities, and recommended practices). Researchers and practitioners should continue to examine importance and confidence ratings as separate constructs.

Second, we found moderate stability between initial importance ratings and importance ratings over the first several sessions, and between initial confidence ratings and confidence ratings over the first several sessions. This offers some credibility to the ratings in this context (i.e., ratings were not haphazard), as only moderate stability is expected knowing that perceived importance and confidence may change as teachers observe change in student behavior and receive feedback on their own behavior.

Lastly, we found evidence that motivational ruler ratings may have utility in predicting change in teacher behaviors during the early phase of consultation. Namely, teachers who were highly confident in session 1 that they could complete their plan nearly doubled their use of five of the six recommended strategies between baseline and session 2. Although teachers with lower confidence showed some improvement in these six behaviors, the rate of improvement was approximately half that of those with high confidence ratings. The pattern was similar for importance ratings. Thus, despite teachers only using ratings from 5 to 10 (on a 1 to 10 scale) and despite the modest correlations between importance and confidence, both importance and confidence ratings were associated with short-cycle changes in several teacher behaviors.

Although this study makes a unique contribution to the literature, the findings should be considered preliminary given the limitations of the sample. First, the sample is small, and the categorical groups of teachers are even smaller. This precluded statistical analyses and examination of each combination of high and low levels of both constructs (importance and confidence). Thus, the findings warrant replication with larger samples to ensure stability. Second, we categorized teachers with ratings of 9 or 10 into the higher group and teachers with ratings of 5 through 8 in the lower group. Although this categorization aligned conceptually with the clinical practice (e.g., how consultants used the ratings to elicit change talk), different categorizations should be considered. Third, other variables (student behavior, teacher characteristics, contextual factors) were not accounted for and warrant investigation given the complexity of factors affecting teacher motivation and behavior. Fourth, the range of the motivational ruler ratings was restricted and skewed towards the upper end of the scale. Additional studies are needed to determine if teachers do not use the low end of the scale or if this range is specific to this sample.

Nonetheless, this study provides the first examination of the psychometric properties of motivational ruler ratings completed by teachers in the context of consultation focused on classroom management interventions. The findings suggest there is adequate variability and stability in ratings, and promise for the short-term predictive validity of the ratings. Thus, researchers should further explore these brief tools as a proxy for motivation for change using outcomes at a variety of time frames and with a larger sample. With replication, such ratings could be used by consultants to individualize consultation to teacher needs and as a way to enhance teacher motivation and implementation, and ultimately, student outcomes.