Instructional Pragmatism: Using a variety of evidence-based approaches flexibly to improve learning

Instructional pragmatism is essential for successfully adapting evidence-based active engagement (EBAE) approaches and involves instructors viewing improving teaching and learning as a process. Promoting and supporting instructional pragmatism is critical for ensuring that instructors are not disheartened if EBAE approaches do not produce desired outcome. Cultivating instructional pragmatism entails supporting instructors so that they recognize the value of being patient and optimistic while adapting EBAE approaches and keep a variety of approaches in their instructional toolbox. Here we illustrate an example of instructional pragmatism in which a quantum mechanics instructor did not give up when an EBAE method involving implementation of a sequence of clicker questions on addition of angular momentum did not yield expected learning outcomes. Encouraging instructors to embrace instructional pragmatism and creating a community of educators who support each other and discuss their implementation of EBAE approaches can go a long way in helping students learn physics.

Instructional pragmatism is essential for successfully adapting evidence-based active engagement (EBAE) approaches and involves instructors viewing improving teaching and learning as a process. Promoting and supporting instructional pragmatism is critical for ensuring that instructors are not disheartened if EBAE approaches do not produce desired outcome. Cultivating instructional pragmatism entails supporting instructors so that they recognize the value of being patient and optimistic while adapting EBAE approaches and keep a variety of approaches in their instructional toolbox. Here we illustrate an example of instructional pragmatism in which a quantum mechanics instructor did not give up when an EBAE method involving implementation of a sequence of clicker questions on addition of angular momentum did not yield expected learning outcomes. Encouraging instructors to embrace instructional pragmatism and creating a community of educators who support each other and discuss their implementation of EBAE approaches can go a long way in helping students learn physics.

I. INTRODUCTION AND BACKGROUND
A major goal of physics courses is to help students learn physics concepts [1][2][3][4][5][6][7][8][9] while also developing their problem solving and reasoning skills [10][11][12][13][14]. Researchers have been involved in developing and evaluating evidence-based active-engagement (EBAE) curricula and pedagogies  to improve student learning, but implementation of these EBAE approaches has been slow. Some major barriers to implementation of the EBAE pedagogies at the college-level include lack of instructor buy-in and their reluctance and/or resistance, partly due to a lack of an institutional reward system for using these evidence-based approaches, the time commitment involved in effectively adapting and implementing them, and instructors' fear that their students may complain [27]. The amount of class time required to implement an EBAE pedagogy, the flexibility with which it can be implemented, the need to train instructors in how to effectively use it, and the architectural constraints of the classrooms may also increase the barrier [27]. Moreover, one major barrier is that even those instructors who adopt and adapt EBAE approaches in their classes are often disappointed when they do not observe the learning gains they expected and quit employing the EBAE methods [27].
The goal of this article is to illustrate an example of instructional pragmatism in which one quantum mechanics (QM) instructor did not lose hope when an EBAE method involving use of clicker questions on addition of angular momentum did not yield the expected learning outcomes on the subsequent posttest. Instead, he remained optimistic, consulted education experts and employed another EBAE method with the same set of students. His flexibility and use of another EBAE pedagogy, "incentives for learning from mistakes" (ILM), was relatively effective in improving his students' performance on an analogous problem posed on a later exam. We argue that supporting and aiding physics instructors to embrace instructional pragmatism can go far in helping their students learn physics.
Instructional Pragmatism Framework: One major cause of the low sustained usage of the EBAE learning tools and pedagogies among the adopters is that the early adaptations in their classes do not necessarily show the same large gains in student learning as those observed by their developers [27]. One reason for this failure to replicate the positive outcomes is that the EBAE approaches must be adapted and refined, e.g., to suit the instructor's style and their students' prior knowledge, skills and motivational characteristics. The effectiveness is limited by the fidelity and bandwidth of implementation of an EBAE approach [27]. The failure to replicate may lead instructors to feel disappointed and quit using the approach. The instructional pragmatism framework hinges on the need to support instructors in order to ensure flexibility and persistence when an EBAE approach is not successful in improving student learning and remind themselves that improving teaching and learning is a process that requires continuous refinement to yield desired outcome. Cultivating instructional pragmatism via a community of like-minded instructors, who support each other as they adapt EBAE approaches and keep several approaches in their instructional toolbox, can go a long way in their sustained implementation. Members of such a community can support each other to be pragmatic and patient about adapting and fine-tuning EBAE approaches that meet the needs of their students and constraints of their classrooms in order to help students master physics concepts.
Background on clicker questions used as an EBAE approach: Clicker questions are conceptual multiple-choice questions typically administered in the classroom to engage students in the learning process and obtain live feedback about their learning [28][29][30][31][32][33][34]. They can be interspersed within lectures to evaluate learning in each class segment, placed at the end of a class or at the beginning of a class to review materials from previous classes. While many clicker questions for introductory [28][29][30][31][32][33][34] and upper-level physics such as QM [35] have been developed, some documented efforts [36] have also focused on development and validation of clicker question sequences (CQSs) [37][38][39][40], e.g., question sequences on a concept that can be used in a few class periods when students learn the concepts and that build on each other to help students organize and extend their knowledge structure.
Background on ILM Pedagogy Used: The illustration of instructional pragmatism of one of the two instructors using CQS as the first EBAE pedagogy here involves one of them also using ILM pedagogy, i.e., giving incentives to students to learn from their mistakes when the earlier implementation of the CQS did not yield the desired outcome. The ILM pedagogy involves instructors explicitly prompting students to learn from their own mistakes by rewarding them for correcting their mistakes [41][42][43][44][45][46].
In a prior ILM study spanning four years with advanced undergraduate physics students taking a QM course, students were given the same four problems in both the midterm exam and final exam (similar to an earlier study in introductory physics [42][43][44][45]). About half of the students [46] were incentivized to correct their mistakes in the midterm exam and earn up to 50% of the points lost on each exam problem. The solutions to the exam problems were provided to all students but only after corrections were submitted to the instructor. Performance on the final exam on the same problems suggests that students who were given incentives to correct their mistakes significantly outperformed those who were not given similar incentives [46].
Organization of This Article: Since CQS was used as the first EBAE pedagogy by both instructors, in section II we discuss the learning goals and in-class implementation of the CQS. In section III, we discuss instructional pragmatism of an instructor who implemented the CQS and did not obtain the desired posttest performance. In particular, the instructor then used the ILM pedagogy to improve student learning. In section IV, we conclude with a discussion and summary that focus on the importance of providing support to instructors.

II. THE EBAE APPROACH INVOLVING CQS
Here we compare and contrast two instructors' teaching approaches to illustrate instructional pragmatism in the context of addition of angular momentum in QM. The first instructional approach that both instructors used was CQS. The learning goals of the CQS included helping students identify the dimensionality of Hilbert space of the spin of two particles, to be able to choose a suitable representation such as "coupled" or "uncoupled" representation and construct a complete set of basis vectors for the product space in the chosen representation and to be able to calculate matrix elements for operators corresponding to observables in these representations [37]. In standard notation, the basis states in the uncoupled representation are eigenstates of Ŝ 1 2 , Ŝ z1 , Ŝ 2 2 and Ŝ z2 and are written |s 1 , m s1 ⟩ ⊗ |s 2 , m s2 ⟩. Here each particle's individual spin and z-component of spin quantum numbers are 1 , 2 and 1 , 2 , respectively. In the coupled representation, the basis states, |s, m s ⟩, are eigenstates of Ŝ 1 2 , Ŝ 2 2 , Ŝ 2 , and Ŝ z where ⃗ = ⃗ 1 + ⃗ 2 .The total spin quantum number, s, and the z-component of the spin quantum number, m s , are for the entire system. The CQS strived to help students use the addition of angular momentum to determine that the total spin quantum number of the system s can range from s 1 + s 2 down to |s 1 − s 2 |, with integer steps in between, where s 1 and s 2 are the individual spin quantum numbers. The z-component of the spin of the composite system is m s = m s1 + m s2 .
Consistent with the learning goals of the CQS [37], pre/posttests were developed and validated [47] to measure comprehension of the concepts related to the addition of angular momentum. On the pretest (after traditional instruction), students were given a system of two spin-1/2 particles with a spin-spin interaction Hamiltonian and a magnetic field-spin interaction Hamiltonian, Ĥ 2 = −μB(̂1 +̂2), and asked to answer these questions: The posttest that students were administered following the implementation of the CQS was analogous to the pretest and asked the same questions as the pretest but for a system of one spin-1/2 particle and one spin-1 particle. The rubrics used for pre/posttests given to students before and after the CQS were used earlier in another context [46]. Questions (a), (b), and (c) were each worth 3 points with partial credit if only some basis states or matrix elements were correct. Question (d) was worth 1 point.
In-Class Implementation of CQS by Instructors: Both instructors (A and B) at the same institution in consecutive years implemented the CQS in a similar manner. In particular, the CQS was implemented with peer discussion [29] in a QM class at a large research university by both instructors after traditional lecture-based instruction in relevant concepts including, but not limited to, those covered by the CQS. Between traditional lecture-based instruction and the CQS implementation, students took a pretest in each class. The first six questions in the CQS were implemented together right after the pretest. The third section of the CQS was implemented at the beginning of the next class. The posttest was administered during the following week to measure the impact of the CQS on student learning. Tables 1 and 2 show pre/posttest performances of students in instructors A and B's QM class (class A and class B, respectively) after traditional lecture-based instruction (pretest) and on posttest after students had engaged with the CQS. The normalized gain is calculated as = ( % − %)/(100% − %) [1] and presented in Tables 1 and  2. Effect size is also calculated in Tables 1 and 2   Although the number of students (N) in each class is small, the pretest scores in Tables 1 and 2 are low. The posttest scores of the two groups in Tables 1 and 2 suggest that the CQS is effective in helping students learn to construct a complete set of basis states (question (a)) and to a limited extent calculate matrix elements for the magnetic field-spin interaction Hamiltonian particularly in class A (question (c)). However, Tables 1 and 2 show that students did not perform well on questions (b) and (d) even after engaging with the CQS. Review of student responses suggests that a major reason for the poor performance on both of these questions, even after the CQS, is due to a majority of students choosing the basis to be the uncoupled representation (since it is the simpler representation for constructing the basis states) and then having difficulty with the matrix elements of the spin-spin interaction Hamiltonian since it is only diagonal in the coupled representation. In question (a), many students in both classes correctly constructed a complete set of basis states, but chose the uncoupled representation.
While the magnetic field-spin interaction Hamiltonian in question (c) is diagonal in both coupled and uncoupled representations, calculating the matrix elements of the Hamiltonian in question (b) in the uncoupled representation is even more challenging since that operator is not diagonal in this basis. Along with a reasonable posttest score for question (a), both class's poor posttest scores on questions (b) and (d) is due to the fact that while students learned to construct a complete set of basis states, many were not versed in calculating the matrix elements of an operator in a representation in which it is not diagonal as in question (b).
For question (d), even after the CQS, many students in both classes incorrectly stated that both Hamiltonians are diagonal in the uncoupled representation they had chosen. Since students were only asked to calculate a single offdiagonal matrix element in question (b), some students who correctly calculated an off-diagonal matrix element in question (b) that was zero concluded incorrectly that the entire ̂1 matrix is diagonal in the uncoupled representation.
Based on think-aloud interviews we conducted with some QM instructors to understand their thought processes, we find that QM experts are more likely to consider whether different operators are diagonal in a representation before choosing a basis to evaluate the two Hamiltonians. They preferred to choose the coupled representation since both Hamiltonians are diagonal in that representation. Interviews with some students suggest that they did not, in general, do this type of metacognition before selecting a basis.

III. INSTRUCTIONAL PRAGMATISM: COMBINING DIFFERENT EBAE PEDAGOGIES
Instructional pragmatism while implementing EBAE pedagogies involves viewing teaching and learning as a process that one should continuously strive to improve. It is an invaluable skill for any instructor to develop and entails staying optimistic, flexible and persistent. This pragmatism can inspire instructors to adapt and refine an EBAE approach or change to a different one to fit the needs of students dynamically as they assess student learning. While EBAE strategies found to be effective at the institutions where they were developed are likely to provide promising results at other institutions, improvement in student outcomes may require adaptations, fine-tuning and several implementations to be effective for a given instructor and their students.
Moreover, cultivation of instructional pragmatism requires that instructors have community support of likeminded educators interested in implementing innovative EBAE pedagogies in order to have a sustained view of teaching and learning as a process and recognize that early implementations may not yield the desired student outcomes. It is beneficial to have some STEM education researchers promoting and supporting instructional pragmatism as part of this type of community of innovative educators. In this community, innovative instructors can regularly discuss their implementation of the EBAE methods and the fact that the implementation must be adapted and refined to suit instructors' teaching style and their students' prior knowledge and skills. They can discuss that an EBAE approach that has been effective elsewhere should not be viewed as a one-size-fits-all panacea. In fact, it is important for instructors to be optimistic and flexible in dynamically adapting and refining EBAE approaches to yield positive outcomes for their students and keep several EBAE instructional tools in their toolbox in case a particular approach does not yield the desired outcome in the context of a given topic in a course for a set of students.
Here we discuss the instructional pragmatism of instructor B who did not give up and consulted with his physics education research colleagues with knowledge of pedagogy after being concerned that a majority of students had not mastered the concepts after his implementation of the CQS. The consultation also focused on the fact that one likely reason for not benefiting from the CQS is that students did not have sufficient initial knowledge (as evidenced by their pretest performance) before engaging with the CQS. When roughly half of the students know the correct answers to the clicker questions, the peer discussion during the implementation of those questions is more effective [28]. They also discussed that one possible reason for the lower preparation evidenced by the pretest may be that class B's instructor did not spend sufficient time before the CQS discussing relevant underlying concepts (e.g., on questions (b) and (d), students in class B performed very poorly on the pretest and for those questions their posttest scores are also below 40%). After this consultation, instructor B decided to use ILM pedagogy, which paid off as discussed below.
In summary, following poor posttest results (Table 2), instructor B displayed instructional pragmatism, consulted with colleagues with knowledge of pedagogy and implemented ILM pedagogy. He returned the posttests to students with scores and incorrect parts marked but without explanations, and asked them to correct their mistakes as homework for up to half of the quiz points they had missed. Unlike the earlier implementation of the ILM pedagogy in QM in which students were asked to correct their mistakes on a midterm exam [46] with a similar half-missed-points incentive (in which all students corrected their mistakes), not all students took advantage of the opportunity instructor B provided possibly because the posttest was a low stakes quiz worth less than 1% of a student's final grade. Table 3 shows the results after 12 of the 19 students made corrections to their posttests. Thus, with instructional pragmatism and implementation of both the CQS and learning from mistakes pedagogies, students in class B who corrected their mistakes demonstrated better learning outcomes (see Table 3).
Moreover, instructor B also gave question (c) as part of the midterm exam. While students in instructor B's class who did not correct their mistakes also performed better on this problem on the exam with a 71% average (since they also had access to the correct solution and had further opportunity to learn concepts before the exam), those who corrected their mistakes performed much better with 85% average, despite their lower average posttest scores initially.

IV. DISCUSSION AND SUMMARY
Physics education researchers have developed many EBAE curricula and pedagogies and documented their effectiveness in certain types of physics classes at some institutions. While there are significant efforts being made to disseminate these tools and pedagogies to instructors at other institutions, their sustained widespread adoption and adaptation in physics classes has been slow. Here we argue that one major reason for the lack of widespread adoption and adaptation of the EBAE approaches is lack of sufficient focus on the critical role of promoting and supporting instructional pragmatism. In particular, physics instructors should be encouraged and supported to be optimistic and internalize the fact that improving teaching and learning is a process. They should strive to continue to improve student learning by adopting and adapting innovative EBAE approaches even if they do not yield stellar outcomes expected. Promoting instructional pragmatism is central to ensuring that instructors do not get discouraged and abandon a promising approach if the earlier implementations did not yield expected student outcomes.
Here we discussed a concrete example of instructor B who displayed instructional pragmatism. Unlike instructor A, who did not use additional EBAE pedagogies to help students learn about addition of angular momentum, instructor B felt the need to provide further support to his class and consulted with a physics education researcher. He pragmatically used the ILM pedagogy and gave students grade incentives to correct their mistakes on the posttest. Students who took advantage of this opportunity and made posttest corrections not only showed gains on the revised posttest, they also performed better on a final exam question that focused on the same concepts.
The mutual support of a community of like-minded STEM educators who are invested in adopting and adapting EBAE approaches cannot be overlooked. It is possible that with this type of a community support, even instructor A in this study would have attempted another EBAE approach when the CQS implementation did not yield positive posttest outcomes (see Table 1). This type of community support to foster instructional pragmatism should include at least some STEM education researchers who can take leadership role in keeping the community focused on instructional pragmatism in the context of implementing EBAE approaches and viewing teaching and learning as a process. The members of such an innovative educator community can meet regularly to discuss how the EBAE approaches in their own STEM classes are working and to learn from and support each other in order to stay focused on the process of adaptation and improvement instead of getting disheartened. This type of discussions with like-minded educators is critical in order for instructional pragmatism to be sustained in a widespread manner. This type of a community can help instructors learn and keep several EBAE approaches in their toolbox.
In summary, instructional pragmatism and being flexible about using various EBAE pedagogies as appropriate to suit students' needs dynamically can go a long way in improving teaching and learning in all physics courses. Ensuring that the EBAE approaches are adapted appropriately for an instructor's style and their students' prior knowledge and skills may take time. Therefore, instructors must be supported to recognize that all EBAE approaches have certain fidelity and bandwidth of implementation in which they will be effective. Instructional pragmatism can be cultivated amongst educators by encouraging and supporting instructors as they innovate and adapt EBAE pedagogies in their classes. In particular, for widespread adoption of EBAE approaches it is necessary to have a community of educators who support each other while they all practice instructional pragmatism and remain optimistic while flexibly adapting EBAE pedagogies to help students learn in their classes.

ACKNOWLEDGMENTS
We thank the NSF for award PHY-1806691