Characteristics of well-propagated undergraduate STEM teaching innovations

The science, technology, engineering, and mathematics (STEM) education communities have developed many innovative teaching strategies and learning materials in the past few decades. While these innovations have significant promise for improving undergraduate STEM education they commonly go unused by other STEM instructors. This study is an inquiry into the relatively small number of teaching innovations that have become widely used in college-level STEM disciplines. The disciplines include biology, chemistry, computer science, engineering, geoscience, mathematics, and physics. The successfully propagated innovations were identified via experts in each discipline and were validated through member checking and searches of relevant literature. Short descriptions of each innovation were developed and, when applicable, sent to the original developer for feedback, and publically-available funding data was collected. In this paper we focus on the “branded” innovations—innovations with a central leadership and core set of practices. Many of these originated in physics, and the majority focus on changes in pedagogy, not content. In addition, most have received funding for at least ten years. These commonalities have implications for the structure of current funding models and the ease of implementing changes in content.


I. INTRODUCTION
Across science, technology, engineering, and mathematics (STEM) disciplines, significant education research focuses on developing teaching innovations that have been shown to improve undergraduate instruction [1].Although many new strategies and materials now exist, most are not widely used by STEM instructors (e.g.[2]).
In order to improve this situation it is important for the STEM education research community to develop a better understanding of how innovations become successfully propagated.The authors are involved in an initiative to develop resources for STEM education researchers who want to develop more effective propagation plans.One of the steps in developing these resources was to identify innovations that are widely used.Elsewhere we describe in depth studies of some of these that helped develop a rich description of the practices and processes that led to their widespread adoption [3], [4].
This paper provides a summary of our analysis of the survey, with attention to the difference between physics and other disciplines.We call these innovations well-propagated instructional strategies and materials (WePISMs).We will discuss some important characteristics of these innovations and the implications for change agents.

II. RESEARCH DESIGN
The study was motivated by a desire to understand the current landscape of widely-used innovations across STEM disciplines.With no clear a priori picture, we used qualitative methods suitable to allow for emergent results.
We carried out the study in three stages as discussed in the following three sections.

A. Initial data collection
The first step in the study was a survey of experts in research-based undergraduate teaching in each of the six disciplines studied (biology, chemistry, engineering, geoscience, mathematics, physics).We identified experts through membership in national committees such as the National Research Council Discipline-based Education Research report committee (NRC DBER), professional society leadership, and our personal networks.Often, we used snowball sampling, asking experts who else would be able to contribute.We contacted 98 experts and received usable responses from 39.
Emails followed a set script, briefly introducing the project and asking experts the following question: "Please respond to this email and identify the five or so 'new' learning materials or teaching strategies that you feel have been most successfully propagated in undergraduate

PERC Proceedings,
[DISCIPLINE].It will be very helpful if you could also include a short explanation of why each was chosen."Most experts responded via email, while others preferred to set up a phone call.The reasons for this were either politically driven (people did not want to accidentally align themselves with one movement or another by "endorsing" it) or time driven (it was simpler to speak about it over the phone.)Phone calls were not recorded, although the innovations named and the basic rationale for including them were written down during the phone call

B. Validating the list of WePISMs
All suggested innovations were included in an initial list.
To validate the list we conducted a search of relevant literature and other artifacts such as project websites and digital libraries.Criteria for being a well-propagated educational innovation included: 1. Used primarily in college settings.Some innovations suggested were developed for K-12, which was not our area of focus.
2. Used primarily as a teaching tool.Some innovations suggested, such as concept inventories, are more commonly used as an educational research tool than a teaching tool (although we realize that there is not a clear line here and our project team has had many discussions about this issue).
3. Evidence that it is used by others.This could be literature support, such as being mentioned in the DBER report [1] or other literature reviews, literature written by instructors who implemented the innovation and reported their experience in an education journal.Other evidence could be being included in a workshop program, or the existence of a conference devoted solely to the innovation.Evidence was also found in the frequency of mentions by experts, and searching the internet for examples of implementations.
With this initial editing of the list completed, we sent the list back to the experts for member checking.Experts often disagreed on items that were included and suggested a few more that could possibly be included, giving opinions or evidence as to why.In addition to member checking with the survey respondents, we solicited feedback on the list from colleagues at the 2013 TUES PI meeting, who had further suggestions.With some new innovations added and others deleted, we went through once again to find other evidence to support that they were indeed used by a broad user base, used in a college setting, and used primarily for teaching.The list (N=46), with descriptions and links to more information, is available at our project website: www.increasetheimpact.com.

C. Categorizing the list of WePISMs
Given that the types of changes varied significantly across the 46 innovations, it was necessary to develop a classification scheme.The research team tried several before deciding on the one presented below (Fig. 1) which categorizes WePISMs based on whether use requires a change in content, pedagogy, or both.Each innovation on the list was coded separately by all six team members and discussed in meetings to come to agreement on its categorization.

FIG 1. The authors' classification scheme of types of educational innovations.
In addition to the type of change required for implementation, the innovations also differed significantly in terms of their level of specificity.For example, some were vague ideas (e.g., active learning) while others were much more specific (e.g., Ranking Tasks).The categories for specificity are described in Table 1.

General
A movement or broad theoretical term in education literature with many possible implementations (e.g.metacognition, active learning)

Recognizable
The innovation has a name which is associated with a set of teaching practices, but has no central leadership (e.g.flipped classroom, think-pair-share)

Branded
The innovation name is associated with a set of teaching practices and has central leadership (e.g.PhET Simulations, Peer Instruction)

A. Characteristics overall FIG 2.
Breakdown of the WePISMs (N=46) by the project type scheme described in Fig. 1.The columns are further broken into level of specificity.
The WePISMs, as shown in Fig. 2, are largely pedagogical changes (61%).This is followed by innovations that do not require a change in pedagogy/content (26%) and innovations that require a change in both pedagogy and content (11%).There was only one innovation reported which represented a change in content only (objects-first learning in computer science).

FIG 3. Discipline use of innovations, as reported by the experts.
The specificity of innovations used varies by discipline (Fig. 3).A chi square test (comparing projected and actual counts) showed a significant difference between physics, engineering, and geoscience in use of branded, recognizable, and general strategies.Engineering had primarily general WePISMs.Physics, in contrast, had few.Geoscience had more recognizable WePISMs than other disciplines.Physics had the largest number of branded innovations.

B. Branded innovations
For the branded innovations, more detailed information could be collected based on key publications and publically available funding data.

Millions
The branded innovations received funding for a period of at least eight years, with most receiving funding for over ten years (Fig. 4).The amount of funding covers a wide range, with the lower end at a half million dollars, with an average of four million dollars (Fig. 5).It is important to note that these are low estimates since most innovations have some funding sources that do not disclose funding amounts.Many of the innovations had more than one funding source (between 2 and 5).Nearly all received funding from Federal sources (mainly the National Science Foundation).Additional funding sources were often the institution where the innovation was developed or private foundations and companies.Notably, computer program innovations such as the PhET Simulations, Geogebra, and ALICE received funding from a large number of sources (20 or more).

IV. CONCLUSIONS
Well-propagated strategies and materials share several commonalities: they are largely changes to pedagogy, and they share a long history of funding, with multiple grants and sources.Physics innovations align with the broader STEM education community in the lack of well-propagated content innovations.While there are efforts to rethink the content, such as the Matter and Interactions textbook [5], efforts such as these are not well-propagated.Physics also aligned with other disciplines with regard to amount of funding and time projects have been funded.
There were disciplinary differences in the use of wellpropagated strategies.Many of the WePISMs originating in physics are branded, contrasted with other disciplines such as geoscience and engineering, which use more general and recognizable innovations.There could be several reasons for this difference.Physics Education Research as a field is slightly older than other STEM education fields, and some of the well-propagated innovations are noted as part of the history of the field [6].It could also be due to disciplinary differences: engineering education is vast and thus may rely on umbrella ideas more heavily than physics.Geoscience is more place-based, so pedagogical solutions for one setting may not transfer to others.
The lack of content innovations and the findings regarding funding have implications for educational developers.The branded innovations were funded for a minimum of eight years, often with multiple sources and multiple grants.This has implications for working within existing funding structures: a typical grant for an education project lasts three or four years, so to get more years of funding means pursuing multiple grants.Developers often think that publishing and presenting results of the work at the end of a three-year grant will mean the innovation reaches others; however the implication here is that in addition to having a good idea, developers need to be willing and able to spend a decade or more working on an innovation and pursuing funding opportunities in order to develop something that can be well-propagated.
The focus on pedagogy may imply a lack of development of content innovations, or it may be that content innovations are not being propagated.Possible barriers to propagation might include disciplinary norms and expectations of content coverage at the departmental and interdepartmental levels.It may be that content reforms require a large amount of cooperation between individuals and departments and thus are slow to be implemented, while innovations focused on pedagogy are more easily adopted into existing institutional structures.It is important for education researchers to ask themselves whether this state is desirable.Although we think of pedagogy as being firmly entrenched in higher education, it may be that content is even more so.
edited by Churukian, Jones, and Ding; Peer-reviewed, doi:10.1119/perc.2015.pr.037Published by the American Association of Physics Teachers under a Creative Commons Attribution 3.0 license.Further distribution must maintain attribution to the article's authors, title, proceedings citation, and DOI.

TABLE 2 .
WePISMs in physics for which funding information was publically available.
5IG5.Amount of funding dollars of branded WePISMs for which information could be found (physics, N=9; other, N=7).