Characterizing problem types and features in physics-intensive PhD research

Benjamin Zwickl1, Anne Leak1, Javier Olivera1, Jarrett Vosburg2, and Kelly Martin3 1School of Physics and Astronomy, Rochester Institute of Technology, 84 Lomb Memorial Drive, Rochester, NY 14623 2Department of Physics and Astronomy, SUNY Geneseo, 1 College Circle, SUNY Geneseo, Geneseo, NY 14454 3School of Communication, Rochester Institute of Technology, 92 Lomb Memorial Drive, Rochester, NY 14623 (Dated: September 19, 2016)


I. INTRODUCTION
Problem-solving receives a significant emphasis across the undergraduate physics curriculum.Most textbooks are full of worked example problems and back-of-the-chapter problems, and a "problem set" is a commonly used synonym for homework.Research on students' development of problem-solving skills, transfer of problem-solving abilities to new contexts, and the effectiveness of particular pedagogical approaches for teaching problem-solving has a long and rich tradition within PER [1].Problem-solving is often identified as a key 21st century skill or transferable skill within the cognitive domain (as opposed to interpersonal or intrapersonal domains) [2] and is sometimes viewed as a subset of critical thinking [3].Within PER, most research on problem-solving has focused on problems posed in introductory-level physics.Although a significant amount of research exists on how experts (i.e., faculty and graduate students) solve these introductory-level problems, there is limited research on the kinds of problems that experts solve as part of their professional scientific work [1].Some research does exist on workplace problems faced by engineers [4].We attempt to fill in that gap by providing a description of problems faced by PhD students as part of physics-intensive PhD research, an emergent framework for classifying problem types, and suggestions for the undergraduate physics curriculum.
There is general consensus about the abstract definition of a problem-the difference between the current state (e.g., knowledge and resources) and a goal state (e.g., a desired outcome).However, actual concrete problems come in an enormous variety of forms.Some problems, such as Sudoku puzzles, have a well-defined form, known solution strategies, and a single correct solution.Other problems, such as minimizing anthropogenic climate change, have an ill-defined end-goal (e.g., which climate parameters should be focused on and over what time scale), constraints (e.g., economic or technological feasibility), and a large number of possible solution paths (e.g., solar energy, energy efficiency improvements, carbon sequestration).In order to characterize the breadth of problems, we drew inspiration from Jonassen's framework that classifies problems according to Learning activity, Inputs, Success criteria, Context, Structuredness, and Abstractness [5].

II. METHODS
During the summer of 2015, we individually interviewed ten graduate students from a private research university with moderate research activity.The semi-structured interview protocol began by asking about students' overall dissertation project, and then followed up with questions about examples of routine, difficult, and important problems they faced during their research.Follow up questions were asked about solutions to these problems if the student did not directly address them.
Students were recruited via email and volunteers were paid $25 as an incentive for the 30-45 minute interview.The students were involved a mix of computational, theoretical and experimental projects in astronomy, astrophysics, optics, condensed matter physics, and microfabrication.Most students that responded were chosen for the study, with some pref-erence given to students that maximized diversity in gender, research focus, and year of study.Although students ranged from their first to sixth year of study, all were actively engaged in research.Participants attended eight different undergraduate institutions.Six were male, and four were female.
The interviews were transcribed and analyzed in NVivo Qualitative Data Analysis software.In a first pass, we identified specific problems, problem attributes, and problemsolving strategies that were discussed in the interview relating to routine, difficult, or important problems.A code definition for problem attributes was iteratively refined until four coders had a high level of agreement across multiple interviews.Problem attributes were defined to be any feature that describes the problem, its end goal, or what made it challenging, which was distinguished from problem-solving strategies that described actions the solver does to solve the problem.All interviews were coded by at least two researchers.Within the 10 interviews, 232 descriptive passages about problems were identified.These passages were then assigned an initial short descriptive open code (e.g., data analysis) that described some salient feature of the problem.The open codes were collapsed into a minimum number of well-defined codes through consensus discussions of the research team.The codes were also grouped into three categories: Context codes (Table I) that describe where the problem was located, Activities codes (Table II) that describe an active aspect of the problem, such as designing, and Features codes (Table III) that describe specific attributes that made the problem challenging, such as complexity.This three dimensional framework of Context, Activities, and Features provides an alternative to Jonassen's problem framework [5] that is well-suited toward teaching and research in physics education.

III. FINDINGS
Each of the three dimensions of the framework (Context, Activity, Features) are discussed within the findings.In addition, because the interview protocol separately addressed routine, difficult, and important problems, we further refine results to contrast routine and important problems.Routine problems included any issues that the participants encountered on a daily basis in their research.Difficult problems were those that the interviewee stated were among the most challenging tasks faced in their research.Important problems included those necessary to complete their thesis research as well as problems considered important to the wider scientific community.For brevity we omit difficult problems from our findings, however we plan to discuss those results in a future publication.
A. Problem context: Prevalence of experiment, computation, and data analysis Physical principles are most compactly and accurately expressed with mathematics.Perhaps because of this, most undergraduate-level problems are also mathematical in nature, and there is substantial research on the role of mathematics in physics problem-solving [6].However, when asked to discuss problems faced in their research, graduate students often described problems outside of mathematics or modeling.Specifically the contexts of Experiment, Programs, Software, and Data Analysis were all very prevalent (Fig. 1).The contexts were part of tasks such as setting up equipment, testing devices, collecting and analyzing data, and programming, which are prevalent in research settings.The problem contexts of Math and Modeling, while frequently occurring in research, represent only a fraction of the breadth of problems encountered in research.

B. Problem context: Social implications of important problems
Again, focusing on Fig. 1, we can see that important problems often involve more social aspects of research, such as collaboration, planning, and sharing results (with the broader scientific community).In fact, the most common important problems students faced were how to share their results with the broader scientific community.One participant said the following, regarding shared results."It's really difficult to get everybody to be in the same mindset and everybody to agree in the end that, yes you're finished, and yes what you have is good data and we don't need to do any more with it, the paper is finally ready to be in publishable form.So I think getting the data ready and then getting all of the people you worked with to agree, has been a struggle." This example highlights that some of the most important problems in PhD research have an intrinsically social component.For these social contexts, students' discussed the relevance of soft skills, such as teamwork, communication, and project management, as fundamental to the solution.

C. Problem activities: Important problems are open-ended
The structure of most undergraduate level problems involves constructing (or designing) some mathematical model or mathematical derivation as part of the solution.In our coding scheme, this would be considered designing a mathematical model, though designing can also be coupled with other contexts (e.g., designing a program or designing an experiment).However, Fig. 2 shows several other activities were common in graduate-level research.These included Troubleshooting, Interpreting Results, Evaluating Options, and Assessing Quality.
The activities associated with the most important problems include interpreting results, assessing the quality of those results, and the evaluating their options.Each of these activities is usually open-ended, involves considering multiple possible solution paths, and has a solution that is unknown to the wider scientific community.

D. Problem features: Important problems with impact
Problem features were characteristics of a problem that made it challenging, and could be coupled with any combination of context and activity.Figure 3 shows the occurrences of those features within the interviews for routine and important problems.For these students, the most important problems tended to be impactful.Solving the problem would matter to society or at least to some members of the scientific community.When discussing how his research could affect the scientific community, one participant explained the following.

Designing
Creating or constructing something (e.g., circuit, output signal, program, hardware/software interface, etc.) to produce an artifact that meets a goal.Troubleshooting Debugging, finding errors, fixing equipment, and correcting mistakes.

Interpreting Results
Understanding and making sense of results to find physical meaning in mathematics and understand the mechanisms that explain the findings.

Evaluating Options
Finding the best solution, model, or approach to optimize results and maximize efficiency.Assessing Quality Determining that results are good enough to move on, convince others, or publish."I think the most difficult problem that I've encountered there is with trying to convince people that what's going on is significant...So I have to convince them that this is something that matters -that's going to affect their measurements, that maybe has already affected their measurements -and can cause them to draw false conclusions, or cause them to throw out data that they could otherwise use." The need to make a difference in the research community, and have their research matter, seemed to drive the sentiment behind these types of important problems.Again as noted before, this problem feature has a clear social dimension and is not merely challenging for its complexity or ill-defined structure.

IV. CONCLUSIONS
Each of our findings suggests implications for the undergraduate physics curriculum.First, we find common problem contexts outside of mathematics and modeling.Professional Second, problem activities are diverse.The acts of troubleshooting, interpreting findings, assessing quality, or weighing multiple options and making a decision are often downplayed in the undergraduate curriculum.It is unreasonable for undergraduate problems to duplicate a PhD thesis, but it may be possible to create problems aligned with particular activities.For example, a problem can be presented in a way that requires students to articulate a set of plausible options, evaluate them, and make a well-reasoned choice.Similarly, students should gain experience with problems where the interpretation of a prediction or of data is emphasizedwhere gaining new physical insight takes priority over simple checks of accuracy (e.g., consistency of units).
Third, problems are not merely cognitive but also social in nature.Our findings show that the most common difficult and important problems involve the student as part of a wider community.To prepare students, we suggest a balance of individualistic and collaborative problem solving activities.Sometimes the greatest challenge is convincing others that your solution is meaningful and correct.When curricula emphasize the social aspect of the problem and solution (e.g., as in cooperative group problem solving), the collaboration is more than a means to improve content learning, but begins to mimic important scientific problems.The most important problems are also those with the most impact to a wider community so it is essential to position solutions or findings within a community.
These findings are only the first steps in an expanded set of problem-solving research that explores the role of representations, expert-novice differences, and explicit teaching of problem-solving strategies across the contexts, activities and features identified in this study.This work is supported by NSF DGE-1432578.

FIG. 2 .
FIG. 2. Activities that result in routine and important problems inPhD-level research.

TABLE I .
Code definitions for problem context.

TABLE II .
Code definitions for problem activities.

TABLE III .
Code definitions for problem features.work often takes place in laboratories or out in the field, and often involves the use of computers and analysis of data.It seems reasonable to provide exposure to a similar diversity of problem contexts in the undergraduate curriculum, especially recognizing the role of laboratory courses as problem-solving environments.Futher, integrating programming and software into undergraduate courses is not a technological fad, but is aligned with what scientists do.Similarly, data analysis could be incorporated throughout lecture and lab courses as students analyze and interpret data related to key principles or canonical models within the course. scientific