Contextualizing problem-solving strategies in physics-intensive PhD research

Anne Leak1, Javier Olivera1, Benjamin Zwickl1, Jarrett Vosburg2, and Kelly Martin3 1School of Physics and Astronomy, Rochester Institute of Technology, 84 Lomb Memorial Drive, Rochester, NY 14623 2Department of Physics and Astronomy, SUNY Geneseo, 1 College Circle, SUNY Geneseo, Geneseo, NY 14454 3School of Communication, Rochester Institute of Technology, 92 Lomb Memorial Drive, Rochester, NY 14623 (Dated: September 15, 2016)


I. INTRODUCTION
Problem-solving is at the heart of undergraduate physics education, and teaching students to develop their problemsolving skills has been an emphasis of PER for several decades [1].Students taking introductory physics classes diverge into a range of careers in STEM and beyond.While the physics content may be of varying relevance to future careers, it is hoped that problem-solving abilities are transferable to the problems these students will face later in their careers and lives.As such, problem-solving is included on a lists of 21stcentury skills that all citizens need [2].
Existing PER on problem-solving shows that experts categorize problems using principles rather than surface features, switch between problem representations effectively, and rely on metacognitive strategies to get "unstuck" and to catch mistakes (e.g., [3]).Many examples of curricula provide explicit expert problem-solving strategies such as the use of multiple representations, a step-wise process (e.g., Isolate the unknown, Substitute, Evaluate) [4], or provide worked examples that follow a standard process (e.g., Identify, Set up, Execute, Evaluate) [5]; however, most PER has restricted attention to problem-solving at the undergraduate level with a limited range of problem solving tasks [1].In this study, we examine problem solving strategies of graduate students who are developing expertise in a variety of research-related tasks.Graduate student problem-solving provides a bridge between the existing body of research in undergraduate physics instruction and professional engagement in problem-solving.
In the cognitive science community, problem-solving is often viewed as interplay between a given problem with the existing knowledge and the individual abilities of the solver.
Yet, research summaries of 21st century skills have emphasized the importance of social interactions and intra-personal skills while problem solving [6].In this paper, we take this wider perspective on problem-solving strategies.Specifically, we explore the following research questions in this paper: 1) What strategies do graduate students report using to solve problems related to their research and 2) In what ways are these strategies used to solve routine problems versus difficult problems they face?

II. METHODS
During the summer of 2015, we interviewed ten PhD students at a private university about their experiences with problem solving as part of their research.We asked students via email if they were interested in participating and paid participants $25 as an incentive for the 30-45 minute interview.The study included participants from their first to sixth year of graduate study.Six were males, and four were females.Students' research specializations spanned a range of physicsintensive projects in astrophysics, astronomy, optics, microfabrication, and materials science, and included a mix of theoretical and experimental methods.
In semi-structured interviews, students were asked to describe their overall research project, then discuss any routine, difficult, and important problems they faced.Students also provided examples of how they solved those problems, which we refer to as strategies.
Throughout the interview transcripts, we identified approximately 370 examples of problem-solving strategies, which we defined as anything the students did themselves in re-sponse to the problems they faced in their research.We also double-coded these as relating to routine, difficult, or important problems depending on which kind of problem they were asked about.Two coders were used across all interviews to ensure inter-rater reliability.In a second iteration of coding, we labeled examples using an emergent coding strategy and collapsed codes into broader categories.For the second pass, two coders jointly negotiated all categories, codes, and definitions before performing frequency counts.

III. FINDINGS
Each problem solving strategy code was categorized into either a general strategy or a problem-driven strategy based on the way it was used by participants as shown in Table I.For example, taking initiative was described by graduate students as something necessary to tackle multiple problems, while applying test cases was mentioned as a way to solve a specific problem such as learning new code.The research protocol asked students to specifically describe problems and strategies related to routine, difficult, and important problems they faced; however, this paper focuses on routine and difficult problems in the interest of space.As shown in Fig. 1, the most commonly reported problem-driven strategies for solving routine problems were applying test cases (5 students) and interpreting findings (5 students).For difficult problems, most students relied on breaking down the problem (5 students).For general strategies, Fig. 2 shows that when faced with routine problems students most commonly search resources (7 students), get help from peers (5 students), and plan ahead (5 students).For difficult problems, students equally used the general strategy get help from peers (5 students), while also applying a variety of other strategies.
To explore these problem-solving strategies in context, we distilled our analysis into findings focusing on three emergent themes: A) students rely on multiple cognitive approaches, B) inter and intra-personal characteristics are helpful for approaching problems, and C) moving beyond routine problem solving requires strategies that connect to a social context.We support these findings with related quantitative data and examples from the perspective of graduate level researchers.

A. Using multiple cognitive strategies
Most problem-solving approaches for introductory physics begin with planning.Due to the constrained nature of many undergraduate physics problems, this often includes pre-existing steps, such as identifying key principles, identifying known and unknown quantities, or drawing a sketch [7].Within PhD-level research, planning ahead was also frequently identified and included strategies like considering expected outcomes in order to determine how to approach a problem.For difficult problems, it was also important for students to be flexible in the approaches they tried and to pursue alternatives as needed.This was most important for solving problems with multiple possible methods or solutions.Additionally, graduate students applied problem-driven strategies such as breaking the problem down into smaller steps.This allowed students to simplify the problem or at least divide large, daunting problems into more manageable subproblems.
One student describes the strategy of taking a complex problem and dividing it into smaller problems, each of which are solvable in a different way."Whenever I have a problem, now my technique is I just want to break the problem into small steps, and then go step by step...For example, I am working on a partial differential equation.First of all, I look up what kind of equation it is...I try that tool.Now I have another equation, so what I can do?It's an eigenvalue problem.Now I can use another tool...So knowing that technique -breaking the problem in steps -that's really helpful."In this case, breaking the problem down was a necessary strategy since each smaller problem required different tools and approaches.This reinforces the need to emphasize planning and sub-division of complex problems as part of the undergraduate curriculum.
Evaluating the quality of results was also identified as an important strategy (see Fig. 1), since it is the step whereby the solver deems a solution is "good enough."In this example, the student compared graduate research to an undergraduate lab course."You need to do this titration... and make sure that you're confident in your results.Whereas in your lab report you can be like, 'Human error.Maybe something happened.Maybe it wasn't the right amount.'But in [the research] lab, if you tell your boss that maybe it wasn't the right amount he's going to be like-" In graduate research, students need to be be able to defend the quality of their results whereas in undergraduate labs, they just need to report that something happened or likely happened to decrease the quality.In undergraduate-level problem-solving, faculty and graders typically evaluate students' work more than the students themselves [7].
Additionally, the use of test cases was useful for routine problems involving troubleshooting program code or experiments.One student said, "There's no answers in the back of the book anymore.So it's you oftentimes need to go make up a number of dummy test cases and make sure my algorithms or my logic makes sense.So, try to find the simplest minimal working example of, I know if I give it this input, here's the output I should get."This graduate student relied on test cases to solve problems since they have known solutions that can be built on to slowly introduce new ideas that do not necessarily have known solutions.Such cognitive strategies, and others discussed in our findings, emphasize that scientific research is open-ended with multiple solution pathways.

B. Relying on inter and intra-personal characteristics
Since graduate level research is rarely done in isolation, inter-personal skills are critical.In fact, some problems had more to do with working with others than with technical issues.In the following example, a student reflected on learning to ask questions as a strategy for solving problems."So that's one thing, it's learning how to get other students to pass on the knowledge to you... demanding, 'I want to learn this thing and you know this thing.Please teach me this thing.'And learning to ask preemptively for help before they leave and everyone's stuck."In this example, it was important for the student to reach out to others and ask for instruction in order to solve problems.Asking questions is a valuable in- terpersonal skill referring to knowing when and how to ask questions to obtain necessary information [8].
While students solved some problems by asking others for help, they also recognized the need to take initiative and learn independently.As one student describes, "Like lots of times, if I had a question about why something isn't working, and there was another grad student that I knew would be able to answer the question, but they weren't available, I would kind of still poke around with it, even if it took me a few hours.Versus I know other people who would just like go home for the day and wait for that person to come in the next day."This student noted that persistence was not a characteristic of all graduate students, but was useful for moving research forward.As another student pointed out, "I guess grad school is more independent.It's really your research, your work, and you have to motivate yourself to move it forward."Motivation and initiative are driven by a sense of research ownership.
Intra-personal strategies were also necessary for solving problems lacking known solutions.As one student attests, "If you encounter a problem...and you don't know how to solve it immediately that shouldn't scare you.You should just calm down and take the appropriate steps to learn how to solve the problem.I feel like the textbook doesn't teach you that, the textbook says, 'Read this problem you have 30 seconds to answer it, or you're going to run out of time to do the rest of the test.'You don't have time to actually think and problem solve, in that sense." C. Moving beyond routine problem solving Fig. 2 shows that students often start with an online search (Search Resources) for solving routine problems.Online searching via Google worked well for problems that needed one quick solution.Though as problems became complex, even searching became tied to a specific science field.One student needed a place to start solving a problem and narrow down various options, "The first thing I did or would do is look online to see if anyone else has solved it already.I would look in the literature and try to see if there's any relevant papers that have come out in the last 30 years or so.Then if there are, then I'd take that down to maybe the last 15, the last 5, and see what the most recent thing might be, and try to use what the science has gone forward in time and see if I can get something that's the most recent." More difficult problems pushed problem solving strategies into a social research context.As one student reflects about graduate work in general, this social environment is illustrated, "In my research group there are, like now, around eight to ten people.And we are working on different group projects...We have a weekly meeting and we discuss those problems.If somebody comes up with a solution, that actually helps the group.Going to peer is a certain thing that we do."For more difficult problems, graduate students often sought help from peers or others in close proximity.Even if these sources did not have exact solutions, they were often able to share ideas that moved research forward.
The social nature of research also posed problems for students; for instance, positioning research findings in the field and ensuring potential impact.Students needed strategies for assessing the quality of their findings and positioning findings in a way that others in the field would find valuable.One student explained that taking logic courses in philosophy helped develop strategies for assessing quality and justifying findings to others."It [taking logic courses] really just gave a way to make definite conclusions and to make conclusions in a way where people couldn't deny them.Especially that's important when you're talking to other people about your research, and when you're writing papers, when you're trying to make sure that everything that you do is entirely defensible."This example emphasizes the need to develop skills for communicating with science communities.

IV. CONCLUSIONS
The identification of an expanded set of problem-solving strategies highlights the need for more sophisticated frameworks of problem solving and additional PER focused on expert-novice differences among scientists engaged in professional-level research.These findings also emphasize the need for incorporating problems that require sophisticated and varied strategies to solve (e.g., planning and evaluation) into the undergraduate curriculum.
The use of inter-personal strategies supports emphasizing team problem solving and group work.Collaborative problem-solving is not just useful for building communication skills, but making better problem solvers.Intra-personal skills (e.g., motivation, persistence, confidence) sustain the problem-solver in the face of adversity.The use of prolonged investigations requiring students to re-address the same problem with different strategies or steps may provide opportunities to develop skills like persistence.Yet, while measures of intra-personal ability exist (e.g., Duckworth's GRIT scale), how to teach these abilities is a largely open question.
Furthermore, graduate students' use of online searching for solving routine problems is remarkably similar to undergraduates' use of online resources for cheating when solving homework problems.What is an authentic problem-solving strategy for researchers becomes an issue of academic dishonesty for students.By providing students with problems that have multiple plausible solution paths or social implications, we may be able to reduce reliance on cheating and provide better preparations for real-world problem-solving.This work is supported by NSF DGE-1432578.

FIG. 1 .FIG. 2 .
FIG.1.Problem-driven strategies used to solve routine and difficult problems in PhD-level research.

TABLE I .
Problem solving strategy codebook.Italicized headings refer to grouped categories of several codes.