Comparative analysis of letters and reports in an upper-division lab

In redesigning the Modern Physics Lab at Strive University, we focused its purpose on developing writing skills. In doing that, we implemented the pedagogical method Letters Home, which offers students the ability to practice communication in the form of letters to experts and non-experts. Students were additionally tasked with writing traditional lab reports. This case study investigates 6 students’ completion of 6 writing assignments (letters and reports) to a real audience. We used the AAPT guidelines to develop a qualitative coding scheme with 8 categories, and we used a linguistic analysis software program called LIWC to evaluate the assignments’ authenticity, clout, tone, and analytical thinking. Our results indicate 6 of the 8 coding categories appear in at least 50% of the data. Also, letters to experts and non-experts indicated similarities in analytical thinking. Authenticity scores were higher for letters to non-experts than experts. Overall, letters and reports are similar in terms of both the AAPT-inspired codes and linguistic dimensions probed by LIWC. The similarities between the letters and lab reports from our study may be due to our curriculum redesign.


I. INTRODUCTION
Physics is an experimental science, and the instructional laboratory experience is one way of learning how to apply empirical reasoning to the physical world [1]. An essential component to any physics curriculum is the laboratory; however, in comparison to lecture courses, pedagogical research on laboratories is sparse. Even more sparse is content about science communication. National reports and academic research consider the ability to communicate physics ideas an essential skill for future physicists. For instance, in 2014, the AAPT Committee on Laboratories published detailed recommendations for undergraduate physics laboratory curricula in which they specify communicating physics as an area of focus. While research on laboratories is sparse, there are several studies that offer insight to the PER community about communication.
For instance, Russell [2] discusses how science disciplines have generalized writing and out-sourced its responsibility to other departments. Carter [3] reinforces Russell's observations by indicating that discipline-based faculty compartmentalized writing and partitioned it from the curriculum. In another study, Stanley and Lewandowski [4] interviewed physics graduate students and studied their lab notebooks to understand how researchers learn scientific documentation skills. They revealed that participants did not attribute acquiring their skills to their courses or to advisory guidance, but instead to informal, experiential environments like research settings.
Most recently, Hoehn and Lewandowski [5] have developed a framework for thinking and understanding the significance of writing within the laboratory.
Lane [6] implemented an activity that incorporated the informal nature of letters to a formal discipline to teach students about communicating physics. Lane calls this activity Letters Home.
To contribute to the field, we have redesigned a course with the goal of educating students about communicating physics through gathering data, analyzing results, and evaluating and presenting their findings in a manner that is understandable. To create a more constructive learning environment, we transformed the upper-division Modern Physics lab at Strive University and adopted learning objectives from AAPT's recommendations. We then used these objectives to align our course goals and curate a curriculum focused on communicating physics. To achieve our objectives, we implemented the activity called Letters Home, which gave students the opportunity to practice communication first in the form of letters to non-experts then to experts. This activity engaged students in the process of collecting and synthesizing physics knowledge, while also exposing them to audience members that are experts and non-experts. Lane [6] writes that traditional lab reports create pedagogical shortcomings, however his observations of Letters Home shows students engaged in the communication process much differently than when using lab reports. This paper will discuss the results from a comparative analysis of writing Letters Home and lab reports to communicate physics in an upper-division lab. In our study, we operationalized the AAPT guidelines to create a framework for evaluating scientific writing in lab courses. To determine how well the learning goals were fulfilled and the influence letters home had on learning, we asked the following research question: How are the content and tone of the letters and lab reports similar or different?

II. COURSE DESIGN
The first phase to designing laboratory courses is identifying learning objectives [7]. In our lab transformation we decided to implement guidance from the report AAPT Recommendations for the Undergraduate Physics Laboratory Curriculum. The report indicated that students will be exposed to all focus areas over their undergraduate experience rather than in a single course [1], and we prioritized 3 of 6 focus areas. We selected focuses that would carry into other levels of physics labs as well as complement each other within this learning environment. Since the Modern Physics course is Strive University's first upper-division lab that majors are required to take, we chose the following AAPT focus areas: constructing knowledge, analyzing and visualizing data, and communicating physics.
Constructing knowledge is considered an overarching goal that spans most undergraduate physics lab curricula. Constructing knowledge entails collecting, analyzing, and interpreting data from observations of the physical world to develop models. The communicative aspects of constructing knowledge include describing experimental observations clearly and accurately while identifying what is important in an experiment. This aspect is also indicated when students develop arguments and identify trends.
Analyzing and visualizing data was selected as another goal because we were interested in teaching student introductory statistical analysis. The communicative aspects of this area include analyzing and displaying data using statistical methods while also interpreting the limitations and uncertainty of the data.
Communicating physics was the cornerstone of most of our lab's transformation. This focus area highlights the importance of representing results and creating ideas for an audience to interpret and evaluate. To prepare students for the rigors of scientific communication, we thought this goal should be addressed as early and often as possible.
In transforming the Modern Physics course, we intentionally scaffolded the schedule, lessons and activities in such a way that students would receive instructional guidance from the beginning and gradually become selfsufficient to not only conduct experiments but communicate their findings to a multi-faceted audience. In the first 6 weeks, students attended three student-instructor discussions every other week for up to 1.5 hours in which the instructor offered guidance about laboratory equipment (i.e. oscilloscopes, multimeters, electronics, etc.), scheduling, computational and statistical analysis, scientific writing, and acclimating to the new learning environment. In the previous course, students were required to complete lab reports; however, for the redesigned course a novel writing activity called Letters Home was implemented with lab reports. A detailed description of our full lab transformation can be found in Ramey [8].
We sought to incorporate an activity that would offer students more opportunities to practice writing. Lane [6] states the Letters Home activity is a more interactive assignment than lab reports because it requires students to articulate the lab events using colloquial vernacular, thereby reinforcing the course's focus areas. We implemented the Letters Home activity as an out-of-class assignment. We tasked students with writing an informal email to a nonexpert (parent, partner, or friend) about that day's experiment. As the semester progresses, students move on to writing an informal email to an expert. This form of scaffolding encourages students to write about science (which most are not accustomed to doing) in a format and to an audience that they may find more comfort. It also primes students to evaluate their results with a consideration to their audiences' knowledge. In the former Modern Physics course, students were required to write lab reports for the instructor. Now, students are writing letters to a person they know. According to Chi [9] the letters home activity is not only constructive but interactive because students are creating a discourse about their experience and findings with an actual audience

III. METHODS
To address our research question, we conducted a mixedmethods comparative analysis on the letters and lab reports from the redesigned course. We used NVivo to conduct an exploratory qualitative analysis for which we created a coding scheme using AAPT's Recommendations as a theoretical framework. We also used Linguistic Inquiry and Word Count (LIWC) to conduct a quasi-qualitative linguistic analysis to characterize the language used in the artifacts. By investigating the characteristics and tone of our artifacts, we are capable of demonstrating students' communication skills within formal and non-formal writing activities.
Strive University (SU) is a large four-year, public doctoral university with the highest level of research activity. According to the American Institute of Physics statistics, the undergraduate physics program at SU is larger than 89% of the U.S.'s physics programs [10]. On average, SU's physics and astronomy department awards approximately 14 bachelor's degrees a year. SU is also recognized as a Hispanic-serving institution, in which the Department indicates approximately 49% of enrolled students are People of Color, and 46% of the students that took the course was People of Color. When this data was collected, the national average of women in undergraduate physics programs was approximately 17%, while the Department indicated 7% of its majors were women and 12% of the students enrolled in this course were women.
In SU's physics department, the Modern Physics course is the first upper-division laboratory in which all majors are required to take (non-majors are allowed to enroll). The semester that we collected the data there were 33 students enrolled. There were 4 letters and 2 lab reports that students were required to complete for a grade. The letters were administered first (where the Letters 1,2, and 3 were written to a non-expert and Letter 4 was written to an expert) followed by two lab reports (which were written to only experts). Therefore, students wrote three assignments to non-experts and three assignments to experts. We decided to do a case study containing 6 students that completed all the assignments. Students were randomly selected from three groups partitioned according to their end-of-semester grade. Group A were students that received an "A" (100-90) in the course, Group B were students that received a "B" (89-80) in the course, and Group C were students that received a "C" or lower (below 80). From each group 2 participants were randomly selected. Therefore, our case study consists of 6 students and 36 artifacts (24 letters and 12 lab reports).
To create a coding scheme that would identify when the focus areas arose in students writing, we blended communicative aspects of constructing knowledge and analyzing and visualizing data with characteristics of communicating physics (as defined by AAPT) and found 5 themes as categories for communication. These categories became our a priori coding scheme that we labeled as such: 1) state a question, 2) describe methods, 3) present evidence, 4) evaluate evidence, and 5) draw evidence-based conclusion(s).
We used NVivo to store and analyze the artifacts. We began with two raters analyzing a set of letters and reports using the a priori coding scheme. The results of our first iteration of coding indicated that categories and definitions needed to be revised to improve inter-rater reliability. After several iterations of coding and numerous team discussions, we came up with a scheme that mapped well onto the data and yielded high inter-rater reliability. About two-thirds of the data mapped on to the following code categories: Question, History, Outline, Experimental Methods, Theoretical Methods, Evidence, Conclusion, and Evaluation & Explanation. Table I gives a definition and an example of the codes used in our analysis.
Cohen's kappa statistic is a common metric for measuring inter-rater reliability when coding qualitative data. A kappa value greater than 0.80 indicates near perfect agreement. Using the final version of our coding scheme, in Table I, we achieved Cohen's kappa of 0.82 when two raters independently coded a subset of letters and reports. From here, a single rater was able to continue analyzing the artifacts. LIWC provides a method of measuring the emotional, cognitive, and structural components of verbal and written forms of communication. With over 6,400 words and approximately 90 variables, LIWC measures an artifact on a scale from 0.0 -100. We used LIWC to characterize the tone and language of letters and reports along four dimension: analytical thinking, clout, authenticity, and emotional tone. Analytical Thinking refers to the degree of formality, logic and hierarchal thinking. Clout is associated with confidence and expertise. Authenticity refers to the degree of honesty, personal investment and disclosure. Emotional Tone is correlated with the style of the artifact (e.g., upbeat, neutral, or hostile). LIWC also offers a sample criteria to assist in describing the artifacts. For instance, artifacts that could be found in NYTimes (published articles) contain language that is highly analytical (92.57) with high clout (68.17); however , they have low authenticity (24.84) and a neutral tone (43.61). And artifacts that would be found in Expressive Writing (emotional writing done by a range of individuals) show high authenticity (78.01) in comparison to analytical thinking (44.88), clout (37.02) and tone (38.60). For more descriptions about LIWC and its sample data please read Pennebaker et. al. [11]. Table II, we infer that the content of letters and reports are similar. Table II shows almost all of the letters and reports include Evidence, Experimental Methods, Theoretical Methods, Conclusion, and Question. In fact, the majority of the letters and lab reports indicate the possibility of the appearance of any one of these categories. There is also an upward trend in the dimensions of Conclusion and Question as activities transition from Letters 1,2,3 (78%) to Reports (92%).

Based on the results in
We also looked at how the codes were assigned within the dataset. We found that more than 50% of the artifacts had six or more dimensions, and all the documents had at least four dimensions. The data also showed only 6% of the artifacts identified all 8 of the argumentation codes. These artifacts were all within the group Letters 1,2,3, which were written to non-experts. While Letters 1,2,3 were the only artifacts to have all 8 categories, 42% of the Lab Reports had at least 7 dimensions. This suggests that, as the semester progressed, students were incorporating more aspects of communication in their writing. This information offered reassurance that our codes were specific and diverse enough to address dimensions outlined by AAPT. Through the course of this paper I will cover the background… purpose, execution and mathematics involved…. Experimental Method Student describes the setup or data taking procedure using words, pictures, or diagrams.
Moreover, there are two Helmholtz coils which are fixed at a distance equal to the radius of each coil…. Theoretical Methods Student describes the models, concepts or equations pertaining to the experiment. This includes derivations.
The way we did this is that we used a right-hand rule that relates the directions of current and magnetic field associated with the current. Evidence Student presents and analyzes data. Data are presented in tables, diagrams, or plots. Analysis includes averages, uncertainties, error propagation, slopes, goodness of fit metrics, data processing or calibration.
The LIWC yielded similar results in students' language within the letters and reports. For instance, Analytical Thinking was very high for all forms or writing (greater than 95%), indicating a high level of formality, logic and hierarchy. Although this was expected of reports, it was surprising for letters because we assumed letters would solicit a more informal writing approach. Also, Clout scores were above neutral indicating artifacts showed levels of confidence.
Clout scores were between types of communication that is formal with a specific topic with a general audience in mind (i.e. NYTimes) and an informal, specified topic engaging a general audience (i.e. Twitter). A more in-depth investigation of Clout revealed those students that received an A in the course scored lower on this variable than those that earned a B or lower.
Our analysis of Authenticity revealed our data had scores lower than that of the sample data presented by Pennebaker et al. While Authenticity measures the degree of honesty, personal investment and disclosure for a text, the score for Letters 1,2,3 is almost double that of Letter 4 and Lab Reports. In fact, students that earned a B or below in the course Authenticity score was double that of students that earned an A. And the Emotional Tone variable indicates that Letter 4 were upbeat and happier than that of Letters 1,2,3 and Lab Reports. This difference could be because Letters 1,2,3 were assigned in the beginning of the semester where students produced more inconclusive results due to lack of knowledge and experience; by Letter 4 all students were able to complete and produce conclusive results to share with an expert. Also, the "accepted" tone taught to students about lab reports is usually neutral and concise; this assumption could encourage students to communicate in a way that reduce bias by stating "just the facts."

V. DISCUSSION AND LIMITATIONS
This study compares letters home and lab reports to see what similarities and differences they have. From our research question we were able to find that the content and tone of letters and lab reports were more similar than they were different. Although the category of Evaluation and Explanation was less frequently coded in the artifacts, it is a higher-order inference skill requiring students to make judgment calls about the quality of the experiment (i.e. errors and uncertainty) and/or identify its limitations. And, although we were able to validate some form of authenticity difference between letters and reports, the tone of language used within the activities was similar. Lane [6] claims letters home "introduces a writing style that is more authentic for the students than the traditional lab report" in which our study found that Letters 1,2,3 did have an Authenticity score that was double that of artifacts written to experts. It seems that audience influences students' personal investment and disclosure. And although students' letters were more authentic than letters, they were not less formal. This could be for a number of reasons, for instance because these are upper-division experimentation our participants may be attempting to still model prose from textbooks or articles. They are still novice which means they have a limited knowledge of the field, therefore modeling experts is a means of learning expert-like skills, even if they may not be necessary.
The results from Table II indicate students are engaging in focus areas mapped to physics communication that were established for the course. Although we did not attempt to change the lab reports, our results show that there is something unique about these lab reports than the ones described by Carter [3] and Lane [6]. The scaffolding approach we took to introduce the lab reports may have acclimated students to a sense of audience that then informed their final assignments. Chi's [9] theoretical framework for activities would define letters home as "interactively" constructive because students are summarizing an experiment to an actual audience for feedback. Based on the similarities in content and language, Chi's equivalent outcome postulate suggest letters and lab reports are similar writing activities.
Although we find similarities between activities that are intended to be different this would also suggest the limitations of our coding scheme. Another limitation comes from the fact we coded only for the presence of various communicative aspects within each document, not their "correctness." Therefore, we are limited to only observing if the code appears and not the quality of data. In future works, we should focus on implementation and audience feedback.

ACKNOWLEDGMENTS
Heather Lewandowski and the PER group at CU -Boulder for the scholastic, financial and moral support in helping this go from a thought to a reality.