Detecting Progression of Scientific Reasoning among University Science and Engineering Students

Scientific reasoning is not a de-contextualized construct; instead, its development and deployment intimately relates to content learning. Drawing on this framework and driven by the instructional goals of many college-level courses that are aimed at both knowledge acquisition and skill development, this study investigates the extent to which undergraduate students’ scientific reasoning skills vary in relation to two aspects of content learning: quantity and domain. Students from different year levels (years 1-4) and two majors (science and engineering) were recruited to complete the Lawson Classroom Test of Scientific Reasoning (LCTSR). These students represented respectively individuals exposed to different quantities and domains of content learning. Results show that, regardless of their major, student reasoning skills measured by the LCTSR remained largely constant across the four years of higher education, with no significant difference between science and engineering. These results call attention to the status quo of undergraduate education and have implications for future improvement.


INTRODUCTION
Besides the goal of promoting student learning of content knowledge, many college-level science and engineering courses are also aimed at advancing student scientific reasoning skills-skills that instructors hope to increase as students learn more course-related topics.A rationale for establishing this goal derives from the argument that students need both domain-specific knowledge and general skills to be competent in their future workplace.[1] This view has been widely accepted and is of little dispute among researchers, educators, administrators and policymakers.However, what has been unclear or perhaps even controversial is whether scientific reasoning can be spontaneously developed through learning content knowledge, and if so, to what extent.
Reasoning skills, according to Lawson [2], are a set of mental strategies, plans or rules used to process information and devise inferences that are beyond direct experiences.In science, such reasoning is often more specifically referred to as generating and evaluating causal mechanisms on the basis of empirical evidence to explain observed phenomena.[2] From this perspective, successful scientific reasoning requires several key skills, such as control of variables, hypothetical-deductive reasoning, proportional reasoning, correlational reasoning, and probabilistic reasoning.[2] Among them, hypothetical-deductive reasoning is considered by Lawson as a quintessential skill that has played a central role in many important scientific discoveries.
Lawson's view of scientific reasoning is by no means comprehensive.As Kuhn [3] pointed out, any conscious and social activity of knowledge seeking can have bearings on reasoning needed for scientific practices.In her view, the ability to coordinate evidence and theory-two cognitively different entities-is key to scientific reasoning.In other words, Kuhn extended the meaning of scientific reasoning from pure cognition to epistemology that involves learners' conceptions about the functions of and the differences between scientific evidence and theories.
That said, what was proposed by Lawson in terms of evidence-based hypothesis generation and evaluation does form a nucleus for the construct of scientific reasoning.Although relatively narrow in the broader context of scientific practices, skills such as formulating and testing causal explanations and control of variables are indeed almost always required for any scientific endeavor.In the Next Generation Science Standards[1], reasoning skills in constructing explanations and designing solutions are unequivocally deemed as an important component of practices that students are expected to engage in through learning sciences and engineering.
Regardless of the fact that there is no universally shared definition on this topic among scholars, it has been broadly recognized that scientific reasoning is not a de-contextualized construct.Instead, according to the Neo-Piagetian theories, [4] the development and deployment of reasoning skills are intimately related to the content domain, educational experience and learning environment in which a learner is situated.
Hence, an important inference from this theoretical basis is that acquisition of content knowledge and development of scientific reasoning should be achieved concurrently through mutually corroborative ties between them.[5] One way to conceptualize their concurrent development is the amount of content learning in relation to scientific reasoning skills.Since successful performance on hypothesis generation and evaluation requires an understanding of the phenomenon of interest, it is conceivable that an increasing body of content knowledge is beneficial for scientific reasoning.A series of empirical studies conducted by Kuhn [3] and Schauble [6] supported this conceptualization.However, this was somewhat confounded by Moore and Rubbo's [7] study, in which they found university students' reasoning measured by the Lawson's Classroom Test of Scientific Reasoning (LCTSR) [8] did not increase after learning a semester of physics.
Another conceptualization of their connections is the influence of learning in different content domains on scientific reasoning.In general, approaches to reasoning can be depicted by two basic models: the science model and the engineering model.[9] The former seeks to understand causal relationships of a phenomenon, whereas the latter is aimed to replicate it as a product.In other words, the science approach emphasizes the "means" whereas the engineering approach focuses on "ends".According to Schauble, [9] learners exposed to science-oriented tasks would be more likely to use valid reasoning strategies to seek causal explanations than those exposed to engineeringoriented tasks.Nevertheless, it is worth noting that science and engineering are not two dichotomous fields.Professionals in engineering often strive to explicate causal mechanisms of a phenomenon, just as much as scientists, in order to best optimize and reproduce the "end" product.

RESEARCH QUESTIONS
Anchored in the above framework of contextualized reasoning, this study investigates two aspects of content learning-quantity and domain thereof-in relation to scientific reasoning.In this study, Chinese university students from different year levels and majors were recruited to complete the LCTSR, representing individuals exposed to various amounts and domains of content learning (see Methods for details).Specifically, we seek to answer the following two questions: (1) To what extent do students' scientific reasoning skills measured by the LCTSR differ across the four year levels of higher education?(2) To what extent do students' scientific reasoning skills differ between science and engineering majors?
Answers to these questions not only help us better understand the progression trend and variations of scientific reasoning in higher education, but also motivate us to seek more effective instructional strategies to promote reasoning skills among students.

METHODS
Our subjects of interest are science and engineering students across the four years of higher education at two Chinese universities.The participants in the study represented nearly all of the target students in these programs (see Table 1).We selected this group of participants primarily because there is a growing interest in Chinese students due to their outstanding performance in recent international assessments.In this study, the LCTSR was used to measure the participants' scientific reasoning skills.As mentioned before, evidence-based hypothesis generation and evaluation, albeit narrow in view of broader scientific practices, taps into the critical skills needed for successful scientific reasoning.The LCTSR was designed to gauge these skills, including control of variables and hypothetical-deductive, proportional, correlational and probabilistic reasoning.The Chinese LCTSR, which was translated by following the same procedures as reported in Ref [10], was administered to 614 students as an in-class quiz in the mid-term week of the 2011 fall semester.Students were given a maximum of 45 minutes to complete the test and were encouraged to provide their honest responses.No incentives or penalties were granted.Upon student consent, responses were collected for analysis.

DATA ANALYSIS AND RESULTS
Since there were eight groups of students from 4 year-levels and 2 majors, we computed the average score of each group for comparisons.Figure 1 shows the results of student overall performance on the LCTSR for all the eight groups.As seen, differing from the concurrent development postulation mentioned before, student scientific reasoning skills measured by the LCTSR remained fairly stable across the 4 years of higher education, a recurrent pattern for both science and engineering majors.Between the two majors, science students consistently outperformed their peers in engineering, regardless of year level.
A two-way Analysis of Variance (ANOVA) further showed that there was no significant interaction between year and major [F(3, 606) = 1.545, p = 0.202].This allowed us to make an aggregate comparison for both main effects.As for the main effect of year, there was no significant difference across the four levels [F(3, 606) = 0.613, p = 0.352], confirming the lack of growth in student scientific reasoning skills.As for the two majors, no significant difference was detected either, hence suggesting science and engineering students possessed comparable reasoning skills.
Given the fact that the LCTSR targeted five subskills of science reasoning, we further looked into student performance on each of these sub-skills individually.As before, the averages of the eight groups were calculated and are depicted in Figure 2. Similarly to the overall result in Figure 1, student performance on each of the five sub-skills demonstrated little progression over the four years of higher education.Also, the cross-year progression lines for science and engineering students were largely entangled, showing no clear distinction between the two majors in terms of their scientific reasoning skills measured by the LCTSR.
We further conducted a two-way ANOVA for each of the five sub-skills.A significant interaction (year × major) was found for all the sub-skills (p values < 0.05) except for control of variables and hypotheticaldeductive reasoning.This precluded broad-brush comparisons and forced us to look into each main effect at their individual levels separately.Although some significant statistics were detected, none of their pair-wise comparisons showed an effect size (Cohen's d) greater than 0.06, a magnitude considerably lesser  than what is typically deemed as a small size of 0.3.This is consistent with the results in Figure 2, as the fluctuations and entanglement of the progression lines revealed no regularities or interpretable patterns.As for the sub-skill of control of variables, no significant interaction [F(3, 606) = 0.814, p = 0.486] or major effect [F(1, 606) = 0.015, p = 0.904] was detected, and the effect of year, albeit statistically significant, was also of a small size (less than 0.06

DISCUSSION
From the Neo-Piagetian perspective, [4] scientific reasoning is closely connected to content learning.One way to conceptualize their intimate connection is the mutual corroboration of knowledge acquisition and skill development.In other words, as the sheer volume of knowledge increases with students' progression into higher levels of education, their scientific reasoning skills are also expected to increase.In our study, students at 4 different year levels of higher education represented individuals who had been exposed to various amounts of content learning.Differing from the expectation, students' scientific reasoning skills measured by the LCTSR remained nearly constant across the entire 4-year period, a recurring pattern for both science and engineering students.
This result called our attention to the status quo of higher education in China.While many college-level science and engineering courses are aimed at promoting student reasoning skills, no tangible results have been obtained to directly bolster this educational goal.In comparison to the previous studies by Moore and Rubbo [7] where students showed little improvement in their reasoning skills after studying one semester of introductory physics, our results revealed a more serious and urgent situation; that is, the entire 4 years of higher education produced little impact on student progress in reasoning skills.It is worth noting that the stagnancy of the trend lines in Figures 1 and 2 were not due to a ceiling effect.Instead, there was still room for improvement.
Another significant finding of this study is the similarity between science and engineering students in their reasoning skills.This is not to say that we have prepared our future professionals in both fields to be equally skillful in evidence-based reasoning.In fact, it is the flat trend lines of both majors, not the growing lines we hoped to see, that made them comparable.In this sense, this similarity only further alerts us to the persistent problems in the current higher education.
We recognize that this is a cross-sectional study.However, our results by and large sketch out the general growth patterns of the Chinese university students' reasoning skills.Indeed, a sense of stability in Chinese secondary/tertiary level curriculum and instruction is indisputable given its long established national standards.Besides, our participating universities have consistently maintained their national rankings and hence have likely attracted and admitted students with similar academic backgrounds over the years.Given these considerations, it is reasonable to take our results as an approximation of progression patterns of students' reasoning skills in Chinese higher education.
While this study is only a first step toward a more systematic investigation of undergraduate education, the results revealed alarming situations that require our concerted actions.Given the need for transforming introductory science and engineering courses, it is crucial that our efforts be aimed at promoting both student knowledge acquisition and skill development.Undoubtedly, this will need methodical improvement in multiple areas, including curriculum, instruction, and assessment.

FIGURE 2 .
FIGURE 2. Student performance on five sub-skills of scientific reasoning measured by the LCTSR.Error bars are standard errors.

FIGURE 1 .
FIGURE 1. Student overall scientific reasoning skills measured by the LCTSR across different years and majors.Error bars represent standard errors.

TABLE 1 .
The numbers of student participants.