We teach a course for elementary education undergraduates that gives students an opportunity to conduct open-ended scientific inquiry and pursue their own scientific questions in much the same way that practicing research scientists do. In this study, we compared what our students say declaratively about the nature of science (NOS) in surveys and interviews with what they do procedurally when engaged in authentic scientific practice. Initially, we were surprised when our students showed very little change on two different validated NOS questionnaires, adhering to seemingly memorized definitions of key NOS vocabulary such as "science" and "experiment." In contrast, on procedural measures of NOS understanding, students developed a decidedly sophisticated approach to answering scientific questions. Our data suggest that students' declarative understandings about the NOS are not a reliable measure of students' ability to engage productively in scientific practices and vice versa. We discuss why this might be and consider the implications of this disconnect on identifying the best approach to NOS instruction and on future science education research.