This study explores both students' ability to make valid inferences from data tables and the effect of students' prior beliefs on that ability. Over 300 introductory physics students participated in one of two experimental conditions. In both conditions, the data filling the tables was identical; however, in the first condition, the tables were presented within a familiar physical context, a method well known to elicit incorrect prior beliefs in many students. In the second, the tables were presented in a generic context. We found that, while most students were able to draw valid conclusion from simple, generic data sets, they were significantly more likely to draw invalid conclusions in the familiar physical context. Closer analysis revealed that, when provided with a physical context, students tended to look at the data less, relying in part on their prior knowledge to draw their conclusions. Interestingly, students in the physical context condition indicated a higher confidence in their responses, despite their lower accuracy.