In this document we describe the results of a series of test-retest studies using the Force and Motion Conceptual Evaluation, a pre/post-test instrument used to measure conceptual understanding and learning gains in introductory physics courses. The subjects took the FMCE once and then again four weeks later, with no formal physics instruction in between. We found the group statistics to be very reliable, indicating that the FMCE is a stable measure of ability level for groups of students. However, the results of individual students reveal high levels of inconsistency. Introductory students tend to answer only 60% of questions the same way at two different testing sessions. After instruction, the subjects still only answered with about 60% consistency, but had higher average scores. We look to conceptual change research for an explanation of why post-instruction students get more questions correct, but maintain the same level of inconsistency.