Many teachers administer a force concept test such as the Force Concept Inventory (FCI) to their students in an effort to evaluate and improve their instructional practices. It has been commonly assumed that looking at class normalized gains allows teachers to compare their courses with other courses. In this paper we present evidence to suggest that the use of class normalized gains alone may not provide a complete picture. We argue that student reasoning ability should also be assessed before between-course comparisons can be made. Assessment of reasoning ability is also useful in identifying students who are at risk. In the following we shall concentrate on the FCI, but we think our conclusions probably apply to physics concept tests generally.