Using data from over 14,000 student responses, we rank incorrect responses on the Force and Motion Concept Evaluation (FMCE). We develop a hierarchy of responses using item response theory and the McNemar-Bowker chi-square test for asymmetry. We use item response theory (IRT) under the assumption that students who score well have a greater understanding of physics than those who do not; therefore, responses that have a greater likelihood of being selected by those who score well are considered better responses. We use the McNemarBowker chi-square test (MB) under the assumption that student understanding is more likely to increase than decrease after an introductory mechanics course. Therefore, more dominant transitions from one answer to another from pretest to posttest indicate that one answer is better than another. We present the results from the IRT and MB analyses, highlighting both agreement and disagreement between the hierarchies of responses generated by each.