We studied a group of middle school teachers as they modified curriculum and developed common formative assessments on force and motion concepts. While discussing assessment goals for student understanding of acceleration, two of the teachers held opposing models about the implications of the sign of acceleration on the direction of an object's motion and whether it was speeding up or slowing down. Failing to resolve the inconsistency between their individual models, the teachers wrote an assessment item for which both models would provide the same correct response, albeit for different reasons. The potential for correct answers for incorrect reasons created ambiguity in the classroom use of the assessment as a formative measure of student understanding. More specifically, the item had limited ability both to refine the teachers' thinking and accurately inform their instruction, interventions, and feedback that would support students in identifying their mistakes.