Student reasoning remains an important topic in physics education. To begin studying reasoning in a simple context we pilot tested short, dichotomous classification questions. These questions present students with a physical situation and ask for classification as possible or impossible, followed by a logical justification and several Likert scale self-assessments. The questions require reasoning, but can be figured out with one key idea, and few logical steps. We developed 16 questions and tested them in clinical interviews with 22 undergraduates who had studied algebra-based or calculus-based physics through electricity and magnetism. Using qualitative and quantitative analysis we find evidence that these questions can help identify differences in students' verbalized explanations that evidence their reasoning. Using these differences we posit testable hypotheses about how subtle differences in the questions can impact students' reasoning. This pilot-testing suggests this type of question may be a useful tool for studying reasoning in physics education.