This paper reports on two experiments in which high performing university students having finished an introductory physics course were asked to pose mechanics questions. In Experiment 1, subjects were given problem situations (i.e. a story line accompanied with a diagram from which problems could be constructed) and asked to generate "textbook-like" problems that could be solved with specific concepts (e.g. conservation of mechanical energy, Newton's Second Law). In Experiment 2, subjects were given Concept Scenarios (i.e. a description of the physics principles and concepts that apply to a problem and the order in which they apply) and asked to generate problems that matched the scenarios. Interviews conducted immediately following the experiment asked the subjects to explain how the problems posed matched either the specified concepts, or the Concept Scenarios. Findings indicate that, when followed by an interview, problem solving is a powerful assessment tool for probing students' understanding of physics concepts, as well as their ability to transfer their knowledge to novel contexts. In many instances, students posed appropriate solvable problems, yet displayed major flaws in conceptual understanding.  This suggests that even good novices are lacking in the way their conceptual knowledge is organized in memory and linked to problem contexts and procedures. Suggestions for using problem posing as a pedagogical tool are presented.