In this work, we introduce the Materials Science Conceptual Evaluation (MSCE). This is a 31 item multiple choice test designed to complement the Materials Concept Inventory (MCI) developed in 2002 by Krause et al. In developing this assessment, we collected data from over 700 students, including over 100 interviews, focused on areas where students struggle or have misunderstandings. There are several main goals of this article. The first is to inform the engineering education community of the existence of this evaluation and its psychometric properties, which support its validity and reliability as an assessment. The second goal is to describe our design and validation process and what we discovered to be the most, and least, important general steps for future engineering education assessment design and validation. Lastly, we will communicate some of the key findings about student difficulties with basic concepts and the required reasoning and graphical skills for an introductory materials science course.