Over the course of four years, we have researched and transformed a key course in the career of an undergraduate physics major--junior-level electricity and magnetism. With the aim of educating our majors based on a more complete understanding of the cognitive and conceptual challenges of upper-division courses, we used principles of active engagement and learning theory to develop course materials and conceptual assessments. Our research results from student and faculty interviews and observations also informed our approach. We present several measures of the outcomes of this work at the University of Colorado at Boulder and external institutions. Students in the transformed courses achieved higher learning gains compared to those in the traditionally taught courses, particularly in the areas of conceptual understanding and ability to articulate their reasoning about a problem. The course transformations appear to close a gender gap, improving female students' scores on conceptual and traditional assessments so that they are more similar to those of male students. Students enthusiastically support the transformations, and indicate that several course elements provide useful scaffolding in conceptual understanding, as well as physicists' "habits of mind" such as problem-solving approaches and work habits. Despite these positive outcomes, student conceptual learning gains do not fully meet faculty expectations, suggesting that it is valuable to further investigate how the content and skills indicative of "thinking like a physicist" can be most usefully taught at the upper division.