Physics departments are increasingly working to improve diversity in graduate programs by using more holistic strategies in their admission and retention practices. However, completion rates for traditionally underserved groups are still problematic. By understanding and accounting for faculty and student divergence on challenges faced in retention, graduate programs will be better equipped to enact change. In this study of two graduate programs (one physics and the other astrophysics), faculty and graduate students were asked why graduate students leave their program, and to identify ways to reduce attrition. While the goal of improving retention was shared between faculty and graduate students, their visions for how retention could be improved were not aligned. Faculty believed that retention could be improved by reforming admissions practices to select for students with attributes critical for success in graduate school, such as resilience. In contrast, the graduate students noted more systemic or socio-cultural factors as impacting retention.