Racial or ethnic isolation may negatively impact physics graduate students' academic and professional experience. An interview study with 16 doctoral students who were considered to be particularly at risk of such isolation found that (i) the students interviewed are mostly not the only member of their racial or ethnic group in their programs; (ii) students have diverse perceptions of who they consider to be part of their racial or ethnic group, such that their unique perspectives need to be taken into account by programs that wish to support them; and (iii) students report negative racial experiences in their programs, such as bias, expectations of service, family and financial difficulties, and political threats. Overall, these findings present both challenges and opportunities for supporting Black and Latinx physics graduate students in ways that are meaningful to them.