Creating equitable learning environments has been an area of significant focus for physics education researchers in recent years. Here we introduce a framework that posits that grade penalty is a measure of academic self-concept and investigate if there are gender differences in grade penalties in physics courses for students majoring in physics. In order to quantify grade penalty, we define grade anomaly as the difference between a student's grade in a course under consideration and their grade point average (GPA) in all other classes thus far. A grade anomaly lower than students expected grade based on their GPA is a grade penalty and higher than expected average grade is a grade bonus. Our framework posits that since women have traditionally been marginalized in physics, female physics majors are more likely to be negatively impacted by a grade penalty in their courses since their academic self-concept as a physics major hinges on them securing a certain grade. In the study presented here, we examine the average grade anomalies across a number of courses for female and male physics majors. We find that these students received grade penalties in almost all physics courses studied, though there were grade bonuses in a few laboratory courses. We also find that in physics courses, on average, women often had larger grade penalties than men, especially in introductory courses. We hypothesize that, because their grade penalties are often larger than men's, women's decisions to pursue a physics major and career may be particularly affected by grade penalties received in their various courses. Furthermore, the grade penalty measure can be easily computed by the physics programs concerned with equity.