Despite numerous studies examining the impact of youth participation in informal physics programs, existing research primarily focuses on short-term outcomes, neglecting the long-term effects. To address this gap, we aim to develop a mixed-methods longitudinal study to understand the long-term impact of youth participation in an informal physics program. This paper focuses on a qualitative pilot study with three prior program participants and serves two purposes: (1) to gauge which elements of the informal physics program may be important to investigate further within the longitudinal study and (2) to further understand the complexities and challenges of utilizing interviews with program participants and alumni as the qualitative approach to the longitudinal study. Findings revealed diverse motivations for program participation, varying levels of program recollection, and a spectrum of impacts on participants. Building upon these findings, our planned longitudinal study will employ both quantitative and qualitative methods to gain a nuanced understanding of the long-term effects of youth engagement in informal physics programs.