Abstract The deployment of facial recognition technologies in high-stakes scenarios has sparked widespread concerns about privacy, reliability, and fairness. A common response to these concerns is the suggestion of adding a human in the loop to provide oversight and ensure fairness and accountability. However, the effectiveness of this approach is not often studied empirically, and the literature shows that humans have biases of their own. In this position paper, we argue for the necessity of empirical studies on human-in-the-loop facial recognition systems. We outline several technical and ethical challenges that arise in conducting such controlled studies and interpreting their results conclusively. Our goal is to initiate a discussion about the best path forward for AI and HCI researchers to work together towards empirical and human-centered approaches to the design and evaluation of human-in-the-loop facial recognition systems. PDF