Abstract In this short paper, we reflect on the lessons that can be learned from the deployment of a risk assessment tool to assist call workers in a child maltreatment hotline. In a paper that will appear at CHI 2020, we found that the humans’ decisions did change when the tool was deployed, but humans were less likely to adhere to the machine’s recommendation when the score displayed was an incorrect estimate of risk. Here, we identify a set of characteristics of the deployment setting that may inform the design of future risk assessment tools and research studies. PDF