Beyond Predictive Algorithms in Child Welfare (2403.05573v1)

Abstract: Caseworkers in the child welfare (CW) sector use predictive decision-making algorithms built on risk assessment (RA) data to guide and support CW decisions. Researchers have highlighted that RAs can contain biased signals which flatten CW case complexities and that the algorithms may benefit from incorporating contextually rich case narratives, i.e. - casenotes written by caseworkers. To investigate this hypothesized improvement, we quantitatively deconstructed two commonly used RAs from a United States CW agency. We trained classifier models to compare the predictive validity of RAs with and without casenote narratives and applied computational text analysis on casenotes to highlight topics uncovered in the casenotes. Our study finds that common risk metrics used to assess families and build CWS predictive risk models (PRMs) are unable to predict discharge outcomes for children who are not reunified with their birth parent(s). We also find that although casenotes cannot predict discharge outcomes, they contain contextual case signals. Given the lack of predictive validity of RA scores and casenotes, we propose moving beyond quantitative risk assessments for public sector algorithms and towards using contextual sources of information such as narratives to study public sociotechnical systems.

• The paper demonstrates that traditional risk assessments inadequately predict non-reunification outcomes by yielding high false positive rates.
• The study applies SVM, random forest, and topic modeling to integrate case narratives, thereby increasing predictive specificity.
• The paper advocates for hybrid, narrative-informed models to capture the complex socio-technical dynamics in child welfare decision-making.

Beyond Predictive Algorithms in Child Welfare: A Quantitative Critique

The paper "Beyond Predictive Algorithms in Child Welfare" presents a critical examination of the role of risk assessments in algorithmic decision-making within the child welfare system (CWS), highlighting the limitations of current predictive models. The authors focus on the United States child welfare sector's reliance on predictive decision-making algorithms, which utilize risk assessments (RAs) as inputs for predictive risk models (PRMs). These PRMs are formulated to identify high-risk cases of child maltreatment, theoretically allowing caseworkers to make more objective and efficient decisions. However, the paper argues for a fundamental reevaluation of these models, emphasizing the value of narrative data in understanding the broader context of child welfare cases.

Key Findings and Methodology

The research investigates whether the integration of case narrative data, specifically caseworkers' casenotes, into PRMs can enhance the models' predictive validity. The paper quantitatively deconstructs two prevalent RAs—AAPI and NCFAS—from a United States CWS agency, comparing the predictive efficacy of models using RA data with those incorporating contextual narratives. The authors employ a combination of support vector machines (SVM) and random forest models as classification tools, utilizing topic modeling to extract thematic insights from narrative data.

Significant findings from the paper include:

1. Inadequacy of Risk Assessments: The paper found that RAs used in building PRMs are ineffective in predicting non-reunification ('NR') outcomes for children, with models displaying high false positive rates. For instance, classifiers built with only RA data failed to accurately identify 'NR' cases, often predicting erroneously these as reunification ('R') outcomes.
2. Value of Narrative Data: Models that incorporated casenotes demonstrated better specificity in identifying 'NR' outcomes compared to RA-only models. Although these models still exhibited limitations, the narrative data provided valuable contextual signals missing from RA scores.
3. Contextual Insights from Casenotes: Through computational text analysis, the paper uncovers that case narratives encapsulate complex, multi-faceted interactions not captured in RA data. These narratives reflect the discretionary work and contextual factors affecting case outcomes, emphasizing the need for a shift from purely quantitative models to hybridized approaches that acknowledge qualitative details.

Implications and Future Directions

The paper critiques the foundational constructs of PRMs in the child welfare sector, suggesting that reliance on RA data for predictive purposes is fundamentally flawed. This critique extends beyond child welfare, posing broader questions about the use of predictive algorithms in public sector decision-making. Highlighting the limitations of current algorithmic approaches, the research advocates for the integration of case narratives to capture the rich, contextual fabric of public sector systems.

The findings support a movement towards contextually aware, human-centered approaches in the design of decision-making algorithms. This shift could mitigate biases inherent in RA data, such as those related to poverty and race, and more accurately reflect the lived realities of those involved in the CWS. The use of narrative data may also provide a more dynamic assessment of risk, accounting for the socio-technical complexities involved in family preservation decisions.

Future developments could focus on refining methods for integrating narrative data into PRMs, exploring novel computational techniques that leverage the nuanced insights provided by casenotes while preserving the interpretability and fairness of predictive models. Additionally, further exploration is required in understanding how algorithmic systems can be designed to support, rather than supplant, the critical judgment of human caseworkers in the child welfare domain.

Conclusion

Overall, this paper provides a compelling argument for reevaluating the use of predictive algorithms in the child welfare system. By critically examining existing models and proposing a more holistic integration of narrative data, the researchers contribute to an evolving discourse on the intersection of algorithmic governance and human-centered decision-making in public sector systems.