Facilitating Human Agency and Oversight in Algorithmic Processes

About the Project

This project began from a Ph.D. proposal submitted by Mr. Kyriakos Kyriakou, a Research Associate at fAIre MRG in CYENS Centre of Excellence as a part of his application for the CYENS Doctoral Training Program (CYENS DTP). Mr. Kyriakou successfully acquired a place in the CYENS DTP and received the relevant Ph.D. Fellowship on September, 2021. The Open University of Cyprus is the host university for the ongoing project and Ph.D. studies. Jahna Otterbacher acts both as an MRG Leader and as a supervisor for the overall duration of the program.

The submitted proposal targets the thematic area “Towards Trustworthy AI: Human Oversight of Algorithms“ with the title “Facilitating Human Agency and Oversight in Algorithmic Processes“.

Project Lead

Kyriakos Kyriakou Research Associate and Ph.D. Student at OUC, CYENS Doctoral Training Program

Doctoral Committee

The doctoral committee consists of the following members:

The Research Problem

The proposed research revolves around the development of a generalized Modular Oversight Methodology (MOM) that can be applied to a set – or ideally any – algorithmic process to aid human agency and oversight. MOM should be customized according to the needs of the application and its context of use. Such applications could include any algorithmic process of a black-box system.

Specific research questions to be addressed are: What kind of properties should be considered for creating a MOM? What are the requirements for the successful application and use of this methodology? How can human computation be used to monitor the behaviors of common systems/processes? As a part of the methodology’s evaluation, one of the questions to address would be “To what degree can the methodology improve end-user trust in the system?”

The area of human oversight is still immature, which underscores the need for proper guidelines, practices and tools to facilitate this process. The authors of the AI4People initiative, suggest that we need to develop a framework to enhance the explicability of AI systems; auditing mechanisms for AI systems to identify unwanted consequences, such as unfair bias; agreed-upon metrics for the trustworthiness of AI products and services; a new EU oversight agency responsible for the protection of public welfare through the scientific evaluation and supervision of AI products, software, systems, or services.; and a European observatory for AI [..] [13].

Sundar argues that it would be beneficial to understand the trade-offs and focus on strategies for negotiating human agency to create synergistic systems that deftly leverage and combine the strengths of machine and human agency [14]. From another point of view, Krafft et al. propose allowing for policy implementation of reporting and oversight procedures as a necessary criterion for re-defining a policy-faced definition of AI [15]. In a similar manner, Wagner reports that very few [technical techniques and governance principles] deal with the specific challenge of human agency [16]. In addition, he examines the extent to which keeping humans in the decision loop—that is, ensuring meaningful human agency in decision-making processes—is necessary to safeguard human rights. Although there are several studies proposing ways of mitigating – implicitly or explicitly – this effect of the lack of human oversight in algorithmic processes [8]–[11], the literature is still fuzzy. What is clear is the need for generalized, practical frameworks and tools, such as the ones we plan to develop, which can be implemented and evaluated empirically.

Previous Work

Previously, we focused on a specific domain of AI, Computer Vision, implementing an award-winning system – OpenTag – that used crowdsourcing to understand human perceptions of Image Tagging Algorithms [12]. The proposed doctoral work will focus on developing a broader notion of the OpenTag methodology, as to be applicable to human oversight for algorithmic processes.

Methodology

Initially, a comprehensive literature review on human agency and oversight, trust and ethics in algorithmic systems will be conducted. Among others, we will address questions such as: What methodologies, approaches, mechanisms and tools currently exist (or are in common)? Which of them are recommended and for which contexts/applications? What’s their scale of actual use and in what areas (e.g., in governmental systems, organizations, individual developers)?

In the next phase, we will study candidate governance mechanisms of algorithmic processes, considering distinct categories of processes and fields of application. We will investigate further the mechanisms proposed in the literature such as the human-in-the-loop (HITL), human-on-the-loop (HOTL), or human-in-command (HIC) [17] and how they can be applied to achieve optimal oversight and trustworthiness, via crowdsourcing. Of course, we will examine mechanisms that can be utilized to aid the process and help us define components of the proposed methodology. In parallel, we will identify factors that might influence human decision-making in the form of agency and oversight of algorithmic behavior. Taking those into consideration, we will report the benefits of applying any of the mechanisms on the overall methodology to see how and to what degree these affect the trustworthiness of the algorithmic processes, including the final decisions made. Hence, we will propose auditing practices for ensuring that an algorithmic process is behaving in a manner that leads to fair justification. On the same line, it would be beneficial to discover the pros and cons of human oversight. In terms of our methodology evaluation, among others, it will be scrutinized for its ability to increase end-users’ trust in / understanding of these processes.

To conclude, we will explore possible interventions for aiding the human decision-making process and craft the necessary tools for facilitating human oversight in algorithmic processes. Based on the findings, and in light of the EU directives for trustworthy and ethical algorithmic systems [18], we will carefully define and propose the aforementioned methodology, accompanied by a set of guidelines and tools for human oversight in algorithmic processes.

Figure 1: An example of a black-box decision support system without human oversight (i.e., without using MOM).

Figure 2: An example of a black-box decision support system with human oversight (i.e., using MOM).

Interested to Collaborate?

References

[1] L. Floridi, “How The Infosphere is Reshaping Human Reality,” Fourth Revolut., p. 265, 2014.

[2] K. Kyriakou, S. Kleanthous, J. Otterbacher, and G. A. Papadopoulos, “Emotion-based Stereotypes in Image Analysis Services,” in Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization, 2020, pp. 252–259.

[3] P. Barlas, K. Kyriakou, S. Kleanthous, and J. Otterbacher, “Social B(eye)as: Human and machine descriptions of people images,” in Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019, 2019, pp. 583–591.

[4] P. Barlas, K. Kyriakou, O. Guest, S. Kleanthous, and J. Otterbacher, “To ‘See’ is to Stereotype,” Proc. ACM Human-Computer Interact., vol. 4, no. CSCW3, pp. 1–31, Jan. 2021.

[5] J. Buolamwini and T. Gebru, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification∗ Joy,” in Proceedings of Machine Learning Research 81, Conference on Fairness, Accountability, and Transparency, 2018, pp. 1– 15.

[6] M. Taddeo and L. Floridi, “How AI can be a force for good,” Science (80-. )., vol. 361, no. 6404, pp. 751–752, Aug. 2018.

[7] D. Danks, “The value of trustworthy AI,” AIES 2019 – Proc. 2019 AAAI/ACM Conf. AI, Ethics, Soc., pp. 521–522, 2019.

[8] B. Nushi, E. Kamar, and E. Horvitz, “Towards Accountable AI: Hybrid Human- Machine Analyses for Characterizing System Failure,” arXiv, no. Hcomp, pp. 126– 135, Sep. 2018.

[9] G. Bansal, B. Nushi, E. Kamar, W. S. Lasecki, D. S. Weld, and E. Horvitz, “Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance,” Proc. AAAI Conf. Hum. Comput. Crowdsourcing, vol. 7, no. 1, p. 19, 2019.

[10] D. R. Honeycutt, M. Nourani, and E. D. Ragan, “Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy,” arXiv, Aug. 2020.

[11] H. Shen and T.-H. K. Huang, “How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels,” arXiv, Aug. 2020.

[12] K. Kyriakou, P. Barlas, S. Kleanthous, and J. Otterbacher, “OpenTag: Understanding Human Perceptions of Image Tagging Algorithms,” in Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing, 2020.

[13] L. Floridi et al., “AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations,” Minds Mach., vol. 28, no. 4, pp. 689–707, 2018.

[14] S. S. Sundar, “Rise of Machine Agency: A Framework for Studying the Psychology of Human–AI Interaction (HAII),” J. Comput. Commun., vol. 25, no. 1, pp. 74–88, 2020.

[15] P. M. Krafft, M. Young, M. Katell, K. Huang, and G. Bugingo, “Defining AI in policy
versus practice,” AIES 2020 – Proc. AAAI/ACM Conf. AI, Ethics, Soc., pp. 72–78,
2020.

[16] B. Wagner, “Liable, but Not in Control? Ensuring Meaningful Human Agency in
Automated Decision-Making Systems,” Policy and Internet, vol. 11, no. 1, pp. 104–
122, 2019.

[17] I. Rahwan, “Society-in-the-loop: programming the algorithmic social contract,” Ethics
Inf. Technol., vol. 20, no. 1, pp. 5–14, Mar. 2018.

[18] E. Commission, “Building Trust in Human-Centric Artificial Intelligence,” 2019