Abstract
Artificial intelligence (AI)/Machine learning (ML) systems have outperformed humans in a variety of sectors like healthcare, autonomous vehicles, criminal justice, banking, and finance. However, the inability to explain their autonomous decisions and actions has created a new challenge in the research community. The need for explainability has shifted the focus of AI research from complex black-box models to explainable and interpretable models. Recently, the topic of explainable AI (XAI) and trustworthy AI (TAI) has become a hotspot and is widely acknowledged by academia, industry, and government. XAI is a research field that aims to make AI/ML results more understandable and explainable to humans. While there are a variety of explainability approaches and methodologies designed for providing explanations and user-friendly decisions, each has its benefits and drawbacks as well as several unsolved challenges. Based on our literature review research, we believe provenance holds great promise for the new state-of-the-art AI/ML solutions. Provenance is increasingly important in AI/ML systems in illustrating the details of workflows and guiding human decision-making. Through this book chapter, we aim to provide an overview of the fundamental concepts of XAI, TAI, and provenance with the latest research and practices. The goal is to highlight the importance of provenance documentation by examining technical approaches in the earth science domain. In the discussion, we propose several directions that AI/ML systems need to pursue. We hope this book chapter not only serves as reference material for future research advances but also encourages experts and professionals from all disciplines to embrace the benefits of provenance, XAI, and TAI.