Large Language Models (LLMs) enable natural language interfaces for data retrieval, but their tendency to hallucinate hinders their adoption in scientific applications that require high accuracy. Unlike deterministic traditional APIs (Application Programming Interface), LLMs introduce inherent uncertainty, posing critical challenges for reliable geoscience data access. This paper presents OpenMindatAI, a system that leverages a role-specialized LLM pipeline and a hybrid validation framework to generate reliable API requests for Mindat, one of the largest open databases in geoscience. A consensus voting mechanism is implemented to enhance reliability by generating multiple requests simultaneously and proceeding only upon majority agreement. The hybrid validation framework combines rule-based syntactic validation and LLM-based semantic validation to further ensure structural correctness and logical consistency. The performance of OpenMindatAI is evaluated using a synthetically generated benchmark dataset conforming to the Berkeley Function Call Leaderboard (BFCL) specifications. As a result, OpenMindatAI achieved an F1-score of 97.08%, significantly outperforming the 83.95% baseline. This work bridges the gap between non-technical users and specialized geoscience data APIs while establishing a foundation for LLM-enhanced data services. Our system serves as a reliable intermediary, enabling external LLM agents to access specialized geoscience data without domain-specific customization, thereby promoting seamless integration of scientific data repositories into the broader LLM ecosystems.
- A hybrid validation framework for LLM-enhanced open data service: the mindat API as a use case
- Jiyin Zhang - University of IdahoWeilin Chen - University of IdahoChenhao Li - University of IdahoXiaogang Ma - University of Idaho
- Earth science informatics, Vol.19(9), 155
- Springer Berlin Heidelberg
- 2126315 / National Science Foundation, United States
- 996989839101851
- Earth and Spatial Sciences; Institute for Interdisciplinary Data Sciences; Computer Science
- English
- Journal article