Abstract
Developing fair and reliable assessment for student-written machine learning (ML) programs is challenging because functionally equivalent scikit-learn pipelines may differ widely in preprocessing design, control structure, and implementation style. Traditional autograders—such as unit testing or output matching—are brittle under this variability and often fail to recognize semantically correct solutions expressed in alternative but valid forms. LearnMatic addresses this challenge by translating Python ML pipelines into a declarative intermediate representation, the Machine Learning Query Language (MQL), and performing comparison at the level of modeling intent rather than surface syntax. The prototype implements this process through a hybrid workflow that combines static code analysis, dataset introspection, MQL normalization and canonicalization, and large-language-model (LLM)-mediated reasoning to determine whether student and instructor submissions encode the same learning task, feature–target mapping, and modeling configuration. This enables equivalence-aware grading and explanatory, concept-focused feedback aligned with instructor expectations.
Rather than implementing full SMT-based verification at this stage, LearnMatic draws inspiration from SQL equivalence and formal-methods research to design an SMT-inspired, LLM-guided semantic comparison framework suited to interactive tutoring and formative assessment. The contribution lies in demonstrating the feasibility and pedagogical value of this equivalence-centered workflow, while establishing a foundation for future extensions incorporating stronger solver-backed guarantees.