WorksPhysics and Astronomy
FacilityAudit-Agent: Beyond Mention-as-Usage for Fine-Grained Facility Contribution Attribution in Astrophysics Publications
Abstract · v1
Observatory bibliographies increasingly need to determine not only whether a facility is named in a paper, but how its data support the scientific argument. We frame this need as semantic contribution attribution on a six-level scale (L0-L5), from non-use and mention-only cases to auxiliary, major, and exclusive data support. We present FacilityAudit-Agent, a framework that builds a page-bounded outline of each PDF, enriches sections with roles and facility mentions, and uses a tool-calling large language model (LLM) to fetch text on demand. Evidence from the sections is combined through an explicit decision tree and a coverage-calibrated confidence score, with uncertain cases routed to human review. The framework supports both single-facility and inventory audits and transfers to new facilities without retraining. On a test set of 39,358 human-annotated paper-facility pairs, FacilityAudit-Agent achieves 89.1% accuracy and 79.8% macro-F1 in single_audit mode, outperforming the other three baselines by 5.8, 8.3, and 12.1 accuracy points, respectively, with the clearest gains at L2 and L5. In inventory_audit mode, it also leads in facility listing and in level assignment among listed pairs. Ablations show multi-hop reading and tree indexing matter most for L3-L5. FacilityAudit-Agent has been deployed at the National Astronomical Data Center, where it is responsible for processing publications related to LAMOST, FAST, and EP.
Review conversation
Reviews are not available for this work yet.