Evaluating LLM-Generated ACSL Annotations for Formal Verification
dependable software systems, yet generating accurate and verifiable specifications for real-world C programs remains challenging. This paper presents an empirical evaluation of automated ACSL annotation generation strategies for C programs, comparing a rule-based Python script, Frama-C’s RTE plugin, and three large language models (DeepSeek-V3.2, GPT5.2, and OLMo 3.1 32B Instruct). The study focuses on oneshot annotation generation, assessing how these approaches perform when directly applied to verification tasks. Using a filtered subset of the CASP benchmark, we evaluate generated annotations through Frama-C’s WP plugin with multiple SMT solvers, analyzing proof success rates, solver timeouts, and internal processing time. Our results show that rule-based approaches remain more reliable for verification success, while LLM-based methods exhibit more variable performance. These findings highlight both the current limitations and the potential of LLMs as complementary tools for automated specification generation.
Tue 30 JunDisplayed time zone: Brussels, Copenhagen, Madrid, Paris change
16:00 - 17:30 | |||
16:00 30mTalk | A New Interaction Concept for Interactive and Autoactive Program Verification FTfJP Wolfram Pfeifer Karlsruhe Institute of Technology (KIT), Mattias Ulbrich KIT, Daniel Drodt Technical University of Darmstadt Pre-print | ||
16:30 30mTalk | Evaluating LLM-Generated ACSL Annotations for Formal Verification FTfJP Arshad Beg Maynooth University, Ireland, Diarmuid O'Donoghue Maynooth University, Ireland, Rosemary Monahan National University of Ireland Pre-print | ||