Microsoft’s AI bug hunter finds Rust flaw that went undetected for nearly 11 years

Microsoft researchers have uncovered a memory-safety bug in Rust’s standard library that had gone undetected for nearly 11 years, alongside other flaws that can trigger undefined behavior even in applications written entirely in safe Rust. Two of the findings affect stable Rust, including a separate issue that could cause applications to access freed memory. The nearly 11-year-old bug involves Rust’s process::exit function. Microsoft’s FORGE Lab researchers found that calling the function from a child thread could cause the runtime to remove the main thread’s alternate signal stack while it was still running. If the main thread subsequently experienced a stack overflow, the operating system could be unable to deliver the signal correctly, resulting in a segmentation fault instead of Rust’s intended stack-overflow handling. “The process::exit signal stack bug was in the standard library for nearly 11 years,” the researchers wrote in a blog post. Microsoft said the researchers found no reliable exploitation path under the conditions they examined and assessed practical exploitation as unlikely. The research used Hyenas, FORGE Lab’s AI-powered vulnerability-scanning system, to investigate unsafe code in Rust’s standard library. Hyenas is the lab’s research version of codename MDASH, Microsoft’s agentic vulnerability-discovery system. Microsoft said some of the bugs were found with help from Hyenas. Researchers then examined the underlying code, reproduced the issues, assessed their security impact, and reported them to Rust maintainers. Microsoft said the practical risk is limited. Each flaw requires unusual conditions to trigger, and a bug in the standard library becomes “a serious vulnerability only if a real program reaches the API in that way,” the researchers wrote. Unsafe code challenges Rust’s safety guarantees Rust’s ownership model and type system are designed to prevent memory-safety problems commonly associated with languages such as C and C++. However, its standard library relies on unsafe code internally for operations that the compiler cannot verify. Components such as Vec and String use these operations but expose them through safe APIs. Microsoft said these APIs must meet a strict soundness requirement: “no matter how safe code uses it, it must be impossible to trigger undefined behavior.” Microsoft grouped the findings into three categories: unsafe code that assumes operations will not panic, code that trusts user-provided implementations to behave correctly, and code that makes incorrect assumptions about the underlying system. The first involved a panic-safety problem in Rust’s MapWindows::clone implementation. The function created a buffer before fully initializing its contents and then called a user-provided clone operation to populate it. If that operation triggered a panic, Rust’s unwinding mechanism could invoke the buffer’s destructor, which would attempt to access uninitialized memory. Microsoft said the reported fix prevents the partially initialized buffer from being destroyed before initialization is complete. Another finding involved str::replace. The researchers found that the function could make an incorrect assumption about a custom implementation of the nightly Pattern trait. A custom implementation could supply a non-ASCII byte where the code expected an ASCII character, causing the replacement operation to produce invalid UTF-8 data. Subsequent string operations could then trigger undefined behavior, Microsoft said. System-level bugs affect stable Rust Microsoft said the first two categories of weaknesses were already being studied in 2021 through Rudra, a static analyzer developed by members of the research team. The standard library had also addressed panic-safety problems as far back as 2015, but similar weaknesses continue to appear in its implementations. The third category involves interactions between Rust and underlying operating systems and C libraries. Besides the process::exit issue, researchers identified a problem in Rust’s CommandExt::exec functionality involving the process-wide environment variable array. Microsoft found that the function could modify this shared state while holding a read lock, potentially allowing other threads to access inconsistent data. Under certain conditions, concurrent operations could leave the environment pointer referencing freed memory. Microsoft said both the process::exit and CommandExt::exec findings affect stable Rust. A proposed fix for the environment-variable issue, which avoids modifying the shared array by calling execve directly, was under review when the blog was published. Two issues reported by Microsoft’s team remained open at publication. The blog also discussed related reports from other researchers and fixes still under review. Findings feed into Microsoft’s security products Microsoft said the research findings can also feed into Microsoft Perception, its multi-agent security system, where specialized agents work across broader security workflows. It would therefore help the company improve the development of MDASH, which brings agentic vulnerability discovery into Microsoft Defender. For example, Microsoft’s Web App Pen Test Agent can use MDASH findings to prioritize testing, assess exploitability against live applications, and provide information to support remediation, the researchers wrote. “Our goal isn’t to simply find more bugs,” the researchers wrote, but to identify meaningful issues and improve the safety of foundational software developers and organizations depend on. The Rust project had not responded to a request for comment from CSO at the time of writing.