Papers for

security testers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Mallocsan detects memory errors in closed-source linux applications

MallocSan: A Memory Safety Tool for Native Closed-Source Applications

Abstract: Memory-corruption errors remain a leading cause of high-impact vulnerabilities in C and C++ software. Deterministic detection, however, remains difficult to deploy: compiler-based sanitizers require source code and control of the build pipeline, hardware-assisted schemes depend on specific platforms, and dynamic binary translation can impose order-of-magnitude slowdowns. This paper presents MallocSan, a heap sanitizer for native, potentially closed-source x86-64 Linux applications that requires no source access, recompilation, or specialized hardware. MallocSan interposes on memory allocation through LD_PRELOAD and embeds an object identifier in the unused high bits of each protected pointer. Dereferencing the resulting noncanonical pointer faults at the offending instruction, which MallocSan decodes and patches at runtime so that subsequent executions perform per-object bounds checks entirely in userspace. Sites that cannot be patched fall back to in-handler emulation or single-stepping, while an optional profile-guided pass rewrites frequently executed residual sites offline. MallocSan also extends identity-based checking to vector gather/scatter instructions and supports policy-scoped coverage. Across seven SPEC CPU 2017 benchmarks, MallocSan outperforms Valgrind Memcheck on five, with a geometric-mean execution-time factor of 4.82x relative to native execution, compared with 18.55x for Memcheck, while preserving substantial parallel scaling on both evaluated multithreaded workloads: 644.nab_s and pigz. On the in-scope Juliet tests, MallocSan detects all seeded violations with no false reports on the good executions. It also detects heap-buffer errors in real-world applications, including a known one-byte overread in LibTIFF's tiffcrop utility.

Tue 29 SeptSoftware Engineering
The gist
Memory errors in software written in C and C++ can cause serious security problems, but finding these errors is hard without access to the program's source code. The authors created MallocSan, a tool that works with Linux programs to catch such errors without needing the source code or special hardware. It works by tracking allocated memory and checking for mistakes while the program runs, causing less slowdown than some other tools. MallocSan was tested on several programs and found errors effectively, even in real-world software.
Open → 2609.38499v1

SkillDRE improves malicious agent skills using staged defense feedback

SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback

Abstract: Agent skills package instructions, executable code, and task-specific resources into reusable artifacts that agents can improve using execution feedback. The same mechanism also enables attackers to evolve malicious skills, making them more effective and less detectable. However, a candidate skill may pass pre-execution scanning yet fail to realize its target under runtime defenses, while a revision that repairs execution may introduce new scanner findings. We introduce SkillDRE, a fully automated framework for evolving complete malicious skill packages through a dual-stage feedback loop. Given a benign task and its associated skills, SkillDRE autonomously constructs and validates a task-conditioned malicious objective and a verifiable judge rule. It then holds both fixed while evolving the skill implementation, with preservation of legitimate task capability. SkillDRE combines scanner-guided evolution with runtime-guided refinement informed by execution outcomes observed under runtime defense. Each runtime-guided revision returns to the pre-execution stage for rescanning and further optimization before re-execution, forming a cross-stage closed loop. Evaluated on SkillsBench across four victim models, SkillDRE achieves an average attack success rate of 45.28%, exceeding the strongest baseline by 40.3%, while its final submitted skills receive no SkillScan findings and largely preserve benign-task performance. These results show that two-stage defense feedback can serve as a useful learning signal for adaptive red teaming and that evaluating either defense stage in isolation can miss the resulting attack capability. Codes is available at https://github.com/whfeLingYu/SkillDRE

Sat 26 SeptCryptography and SecurityArtificial Intelligence
The gist
Some computer programs called agents use skill packages to perform tasks. These skills can be attacked and improved by hackers who try to sneak harmful code past security checks. The authors present SkillDRE, a system that cleverly changes these skills using two levels of feedback: one before the code runs and one during its execution. This helps attackers make their harmful skills work better without being detected, while still doing the original good task. Their tests show SkillDRE is more effective than previous methods at bypassing defenses.
Open → 2609.32400v1