Home
Report on the analysis of malware evasion techniques

Report on the analysis of malware evasion techniques

July 9, 2025

Malware analysis today faces increasingly intelligent and adaptive malware. Their goal: to thwart monitoring tools, virtual machines, sandboxes, and analysts themselves. This report is based on information provided by MITRE ATT&CK, an in-depth analysis of the evasion techniques used by malware, and feedback shared at the 12th edition of Botconf in 2025. It highlights the most commonly used methods for bypassing analysis, as well as effective approaches to neutralize them.

Module 1 – Anti-VM and Anti-Sandbox Techniques

The first hurdle that malware will often try to overcome is detecting the environment in which it is running. By identifying whether it is running in a virtual machine or an automated sandbox, it can then decide to adopt harmless behavior or simply stop. This is a simple but effective way to avoid detection by researchers and automated security tools.

Detection is usually performed by searching for certain artifacts: registry keys, files, services, or processes specific to a hypervisor. For example, names such as VBoxGuest, vmtoolsd.exe, vm3dservice, or vboxservice are common targets. Tools such asProcmon allow these searches to be observed in real time.

But malware isn't limited to these signatures. It also inspects system parameters such as hard drive size, available RAM, and thenumber of processor cores. Since these characteristics are often low in test environments, they are reliable indicators for malware.

Another tactic is to list the processes that are currently running. If the malware detects a process linked to a sandbox, debugging tool, or security product, it may decide not to continue. This technique is used in particular by theBumblebee malware family, which interrupts its execution if it detects procmon.exe, processhacker.exe, or wireshark.exe.

Finally, some malware checks for network activity or user history to ensure that it is not in a sterile environment. This may involve checking the browser cache or recently opened files. If nothing is found, it concludes that it is being monitored.

To circumvent these techniques, analysts have several options available to them:

Modify or hide artifacts from virtual machine editors (files, registry keys, services).
Use tools such as VBoxCloak that automate the concealment of a VirtualBox environment.
Simulate user activity by opening files, launching legitimate applications, or filling the browser cache.
Simulate a credible network connection using tools such as FakeNet-NG, which intercept and respond to malware traffic in a controlled manner.

When it comes to more advanced bypassing, x64dbg can be used to dynamically patch malware instructions in order to skip checks or modify their results (for example, forcing a cmp to always succeed). Similarly, static analysis in IDA Pro can identify suspicious calls to RegOpenKeyEx, FindFirstFileW, GetDiskFreeSpaceEx, etc.

Key takeaways

Malware searches for clues to a virtual environment: registry keys, file paths, suspicious processes.
Ils peuvent mesurer des ressources matérielles minimales pour éviter les environnements de test (ex. < 4 Go RAM, < 80 Go disque).
Checking processes or the absence of user activity allows automated sandboxes to be detected.
In the absence of network connectivity, malware often stops running: FakeNet allows you to simulate a realistic environment.
Tools such as VBoxCloak and VMwareCloak hide traces of virtualization.
The reverse engineer can modify paths, values, or skip critical instructions via IDA or x64dbg.

Module 2 – Anti-Debugging Techniques

Once the virtualization environment has been bypassed, the malware will attempt to determine whether it isbeing actively debugged. This phase is particularly critical for the analyst, as the malware will attempt to self-destruct, mask its behavior, or trap the analyst.

Windows API calls such as IsDebuggerPresent or CheckRemoteDebuggerPresent are often used to detect a debugger. These functions read values in thePEB (Process Environment Block) structure, including the BeingDebugged field. It is also common to check the NtGlobalFlag flag in the PEB.

But the techniques don't stop there. Some use calls such as OutputDebugString, NtQueryInformationProcess, or low-level system calls. Others employ temporal methods such as:

GetTickCount, which returns the time since the system started (useful for detecting a recent startup).
GetLocalTime, which allows you to implement "time bombs" (e.g., only executes after June 1, 2025).
RDTSC (Read Time-Stamp Counter), which measures the time between two instructions, revealing interruptions caused by the debugger.

Malware may also attempt to deliberately delay execution using Sleep, NtDelayExecution, or stalling loops. This delay can exceed the execution window
of a sandbox, making the malicious behavior invisible. Finally, some techniques aim to detect software breakpoints (0xCC) or software hooks (0xE9) on critical functions. To do this, the malware can scan the memory using VirtualQuery or ReadProcessMemory.

There are several solutions to these mechanisms:

Dynamically modify instructions via x64dbg (e.g., replace "je" with "jmp").
Change the system date or let the system run for a few minutes before scanning.
Use ScyllaHide, an x64dbg plugin that can automatically hide the debugger.
Prefer hardware breakpoints, which are more discreet, when malware detects software breakpoints.

Key takeaways

Malware uses IsDebuggerPresent, NtQueryInformationProcess, GetTickCount, and RDTSC to detect debuggers.
Time bombs or ignition time checks allow execution only in real contexts.
RDTSC measures CPU cycles: any abnormal latency may indicate the presence of a debugger.
Delays (Sleep) are used to escape automated sandboxes that generally run for a short period of time: their value can be modified.
Software breakpoints (0xCC) and hooks (0xE9) can be detected via memory inspection.
The use of ScyllaHide or dynamic patches is essential to overcome these obstacles.
It is possible to force a jump (jmp) or simulate a system value by modifying the registers.

Module 3 – Indirect System Calls and MutationGate

This module covers more advanced and recent techniques designed to evade behavioral detection systems such as EDRs (Endpoint Detection & Response). Rather than directly calling classic Windows API functions, malware will generate its own system calls indirectly in order to fly under the radar.

Indirect System Calls

The indirect syscall approach, popularized by SysWhispers3, allows hooks placed by EDRs around sensitive functions (such as NtOpenProcess or NtAllocateVirtualMemory) to be bypassed. By manually constructing the system call—initializing registers, placing the function identifier (SSN) in EAX, and executing the syscall instruction—the malware avoids traditional monitoring points.

SysWhispers offers several methods:

Jumper: jumps directly to the syscall instruction in ntdll.dll.
Embedded: integrates the syscall directly into the binary.
Egg-hunter: code deliberately "broken" but repaired in memory by a loader.
Jumper-randomized: adds randomness to the choice of the function called.

These techniques make analysis complex, as no API is visible in the import table and the function names are not present.

MutationGate: an even more discreet evasion technique

The technique described as MutationGate, recently documented in the community, introduces an even more stealthy method. It uses vectored exception handlers (VEH) to dynamically hijack system calls. The idea is simple but brilliant: the malware calls an innocent function (such as NtDrawText) on which a hardware breakpoint is placed. When it reaches this point, an exception is raised. The associated handler intercepts the exception and modifies the RAX register on the fly, injecting the SSN of the real desired function (
, for example NtOpenProcess) just before the syscall is executed.

This mechanism allows the real intent of the code to be completely hidden:

No trace in the imports table.
No direct calls to dangerous functions.
Hiding behavior until the last moment, within the handler.

During analysis, only a detailed inspection of the VEH handler can identify the true behavior. The malware often uses several hardware breakpoints (up to 4) to manage different call redirections.

Key takeaways

Indirect syscalls allow EDR hooks to be bypassed by directly calling the syscall via custom stubs.
MutationGate goes further by injecting the SSN into the RAX register at the time of an exception, making the system call undetectable.
It is possible to analyze these behaviors using x64dbg and IDA, by placing breakpoints on critical instructions (syscall, RAX, VEH handlers).

Conclusion

Through these three modules, we see that malware evasion techniques are no longer purely defensive, but are becoming proactive. They aim to disrupt analysts, deceive automated tools, and manipulate the lower layers of the system to remain stealthy. To counter these techniques, analysts must equip themselves with a rigorous methodology, the right tools, and, above all, a thorough understanding of the internal mechanisms of Windows.

This report illustrates the technological and strategic evolution of malware. From simple virtualization tests to system calls obfuscated by exception handlers, evasion techniques are becoming increasingly complex, requiring analysts to remain constantly vigilant and constantly update their methods.

However, there are solutions for each level of evasion: environment concealment, dynamic code modification, network emulation, etc. This expertise is now crucial in areas such as incident response, threat hunting, and post-mortem analysis.