What changes this picture is the rise of agents. Once a model can use tools in a loop, inspect intermediate results, and continue analysis on its own, it can take over meaningful parts of the workflow. We already see that across tasks such as solving CTF challenges, generating fuzzing harnesses, malware analysis, and exploit-generation experiments. However, while agents are already useful on their own, they become substantially more effective once they are given the right environment, tooling, and guidance.

In this post, we show how to build such an agentic malware-analysis pipeline in practice. We demonstrate this on a concrete malware case study in which some functionality is easy to identify from strings and APIs, while other parts only reveal themselves after deeper analysis. We first introduce the basic ingredients of agentic workflows, then look at what a strong general-purpose agent can find in the sample on its own, and finally compare that to the same agent embedded in a structured workflow. This lets us compare what already works, what the pipeline improves, and which limitations remain.

The companion repository with the analysis environment, pipeline, and sample setup is available here.

Agents

For the purposes of this post, we can use a simple definition: an agent is an LLM that can use tools in a loop to achieve a goal. Instead of answering once inside a single prompt, it can inspect tool output, decide what to do next, execute another step, and continue until it reaches a result. That is the key shift from LLM-assisted reverse engineering to agentic workflows.

The basic loop is straightforward. The model receives a task, chooses a tool, sees the result, and repeats this process until it can produce a final answer. In simplified pseudocode, such a reverse-engineering agent could look as follows:

# define tools
tools = {
    "strings": strings,
    "disasm": disasm,
    "decompile": decompile,
}

# set LLM instructions
msgs = [
    {
        "role": "system",
        "content": (
            "You are a reverse-engineering agent. "
            "Use tools to analyze binaries."
        ),
    },
    {
        "role": "user",
        "content": (
            "Figure out what function sub_401230 does."
        ),
    },
]

# agent loop
while True:
    r = llm(msgs, tools=tools)

    if r.type == "final":
        print(r.content)
        break

    result = tools[r.tool_name](**r.tool_args)
    msgs += [
        r,
        {
            "role": "tool",
            "name": r.tool_name,
            "content": str(result),
        },
    ]

The agent can retrieve strings, disassembly, and decompiler output, decide which tool to call next, observe the result, and iteratively refine its understanding until it can answer the question. Even this simple setup is already enough to turn a model from a passive assistant into an active analysis loop.

While there are good reasons to implement such loops from scratch, and official examples from both OpenAI and Anthropic show this approach directly, in practice many people will use command-line agents such as OpenAI’s Codex CLI, Anthropic’s Claude Code, or open-source agents like opencode. These tools already handle the interactive loop, tool execution, and session state for us.

For reverse engineering, the next question is how we connect the right analysis tooling and how we guide the workflow. The Model Context Protocol (MCP), as defined in the official specification, standardizes how agents discover and call external tools. Through MCP, an agent can query a disassembler, collect metadata, inspect cross-references, or run helper scripts without us manually copying results back and forth. Projects such as GhidraMCP and ida-pro-mcp make this kind of integration directly available for Ghidra and IDA Pro.

Agent skills, on the other hand, are reusable, task-specific workflows for agents. Their main purpose is to tell the agent how to approach a task. In practice, a skill can include a workflow, supporting scripts, file templates, examples, and concrete expectations for the output. This is useful across many reverse-engineering tasks: for a crackme, we may want the agent to first identify the relevant input checks and success paths before it starts patching the binary. For obfuscated code, we may want it to first classify the protection, determine where and how it appears in the binary, and then derive a suitable workflow and tooling strategy to deobfuscate it. In malware analysis, we may want the agent to first collect broad evidence such as strings and API imports, identify the most relevant points of interest, and only then inspect the surrounding code regions in the disassembler.

However, tool access and workflow guidance still do not solve the main practical limitation of unstructured agent runs: the context window. This is the temporary amount of information an agent can keep in view during a session, including chat history, tool output, notes, and intermediate observations. In reverse engineering, this window fills up quickly. Large amounts of disassembly or decompiler output, especially for large functions or groups of related functions, can already consume a significant fraction of the available context. Once the context becomes too large, the agent usually compacts it into a summary. As a consequence, details may be lost, the analysis may drift, work may be repeated, or the agent may simply get lost.

To deal with this limited context, two things are especially useful in practice: subagents and planning. With subagents, the main agent can offload focused work to a child agent with its own context window, such as analyzing a single function or tracing references to a string. Anthropic documents this pattern explicitly as subagents, while OpenAI discusses comparable delegation setups as multi-agent orchestration. Planning helps with a different part of the same problem: it does not give the agent more context, but it can help it spend that context more deliberately. For example, Claude Code exposes this explicitly as Plan Mode, where the agent first explores the problem with read-only operations and works out which evidence to collect and which leads are worth following.

Still, neither subagents nor planning solve the underlying issue. If the context gets compacted, the agent may still lose track of prior findings because it has no persistent memory, no stable notes to recover from, and no workflow it can resume reliably.

With these foundations in place, let us now look at what a strong agent can do with tool access alone. Afterwards, we will build a pipeline around the same setup and examine how much that changes the depth and reliability of the analysis.

Baseline: Unstructured Agent-based Malware Analysis

To get a feeling for what a strong general-purpose agent can already do on its own, we start with a simple setup: direct access to a disassembler, but no task-specific workflow, no persistent case file, and no additional guidance beyond the analysis prompt. In other words, the agent can drive the tools, but it has to structure the investigation on its own. That setup is already useful in practice, and we should expect it to produce a decent first triage and some concrete leads. The more interesting question is where it starts to flatten out once the sample requires deeper reconstruction.

As our test case, we use mfc42ul.dll from the German “Staatstrojaner” (federal Trojan horse) case publicized by the Chaos Computer Club (CCC). The sample is a good fit because some of its functionality is relatively easy to spot from strings, imports, and a quick pass through the code: screenshots, Skype interception, persistence, and covert communication. But it also contains deeper logic that is much harder to recover from a shallow analysis, including process-aware activation, command dispatching, protocol details, hardcoded C2 information, and the statically embedded AES key. That makes it a useful test case: it lets us judge not only whether the agent finds something interesting, but also how far the analysis actually goes.

In our initial run, we use OpenAI’s Codex CLI with gpt-5.4 xhigh and give it access to my headless Binary Ninja MCP server binary-ninja-headless-mcp. We then ask the agent to analyze the sample via the MCP server and produce a detailed overview of the malware, its features, and the corresponding code locations. At this point, the agent has everything it needs to inspect the binary and drive the analysis on its own:

› Analyze the malware in examples/samples/mfc42ul.dll
  Give me a detailed overview of the sample’s functionality and features,
  together with the corresponding code locations. Use the MCP server to
  perform the analysis.

One first useful observation is that the agent does not behave randomly. Even without additional workflow guidance, it starts with a fairly sensible triage strategy: open the sample, collect high-level metadata, identify likely behavior from strings and imports, and then follow those leads into the code. The following excerpt shows how the run begins:

• I’m analyzing examples/samples/mfc42ul.dll with the Binary Ninja MCP tools.

• Updated Plan
  └ Static malware triage and behavior reconstruction for the DLL using Binary
    Ninja MCP.
    □ Open the DLL in Binary Ninja and collect high-level metadata.
    □ Identify likely behavior clusters from strings/xrefs.
    □ Trace the main execution paths and map capabilities to concrete code locations.

• Called
  └ binary_ninja_headless_mcp.session.open({...})
    ok session_id=...

As the run continues, the same pattern remains visible: broad metadata collection first, then pivots off suspicious strings and imports, and only gradually deeper inspection of the code.

• The file is a 32-bit PE DLL with a large code surface, so I’m doing a fast
  triage pass before drilling into behavior. I’m collecting exports,
  imports, and structural metadata now, then I’ll pivot off suspicious
  strings and entry-point code.

This is already a useful and fairly reasonable starting point. The agent is not guessing blindly; it is using the tooling in a disciplined way and follows a plausible initial workflow. At the same time, the limitation is already visible: the analysis is driven primarily by obvious pivots such as strings, imports, and surface-level structure. That is often enough for a first triage, but not necessarily enough to recover the deeper logic of the sample.

After roughly 15 minutes and 268 tool calls, the agent produced a first high-level summary of the sample. The result is already quite useful for an initial triage. The agent:

• identifies the dropped driver winsys32.sys, the corresponding installation logic, and the driver device \\.\KeyboardClassC, and ties them to functions such as sub_1000e8a0 and sub_10003a40,

• notices the Skype-related markers and target process names, and links them to nearby control logic around sub_10007490,

• recovers the proxy-discovery logic in sub_1000ead0, including registry-based handling of Internet Explorer proxy settings,

• identifies a proxy-aware outbound transport based on HTTP CONNECT, together with the protocol marker C3PO-r2d2-POE, in sub_10006790, and

• finds a secondary payload path around sub_1000d470, including temporary-file staging, execution, and later cleanup.

This is already a meaningful speedup for the initial analysis: the agent produces a useful high-level overview of the sample’s features and maps them to concrete code locations. At the same time, the result remains fairly shallow. It does not identify the cryptographic code, recover the hardcoded key, or reconstruct the deeper protocol logic. In other words, it is a good first triage, but not yet a deep analysis.

Another practical problem is that the run nearly exhausts the available context window. By the end, only a small fraction of the context remains available. This makes the setup brittle: even when the agent finds useful leads, it has limited ability to preserve and build on them once the investigation grows larger or requires repeated deep dives into related code. Large chunks of disassembly and decompiler output consume context quickly, and once earlier findings are compacted into summaries, details can easily be lost.

So the baseline is already helpful, but mostly as triage. It points us to relevant functionality, recovers several visible features, and gives us concrete starting points for manual inspection. What it does not yet provide is a deeper reconstruction of how the malware is organized internally.

To see whether this is mostly a Codex-specific limitation or a more general pattern, we can repeat the same experiment with another strong agent. Using Anthropic’s Claude Code with Opus 4.6 and high effort, the overall picture improves somewhat. In a similar amount of time, Claude recovers a few additional details:

• the audio-recording pipeline, including dual-channel capture and the related waveIn* handling,

• the SNMP-based host fingerprinting together with the VMware MAC-prefix filter in the anti-analysis logic, and

• the logic that reads the DLL’s own filename at runtime, suggesting that the sample checks whether it is still running under the expected module name.

At the same time, the overall pattern remains the same: Claude also gives a useful first overview, but it still does not really reconstruct the deeper logic of the sample.

Overall, these experiments suggest that changing the model helps only to a limited extent; it does not solve the main problem. As we will see, what matters much more is whether the agent has clear guidance, a structured plan, and a way to continue the analysis even after context-window resets. Let us now look at how to build that into the workflow.

Agentic Analysis Pipeline

The earlier experiments already show the main limitations that the workflow needs to address. A strong general-purpose agent with access to a binary-analysis engine can produce a useful first triage: it recovers visible functionality, finds sensible pivots, and maps obvious behavior to code locations. In our case, that was enough to identify some of the easier-to-spot features, but not the cryptographic code, the deeper protocol logic, or the internal organization of the sample. At the same time, the run was already approaching the limits of the context window.

To address these problems, we need a workflow that gives the agent a clear analytical structure and a persistent external memory it can return to. In the end, our goal is to make the analysis more repeatable, resumable, and capable of going deeper than unstructured tool-driven exploration without exhausting the context window unnecessarily. Therefore, we can now define four design goals for the workflow:

1. Start with broad fingerprinting. The agent should first collect a wide range of signals—strings, API imports, symbol names, crypto indicators, points of interest, and other easy-to-extract artifacts—to build a rough high-level picture of the sample before drifting into deeper analysis.

2. Keep raw evidence separate from higher-level reasoning. Raw artifacts (e.g., strings), extracted signals, planning results, hypotheses, and interpretations should not all be mixed together. This lets the agent work from structured summaries without losing access to the underlying evidence.

3. Follow a clearly staged process. The workflow should make explicit what has already been done, what comes next, which questions remain open, and what every stage is expected to produce. In implementation terms, stages are defined by templates, expected output artifacts, and lightweight validators that check whether the required files and fields are present before the workflow continues. This ensures that the workflow produces explicit intermediate artifacts that later stages can build on directly.

4. Externalize state to disk. Notes, artifacts, progress, hypotheses, and reports should be written to persistent storage so that the analysis can continue even after the context window fills up.

We address these goals by structuring the agent around a staged workflow that roughly mirrors how a human analyst would approach a malware sample: first collecting broad signals such as strings, API imports, and signatures; then building a rough picture of the sample; forming initial hypotheses; prioritizing the most promising leads; and finally validating or falsifying those hypotheses through deeper analysis. To keep this workflow manageable, we separate responsibilities across three roles: the orchestrator, the planner, and the reporter.

The orchestrator coordinates execution, initializes the case directory, checks artifact completeness, and manages phase transitions. The planner turns intermediate evidence into hypotheses, priorities, and concrete deep-analysis tasks. The reporter turns the accumulated artifacts into a consumable write-up, separates observed facts from inferred claims, and records uncertainty and next steps. During exploration, the orchestrator can also delegate focused collection tasks to subagents and merge their outputs back into the case state.

At a high level, the workflow looks as follows:

1. Broad fingerprinting and triage: Collect cheap signals such as file metadata, hashes, entropy-related hints, strings, imports, symbol names, YARA matches, capa results, compiler or packer hints, crypto-related indicators, and other obvious points of interest to build an initial picture of the sample.

2. Filtering and grouping: Filter the raw output, group related findings, and surface the strings, imports, capability clues, and code regions that look most relevant.

3. Hypothesis building: Use those findings to form initial ideas about what the sample does, which major components it seems to contain, and which leads are worth following.

4. Internal structure mapping: Build a working map of how the sample appears to be organized internally: which parts seem to handle persistence, communication, collection, injection, or other major tasks, and how they fit together.

5. Deep-analysis planning: Select and prioritize the functions, components, features, and hypotheses that are the best candidates for deeper inspection with the binary-analysis tooling.

6. Targeted deep analysis: Inspect those high-priority targets with the binary-analysis tooling to validate or revise earlier assumptions and recover the missing low-level logic.

The key improvement comes from imposing a structured workflow on the analysis. Instead of letting the agent interact with tools in an open-ended way, the workflow organizes the process into explicit stages with concrete intermediate artifacts. Raw evidence is collected first, then filtered and grouped, and then turned into hypotheses, structure maps, and plans for deeper inspection. As new evidence emerges, the agent can revise earlier assumptions and update the case state, while continuing from the artifacts produced so far.

To support this, the pipeline uses a persistent on-disk case directory for every sample. After each major step, artifacts, notes, ranked observations, hypotheses, plans, and reports are updated and stored. In effect, this directory becomes the agent’s externalized working memory. If the context window gets compacted, the agent can reread prior artifacts and continue from the current state. This also makes the workflow inspectable and reproducible, since intermediate results are no longer hidden inside transient model context.

We implemented this pipeline as a portable analysis environment that combines helper scripts, a reusable orchestration skill, and MCP-connected disassembler tooling into a single workflow. The agent runs inside a dedicated Docker container defined by the Dockerfile, with access to the available tools, scripts, and MCP servers. The workflow is packaged as a malware-analysis-orchestrator skill for Claude Code together with a matching Codex variant, and can use either our Binary Ninja Headless MCP server or Ghidra Headless MCP server for static analysis. The setup is designed to remain generic across agents, binary formats, and analysis backends. The full pipeline is available on GitHub.

With that in place, we can rerun the same mfc42ul.dll sample under the structured workflow and compare the result to the earlier analysis.

Evaluation

In the following, we focus on the Claude Code runs, since they produced the clearest write-up for this section. The runtime remained in roughly the same range as before. Codex improved in the same direction once the workflow was enabled as well, but we omit its output here because it would not change the overall conclusion.

Before turning to the results, let us briefly recall what is present in mfc42ul.dll. Some features—such as screenshot capture, Skype interception, persistence, and proxy handling—were relatively easy to identify from strings, API imports, and a shallow code pass. However, the sample also contains deeper logic and embedded data that are much harder to recover, including process-aware activation, command dispatching, protocol details, hardcoded C2 information, and the statically embedded AES key. These are exactly the parts where the structured workflow has to prove its value.

To evaluate this, we slightly expand the earlier query and explicitly instruct the agent to use the malware-analysis-orchestrator skill:

❯ Analyze the malware in examples/samples/mfc42ul.dll -- Give me a detailed
  overview of the sample's functionality and features, together with the
  corresponding code locations. Use the skill
  /agent/agent_helpers/claude/skills/malware-analysis-orchestrator/ for
  analysis.

After roughly 17 minutes, the workflow completes its first pass. The agent populates all 13 required case files, identifies 13 functional components, records 10 behavior hypotheses, and assigns HIGH confidence to 8 of them. Even before any dedicated deep-validation step, the resulting overview is already materially richer than the earlier triage-oriented summary. To keep the presentation manageable, we show a slightly shortened version of the first-pass output below:

● Analysis complete. All 13 artifacts populated. Here is the detailed overview:

  1. Process-Aware Initialization (sub_1000f590 @ 0x1000f590)
  2. Persistence — AppInit_DLLs (sub_1000d680 @ 0x1000d680)
  3. Persistence — Kernel Driver Service (sub_10003a40 @ 0x10003a40)
  6. C2 Communication (sub_10006790 @ 0x10006790)
  9. Skype VoIP Interception (sub_10007720 @ 0x10007720)
  14. Central Orchestrator (sub_1000ecc0 @ 0x1000ecc0)

  All artifacts are in status/000-mfc42ul.dll/. The analysis identified
  13 functional components, 10 behavior hypotheses (8 HIGH confidence),
  56 capa capabilities, and produced comprehensive IOCs for detection engineering.

Already at this stage, the workflow no longer stops at visible features alone. It begins to group functionality into components, map those components to specific code regions, and point toward the sample’s internal control structure. That is already a qualitatively different kind of result.

The difference is not just that the workflow finds more. It also writes much more than a short final summary. Instead, the run produces a detailed set of intermediate artifacts for hypotheses, component mapping, planning, prioritization, and reporting. The following files are just a few examples of the case state the workflow yields:

• 05_behavior_hypotheses.md records the current hypotheses, the supporting evidence, their confidence, and the checks that are still missing.

• 08_deep_analysis_plan.md turns those hypotheses into concrete follow-up tasks, with target functions, goals, and expected findings.

• 09_priority_queue.md tracks which open questions matter most, why they matter, and what depends on resolving them.

• 10_reporting_draft.md brings the current findings, code references, IOCs, and remaining gaps into a draft report that the agent can return to later.

From a higher-level perspective, the earlier run compares to the structured workflow as follows:

Most importantly, the first pass no longer leaves us with just a short summary. Because it produces ranked hypotheses and a persistent case directory, we can continue directly from that state rather than restarting the investigation. We therefore ask the agent to validate the 8 HIGH-confidence hypotheses. Roughly 30 minutes later, the workflow produces a much deeper second pass:

Validation Results

H1: Full-Featured Surveillance RAT — VALIDATED
- Discovered complete command dispatch vtable at 0x1003d7d0 with 10 remotely-callable commands
- Mapped orchestrator vtable at 0x1003d7f8 with C2 data loop, heartbeat (byte 0x26), fingerprinting

H3: Process-Aware Module Activation — VALIDATED
- Confirmed GetModuleFileNameA + case-insensitive path matching in sub_10003990
- Full init chain: DllMain → process detect → module alloc → orchestrator

H6: C2 with AES-128 Encryption — VALIDATED
- Protocol vtable at 0x1003d330: encrypt+send, recv+decrypt, raw I/O via WinSock send()/recv()
- Encrypt: sub_10012130, Decrypt: sub_10013260, Key expansion: sub_10014410
- Wire format: C3PO-r2d2-POE → 4-byte length → AES-128-ECB encrypted payload

Three improvements stand out here. First, the workflow recovers the internal command dispatch rather than stopping at the sample’s visible high-level capabilities. Second, it validates the process-aware activation logic and reconstructs the initialization chain from DllMain into the orchestrator. Third, it begins to recover the protocol layer itself, including the protocol vtable, the send and receive paths, and the cryptographic routines that are used together with the C2 communication.

A closer look at the recovered command-dispatch table shows the kind of internal structure that the deeper pass can uncover:

Taken together, this is a substantial step up from the earlier runs. The workflow does not just recover a longer list of visible features; it begins to expose the malware’s internal organization: how capabilities are dispatched, how activation depends on the host process, and where protocol handling and cryptographic routines sit in relation to the C2 communication. At this point, the agent already leaves behind a much more useful working model of the sample than a conventional first-pass triage.

The result is still not perfect. In this run, the agent did not recover the hardcoded C2 server or the embedded AES key, and it reported AES-128-ECB even though the sample actually uses AES-256-ECB. However, this miss should be interpreted in context. The workflow did identify the relevant protocol path, the wire format, the use of ECB mode, and the cryptographic routines involved in the C2 communication. In other words, it did not miss the protocol layer entirely; its reconstruction remained incomplete at some of the finer details. At the same time, the run remains non-deterministic: a rerun may recover additional details, emphasize different parts of the sample, or make different mistakes. The result should therefore not be treated as a final report, but as an intermediate state.

Even with these gaps, the improvement is substantial. The structured workflow pushes the agent well beyond loose triage and leaves behind a much stronger intermediate state: a case directory with evidence, ranked hypotheses, priorities, and partially validated internal structure. In practice, that can save an analyst hours to days of early-stage work. Instead of spending most of the time collecting obvious clues and organizing first impressions, we can focus on validating deeper findings, resolving the remaining unknowns, and correcting the places where the agent is still wrong.

Because the workflow preserves explicit case state, follow-up analysis also becomes much sharper. The analyst can guide the agent toward a specific hypothesis, subsystem, or unresolved question, and the agent can continue from the accumulated evidence rather than starting over. It can also annotate the analysis database itself—for example by renaming symbols, attaching notes, and marking relevant functions—so that the human analyst can continue from a better-structured and better-documented starting point.

Closing Remarks

As we’ve seen, agents are already changing reverse engineering in a significant way. In this post, even the unstructured run was already useful as a first triage: it recovered visible functionality, pointed to relevant code locations, and produced concrete leads for follow-up. The real improvement, however, came from giving the agent a structured workflow, persistent case state, and the right analysis tooling. This also shows where agents are already particularly strong: scaling the mechanical parts of analysis, collecting evidence, correlating artifacts, generating scripts, and producing a first structured understanding of a binary far faster than a human could.

And we are still early. Models, tools, and agentic workflows are improving quickly, and the current generation already shows how much repetitive reverse-engineering work can be automated. Over time, this will likely push human analysts away from manually performing every step themselves and more toward guiding workflows, validating results, refining assumptions, and deciding where deeper analysis is needed.

But that does not make human expertise obsolete. Reverse engineering remains an adversarial task: binaries may be unusual, intentionally deceptive, or heavily obfuscated. In those settings, a plausible answer is not the same as a correct one. Agents can help explore and accelerate, but they still struggle when the analysis requires novel reasoning, strong validation, or a change in strategy.

Obfuscation is a good example. Even agentic workflows that perform well on triage and structured analysis can still break down once strong control-flow obfuscation, virtualization, custom packing, or non-standard data encodings enter the picture. At that point, the challenge shifts from simply driving tools to understanding how the protections work, testing hypotheses against the binary, and designing the right custom tooling or deobfuscation workflow. An LLM can help with that process, but the direction still has to come from the analyst.

In that sense, agents do not remove the need for reverse-engineering expertise; they raise the bar for where that expertise matters most. And if there is one area where this becomes especially visible, it is deobfuscation. For analysts who have to deal with protected or obfuscated binaries in practice, I cover many of these workflows in my hands-on training Software Deobfuscation Techniques.

In reverse engineering, API calls provide crucial insights into a program’s behavior. They often perform high-level operations like file access, network communication, string operations or memory management. They help us reverse engineers to build a better understanding of what a program does and how it works. In the context of vulnerability research, we can locate areas of code that are commonly associated with vulnerabilities, like functions related to memory management or user input processing. For malware analysis, we can learn how a malware sample interacts with the operating system, networks, files, etc.; this can help us to understand its purpose and functionality.

However, in some cases, API functions are hard to identify in binaries: for example in statically-linked executables and in embedded firmware. In statically-linked binaries, the code of standard libraries and third-party libraries is incorporated directly into the binary. Since there is no clear separation between the user code and the libraries, without symbols, it is difficult to identify which functions belong to the program and which are from libraries. In the case embedded reverse engineering, firmware is often designed to directly interact with the hardware of a specific device; therefore, it may contain unique API functions which contain similar functionalities as in standard libraries, but are not present in those. Sometimes, manufacturers even use their own proprietary standard library implementations; without access to the source code or documentation, it can be nearly impossible to identify these functions.

In this blog post, we will initially discuss common methods for detecting API functions; then, we’ll evaluate the heuristic and showcase additional use cases for malware analysis. So, if you are curious how to detect API functions in various reversing scenarios, stay along. If you want to play around with the heuristic on hour own, you can use my Binary Ninja plugin obfuscation_detection.

Methods to Detect API Functions

Without symbol information, it remains challenging to identify API functions in statically-linked binaries and embedded firmware for the reasons mentioned above. In practice, two common approaches rely on signature matching and cross-referencing with known libraries.

The concept behind signature matching is straightforward: it involves searching for specific byte sequences—or signatures—of known API functions within a binary. These signatures, stored in a database, each act as a distinct fingerprint for an associated API function. During analysis, the binary is scanned and the byte sequences of functions are compared to the signatures from the database. When a match occurs, it signifies the presence of that particular API function in the binary. Several systems employ this function signature method, such as IDA Pro’s Fast Library Identification and Recognition Technology (FLIRT) and Binary Ninja’s Signature Kit Plugin.

However, the efficacy of signature matching heavily depends on the accuracy of the signature database. This database must include signatures corresponding to the specific library version used in the binary; furthermore, it must be compiled for an identical platform (operating system & CPU architecture) using similar compiler settings. Variations can cause significant differences and lead to mismatches, limiting the usability of such systems in practice.

The second method, cross-referencing with known libraries, employs a comparable strategy. In this method, binary diffing techniques (e.g., BinDiff) are used to compare the binary against different versions of known libraries to locate similarities. When a match is found, the function name and type information from the known library can be transferred to the binary under investigation. Since this method considers additional features like control-flow graph structures and call hierarchies, it allows slight differences in library versions and compiler settings without causing a complete mismatch; this way, it potentially overcomes some limitations of the signature matching approach.

As a downside, this method often carries a significant degree of uncertainty due to the reliance on feature matching, which can lead to false positives and negatives. Furthermore, cross-referencing is still constrained by the availability of a corresponding library. If the binary was compiled with an uncommon or proprietary library for which there is no available version to compare against, these methods would not be effective.

In the following, we will take a look at another simple but effective heuristic, which has proven to be very efficient in many reversing scenarios.

Heuristic: Frequently Called Functions

One heuristic approach with notable potential is the analysis of call frequency: identifying functions within a binary that are invoked frequently by others. This simple technique can yield substantial insights: functions with high call frequencies are often of significant interest in the reverse engineering process, since they often represent core functionalities that the software leans heavily upon.

Often, these frequently called functions are crucial API functions, facilitating a wide range of fundamental operations within a program. Examples include string operations (e.g., strlen, strcmp), memory management functions (e.g., malloc, free, memcpy, and memset), file access methods (e.g., open, close), and network communication (e.g., send, receive). These API functions play pivotal roles within the binary, making them frequent call targets and providing valuable insights into the binary’s operation. Therefore, identifying these high-frequency call targets can offer valuable footholds for understanding the inner workings of the binary.

To implement this heuristic in your disassembler of choice, intuitively, we sort all functions by their number of independent callers in descending order. To focus on the most relevant findings, one might choose to concentrate, for example, on the top 10% of these functions.

In BinaryNinja, this operation can be achieved within a few lines of Python code:

def find_most_frequently_called_functions(bv):
    print("Frequently Called Functions")

    # print top 10% (iterate in descending order)
    for f, num_callers in get_top_10_functions(bv.functions, lambda f: len(f.callers)):
        print(
            f"Function {hex(f.start)} ({f.name}) is called from {num_callers} different functions.")

In the code above, the function find_most_frequently_called_functions takes a binary view bv (a representation of the analyzed binary) as input, and applies the get_top_10_functions function to the list of all functions within the binary. This function ranks the API functions based on the number of independent callers; the result is then printed for the top 10% of functions. This approach surfaces those functions that are most frequently called, making them prime candidates for further investigation during the reverse engineering process.

Similar implementations can be realized in IDA or Ghidra: In IDA, the corresponding functions can be found by iterating over cross-references to a particular function; in Ghidra, the getCallingFunctions method of a function object can be utilized to identify the callers.

However, while our outlined heuristic offers an intuitive approach for identifying API functions in binaries, let us now investigate whether we can validate its effectiveness in various real-world reverse engineering scenarios.

Evaluation

To evaluate the effectiveness of our heuristic in real-world scenarios, we will apply it to a set of diverse case studies:

• the ls command from the Linux Coreutils package,
• a statically-linked malware sample from the XOR DDos family,
• an embedded firmware and
• another malware sample from the PlugX family.

For consistency and comparability, we will limit our analysis to the top 10 identified functions in each case. We anticipate that these functions will reveal commonly invoked API calls. Moreover, our heuristic may also uncover critical functions that underpin the key functionalities or logic of the binary under analysis. In these cases, we’ll delve deeper to understand their roles and operations. With this said, let’s plunge into the results!

Coreutils

Our first case for evaluation is the ls command from the Coreutils package, well-known for generating command line directory listings. Since this binary is dynamically linked, API functions appear as wrappers that enable jumps to functions not included in the binary; at load time, the exact function locations are resolved by the operating system during the dynamic linking phase.

Upon applying our heuristic, we obtain the following results:

Function 0x100007528 (_putchar) is called from 28 different functions.
Function 0x100007522 (_printf) is called from 23 different functions.
Function 0x100007558 (_snprintf) is called from 12 different functions.
Function 0x1000074bc (_getenv) is called from 10 different functions.
Function 0x10000755e (_strcmp) is called from 9 different functions.
Function 0x1000069f0 (sub_1000069f0) is called from 9 different functions.
Function 0x100007582 (_strlen) is called from 8 different functions.
Function 0x100007564 (_strcoll) is called from 8 different functions.
Function 0x10000745c (_err) is called from 8 different functions.
Function 0x1000073fc (___error) is called from 8 different functions.

A quick scan at the results reveals that 9 out of the top 10 identified functions relate directly to API calls. These include operations related to string manipulation (such as _strcmp, _strlen, _strcoll, _snprintf), output to stdout (_putchar, _printf), environment variable parsing (_getenv), and error handling (_err, ___error).

Interestingly, one function in the top-10 list, sub_1000069f0, does not appear to be a typical API function. However, a closer examination reveals that this function acts as a wrapper for the API function _putchar:

1000069f0  55                 push    rbp {__saved_rbp}
1000069f1  4889e5             mov     rbp, rsp {__saved_rbp}
1000069f4  e82f0b0000         call    _putchar
1000069f9  31c0               xor     eax, eax  {0x0}
1000069fb  5d                 pop     rbp {__saved_rbp}
1000069fc  c3                 retn     {__return_addr}

In summary, all of the top 10 identified functions are API-related.

XOR DDos

Our second case study involves a statically-linked malware sample from the XOR DDos family. As indicated by its name, this Linux-targeting malware executes DDos attacks and employs XOR-based encryptions (for strings and C&C server communication). This particular sample retains all its symbols; therefore, it serves as an excellent evaluation target, as we can use these function names to easily confirm whether the identified functions are API functions. The output of our heuristic test on this sample is as follows:

Function 0x8065320 (free) is called from 298 different functions.
Function 0x8066a70 (memcpy) is called from 191 different functions.
Function 0x80662b0 (strlen) is called from 184 different functions.
Function 0x80669b0 (memset) is called from 174 different functions.
Function 0x8063d30 (__libc_malloc) is called from 151 different functions.
Function 0x8053810 (__lll_unlock_wake_private) is called from 148 different functions.
Function 0x8053700 (__lll_lock_wait_private) is called from 122 different functions.
Function 0x8060080 (ptmalloc_init) is called from 114 different functions.
Function 0x80569b0 (__strtol_internal) is called from 99 different functions.
Function 0x80661f0 (strcmp) is called from 93 different functions.

The most frequently called function is free, an API function which deallocates memory on the heap. Other frequently invoked API functions are also related to memory management (__libc_malloc), data movement (memcpy), memory initialization (memset) and string operations (strlen, __strtol_internal, strcmp). Moreover, the appearance of API functions like __lll_unlock_wake_private, __lll_unlock_wake_private and ptmalloc_init among the top-called functions suggests that the malware extensively employs multi-threading. This behavior aligns well with the nature of the sample, which establishes a multitude of network connections to execute DDoS attacks. Once again, all identified functions are API functions.

Embedded Firmware

For our next case study, we delve into embedded firmware reverse engineering. We examine a non-further specified demo application designed for an embedded microcontroller operating on an ARMv8M architecture. This application implements hardware initialization routines as well as basic network communication logic relying on cryptographic techniques. Instead of using a standard library, the firmware employs a custom statically linked library. We’ve compiled the firmware with the highest level of optimization, while retaining function symbols for easier analysis.

These are the results:

Function 0x8081cd8 (memcpy) is called from 379 different functions.
Function 0x808140c (vLoggingPrintfError) is called from 343 different functions.
Function 0x8081d26 (memset) is called from 338 different functions.
Function 0x80625a4 (sys_assert) is called from 307 different functions.
Function 0x80785ae (DbgConsole_Printf) is called from 250 different functions.
Function 0x8040738 (multiply_casper) is called from 247 different functions.
Function 0x807e2a0 (ulSetInterruptMask) is called from 160 different functions.
Function 0x807fd28 (CASPER_MEMCPY) is called from 139 different functions.
Function 0x80821e0 (__mbedtls_mpi_free_veneer) is called from 102 different functions.
Function 0x80511b8 (mbedtls_platform_zeroize) is called from 87 different functions.

The heuristic reveals a wild mix of the most frequently called functions within the embedded firmware: At the top of the list, we find memcpy and memset, reflecting an extensive amount of memory manipulation operations throughout the firmware. This is followed by vLoggingPrintfError and DbgConsole_Printf, which are associated with console output functions, implying a logging or debugging system implemented within the firmware, while sys_assert hints towards a custom error-handling mechanism.

The firmware further includes custom routines to handle memory-related tasks, such as CASPER_MEMCPY, __mbedtls_mpi_free_veneer and mbedtls_platform_zeroize. These functions belong to the cryptographic libraries CASPER and mbedTLS, providing optimized or secure alternatives to standard memory operations. Notably, mbedtls_platform_zeroize offers a secure means to erase sensitive data from memory, a crucial function in cryptographic contexts.

Additionally, multiply_casper provides an optimized version for cryptographic multiplications. Furthermore, ulSetInterruptMask appears to be related to the firmware’s interrupt handling system, a common element in microcontroller-based applications.

In summary, our heuristic provides insights into the firmware’s core functionality by highlighting a mixture of standard library functions, application-specific error handling routines, console output functions, custom cryptographic implementations, and functions related to interrupt management within the firmware.

PlugX

For our final experiment, we take a look at a sample from the PlugX malware family. Compared to the other case studies, due to the lack of function symbols, we have significantly less guidance. To put it differently, we solely rely on our heuristic and manual analysis.

Here are the results obtained from the PlugX sample:

Function 0x10001620 (sub_10001620) is called from 1253 different functions.
Function 0x10001590 (sub_10001590) is called from 1253 different functions.
Function 0x1002cc5e (__seterrormode) is called from 320 different functions.
Function 0x1002cc4e (sub_1002cc4e) is called from 219 different functions.
Function 0x1002d530 (__free_base) is called from 141 different functions.
Function 0x100302f0 (sub_100302f0) is called from 90 different functions.
Function 0x10001530 (sub_10001530) is called from 73 different functions.
Function 0x10003af0 (sub_10003af0) is called from 63 different functions.
Function 0x10004360 (sub_10004360) is called from 61 different functions.
Function 0x10003cb0 (sub_10003cb0) is called from 31 different functions.

Given these results, two observations stand out: Firstly, the only two identifiable API functions are __seterrormode and __free_base; these are responsible for error handling and memory management. Secondly, two functions—sub_10001620 and sub_10001590—are called with a significantly high frequency compared to all other functions in the binary.

Analyzing the calling patterns of these functions reveals a fascinating interplay: Whenever sub_10001620 is invoked, sub_10001590 has always been executed just a few lines before. Even more, the input parameters of sub_10001590 are string values corresponding to Windows APIs, such as kernel32. Finally, we also note that the return value of this function is fed as input to sub_10001620, indicating that the output of the one function serves as input to the other; if we put everything together, we can represent a typical instance of this call structure as follows:

sub_10001620(0, sub_10001590("kernel32"), 0xffc97c1f)

We now understand why both functions are invoked an equal number of times: They are interdependent. Yet, upon closer inspection, we make another noteworthy discovery. The third parameter of the outer function—the constant 0xffc97c1f—in combination with the string kernel32, suggests that this sequence of function calls represents an API hashing routine, a mechanism often utilized by malware to obfuscate API calls from static analysis. This mechanism works by traversing a series of API function names represented as strings, computing their hash values and subsequently comparing the computed hash values to a pre-computed constant during runtime. If a match is found, the corresponding API function is imported and executed.

Without going too much into detail, further analysis reveals that sub_10001590 acts as an obfuscated call to LoadLibraryA, which loads a specified Windows library into the address space of the calling process. Subsequently, sub_10001620 implements a slightly altered version of the CRC32 algorithm, which compares the computed CRC32 hash to the constant input (0xffc97c1f). This workflow perfectly matches the modus operandi of API function hashing.

Using this knowledge, we can now represent the function calls as follows:

crc32(0, LoadLibraryA("kernel32"), 0xffc97c1f) ; "GetProcAddress\x00"

In this context, the constant 0xffc97c1f corresponds to the hash of the GetProcAddress function, which is loaded from kernel32.

To conclude this case study, our heuristic once again identified all key API functions within the analyzed malware sample. Although it doesn’t pinpoint specific API functions as explicitly as it does in the other case studies, the heuristic confirms the underlying assumption that API functions are called frequently. In this particular case, an API hashing routine—designed to import hidden API functions—emerged as the most frequently called function structure in the entire malware sample. As a consequence, the heuristic pinpoints code locations which provide a deep understanding of the malware’s behavior; therefore, it can serve as a potent tool in malware analysis.

In a broader perspective, we can say the heuristic is very effective in detecting frequently used API functions. Even in instances where highlighted functions are no API calls, they often expose other integral functionalities of the binary—for example, cryptographic operations in the embedded firmware sample. The heuristic proves to be not just simplistic and easy to implement, but also extremely efficient. It provides deep insights into binaries, irrespective of the reverse engineering scenario at hand, be it malware analysis, embedded firmware reverse engineering or the analysis of statically-linked executables; it is an invaluable tool in our reverse engineering arsenal.

Closing Remarks

In a series of previous blog posts, I’ve discussed various efficient, architecture-agnostic heuristics that allow us to pinpoint obfuscated code in large binaries. Throughout these blog posts, I’ve often highlighted that these heuristics have a wider scope, proving valuable across a diverse range of reverse engineering scenarios. These observations inspired me to present at REcon, showcasing the versatility of these heuristics in a plethora of contexts. This blog post marks another significant step on that path.

Interestingly, the heuristic introduced today represents a clear departure from previous ones, as it supports a wide range of reverse engineering scenarios, while it is the first not specifically designed to pinpoint obfuscated code. With the same goal of providing useful tools and insights for the entire reverse engineering community, I’ve updated the documentation of my obfuscation detection plugin to offer more intuitive use cases and insights for everyday reverse engineering tasks. If you’re interested, I encourage you to check it out.

My professional work does not only span the domain of code (de)obfuscation—a topic I deeply explore in my training sessions—but also traverses the landscapes of embedded security and malware analysis. Therefore, you can expect the introduction of more potent heuristics designed for a broad spectrum of reverse engineering scenarios over time. So, stay tuned for more exciting things to come. Until our next encounter, thanks for all the fish! :)

Similar to previous blog posts in which we developed a heuristics to pinpoint control-flow flattening and discussed various complexity metrics to pinpoint obfuscation in general, the implementation of these techniques has been added to my Obfuscation Detection plugin for Binary Ninja. As for the other heuristics, the implementation is architecture-agnostic, although it contains some architecture-specific optimizations for x86, x86-64, ARM32 and AARCH64.

In the following, we first familiarize ourselves with the foundations of n-gram analysis and its use cases. Then, we discuss how to perform statistical analysis of assembly code and develop a heuristic to identify uncommon instruction sequences. Afterward, we explore some similarities and differences between the most common CPU architectures. Finally, we evaluate the heuristic by identifying and analyzing obfuscated functions in malware, a Windows kernel module, an anti-cheat software and a mobile DRM system.

n-gram Analysis

The term n-gram originates from the field of computational linguistics. n-grams are used to analyze the underlying structures of a language. For this, we partition a text into fixed-size chunks (so-called grams of size n) and count the number of occurrences of the individual chunks in the text. If the underlying text sample is large enough, we retrieve a representative frequency distribution of the language that gives us insights into common patterns of the underlying language, such as the most common letters or most common words of the language. Furthermore, it allows us to predict which word most likely follows after a given word; a characteristic which is used to support users while typing messages on mobile phones by making suggestions based on the current writing context.

To illustrate this, let us count all 1-grams in the following text:

This is a test.

First, we normalize the text by representing it as a continuous sequence of lower-case letters only:

thisisatest

For the English language, 1-grams are equivalent to individual letters. Therefore, we just count the number of occurrences of all individual letters:

• t: 3
• h: 1
• i: 2
• s: 3
• a: 1
• e: 1

As we can see, the letters ‘t’ and ‘s’ are the most common, ‘a’ and ‘e’ the least common letters. If the sample text was larger—for example, 1,000 words—we would obtain a representative frequency distribution of the English language, in which ‘e’ is the most common letter with 13%, followed by ‘t’ with 9.1% and ‘a’ with 8.2%. These tables are used, for example, in cryptanalysis to break substitution ciphers: By performing a frequency analysis of the ciphertext, we can map the distribution of letters in the ciphertext to the distribution of letters of the English language. As a consequence, we can undo the substitution.

While such 1-grams can already be used to break substitution ciphers, statistical analysis becomes even more powerful if it also considers the context of the letters, for example which letter follows before and afterward in the text. To achieve this, we increase the size of the chunks (the number n) and count these chunks in a sliding window approach. For example, if we want to compute all 2-grams in the text above, we count the chunks as follows:

• th: 1
• hi: 1
• is: 2
• si: 1
• sa: 1
• at: 1
• te: 1
• st: 1

As we can see, ‘is’ is the most-common 2-gram, while all other 2-grams only occur once.

We could continue this approach and compute all 3-grams, 4-grams, 5-grams etc. Now the question is: What n should be used for statistical analysis? While there is no definite answer to this, we can say that the larger the n, the more context we consider. While 1-grams don’t consider any context, 7-grams and more usually tend to overfit to the input text and lose universality. In my experience, 3-grams or 5-grams provide a good trade-off between considering the context and keeping universality. In the following, we perform all n-gram analyses on the assembly level as 3-grams.

Statistical Analysis of Assembly Code

Similarly to natural languages, we can perform a statistical analysis on machine code and assembly code to identify underlying patterns and make predictions based on these patterns. For example, based on the byte frequency of individual bytes in the binary, we can differentiate between code and data; furthermore, we can identify packed code by computing the entropy of individual sections. Closer to the n-gram analysis discussed above, statistical analysis also supports the reverse engineering of unknown instruction set architectures: By mapping byte distributions to common instruction sequences, it is possible to identify the byte representation of the most common instructions, such as the architectural equivalent of mov or call instructions.

In our case, as stated above, the goal is to identify uncommon instruction sequences in binaries. For this, we rely on the disassembly output of one of the major disassemblers, such as IDA Pro, Ghidra or Binary Ninja. Now, we need to accomplish two things:

1. We need to compute common 3-gram frequencies of assembly code as ground truth.
2. We need a heuristic to identify code patterns which deviate from the most common 3-grams.

For the sake of simplicity, we rely on x86-64 bit assembly to illustrate how to achieve these goals. Later on, we discuss how such a heuristic can be implemented in an architecture-agnostic way.

Obtaining a Ground Truth

To obtain a ground truth, we need to compute common 3-grams of the assembly language. However, to avoid overfitting to specific binaries or architectures, we consider instruction opcodes only and omit instruction details such as concrete registers or memory addresses. For example, consider the following code sequence:

push    rbp
mov     rbp, rsp
push    r14
push    rbx
test    rdi, rdi

To perform a 3-gram analysis, we first remove all the instruction details and obtain the following sequence:

push
mov
push
push
test

Based on this, we can compute the following 3-grams:

• push ; mov ; push
• mov ; push ; push
• push ; push ; test

If we perform this analysis over a large set of binaries from different categories (for example compilers, debug tools, multimedia libraries etc.), we get a representative collection of 3-grams, from which we can use, for instance, the 1,000 most common 3-grams as a ground truth.

Identification of Uncommon Instruction Sequences

To identify uncommon instruction sequences, we intuitively want to find code locations whose instruction sequences highly deviate from the generated ground truth. However, for this, we need a well-defined window in which we perform the analysis. The most straightforward approach is to perform this analysis on the function level. As a consequence, we re-phrase our goal as follows:

Our goal is to identify functions whose instruction sequences deviate the most from the generated ground truth. For this, we perform the following steps for each function:

1. We count the number of 3-grams in the function which are not part of the ground truth.
2. We divide this number by the total number of 3-grams in the function.

This way, we can assign a score to each function; this score represents the relative amount of 3-grams which are not contained in the ground truth. The higher the score, the more do the instruction sequences in the function deviate from the ground truth.

Given that each function has a score representing the deviation from the ground truth, we can sort all functions by their score—analogous to the other detection metrics—and manually inspect the top 10% of these functions.

Architecture-agnostic Implementation

To realize this heuristic in an architecture-agnostic way, the implementation relies on Binary Ninja’s low-level intermediate language (LLIL). This intermediate language is a common representation of assembly code for all architectures supported by Binary Ninja. During analysis, Binary Ninja lifts assembly instructions into this LLIL, which will then be used for further analysis passes, up to its decompiler output.

To exemplify, the architecture-dependent x86-64 sequence

mov
mov
mov

translates to LLIL as follows:

LowLevelILOperation.LLIL_SET_REG
LowLevelILOperation.LLIL_SET_REG
LowLevelILOperation.LLIL_SET_REG

where the operation LLIL_SET_REG represents an architecture-independent assignment to a register.

We can use this implementation to detect uncommon instruction sequences on all architectures supported by Binary Ninja. However, this approach has a downside: Since it relies on LLIL, it enforces the lifting of all instructions in the binary, which increases the analysis time. On the contrary, an analysis on the assembly level would be significantly faster, but requires a ground truth for each architecture.

As a consequence, the implementation is a trade-off between universality and speed: For the architectures x86, x86-64, ARM32 and ARM64, there are custom ground truths and analyses based on the disassembly opcodes; for all other architectures, the implementation relies on Binary Ninja’s LLIL.

To build a representative ground truth for a wide variety of common applications, the Linux coreutils & binutils, compiler GCC, debugger GDB, image manipulation program GIMP as well as the multimedia player mplayer were used. To cover 3-gram variations of different architectures, all projects have been compiled for the four architectures mentioned above and the according 3-grams have been computed.

Similarities and Differences between Architectures

Before we evaluate the heuristic on various real-world scenarios, let’s take some time and explore some of the similarities and differences between different architectures for fun. However, keep in mind that the following observations only scratch the surface and try to give a better intuition. That said, let us take a closer look at the computed ground truths.

At a first glance, we can see that, across all architectures, the most common instruction sequences are data movement chains, sometimes combined with arithmetic operations or function calls. Furthermore, we can see that all architectures include 3-grams of architecture-specific push and pop operations, most to be found in function prologues and epilogues. In addition, they all include several architecture-specific add chains, due to offset calculations and pointer arithmetics.

Some of the differences, however, are quite more interesting. For instance, the ARM-based architectures contain significantly more load/store operations; the reason for that is that, in most cases, these architectures read and write memory using dedicated instructions, while the x86 and x86-64 architectures can access memory in nearly all data movement and arithmetic operations. Another difference can be seen between the x86 and the x86-64 architecture: The 32-bit architecture contains significantly more push ; call chains. This is due to differences in the calling conventions: the 32-bit architecture uses the stack to pass function arguments, while the 64-bit architecture mostly uses registers.

In short, we can say that many low-level patterns such as function management, calls, data movement and pointer arithmetic are architecture-agnostic, while there are architecture-specific concepts which lead to differences in the most common instruction patterns. This might also have an impact on the architecture-agnostic ground truth in LLIL, if the boundary for the most common 3-grams is set too low. For this reason, the ground truth in LLIL stores the 3,000 most common LLIL 3-grams obtained for all architectures, while the architecture-specific implementations only store the 1,000 most common 3-grams.

Finally, let us now look at some real-world binaries and evaluate what kind of instruction sequences can be identified by the heuristic.

Evaluation

To understand what kind of code locations these heuristics identify, we take a closer look at several binaries that are only partially obfuscated; this means, we use binaries which contain mostly compiler-generated code, but are obfuscated in specific parts to hide underlying computations. In the following sections, we analyze a malware sample, a module from the Window kernel, an anti-cheat system and a mobile DRM system. Note that some of these binaries will not be specified further; however, although we omit some details, the discussion will still provide some intuition and strategies of what to expect and how to proceed. So, to reproduce the results, feel free to pick the anti-cheat software or DRM system of your choice and follow along. Now, let’s get into this and discuss the results.

Malware

The chosen malware sample belongs to the emotet family and has been already described in a previous blog post, in which we analyzed its control-flow flattening implementation. If we use the heuristic to identify uncommon instructions sequences, it identifies 14 functions; while some of them implement control-flow flattening, the majority (11) of them share a structure similar to the following code snippet:

push    ecx {var_4_6}
mov     dword [esp {var_4}], 0x8f8e
shl     dword [esp {var_4_1}], 0x3  {0x47c70}
xor     dword [esp {var_4_2}], 0xe2f0cbde  {0xe2f4b7ae}
xor     dword [esp {var_4_3}], 0x3a546a44  {0xd8a0ddea}
add     dword [esp {var_4_4}], 0xffff377d  {0xd8a01567}
or      dword [esp {var_4_5}], 0xe5ad4c54  {0xfdad5d77}
xor     dword [esp], 0xfdad5c37  {0x140}
mov     eax, dword [esp]  {0x140}
pop     ecx  {0x140}
retn     {__return_addr}

As we can see, the function initializes some variables on the stack. If we analyze where and how these functions are called, we learn that they are part of the control-flow flattening obfuscation and used to update the state variable which encodes the control flow.

Overall, every function identified by the heuristic is obfuscated or pinpoint a helper function managing the obfuscated state.

Windows Kernel

Next, let us take a closer look at ci.dll, the Windows kernel module responsible for code integrity and file authentication. As documented by the Airbus Security Lab, this kernel module implements an AES white-box, which is protected by a virtualization-based obfuscator, the Warbird virtual machine. Note that the concept of virtualization-based obfuscation (also known as virtual machines or VMs) and how to break them has been covered in a previous blog post.

With our heuristic, overall we identify 206 out of 2,065 functions (the top 10%, as described above). Nearly half of these functions are related to cryptographic operations, as we can deduce from by their symbol names (for example SymCryptSha256AppendBlocks_shani); however, 125 of them follow a different pattern: While their symbol name does not reveal any information, their overall code structure is similar to the following code snippet:

mov     r10, rcx
lea     r9d, [rdx+0x62d7be82]
lea     eax, [rdx+0x3853ae71]
movzx   ecx, r9b
xor     eax, r9d
xor     rcx, 0x76
add     eax, 0x64cf8a2f
mov     r8, rdx
shr     r8, 0x20
sub     r8d, edx
shl     rax, 0x20
mov     rcx, qword [r10+rcx*8]
add     r8d, 0x4547fd6
shl     r8, 0x20
or      rax, 0x5d304937
or      r8, 0x1d570d9f
cmp     qword [r10+0x128], rcx
cmove   rax, r8
retn     {__return_addr}

These functions implement handlers of the Warbird VM. As we can see, the handler encodes its data flow using constant unfolding (e.g., writing x + 5 as x + 10 - 5) and conditional moves. However, if we take a closer look we notice [r10+rcx*8], which might access a 64-bit jump table at address r10 to which the offset rcx is added, probably computing the address of the next VM handler.

In summary, we can say that the heuristic identified uncommon instructions sequences; 61% of these functions are part of the Warbird VM, while the others implement cryptographic operations.

Anti-Cheat

After having examined a malware sample and a Windows kernel module, let us now look into an anti-cheat software, which is used to detect cheating attempts in online games. Due to its size, this time, the heuristic finds 3,433 out of 34,321 functions in total. While we do not want to manually inspect 3,433 functions, we can immediately identify a common structure by exploring some of these:

pop     rsp
lea     rsi, [rel 0x141f2c3e6]
mov     rbx, r11
movq    rdi, xmm6
sub     r9, 0x4
movq    xmm7, rbx
movq    rbp, xmm7
mov     qword [rbp], rdi
movsxd  rax, dword [rsp]
add     rsp, 0x4
add     rsi, rax
jmp     rsi

We have hundreds of identified functions which end in indirect jumps and compute the target address by adding an offset to a fixed constant. This seems to be some kind of control-flow encoding; however, if you spend more time with virtualization-based obfuscation, this pattern might also remind you of a technique which is known as threaded code: At the end of each VM handler, the dispatcher is inlined and computes the address of the next handler. While we cannot know for sure without further analysis, there is a good chance that the anti-cheat software is protected via virtualization-based obfuscation, which has been located by the heuristic.

Mobile DRM

Finally, let us take a look at a mobile DRM system which plays encrypted multi-media content. Given its size, we again identify a large number of functions (approximately 3,000) with our heuristic. This time, we have a hard time manually inspecting the functions; randomly looking at some function does not reveal any patterns with regards to similar code structures, graph layouts or others. Without further knowledge, way more analysis effort is required to get a better understanding.

However, this does not mean that the heuristics did not identify interesting functions. For example, we identified a lot of functions using a wide amount of floating-point instructions, which it not uncommon for multi-media applications. Furthermore, we pinpointed functions making extensive usage of hardware-based encryption instructions, indicating that these functions might perform some kind of content decryption or key derivation. Finally, we also identified some functions protected with arithmetic obfuscation or mixed Boolean-Arithmetic (MBA) expressions, indicating that we are looking at the right places.

Overall, we can summarize that the heuristic reliably pinpoints uncommon instruction sequences; in certain binaries, this is often obfuscated code. Furthermore, we have seen that choosing the top 10% of all functions might be a bit overwhelming for large binaries. In some cases, the sheer amount of functions can be managed by detecting repetitive obfuscation patterns, which allows an effortless classification of the identified functions. In other cases, significantly more effort is required. However, even in these cases, the heuristic can be useful. Digging into large binaries is always like looking for a needle in a haystack; each heuristic performing a reasonably helpful pre-selection speeds up reverse engineering.

Closing Remarks

In this blog post, I tried to give an intuition for the power of statistical analysis, not only in the context of code obfuscation. I’ve always been fascinated by statistical analysis and its ability to reveal underlying structures, which makes it a good fit for reverse engineering. Although I assumed after my last blog post on obfuscation detection that this series would be finished, I came up with the idea in a discussion with friends and implemented a prototype; after some experiments, I implemented it properly in an architecture-agnostic way. While the results certainly overlap (at least partially) with the other heuristics, it is the first which does not rely on complexity metrics, adding another dimension to the analysis. Furthermore, it is fun to explore the differences and oddities between architectures, while learning that overall, they are very similar on a higher level.

A further perspective of this post is to show how to classify results identified by such heuristics. While we stopped analysis rather quickly, the intention was to provide some insights and strategies how one can reason about obfuscation, starting from scratch. Further analysis until deobfuscation and understanding of the underlying protected code is an iterative process, which I partially covered in other blog posts, such as the post on breaking virtualization-based obfuscation and the post on simplifying MBA-based obfuscation. Furthermore, I cover such topics and strategies in-depth in my training classes on code deobfuscation.

((x ^ y) + (2 * (x & y))) + ((x ^ z) + (2 * (x & z)))

What does it compute? You will see, it’s not very complicated; it is (x + y) + (x + z).

What we can observe here is one of the most sophisticated code obfuscation techniques which is known as mixed Boolean-Arithmetic (MBA). MBAs arithmetically encode semantically simple operations in complex arithmetic expressions to hide the underlying computations. In general, it is extremely difficult to simplify such expressions, since there are endless possibilities to construct them. Nevertheless, recent research advanced in breaking certain instances of MBAs by using algebraic simplifications, program synthesis, machine learning and the exploitation of mathematical characteristics. However, one of the most generic and powerful attacks on MBAs is known as QSynth (official framework: QSynthesis) and has been introduced by Robin David and Luigi Coniglio. Earlier this year, I published my MBA deobfuscation framework msynth which incorporates similar ideas; it is specifically designed to work on real-world binaries and can simplify most MBAs found in the wild.

From this blog post, you will get a better understanding of the ideas behind msynth and learn how to use it to simplify MBA expressions in real-world binaries. Who knows, it may assist your day-to-day malware analysis and code deobfuscation tasks or just give you the ability to play around with MBA deobfuscation.

Simplification Approach

Given a complex expression like the one above, our goal is to find a shorter expression that exhibits the same input-output behavior. In our case, (x + y) + (x + z) is such a simpler expression: No matter which inputs we assign to x, y and z, both expressions always compute to the same results. To find such expressions, intuitively, we want to identify parts of the complex expression—so-called subexpressions—and repeatedly replace them with simpler representations. However, to keep this process efficient, we first collect a large number of relationships between complex and simple expressions. In short, our simplification approach uses two components:

• A database of arithmetic simplification identities
• An algorithm which recursively walks over all subexpressions

The database can be understood as a pre-computed lookup table mapping input-output behavior to the smallest possible expression. For example, if we know that the input tuple (10, 20) evaluates to 30 (written as (10, 20) -> 30) and (10, 10) evaluates to 20 (written as (10, 10) -> 20), then we can map this behavior to an addition of two variables, for example x + y. However, we would not map this behavior to an expression such as x + y + 5 - 5, since it is not the shortest representation.

The simplification algorithm recursively walks over all subexpressions–from the largest to the shortest—and evaluates the subexpression’s input-output behavior. Afterward, we query the database whether a shorter entry exists for this behavior. If so, we replace the current subexpression with the one from the database and stop; otherwise, we continue and go to the next subexpression. We repeat this process, until we cannot simplify any more subexpressions.

If this sounds confusing, don’t worry: Let us illustrate the interplay between algorithm & database and gradually simplify the expression above. First, let us assume we have the following lookup table, in which each input-output pair uses an input tuple of length 3 and in which the expression can use the variables x, y and z:

(10, 20, 30) -> 30: x + y
(10, 20, 30) -> 40: x + z

The database contains two entries; one entry maps an input vector to x + y, another one to x + z.

To simplify the complex expression, we now derive all of its subexpressions, sort them by their lengths in descending order and iteratively walk over the following list:

((x ^ y) + (2 * (x & y))) + ((x ^ z) + (2 * (x & z)))
(x ^ y) + (2 * (x & y))
(x ^ z) + (2 * (x & z))
2 * (x & y)
2 * (x & z)
x ^ y
x & y
x ^ z
x & z
x
y
z

Starting with the original expression, we evaluate it with (10, 20, 30) and obtain 70. Since (10, 20, 30) -> 70 is not in our database, we continue with the next subexpression, (x ^ y) + (2 * (x & y)). This time, it evaluates to 30, giving us (10, 20, 30) -> 30. This entry maps to x + y. Therefore, we replace the subexpression in the complex expression with x + y, which leads us to:

(x + y) + ((x ^ z) + (2 * (x & z)))

Then, we re-start the algorithm, now beginning with the shorter expression. Once we reach the subexpression (x ^ z) + (2 * (x & z)), we evaluate it to (10, 20, 30) -> 40. If we perform a database lookup for this entry, we learn that it is equivalent to x + z. As a consequence, we again replace the subexpression and obtain:

(x + y) + (x + z)

In a final run, we re-start the process one last time. However, since we cannot simplify any further subexpressions, the algorithm terminates.

This example illustrates the deobfuscation process quite well. However, to be usable in real-world scenarios, we have to take care of a few things:

• If the pre-computed database contains only a handful of entries, simplification will not be very effective. In general we can say: The more entries the pre-computed database contains, the higher the success rate. To derive more database entries, we can perform an exhaustive search over all possible combinations of variables and mathematical operations up to a certain upper bound. Additionally we can generate random expressions of higher complexity and also add them to the database.

• It is error-prone to use only a single input-output pair for database lookups. Consider the following example: The input-output pair (2, 2) -> 4 maps to more than one expression; it holds for x + y, x + x, x * y and others, which all have the same length. To circumvent this and increase the accuracy, we have to use a higher number of input-output pairs. In practice, numbers between 10 and 30 are sufficient. While higher numbers further increase the accuracy, they slow down the simplification process enormously. To exclude false positives entirely, we can use an SMT solver to check if the subexpression and the matching expression from the database are equivalent.

• Expressions on the binary level often use up/downcasts to realize, for example, 32-bit operations on 64-bit architectures. To simplify such arithmetic constructs, we have to include type casts in our pre-computed lookup tables.

With that in mind, let us now dive into msynth and use it to break some MBAs.

In Practice

msynth is an MBA deobfuscation framework built on top of the Miasm reverse engineering framework, which we already used in my last blog post on VM-based deobfuscation. It ships with a pre-computed database (referred to as oracle) including common constants and type casts. To simplify MBAs in binaries, msynth can be combined with Miasm’s symbolic execution engine to derive expressions from assembly code. Example scripts for this can be found in msynth’s repository and in the repository of one of my code deobfuscation workshops.

Let us now first have a look at how we can use msynth to simplify arbitrary MBAs. Then, we combine it with symbolic execution to break an MBA on the binary level. Afterward, we examine how msynth can be used to deobfuscate MBAs in malware samples. Finally, we see that msynth can even be useful in analysis scenarios where the code is not necessarily obfuscated.

MBA Simplification

To simplify an MBA expression, we need msynth’s Simplifier class which has to be initialized with a file path to the pre-computed database. Then, we can simplify an expression in Miasm IR by calling simplify. Overall, this only takes three lines of code:

from msynth import Simplifier

# initialize simplifier
simplifier = Simplifier(oracle_path)
# simplify expression
simplified = simplifier.simplify(expression)

If we add some more lines, we turn the snippet into a fully-working script that simplifies our MBA from before (a similar script which can be used for own experiments can be found here):

from miasm.expression.expression import *
from msynth import Simplifier

def mba(size):
    # ((x ^ y) + (2 * (x & y))) + ((x ^ z) + (2 * (x & z)))
    x = ExprId("x", size)
    y = ExprId("y", size)
    z = ExprId("z", size)
    return ((x ^ y) + (ExprInt(2, size) * (x & y))) + ((x ^ z) + (ExprInt(2, size) * (x & z)))

# path to pre-computed lookup table
oracle_path = "./oracle.pickle"
# initialize simplifier
simplifier = Simplifier(oracle_path)
# simplify expression
simplified = simplifier.simplify(mba(32))

# print results
print(f"original: {mba(32)}")
print(f"simplified: {simplified}")

The output looks as follows:

original: (x ^ y) + 0x2 * (x & y) + (x ^ z) + 0x2 * (x & z)
simplified: y + z + x * 0x2

msynth does not only simplify the MBA, the resulting expression is even shorter than (x + y) + (x + z). This is due to Miasm’s symbolic execution engine which simplifies x + x to x * 0x2. If we aim at simplifying more complex MBAs, we can easily modify the MBA functions and for example receive:

original: ((x ^ 0xFFFFFFFF) | z) + -(((((((0x1 + z + -0x1) | (x ^ 0xFFFFFFFF)) + -(x ^ 0xFFFFFFFF)) ^ 0xFFFFFFFF) & z) + z + (z & (z ^ 0xFFFFFFFF))) ^ 0xFFFFFFFF) + (x & ((x & y) + (x & y) + (x ^ y))) + (x & ((x & y) + (x & y) + (x ^ y))) + (x ^ ((x & y) + (x & y) + (x ^ y))) + 0x1 + ((((y + ((y ^ 0xFFFFFFFF) & z)) | (y + ((y ^ 0xFFFFFFFF) & z) + (((y + ((y ^ 0xFFFFFFFF) & z)) ^ 0xFFFFFFFF) & z))) + -(y & (y + ((y ^ 0xFFFFFFFF) & z)))) ^ 0xFFFFFFFF) + --x + ((y + ((y ^ 0xFFFFFFFF) & z) + (((y + ((y ^ 0xFFFFFFFF) & z)) ^ 0xFFFFFFFF) & z)) & (z ^ 0xFFFFFFFF)) + ((x ^ 0xFFFFFFFF) | z) + -(((((((0x1 + z + -0x1) | (x ^ 0xFFFFFFFF)) + -(x ^ 0xFFFFFFFF)) ^ 0xFFFFFFFF) & z) + z + (z & (z ^ 0xFFFFFFFF))) ^ 0xFFFFFFFF) + (x & ((x & y) + (x & y) + (x ^ y))) + (x & ((x & y) + (x & y) + (x ^ y))) + (x ^ ((x & y) + (x & y) + (x ^ y))) + 0x1 + ((((y + ((y ^ 0xFFFFFFFF) & z)) | (y + ((y ^ 0xFFFFFFFF) & z) + (((y + ((y ^ 0xFFFFFFFF) & z)) ^ 0xFFFFFFFF) & z))) + -(y & (y + ((y ^ 0xFFFFFFFF) & z)))) ^ 0xFFFFFFFF) + --x + ((y + ((y ^ 0xFFFFFFFF) & z) + (((y + ((y ^ 0xFFFFFFFF) & z)) ^ 0xFFFFFFFF) & z)) & (z ^ 0xFFFFFFFF)) + ((x ^ 0xFFFFFFFF) | z) + -(((((((0x1 + z + -0x1) | (x ^ 0xFFFFFFFF)) + -(x ^ 0xFFFFFFFF)) ^ 0xFFFFFFFF) & z) + z + (z & (z ^ 0xFFFFFFFF))) ^ 0xFFFFFFFF) + (x & ((x & y) + (x & y) + (x ^ y))) + (x & ((x & y) + (x & y) + (x ^ y))) + (x ^ ((x & y) + (x & y) + (x ^ y))) + 0x1 + ((((y + ((y ^ 0xFFFFFFFF) & z)) | (y + ((y ^ 0xFFFFFFFF) & z) + (((y + ((y ^ 0xFFFFFFFF) & z)) ^ 0xFFFFFFFF) & z))) + -(y & (y + ((y ^ 0xFFFFFFFF) & z)))) ^ 0xFFFFFFFF) + --x + ((y + ((y ^ 0xFFFFFFFF) & z) + (((y + ((y ^ 0xFFFFFFFF) & z)) ^ 0xFFFFFFFF) & z)) & (z ^ 0xFFFFFFFF))
simplified: z * 0x3 + (x + y) * 0x6

MBA Challenge

Since we know how to simplify MBAs, we go a step further and combine it with Miasm’s symbolic execution engine. For this, we rely on the script symbolic_simplification.py from msynth’s repository as an example. Provided with a binary, an address and a path to the pre-computed database, the script symbolically executes a single basic block at the given address and simplifies all derived expressions. To use it, we have to call it as follows:

python symbolic_simplification.py <binary> <address> <path to database>

Let us try this on an artificial MBA challenge which is inspired by a commercial protector. In particular, let us simplify a function-level MBA which returns the result in rax and uses rdi, rsi and rdx as inputs:

00001200  89f1               mov     ecx, esi
00001202  f7d1               not     ecx
00001204  21d1               and     ecx, edx
00001206  01f1               add     ecx, esi
00001208  4189c8             mov     r8d, ecx
0000120b  21ce               and     esi, ecx
0000120d  41f7d0             not     r8d
00001210  8d043e             lea     eax, [rsi+rdi]
00001213  4121d0             and     r8d, edx
00001216  f7d2               not     edx
00001218  4101c8             add     r8d, ecx
0000121b  4421c2             and     edx, r8d
0000121e  4409c1             or      ecx, r8d
00001221  01d0               add     eax, edx
00001223  29c8               sub     eax, ecx
00001225  c3                 retn     {__return_addr}

Let’s also have a look at its decompilation:

uint64_t mba3(int32_t arg1, int32_t arg2, int32_t arg3) {

    int32_t rcx_2 = (not.d(arg2) & arg3) + arg2
    int32_t r8_3 = (not.d(rcx_2) & arg3) + rcx_2
    return zx.q((arg2 & rcx_2) + arg1 + (not.d(arg3) & r8_3) - (rcx_2 | r8_3))
}

From looking at the expression, we don’t get any clue what it computes. However, if we symbolically execute it with Miasm and simplify it with msynth, we get the following output (for the return value in rax):

before: {(((((((RSI[0:32] ^ 0xFFFFFFFF) & RDX[0:32]) + RSI[0:32]) ^ 0xFFFFFFFF) & RDX[0:32]) + ((RSI[0:32] ^ 0xFFFFFFFF) & RDX[0:32]) + RSI[0:32]) & (RDX[0:32] ^ 0xFFFFFFFF)) + -(((((((RSI[0:32] ^ 0xFFFFFFFF) & RDX[0:32]) + RSI[0:32]) ^ 0xFFFFFFFF) & RDX[0:32]) + ((RSI[0:32] ^ 0xFFFFFFFF) & RDX[0:32]) + RSI[0:32]) | (((RSI[0:32] ^ 0xFFFFFFFF) & RDX[0:32]) + RSI[0:32])) + (RDI + {(((RSI[0:32] ^ 0xFFFFFFFF) & RDX[0:32]) + RSI[0:32]) & RSI[0:32] 0 32, 0x0 32 64})[0:32] 0 32, 0x0 32 64}


[snip]

simplified: {-RDX[0:32] + RDI[0:32] + RSI[0:32] 0 32, 0x0 32 64}

We clearly see that the the simplified code performs an addition of rdi and rsi and subtracts rdx. However, these operations are only performed on the register’s lower 32-bit (denoted by [0:32]); thus, the final result is extended (with zeros) to 64-bits. If we want to clean up the decompiler output, we could rewrite it as follows:

uint64_t mba3(int32_t arg1, int32_t arg2, int32_t arg3) {
    return zx.q(arg1 + arg2 - arg3)
}

Malware Analysis

Leaving the artificial examples behind, let us now use msynth to support us in the process of malware analysis. Often, malware is obfuscated with frameworks such as Obfuscator-LLVM, which commonly deploy MBAs. For example, the FinSpy sample—which we already used in a prior blog post—uses MBAs to to hide data movements based on the conditional move instruction cmov. If we can simplify the MBA, we know whether the data will be conditionally moved or not.

To deobfuscate MBAs within a basic block, we call the script as we did in the example before:

$ python symbolic_simplification.py samples/finspy 0x4035af oracle.pickle

After we symbolically executed the basic block at 0x4035af and simplified it with msynth, we get the following output:

before: (((@32[@64[0x614F40]][31:32]?({@32[@64[0x614F40]] 0 32, 0xFFFFFFFF 32 64},{@32[@64[0x614F40]] 0 32, 0x0 32 64})) * ((@32[@64[0x614F40]] + 0xFFFFFFFF)[31:32]?({@32[@64[0x614F40]] + 0xFFFFFFFF 0 32, 0xFFFFFFFF 32 64},{@32[@64[0x614F40]] + 0xFFFFFFFF 0 32, 0x0 32 64})))[0:32] & 0x1)?({(((({((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8} ^ 0x1) | 0x1) ^ 0x1) & 0x1) | {((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8} 0 8, 0xFFFFFF 8 64},{({((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8} ^ 0x1) | {((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8} 0 8, 0xFFFFFF 8 64})

[snip]

simplified: 0xFFFFFF01

While the original symbolic expression is long and complex, msynth simplifies it to the constant 0xFFFFFF01. In another example, we can simplify a similarly complex expression to the constant 0x0:

before: ((((@32[@64[0x614F40]][31:32]?({@32[@64[0x614F40]] 0 32, 0xFFFFFFFF 32 64},{@32[@64[0x614F40]] 0 32, 0x0 32 64})) * ((@32[@64[0x614F40]] + 0xFFFFFFFF)[31:32]?({@32[@64[0x614F40]] + 0xFFFFFFFF 0 32, 0xFFFFFFFF 32 64},{@32[@64[0x614F40]] + 0xFFFFFFFF 0 32, 0x0 32 64})))[0:32] & 0x1)?(((((({((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8} ^ 0x1) | 0x1) ^ 0x1) & 0x1) | {((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8}) & 0x1,(({((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8} ^ 0x1) | {((@32[@64[0x614F90]] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)) & (@32[@64[0x614F90]] ^ 0xA))[31:32] ^ (@32[@64[0x614F90]] + 0xFFFFFFF6)[31:32] 0 1, 0x0 1 8}) & 0x1))?(0x0,0x1)

[snip]

simplified: 0x0

Complex Code

Sometimes we have to analyze code which is not necessarily obfuscated but so complex, for example due to aggressive compiler optimizations, that we don’t have any clue of what is going on. In these cases, msynth can also support us in our analysis by removing at least some complexity in the expressions.

As an example, let us use the Windows kernel ntoskrnl.exe which we already used in a previous blog post to detect PatchGuard. If we take a closer look at PatchGuard’s initialization function, we find a repetitive pattern that is rather complex. This pattern might be guarded by MBAs, but it might also be just a piece of weird code whereof we currently can’t determine the functionality, given our current level of knowledge. However, even in this case, msynth allows us to massively simplify the expression:

before: {((({(({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[0:32] ^ ({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[64:96]) >> 0xF 0 32, 0x0 32 64} * 0x4EC4EC4F)[32:64] >> 0x3)[31:32]?({({(({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[0:32] ^ ({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[64:96]) >> 0xF 0 32, 0x0 32 64} * 0x4EC4EC4F)[32:64] >> 0x3 0 32, 0xFFFFFFFF 32 64} * 0x1A,{({(({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[0:32] ^ ({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[64:96]) >> 0xF 0 32, 0x0 32 64} * 0x4EC4EC4F)[32:64] >> 0x3 0 32, 0x0 32 64} * 0x1A))[0:32] 0 32, 0x0 32 64}

[snip]

simplified: {(({(({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[0:32] ^ ({RDI 0 64, 0x0 64 128} * {((RAX | (RDX << 0x20)) >>> 0x3) ^ (RAX | (RDX << 0x20)) 0 64, 0x0 64 128})[64:96]) >> 0xF 0 32, 0x0 32 64} * 0x4EC4EC4F)[32:64] >> 0x3) * 0x1A 0 32, 0x0 32 64}

Although the simplified expression is still complex and doesn’t tell us much without further analysis, the expression is significantly better to read due to its eliminated subexpressions.

Overall, the aforementioned scenarios are only example usages for msynth. In practice, you may stumble upon a wide variety of MBAs or other complex expressions of which you suspect shorter representations to exist. Try it on your own; msynth does not solve all of your problems, but it may be a useful tool that supports your analysis.

Closing Remarks

I wrote msynth with the goal to create a stable and usable MBA deobfuscation framework that supports me in my day-to-day reverse engineering tasks; it has been used in audits of various obfuscation schemes as well as in the analysis of some malware samples; furthermore, my students use it regularly in my code deobfuscation training to break sophisticated protection schemes such as virtual machines with MBA-obfuscated handlers. msynth isn’t perfect (especially in terms of performance for very large expressions), but it will improve over time. Furthermore, it is a playground for me (and hopefully others) to test and develop novel deobfuscation techniques which will be integrated over time. So, have fun with it and stay tuned!

While we cover only a basic implementation of virtualization-based obfuscation, we focus on a generic approach to build disassemblers that scale for most virtual machines that can be found in the wild. As a leading example for this blog post, we will use a simple virtual machine that protects a Fibonacci implementation using the Tigress C obfuscator. In the end, we will also have a look at how the original code can be reconstructed from the VM disassembly.

If you would like to play around with the code (based on Miasm) and samples, you can find them here.

Virtual Machines

Virtual machines (VM) belong to the most sophisticated obfuscation schemes, since they create an additional layer of complexity for reverse engineers: By introducing a custom instruction set architecture (ISA) in software (similar to the Java virtual machine), they hide the original code in a sequence of bytes (the so-called bytecode) that is interpreted at runtime. In other words, the bytecode is an array of bytes that encodes the to-be-protected program as machine code; without further knowledge of the virtual machine implementing the custom instruction set architecture, the bytes aren’t meaningful for a reverse engineer. To give them a meaning, we have to fully understand the VM and its execution flow, such that we can build a disassembler for the custom instruction set and decode the bytecode.

To achieve this, let us start with opening a virtual machine in the disassembler of our choice. For the simplest virtual machines, we see a control-flow graph like the following:

We see a control-flow graph that has a central basic block—the dispatcher—which directs the control flow to individual basic blocks, which jump back to the dispatcher. While this reminds us of control-flow flattening, there is a fundamental difference: In the case of control-flow flattening, the individual basic blocks have a 1:1 mapping to the original code. However, for virtual machines, these basic blocks are known as handlers and implement the individual instruction semantics of the custom instruction set architecture, such as virtual addition or push/pop instructions.

On a high-level, the virtual machine operates as follows: Starting at the entry basic block, it backups the native CPU registers and initializes the VM state. Afterward, it walks over the bytecode array, iteratively fetches some bytes, decodes the corresponding instruction and executes the handler that implements the virtual instruction. The fetch-decode-execute (FDE) process is performed in a loop until the VM reaches a specific handler that restores the native CPU registers and leaves the VM. This handler is called the VM exit. Given this context, we can annotate the control-flow graph as follows:

Internally, virtual machines often use additional data structures to preserve their internal state (like storing native registers and intermediate values). Two of the most important data structures are the virtual instruction pointer and (sometimes) the virtual stack pointer. Similar to a native instruction pointer register (such as rip on x86-64), the virtual instruction pointer points to the current instruction in the bytecode and is used to decode operands and keep track of the VM execution flow. Analogously, the virtual stack pointer keeps track of a VM-internal stack that may be used to store intermediate values.

Overall, we can understand a VM as an obfuscated function with a custom calling convention: It takes some parameters as inputs and calculates one or more outputs. To preserve the outer execution context, it makes use of a prologue and epilogue (VM entry and exit) that saves and restores the native CPU state. Internally, it operates on an undocumented state and interprets a sequence of bytes that represents the protected code. To reconstruct the underlying code, we have to locate the virtual machine’s inputs/outputs, understand its internal state and instruction set. Then, we can write a disassembler for the custom instruction set architecture which decodes the bytecode into human-readable pseudocode. In the following, we will have a look at how we can write a generic disassembler based on symbolic execution. Beforehand, we make a short excursion into symbolic execution.

Symbolic Execution

Symbolic execution is a program analysis technique which allows us to symbolically evaluate and summarize assembly code. These summaries provide concrete insights into the semantics of executed instructions.

To symbolically execute assembly code, we first lift it into an intermediate representation. Afterward, we evaluate the code assignment by assignment and track the individual register and memory assignments in a hashmap that is referred to as symbolic state. To propagate the data flow between the instructions, we always use the latest register/memory definitions from the symbolic state. For example, consider the following assembly code and the corresponding assignments:

mov rax, rbx                  ; rax := rbx
add rax, 0x20                 ; rax := rax + 0x20
add rbx, rax                  ; rbx := rbx + rax
xor rcx, rbx                  ; rcx := rcx ^ rbx

Let us assume that the initial symbolic state maps all registers to themselves:

rax: rax
rbx: rbx
rcx: rcx

After the first instruction, we update the symbolic state such that rax now maps to rbx. If we evaluate the second instruction, we propagate this information and assign state[rax] + 0x20 = rbx + 0x20 to rax. After we symbolically executed all instructions, we have the following symbolic state:

rax: rax + 0x20
rbx: rbx + (rax + 0x20)
rcx: rcx ^ (rbx + (rax + 0x20))

The symbolic state reveals the underlying semantics of a sequence of code in a comprehensive manner. If we want to enrich symbolic analysis with (partially) concrete values, we can provide them to the initial symbolic state and perform a concolic execution. For example, if we initialize rax with 0x10, we obtain

rax: 0x30
rbx: rbx + 0x30
rcx: rcx ^ (rbx + 0x30)

In a purely concolic scenario, symbolic execution can be understood as an emulator for assembly code. However, if we keep at least some registers/memory locations symbolic, we get a very powerful technique which allows us to automatically explore virtual machine interpreters. In the following, we will make use of this to build a generic disassembler.

Writing a Disassembler

Our goal is to design a generic disassembler approach that works for most virtual machines in the wild, no matter their complexity. Starting from scratch with nearly no prior knowledge of the virtual machine, we interactively add more and more knowledge until we have a fully-fledged disassembler. For this, we build a system which symbolically explores the virtual machine and enrich the output with useful information about the symbolic state. On a high-level, the approach consists of two steps:

1. We build a symbolic executor that follows the VM execution flow from the VM entry to the VM exit.
2. Afterward, we add callbacks to interesting code locations (such as VM handlers) and dump useful information from the symbolic state.

Building such a symbolic executor is an interactive process: Starting at the VM entry, we symbolically follow the execution flow until the symbolic executor stops. This is always the case if the VM does a control-flow transfer that relying on symbolic values. Then, the symbolic executor has insufficient information to derive a concrete value. In these cases, we manually inspect why this happens and add additional knowledge to the symbolic state. Often, these are either missing information about the bytecode or (conditional) jumps which depend on user input (inputs to the VM from the outside). For the former, we concretize the symbolic memory around the bytecode; for the latter, we can hardcode specific input values.

Once the symbolic executor follows the execution from VM entry to exit without any interruption, we can manually reverse engineer specific VM components and add more and more knowledge about the VM to the executor. Typically, we start by dumping the virtual instruction pointer and track all the executed handlers. Afterward, we manually inspect individual handlers, understand their semantics and add this information to the symbolic executor. In the end, we receive a fully-fledged disassembler which allows us to dump concrete values at runtime, print and optimize the disassembly or reconstruct the control-flow graph. In theory, it is even possible to perform a devirtualization and emit equivalent x86-64 code, although this may take quite some time to implement correctly.

In Practice

After we discussed the generic approach to build a disassembler based on symbolic execution, we will now exemplify the process on a simple virtual machine. For a better understanding, we will first give a detailed overview of the virtual machine. Afterward, we explore the process from symbolically following the execution flow to building a fully-fledged disassembler.

Sample

Our sample of choice protects an implementation of Fibonacci with a virtual machine whose control-flow graph is comparable to the one above. The virtual machine is stack-based; it uses a dedicated stack to store intermediate values; for example, a calculation such as 5 + 8 is performed by the following pseudocode:

push 5
push 8
add

The VM entry is at 0x115a. The virtual machine uses rdx as the virtual instruction pointer and rcx as the virtual stack pointer; the bytecode is located at 0x4060. The handler 0x11a9 performs a stack-based addition:

000011a9  4883c201           add     rdx, 0x1
000011ad  8b01               mov     eax, dword [rcx]
000011af  0141f8             add     dword [rcx-0x8], eax
000011b2  4883e908           sub     rcx, 0x8
000011b6  eb4f               jmp     0x1207

For this, it fetches two arguments from the stack—[rcx] and [rcx-0x8]—adds them and stores them on the top of the stack. It also increments the virtual instruction pointer in 0x11a9 and the virtual stack pointer in 0x11b2. Overall, the virtual machine has the following 11 handlers:

• 0x129e: CMPBE (compare below equal)
• 0x1238: PUSHFROMVAR (load integer from local variable and push onto stack)
• 0x126d: CHECKZERO (if constant == 0x0 then push var_c on top of stack)
• 0x11c4: PUSHPTR (push pointer to local variable)
• 0x1262: GOTO (jump to an address)
• 0x11a9: ADD (add two dwords)
• 0x1245: VMEXIT (leave the VM)
• 0x11f1: CMPE (compare if two values are equal)
• 0x11e1: PUSH (push constant onto stack)
• 0x1281: JCC (conditional jump)
• 0x1226: POPTOVAR (assign value to local variable)

To perform the Fibonacci calculation, it expects an input parameter in rdi representing the n-th Fibonacci number to calculate; after the VM exit, rax holds the output (the n-th Fibonacci number).

If you want to reverse engineer the virtual machine on your own, use the provided information as a guidance. However, let us now forget these analysis details and start writing the disassembler without any prior knowledge.

Following the VM Execution Flow

To build a symbolic executor that follows the VM execution flow from the VM entry to the exit, we use the script follow_execution_flow.py. It hardcodes the VM entry at 0x115a and uses the following loop to symbolically follow the execution flow:

# init worklist
basic_block_worklist = [ExprInt(start_addr, 64)]

# worklist algorithm
while basic_block_worklist:
    # get current block
    current_block = basic_block_worklist.pop()

    print(f"current block: {current_block}")

    # symbolical execute block -> next_block: symbolic value/address to execute
    next_block = sb.run_block_at(ira_cfg, current_block)

    print(f"next block: {next_block}")

    # is next block is integer or label, continue execution
    if next_block.is_int() or next_block.is_loc():
        basic_block_worklist.append(next_block)

In short, it implements a worklist algorithm that iteratively executes the next basic block until the calculation of the next basic block relies on symbolic values. Initially, the output looks as follows:

current block: 0x115A
next block: 0x1207
current block: 0x1207
next block: 0x120A
current block: 0x120A
next block: (@8[0x4060] == 0x80)?(0x129E,0x1212)

We successfully executed the basic blocks 0x115A, 0x1207 and 0x120A. However, the next basic block relies on a memory value that the symbolic executor cannot resolve: @8[0x4060] (which can be understood as an 8-bit memory read from the address 0x4060). Let’s have a look at the corresponding disassembly:

00001178  488d15e12e0000     lea     rdx, [rel data_4060]
[...]
00001207  0fb602             movzx   eax, byte [rdx]
[...]
0000120a  3c80               cmp     al, 0x80

We notice that the comparison with al relies on the memory dereference [rdx], which again relies on the address 0x4060. In other words, the symbolic executor propagated the memory read to the byte comparison: @8[0x4060] == 0x80. Semantically, the virtual machine reads the first byte from the bytecode and dispatches it to find the first handler. To allow the symbolic executor to perform the dispatching, we empower it with knowledge of the bytecode by adding it to the symbolic memory:

# add bytecode to symbolic memory -- start address and size (highest address - lowest address)
sym_address, sym_value = constraint_memory(0x4060, 0x4140 - 0x4060)
sb.symbols[sym_address] = sym_value

Afterward, the symbolic executor runs successfully. The next time it stops it looks as follows:

[...]
current block: 0x120A
next block: 0x1212
current block: 0x1212
next block: 0x1218
current block: 0x1218
next block: 0x121C
current block: 0x121C
next block: 0x121E
current block: 0x121E
next block: 0x1281
current block: 0x1281
next block: RDI[0:32]?(0x1298,0x1286)

This time, we stop at a conditional jump that relies on rdi. After some manual investigation, we come to the conclusion that this is an input parameter to the VM. Let us initialize it with 0 as follows:

# constraint VM input (rdi, first function argument). The value in `ExprInt` represents the function's input value.
rdi = ExprId("RDI", 64)
sb.symbols[rdi] = ExprInt(0, 64)

If we re-run again, the symbolic executor stops here:

[...]
next block: 0x1245
current block: 0x1245
next block: 0x125A
current block: 0x125A
next block: @64[RSP]

Now, the next basic block is an address taken from the stack; a typical pattern for a ret instruction. Indeed, if we look at the corresponding handler we see that it leaves the VM; we found the VM exit. In other words, our symbolic executor now runs from the VM entry to the VM exit. If we have a closer look at the VM exit, we also see the following assembly line:

00001245  8b01               mov     eax, dword [rcx]

Here, we use the virtual stack pointer rcx and load a value into rax. In other words, we found the VM output. To verify our assumption, we can add the following code lines after the worklist algorithm to print the VM output:

# dump VMs/functions' return value -- only works if SE runs until the end
rax = ExprId("RAX", 64)
value = sb.symbols[rax]
print(f"VM return value: {value}")

If we re-run the script, we receive 0 as output; if we change the input value in rdi to 10, we get as output 55. In other words, the symbolic executor correctly emulates the VM execution for the provided inputs and calculates the same output as the binary (remember that it calculates the n-th Fibonacci number).

In summary, we have not only built a symbolic executor following the VM execution flow from the VM entry to its exit; we were also able to emulate the whole VM and inspect its input and output. In the following, we will extend the analysis and enrich the symbolic emulator with information about the individual VM handlers.

From Symbolic Execution to VM Disassembly

In the next step, we want to turn the symbolic executor into a fully-fledged disassembler. For this, we want to add callbacks for the individual handlers which dump additional information from the symbolic state. To realize this, we first start by manually creating a list of handlers. Then, we add the callback in the worklist algorithm as it can be seen in vm_disassembler.py:

# worklist algorithm
while basic_block_worklist:
    # get current block
    current_block = basic_block_worklist.pop()

    # if current block is a VM handler, dump handler-specific knowledge
    if current_block.is_int() and int(current_block) in VM_HANDLERS:
        disassemble(sb, current_block)
    [...]

If the current symbolically executed basic block is the beginning of a VM handler, we call the function disassemble which takes the handler and the symbolic state as input. Within the function, we dispatch the individual handler addresses and dump handler-specific information. For now, let us just create a template and print the virtual instruction pointer:

    # fetch concrete value of current virtual instruction pointer
    vip = sb.symbols[ExprId("RDX", 64)]

    # catch the individual handlers and print execution context
    if int(address) == 0x129e:
        print(f"{vip}: handler {address}")
    [...]

If we execute the script, we get the following handler trace:

0x4060: handler 0x11E1
0x4065: handler 0x11C4
0x406A: handler 0x1226
0x406B: handler 0x11E1
0x4070: handler 0x11C4
0x4075: handler 0x1226
0x4076: handler 0x11C4
0x407B: handler 0x1238
0x407C: handler 0x11C4
0x4081: handler 0x1226
0x4082: handler 0x1262
0x4087: handler 0x11E1
0x408C: handler 0x126D
0x4091: handler 0x1238
0x4092: handler 0x11F1
0x4093: handler 0x1281
0x409D: handler 0x11E1
0x40A2: handler 0x11C4
0x40A7: handler 0x1226
0x40A8: handler 0x1262
0x40AD: handler 0x1262
0x4136: handler 0x11C4
0x413B: handler 0x1238
0x413C: handler 0x1245

Looking at the trace, we see that some handlers are executed more than once. To give the instruction trace more meaning, we can now manually inspect the individual handlers and replace the output with handler-specific information. For example, we could start and replace the last line handler 0x1245 with VMEXIT. This process might be time-consuming; however, it becomes easier the more handlers we already know since we permanently improve our understanding of how the VM works internally. In the end, we can enrich the disassembler as it can be seen in vm_disassembler_final.py. Its final output is:

0x4060: PUSH 0x0
0x4065: PUSHPTR var_0x4 (push pointer to local variable var_0x4)
0x406A: POPTOVAR (assign value to local variable)
0x406B: PUSH 0x1
0x4070: PUSHPTR var_0x8 (push pointer to local variable var_0x8)
0x4075: POPTOVAR (assign value to local variable)
0x4076: PUSHPTR var_0x8 (push pointer to local variable var_0x8)
0x407B: PUSHFROMVAR (load integer from local variable and push onto stack)
0x407C: PUSHPTR var_0xC (push pointer to local variable var_0xC)
0x4081: POPTOVAR (assign value to local variable)
0x4082: GOTO 0x4087
0x4087: PUSH 0x0
0x408C: if 0x0 == 0x0 then push var_c (rdi) on top of stack
0x4091: PUSHFROMVAR (load integer from local variable and push onto stack)
0x4092: CMPE
0x4093: conditional jump
0x409D: PUSH 0x0
0x40A2: PUSHPTR var_0x10 (push pointer to local variable var_0x10)
0x40A7: POPTOVAR (assign value to local variable)
0x40A8: GOTO 0x40AD
0x40AD: GOTO 0x4136
0x4136: PUSHPTR var_0x10 (push pointer to local variable var_0x10)
0x413B: PUSHFROMVAR (load integer from local variable and push onto stack
0x413C: VMEXIT

Given this detailed VM disassembly output, we can now have a closer look at the VM disassembly, detect patterns, simplify the output and reconstruct the high-level code.

Code Reconstruction

To reconstruct the high-level code for the observed VM trace, let’s first have a closer look at some instruction patterns:

0x4060: PUSH 0x0
0x4065: PUSHPTR var_0x4 (push pointer to local variable var_0x4)
0x406A: POPTOVAR (assign value to local variable)

This pattern occurs several times; it pushes a constant onto the stack, afterward a local variable offset. Finally, it assigns the value to the local variable. In short, this pattern can be simplified to var_0x4 = 0x0.

0x4076: PUSHPTR var_0x8 (push pointer to local variable var_0x8)
0x407B: PUSHFROMVAR (load integer from local variable and push onto stack
0x407C: PUSHPTR var_0xC (push pointer to local variable var_0xC)
0x4081: POPTOVAR (assign value to local variable)

This instruction pattern first pushes a pointer to a local variable onto the stack, afterward it fetches its value. Then, we push another pointer to a variable and assign it the value from the stack. In short, it can be written as var_0xc = var_0x8.

Let’s also have a closer look at the conditional jump:

0x4087: PUSH 0x0
0x408C: if 0x0 == 0x0 then push var_c on top of stack
0x4091: PUSHFROMVAR (load integer from local variable and push onto stack)
0x4092: CMPE
0x4093: conditional jump

We first push 0x0 onto the stack. Then, we compare 0x0 == 0x0 and if this is true (which it is in this case), we put var_c onto the stack. We didn’t explore it in depth, but var_c holds the value of rdi, the input to the VM. This way, we put rdi onto the stack. Finally, we compare the two stack values if they are the equal: if rdi == 0x0: [...].

If we do this for all instruction patterns and remove the gotos, we receive a cleaned output:

; var_0x4 = 0x0
0x4060: PUSH 0x0
0x4065: PUSHPTR var_0x4 (push pointer to local variable var_0x4)
0x406A: POPTOVAR (assign value to local variable)

; var_0x8 = 0x1
0x406B: PUSH 0x1
0x4070: PUSHPTR var_0x8 (push pointer to local variable var_0x8)
0x4075: POPTOVAR (assign value to local variable)

; var_0xc = var_0x8
0x4076: PUSHPTR var_0x8 (push pointer to local variable var_0x8)
0x407B: PUSHFROMVAR (load integer from local variable and push onto stack
0x407C: PUSHPTR var_0xC (push pointer to local variable var_0xC)
0x4081: POPTOVAR (assign value to local variable)

; push rdi (== 0) onto stack
0x4087: PUSH 0x0
0x408C: if 0x0 == 0x0 then push var_c on top of stack
0x4091: PUSHFROMVAR (load integer from local variable and push onto stack

; if rdi == 0
0x4092: CMPE
0x4093: conditional jump

; var_0x10 = 0x0
0x409D: PUSH 0x0
0x40A2: PUSHPTR var_0x10 (push pointer to local variable var_0x10)
0x40A7: POPTOVAR (assign value to local variable)

; return var_0x10
0x4136: PUSHPTR var_0x10 (push pointer to local variable var_0x10)
0x413B: PUSHFROMVAR (load integer from local variable and push onto stack

0x413C: VMEXIT

If we also remove the VM disassembler output and re-write our comments in pseudocode, we obtain:

var_0x4 = 0x0
var_0x8 = 0x1
var_0xc = var_0x8

if rdi == 0:
    var_0x10 = 0x0
    return var_0x10
else:
    print("not seen in VM trace")

We might see that it is the initialization of the Fibonacci function where we initialize two values and compare the input to 0x0. Since we initialized rdi with 0x0 in the symbolic executor, the VM execution triggers exactly this case. To dispel the last doubts, we can also compare it with the original, unprotected code:

unsigned int fib(unsigned n) {
  unsigned a=0;
  unsigned b=1;
  unsigned int s = b;

  if (n == 0) {
    return 0;
  }
  [...]

As we can see, our reconstruction nearly fits the original code snippet line by line. To reconstruct the missing parts of the original code, we can assign other values to rdi and re-run the whole process.

This has been a long journey, but in the end we analyzed a virtual machine, wrote an interactive disassembler based on symbolic execution and reconstructed the original code based on the VM disassembly.

Setting the Scene

In this post, we discussed the fundamentals of virtualization-based obfuscation and learned a generic approach on how to build disassemblers for virtual machine implementations using symbolic execution. Afterward, we put it into practice on a simple virtual machine for which we not only constructed a disassembler, but also recovered parts of the protected high-level code. Overall, the whole process has been an interplay between manual analysis and automation; especially the manual work required a lot of time. In general, large parts of this time-consuming manual work can be simplified by using heuristics, pattern matching and other analysis techniques; we could, for example, detect the handlers automatically and also extract their semantics. This may be a topic for a future post.

In the wild, there exist a plethora of virtualization-based obfuscators of varying complexity that are deployed in malware, commercial applications and even in the Windows kernel. Naturally, the example code from this post will break horribly on such samples: function calls, indirect calls, jump tables, external/API calls, additional layers of obfuscation and others must be treated in special ways. Nevertheless, the core concepts remain the same.

If you would like to dive deeper into these topics, learn how to combine this approach with other deobfuscation techniques such as compiler optimizations, taint analysis, SMT solving & program synthesis or if you would like to know how you can handle the aforementioned special cases, then have a look at my training class on code deobfuscation techniques.

In this blog post, we will have a look at some generic heuristics that allow us to quickly identify interesting code parts. For this, we first discuss the general idea to detect code obfuscation. Afterward, we have a closer look at different heuristics and what they identify. In the end, we investigate how the heuristics work on (partially) obfuscated binaries.

We implemented all heuristics in a Binary Ninja plugin called Obfuscation Detection. The repository also contains (most of the) sample binaries we use for evaluation. If you would like to play around with it, check it out.

Detection Heuristics

Our goal is to develop heuristics that pinpoint code which shares similar characteristics to obfuscated code. Early in the reverse engineering process when we want to get a better overview over the binary, we can use these heuristics to spot interesting code locations that we can inspect manually. Even if these code locations are not obfuscated, they are still relevant for reverse engineers since they often implement complex dispatching routines, cryptographic algorithms or other important program logic.

One way to look at code obfuscation is that it tries to impede reverse engineering by artificially increasing the code’s complexity. Therefore, we can identify obfuscated code by looking for complex code, such as functions with large basic blocks or control-flow graphs. Another way to look at code obfuscation is that it tries to confuse reverse engineers by playing with their assumptions and analysis tools. This way, we can look for anomalies in our analysis tooling, such as overlapping instructions or meaningless disassembly.

If we want to apply such heuristics to large binaries (e.g., several hundreds of megabytes in size), they have to be efficient and provide a minimal analysis overhead. As a consequence, we rely on data points that are easy to obtain and heuristics that are cheap to compute.

In practice, it is useful to apply several heuristics independently: While different heuristics may find the same code locations, they may also find other locations since they look for different characteristics. In the following, we will get to know three such heuristics that accomplish the aforementioned requirements.

Complex Functions

Intuitively, large functions implement a complex program logic such as file parsing, network protocols, dispatching routines or cryptographic algorithms. If functions are large due to cod obfuscation, they often contain dead code, (nested) opaque predicates or control-flow flattening.

To determine a function’s complexity, we could, for example, count its number of instructions, basic blocks or bytes. However, in all of these cases, we ignore the function’s branch characteristics: We ignore if the function consists contains loops or (nested) branches. A more generic way to measure a function’s complexity is to measure the complexity of its control-flow graph; for this, we can calculate its cyclomatic complexity. In short, the cyclomatic complexity measures the number of independent paths in a function and is calculated by #edges - #blocks + 2. If the control-flow graph has only a single basic block and no edges, we get 0 - 1 + 2 = 1; if it contains five basic blocks and eight edges, we get 8 - 5 + 2 = 5. The second control-flow graph is more complex. We can implement this heuristic within a few lines in Binary Ninja:

def calc_cyclomatic_complexity(function):
    # number of basic blocks
    num_blocks = len(function.basic_blocks)
    # number of edges in the graph
    num_edges = sum([len(b.outgoing_edges) for b in function.basic_blocks])
    
    return num_edges - num_blocks + 2

To build a meaningful heuristic that is based on the cyclomatic complexity, we can sort all functions by their complexity and print the upper 10%:

def find_complex_functions(bv):
    # sort functions by cyclomatic complexity
    sorted_functions = sorted(bv.functions, key=lambda x: calc_cyclomatic_complexity(x))

    # bound to print only the top 10%
    bound = math.ceil(((len(bv.functions) * 10) / 100))
    
    # print top 10% (iterate in descending order)
    for f in list(reversed(sorted_functions))[:bound]:
        print(f"{hex(f.start)}: {calc_cyclomatic_complexity(f)}")

Overall, the function’s cyclomatic complexity provides a good heuristic to fingerprint complex graphs. However, it ignores the size of basic blocks and the number of instructions in the function; data points which are interesting to pinpoint other code constructs.

Large Basic Blocks

Large basic blocks guarantee that a sequence of code is executed in a row. Often, they implement complex calculations, (unrolled) cryptographic algorithms or initialization routines. For obfuscated code, large basic blocks often contain dead code, initialize virtual machines or hide simple arithmetic calculations in complex arithmetic encodings.

To pinpoint large basic blocks, we could sort all basic blocks in the binary by their number of instructions. However, this might create a lot of noise, especially if there are a lot of large blocks within the same function. Alternatively, we can consider large basic blocks in a function context by calculating the average number of instructions per basic block, relative to the number of basic blocks in the function:

# instructions in the function
------------------------------
# basic blocks in the function

The average number of instructions is high if a function contains only a single large basic block or if a function contains several larger blocks. This way, we can again pinpoint complex code on the function level.

In Binary Ninja, we implement this as follows:

def calc_average_instructions_per_block(function):
    # number of basic blocks
    num_blocks = len(function.basic_blocks)
    # number of instructions
    num_instructions = sum(
        [b.instruction_count for b in function.basic_blocks])
    
    return num_instructions / num_blocks

Similar to the previous heuristic, we can sort all functions based on their average basic block size and print the upper 10%.

def find_large_basic_blocks(bv):
    # sort functions by average basic block size
    sorted_functions = sorted(bv.functions, key=lambda x: calc_average_instructions_per_block(x))

    # bound to print only the top 10%
    bound = math.ceil(((len(bv.functions) * 10) / 100))
    # print top 10% (iterate in descending order)
    for f in list(reversed(sorted_functions))[:bound]:
        print(f"{hex(f.start)}: {math.ceil(calc_average_instructions_per_block(f))}.")

Instruction Overlapping

Up until now, the developed heuristics aimed at detecting complex code. However, sometimes code obfuscation tries to confuse disassemblers by introducing opaque control transfers to addresses that are in the middle of valid instructions. This way, the disassembler does not know how to proceed and build the control-flow graph, since two instructions overlap. In non-obfuscated code, this can also happen in cases where the disassembler mistakenly interprets data as code, therefore creates meaningless disassembly.

To detect instruction overlapping, we can simply walk over all instructions and store the addresses of all bytes that belong to the corresponding instruction in a set. If we check before inserting a byte, whether it is already part of the set, we know that instructions do not overlap. Otherwise—if an address is already in the set before insertion—we found two instructions overlap. In this case, we can print the function the instruction belongs to. If we want to go a step further, we can also highlight the corresponding instructions in the disassembler.

Compared to the other heuristics in which we walked over all functions and interacted with the graph API, we now have to walk over all function bytes in the program, making the implementation and runtime overhead more expensive (we omit the code listing for readability). However, in practice, the overhead is still acceptable for large binaries (up to several minutes).

Evaluation

To get a feeling on how good these heuristics work, we will evaluate them on three binaries: two malware samples and the Windows kernel. Intuitively, we expect to find obfuscated code as well as other interesting program logic. So, let’s dig in and have a look how the heuristics perform on the individual samples.

Emotet

The Emotet sample sample uses a custom implementation of control-flow flattening. While most of it functions are obfuscated, the sample contains also non-obfuscated code.

Cyclomatic Complexity
Function 0x4063f0 (sub_4063f0) has a cyclomatic complexity of 76.
Function 0x4012a0 (sub_4012a0) has a cyclomatic complexity of 36.
Function 0x405800 (sub_405800) has a cyclomatic complexity of 35.
Function 0x402b60 (sub_402b60) has a cyclomatic complexity of 35.
Function 0x409e20 (sub_409e20) has a cyclomatic complexity of 31.
Function 0x404f50 (sub_404f50) has a cyclomatic complexity of 29.
Function 0x40a4b0 (sub_40a4b0) has a cyclomatic complexity of 27.
Function 0x402210 (sub_402210) has a cyclomatic complexity of 26.
Function 0x4025a0 (sub_4025a0) has a cyclomatic complexity of 24.
Function 0x40a9d0 (sub_40a9d0) has a cyclomatic complexity of 22.
Function 0x409530 (sub_409530) has a cyclomatic complexity of 22.
[snip]

We see that one function is significantly more complex than all others; the others, however, are in a comparable dimension. Most of the functions (including the most complex one) are obfuscated via control-flow flattening; the functions that are not obfuscated implement some sequential dispatching logic.

Large Basic Blocks
Basic blocks in function 0x405e40 (sub_405e40) contain on average 19 instructions.
Basic blocks in function 0x405d20 (sub_405d20) contain on average 19 instructions.
Basic blocks in function 0x405e00 (sub_405e00) contain on average 16 instructions.
Basic blocks in function 0x405c90 (sub_405c90) contain on average 13 instructions.
Basic blocks in function 0x404ee0 (sub_404ee0) contain on average 13 instructions.
Basic blocks in function 0x405ea0 (sub_405ea0) contain on average 12 instructions.
Basic blocks in function 0x405db0 (sub_405db0) contain on average 12 instructions.
Basic blocks in function 0x405d70 (sub_405d70) contain on average 12 instructions.
[snip]

The large basic block heuristic does not produce any anomaly; all the values are in a comparable range. If we dig into the individual functions, we see that most of them share a similar structure: They are single-basic block functions that set initialize memory values. If we analyze the functions’ usages, we learn that they are called from obfuscated functions to update the control-flow flattening state:

int32_t sub_405c90() {
    int32_t ecx
    int32_t var_4_4 = ecx
    int32_t var_4 = 0x2224
    int32_t eax
    int32_t edx
    edx:eax = mulu.dp.d(0xcccccccd, 0x7acbf5eb)
    return ((edx u>> 6) - 0xc617) ^ 0x1882c5b
}

The instruction overlapping heuristic does not find any overlapping instructions. So far, both complexity heuristics pinpoint different code locations, effectively clustering the obfuscated code in complex state machines and helper functions.

Adylkuzz

The Adylkuzz sample is protected by the VMProtect obfuscator. VMProtect is a virtualization-based obfuscator that heavily relies on opaque predicates thwarting disassemblers with instruction overlapping/disalinged control flow. Furthermore, it uses dead code to bloat its code size.

Cyclomatic Complexity
Function 0x70c597 (sub_70c597) has a cyclomatic complexity of 79.
Function 0x5c0821 (sub_5c0821) has a cyclomatic complexity of 76.
Function 0x6ff664 (sub_6ff664) has a cyclomatic complexity of 75.
Function 0x70b66e (sub_70b66e) has a cyclomatic complexity of 74.
Function 0x6ff79b (sub_6ff79b) has a cyclomatic complexity of 74.
Function 0x70feea (sub_70feea) has a cyclomatic complexity of 73.
Function 0x709927 (sub_709927) has a cyclomatic complexity of 73.
Function 0x5c36db (sub_5c36db) has a cyclomatic complexity of 60.
Function 0x6fefe5 (sub_6fefe5) has a cyclomatic complexity of 41.
Function 0x5c0bfc (sub_5c0bfc) has a cyclomatic complexity of 31.
Function 0x7086ab (sub_7086ab) has a cyclomatic complexity of 28.
Function 0x703be1 (sub_703be1) has a cyclomatic complexity of 28.
Function 0x70dd19 (sub_70dd19) has a cyclomatic complexity of 27.
Function 0x6f9fd3 (sub_6f9fd3) has a cyclomatic complexity of 27.
Function 0x6e53bf (sub_6e53bf) has a cyclomatic complexity of 27.
Function 0x6fbbed (sub_6fbbed) has a cyclomatic complexity of 26.
Function 0x70ac91 (sub_70ac91) has a cyclomatic complexity of 24.
[snip]

Most of the identified functions with a high cyclomatic complexity are garbage, since the disassembler produces an invalid disassembly due to overlapping instructions. However, the few valid functions initialize the VM and import the hidden API calls (via LoadLibraryA).

Large Basic Blocks
Basic blocks in function 0x70d941 (sub_70d941) contain on average 112 instructions.
Basic blocks in function 0x6baa2e (sub_6baa2e) contain on average 104 instructions.
Basic blocks in function 0x5becad (sub_5becad) contain on average 67 instructions.
Basic blocks in function 0x6ba981 (sub_6ba981) contain on average 59 instructions.
Basic blocks in function 0x5b98a3 (sub_5b98a3) contain on average 49 instructions.
Basic blocks in function 0x5b37a2 (sub_5b37a2) contain on average 49 instructions.
Basic blocks in function 0x6f836d (sub_6f836d) contain on average 48 instructions.
Basic blocks in function 0x5b7ac6 (sub_5b7ac6) contain on average 48 instructions.
Basic blocks in function 0x5ba8e9 (sub_5ba8e9) contain on average 46 instructions.
Basic blocks in function 0x5b1236 (sub_5b1236) contain on average 45 instructions.
Basic blocks in function 0x6fe79c (sub_6fe79c) contain on average 44 instructions.
[snip]

For the large basic block heuristic, the results are a bit different: While the functions with the highest scores are also garbage, many other identified functions implement the instruction semantics handler of the virtualization-based obfuscation (VM handler) within a single basic block.

Overlapping instructions in function 0x5bedbd (sub_5bedbd).
Overlapping instructions in function 0x5bf05a (sub_5bf05a).
Overlapping instructions in function 0x5bf4d6 (sub_5bf4d6).
Overlapping instructions in function 0x5bf7de (sub_5bf7de).
Overlapping instructions in function 0x5c0125 (sub_5c0125).
Overlapping instructions in function 0x5c01b5 (sub_5c01b5).
Overlapping instructions in function 0x5c0363 (sub_5c0363).
Overlapping instructions in function 0x5c03bf (sub_5c03bf).
Overlapping instructions in function 0x5c0821 (sub_5c0821).
Overlapping instructions in function 0x5c0bfc (sub_5c0bfc).
Overlapping instructions in function 0x5c1003 (sub_5c1003).
Overlapping instructions in function 0x5c1447 (sub_5c1447).
Overlapping instructions in function 0x5c1563 (sub_5c1563).
[snip]

As indicated by the results of the other heuristics, the instruction overlapping heuristic identifies a magnitude of functions that contain overlapping instructions and produce incorrect disassembly. While many functions can be immediately ignored since they are only garbage, some functions might contain valid instructions; in those functions, only parts of the disassembly are broken. However, by purely static analysis, it is hard to tell if the instructions are valid or not.

In summary, we can say that all heuristics identified garbage code. If we remove all functions that are identified by the instruction overlapping heuristic from the results of the other heuristics, we can again group the identified functions into two categories: The complex functions perform VM-related initialization routine or decrypt API calls, while the large basic block heuristic pinpoints the VM handlers.

Windows Kernel

After evaluating the heuristics on obfuscated malware samples, let us have a look on how they work on a commercial real-world application: the latest version of the Windows kernel, ntoskrnl.exe (11 MiB, MD5: c9d2f9ada42052c2a34cb3e0743caf48). While most parts of the Windows kernel are not obfuscated, it contains an anti-tamper protection called PatchGuard which employs a lightweight obfuscation by Microsoft’s in-house obfuscation framework Warbird.

Function 0x140a1bee4 (sub_140a1bee4) has a cyclomatic complexity of 2964.
Function 0x1409f7010 (FsRtlMdlReadCompleteDevEx) has a cyclomatic complexity of 2371.
Function 0x1403da6d0 (sub_1403da6d0) has a cyclomatic complexity of 1506.
Function 0x1405d3a40 (PropertyEval) has a cyclomatic complexity of 718.
Function 0x14069fcf0 (NtSetInformationProcess) has a cyclomatic complexity of 642.
Function 0x14068ecb0 (ExpQuerySystemInformation) has a cyclomatic complexity of 435.
Function 0x14066bc78 (SPCall2ServerInternal) has a cyclomatic complexity of 414.
Function 0x14022fba0 (MmCheckCachedPageStates) has a cyclomatic complexity of 318.
Function 0x140a0b0fc (sub_140a0b0fc) has a cyclomatic complexity of 281.
Function 0x1406a9da0 (NtSetSystemInformation) has a cyclomatic complexity of 274.
Function 0x140675a50 (IopParseDevice) has a cyclomatic complexity of 271.

We notice that the first three functions have a very high complexity; afterward, the values drop quickly. Public research about PatchGuard’s internals are not very well documented, so it’s hard to tell what these functions do. However, the first three functions are definitely related to PatchGuard: The first one is related to PatchGuard’s initialization routine, while the second (FsRtlMdlReadCompleteDevEx) is known to perform some PatchGuard related checks. The third function is called by KiFilterFiberContext, which is also a known PatchGuard function.

Large Basic Blocks
Basic blocks in function 0x140a62e04 (SepInitSystemDacls) contain on average 491 instructions.
Basic blocks in function 0x1403e8f74 (SymCryptSha256AppendBlocks_ul1) contain on average 236 instructions.
Basic blocks in function 0x1404bdc60 (HalpRestoreHvEnlightenment) contain on average 147 instructions.
Basic blocks in function 0x140a55cb8 (MiInitializeDummyPages) contain on average 133 instructions.
Basic blocks in function 0x1409a7744 (HalpBlkInitializeProcessorState) contain on average 103 instructions.
[snip]

The functions with the largest average basic block size do not seem to be related to PatchGuard. Instead, based on their function names, they initialize different data structures (SepInitSystemDacls, HalpRestoreHvEnlightenment, MiInitializeDummyPages and HalpBlkInitializeProcessorState) or implement cryptographic algorithms (SymCryptSha256AppendBlocks_ul1).

While the instruction overlapping heuristic also pinpoints some functions, they can be ignored: The results are all false positives, since the disassembler wrongly interprets data as code.

Overall, we can summarize the experiments and say that the heuristics pinpoint all kinds of interesting code parts, no matter if the code is obfuscated, implements a complex state machine, initialization routines or cryptographic algorithms. As we have seen for Emotet and the Windows kernel, it can be beneficial to pay special intention to peaks in the values. Furthermore, we observed that heuristics often produce different results, but can also identify the same code locations.

Setting the Scene

In a previous blog post, we introduced a heuristic to detect control-flow flattening and state machines in binaries. This time, we developed more generic heuristics to pinpoint code obfuscation and complex code. While the heuristics are different in their nature, they all are easy to implement, cheap to calculate and exploit characteristics that are shared by obfuscated as well as interesting non-obfuscated code.

As part of our day-to-day reverse engineering, we can use these heuristics to get an initial overview over the binary; we can spot which code areas might be worth a closer look. If the code is obfuscated, we then can try to understand the context in which the obfuscation is embedded. Afterward—if we want to better understand the obfuscated code—we may look for patterns and come up with a strategy to automatically remove the obfuscation (as we’ll do in my code deobfuscation training classes).

Today, we take a closer look at this process and automate the decryption of strings for a malware sample from the Mirai botnet. Mirai is a malware family that hijacks embedded systems such as IP cameras or home routers by scanning for devices that accept default login credentials. To impede analysis, Mirai samples store those credentials in an encoded form and decode them at runtime using a simple XOR with a constant. In the following, we first manually analyze the string obfuscation. Afterward, we use Binary Ninja’s high-level intermediate language (HLIL) API to get all string references and decrypt them.

If you would like to try it on your own, you’ll find the code and the used malware sample on GitHub. To better understand Mirai, you can also have a look at its leaked source code.

Manual Analysis

In static malware analysis, one of the first things to do is to have a closer look at the identified strings, since they often reveal a lot of context. In this sample, however, we mostly see strings like PMMV, CFOKL, QWRRMPV and others. At first glance, they don’t make much sense. However, if we have a closer look at how they are used in the code, we notice something interesting: They are repeatedly used as function parameters for the function sub_10778. (The corresponding function calls can be found here in the leaked source code.)

sub_10778("PMMV", &data_1616c, 0xa)
sub_10778("PMMV", "TKXZT", 9)
sub_10778("PMMV", "CFOKL", 8)
sub_10778("CFOKL", "CFOKL", 7)
sub_10778("PMMV", &data_16184, 6)
sub_10778("PMMV", "ZOJFKRA", 5)
sub_10778("PMMV", "FGDCWNV", 5)
sub_10778("PMMV", 0x1619c, 5)  {"HWCLVGAJ"}
sub_10778("PMMV", &data_161a8, 5)
sub_10778("PMMV", &data_161b0, 5)
sub_10778("QWRRMPV", "QWRRMPV", 5)

Based on this, we can assume that the passed strings are decoded and further processed in the called function. If we inspect the decompiled code of the function, we identify the following snippet that operates on the first function parameter arg1. For the second parameter arg2, we can find a similar snippet.

uint32_t r0_3 = sub_12c90(arg1)
void* r0_5 = sub_14100(r0_3 + 1)
sub_12d0c(r0_5, arg1, r0_3 + 1)
if (r0_3 s> 0) {
    char* r2_3 = nullptr
    do {
        *(r2_3 + r0_5) = *(r2_3 + r0_5) ^ 0x22
        r2_3 = &r2_3[1]
    } while (r0_3 != r2_3)
}

The code first performs some function calls using arg1, goes into a loop and increments a counter until the condition r0_3 != r2_3 no longer holds. Within the loop, we notice an XOR operation *(r2_3 + r0_5) ^ 0x22, where *(r2_3 + r0_5) seems to be an array-like memory access that is xored with the constant 0x22. After performing a deeper analysis, we can clean up the code by assigning some reasonable variable and function names.

uint32_t length = strlen(arg1)
void* ptr = malloc(length + 1)
strcpy(ptr, arg1, length + 1)
if (length s> 0) {
    char* index = nullptr
    do {
        *(index + ptr) = *(index + ptr) ^ 0x22
        index = &index[1]
    } while (length != index)
}

Now, we have a better understanding of what the code does: It first calculates the length of the provided string, allocates memory for a new string and copies the encrypted string into the allocated buffer. Afterward, it walks over the copied string and decrypts it bytewise by xoring each byte with 0x22. This is also in line with the decryption routine of the original source code.

In other words, strings are encoded using a bytewise XOR with the constant value 0x22. If we want to decode the string PMMV in Python, we can do this with the following one-liner.

>>> ''.join([chr(ord(c) ^ 0x22) for c in "PMMV"])
'root'

We walk over each byte of the string, get its corresponding ASCII value via ord, xor it with 0x22 and transform it back into a character using chr. In a final step, we join all characters into a single string.

After we manually analyzed how strings can be decrypted, we will now automate this with Binary Ninja.

Automated Decryption

To automate the decryption, we first have to find a way to identify all encoded strings. In particular, we have to know where they start and where they end; in other words, we aim to identify all encrypted bytes. In the second step, we can decrypt each byte individually.

Beforehand, we noticed that the encoded strings are passed as the first two parameters to the function sub_10778. To obtain the encoded strings, we can exploit this characteristic by searching for all function calls and parse all passed parameters. Using Binary Ninja’s high-level intermediate language (HLIL) API, we can realize this within a few lines of code.

# get function instance of target function
target_function = bv.get_function_at(0x10778)
# set of already decrypted bytes
already_decrypted = set()

# 1: walk over all callers
for caller_function in set(target_function.callers):
    
    # 2: walk over high-level IL instructions
    for instruction in caller_function.hlil.instructions:
    
        # 3: if IL instruction is a call
        #    and call goes to target function
        if (instruction.operation == HighLevelILOperation.HLIL_CALL and
            instruction.dest.constant == target_function.start):
                
            # 4: fetch pointer to encrypted strings
            p1 = instruction.params[0]
            p2 = instruction.params[1]
            
            # 5: decrypt strings
            decrypt(p1.value.value, already_decrypted)
            decrypt(p2.value.value, already_decrypted)

After fetching the function object of the targeted function sub_10778, we walk over all functions calling sub_10778. For each of these calling functions (referred to as callers), we need to identify the instruction that performs the call to sub_10778. In order to do this, we walk over the caller’s HLIL instructions; for each instruction, we then check if its operation is a call and if the call destination is the targeted function. If so, we access its first two parameters (the pointers to the encoded strings) and pass them to the decryption function. Since some strings—such as PMMV—are used as parameters multiple times, we ensure that we only decrypt them once. Therefore, we collect the addresses of all bytes that we already have decrypted in a set called already_decrypted.

Up until now, we identified all parameters that flow into the decryption routine. The only thing left to do is to identify all encrypted bytes and decrypt them. Since each parameter is a pointer to a string, we can consider it as the string’s start address. Similarly, we can determine the string’s end by scanning for terminating null bytes.

def decrypt(address, already_decrypted):
    # walk over string bytes until termination
    while True:
        # read a single byte from database
        encrypted_byte = bv.read(address, 1)

        # return if null byte or already decrypted
        if encrypted_byte == b'\x00' or address in already_decrypted:
            return

        # decrypt byte
        decrypted_byte = chr(int(encrypted_byte[0]) ^ 0x22)
        
        # write decrypted byte to database
        bv.write(address, decrypted_byte)

        # add to set of decrypted addresses
        already_decrypted.add(address)

        # increment address
        address += 1

Taking the string’s start address as input, we sequentially walk over the string until we reach a byte that terminates the string or that was already decrypted. For each byte, we then transform it into an integer, xor it with 0x22, encode it as a character and write it back to the database. Afterward, we add the current address to the set already_decrypted and increment the address.

Finally, we have all parts together: We walk over all function calls of the string decryption function, parse the parameters for each call and decrypt all the strings in Binary Ninja’s database. If we put everything into a Python script and execute it, the decompiled code from above contains all strings in plain text.

sub_10778("root", "xc3511", 0xa)
sub_10778("root", "vizxv", 9)
sub_10778("root", "admin", 8)
sub_10778("admin", "admin", 7)
sub_10778("root", "888888", 6)
sub_10778("root", "xmhdipc", 5)
sub_10778("root", "default", 5)
sub_10778("root", 0x1619c, 5)  {"juantech"}
sub_10778("root", "123456", 5)
sub_10778("root", "54321", 5)
sub_10778("support", "support", 5)

As a result, the decompilation reveals much more context information. By googling some of the strings, we learn that the parameters are username/password tuples of default login credentials.

Setting the Scene

Automation allows us to spend less time with tedious and repetitive reverse engineering tasks. In this post, I tried to emphasize the thought process behind automation on the example of decrypting strings in malware. Starting with manual analysis, we first pinpointed interesting behavior: encrypted strings used as function parameters. Then, we put it into context by digging into the function, and learned that the strings are decrypted inside. By noticing a recurring pattern—that the function is called several times with different parameters—we developed an idea of how to automate the decryption. By using Binary Ninja’s decompiler API, we walked over all relevant function calls, parsed the parameters and decrypted the strings. In the end, 20 lines of code sufficed to improve the decompilation and achieve a much better understanding of the malware sample.

Even if you are just starting out, I encourage you to get familiar with the API that your tool of choice exposes, and to automate some of the tedious tasks you encounter during your day-to-day reversing. It is not only fun; reverse engineering also becomes so much easier.

• Presentation on control-flow analysis

• Code for graph analysis in Miasm

Graph Analysis in Miasm

For playing around with control-flow graphs, dominance relations and loop detection, we take the following graph as an example:

As described in my previous blog post on automated detection of control-flow flattening, this graph has an entry node a and a loop between c, e and g. To automatically determine all entries, dominance relations and loops, we can use Miasm’s DiGraph class:

from miasm.core.graph import DiGraph

# define edges
edges = [
    ("a", "b"),
    ("a", "c"),
    ("b", "d"),
    ("c", "d"),
    ("c", "e"),
    ("d", "f"),
    ("e", "f"),
    ("e", "g"),
    ("g", "c"),
]

# init graph
g = DiGraph()

# add edges
for x, y in edges:
    g.add_edge(x, y)

# walk over graph entries
for entry in g.heads():
    # dominators
    dominators = g.compute_dominators(entry)

    # dominator tree
    dominator_tree = g.compute_dominator_tree(entry)

    # natural loops
    loops = g.compute_natural_loops(entry)

First, we initialize a DiGraph instance and add all edges to the graph. Then, we walk over all graph entry nodes and compute the dominance relations, the dominator tree and all loops in the graph. With some additional code for pretty printing (omitted here due to readability), we get the following output:

Graph entry: a


Dominators:
a: a
b: a, b
c: a, c
d: a, d
e: a, c, e
f: a, f
g: a, c, e, g


Dominator Tree:
a -> b
a -> c
a -> d
a -> f
c -> e
e -> g


Natural Loops
g -> c: {a, c, e, g}

To apply these techniques to real-world binaries, we just wrap around some disassembling logic:

# init symbol table
loc_db = LocationDB()

# open the binary for analysis
container = Container.from_stream(open(file_path, 'rb'), loc_db)

# cpu abstraction
machine = Machine(container.arch)

# init disassemble engine
mdis = machine.dis_engine(container.bin_stream, loc_db=loc_db)

# disassemble the function at address and get CFG
asm_cfg = mdis.dis_multiblock(start_address)

The code takes a binary as input (provided via file_path), initializes the disassembly engine and disassembles a function at start_address. Then, the resulting asm_cfg is a DiGraph instance holding the function’s control-flow graph. The entire script that re-implements the heuristic to detect control-flow flattening in Miasm can be found here.

Therefore, I would like to use this post to introduce a heuristic that automatically identifies one of the most common obfuscation techniques, control-flow flattening. However, before we approach this, we will have a look together at how reverse engineers use control-flow graphs, the impact of control-flow flattening on reverse engineering and a tiny bit of graph theory. Afterward, we have all the fundamentals to build a heuristic that detects flattening’s underlying structure and implement it within a few lines of Python. In the end, we investigate how the heuristic performs on obfuscated and non-obfuscated code and discuss its limitations.

If you would like to play around with it, you’ll find the code and all binaries on GitHub.

Control-flow Reconstruction

In many cases, we can reconstruct a function’s control flow without any code analysis, just by simply looking at its control-flow graph. For instance, consider the following graph:

As we can see, basic block a has two outgoing edges: one to b and one to c. This is a typical pattern for a conditional branch, where we jump to one basic block (let’s say to b) in the true case and to the other one (to c) in the false case. While b and c most likely perform different calculations, their control flow joins in the basic block d. On the high-level, this is a typical if-then-else pattern. Thus, we can recover the control flow without reading a single assembly line:

/* type */ func(/* parameters */) {
    /* block A */
    if (/* jump condition */) {
        /* block B */
    } else {
        /* block C */
    }
    /* block D */
}

After we recovered the high-level structure, we can reconstruct the code itself, basic block per basic block. In this case, let’s just assume that we recover the following function:

int check(int x) {
    /* block A */
    int r;
    if (x < 10) {
        /* block B */
        r = 20;
    } else {
        /* block C */
        r = 30;
    }
    /* block D */
    return r;
}

The function check takes a parameter x as input and checks if it is smaller than 10. In this case, the function returns 20, otherwise 30. This was easy, huh? Well, let’s have a look at control-flow flattening then.

Control-flow Flattening

Control-flow flattening is a code transformation that removes structure from a control-flow graph. As a result, the control flow cannot be easily recovered as before. For instance, consider the following graph:

If we apply the same method to recover the control flow, we now only see that the blocks a, b, c and d are connected with a block labeled dispatcher. However, we do not know how these blocks relate to each other:

• Is block a visited before or after block b?
• Is there any conditional branch?
• Is there any loop?

In other words, we do not get any meaningful information by looking only at the graph, since all blocks are on the same level; they are flattened.

In its most basic form, control-flow flattening has a central basic block—the dispatcher—that directs the control flow to the individual blocks. This is realized by a state variable that tracks which block will be visited next. The entry initially sets the state variable to the first basic block—in this case a; then, each block updates the state variable in correspondence to the underlying program logic. In other words, control-flow flattening can be considered as a state machine in which each basic block represents a single state.

On the code level, this can be realized by an endless loop that dispatches a state variable in a switch statement. For example, we can represent our function check as follows:

int flattened(int x) {
    /* entry */
    int r;
    int state = 0;

    /* dispatcher */
    while (1){
        switch(state) {
            case 0: {
                /* block A */
                if (x < 10) { 
                    state = 1;
                } else {
                    state = 2;
                }
                break;
            }
            case 1: {
                /* block B */
                r = 20;
                state = 3;
                break;
            }
            case 2: {
                /* block C */
                r = 30;
                state = 3;
                break;
            }
            case 3: {
                /* block D */
                return r;
            }
        }
    }
}

Remember, each block represents a state that is identified via a variable (here: state). After initializing it with 0, we enter the dispatching loop until we resolve the case for 0. The first state—basic block a—checks if x is smaller than 10 and sets state according to the original control flow: if the condition is true, state becomes 2, which directs the control flow to block b, otherwise to c (by setting state to 3). Afterward, both blocks b and c then set the state variable to 3, realizing the joined control flow in block d.

While we cannot recover the control flow by simply looking at the graph, control-flow flattening can often be broken in a simple manner: by looking at the code and analyzing the state transitions. In static analysis scenarios, this works as follows: First, we identify the state variable. Second, we look how it is initialized and follow the dispatching logic until we find the corresponding block. Since this block updates the state variable in accordance to the control flow, we know which state will be executed next. If we repeat this process for each block and track the updates to the state variable, we can consecutively reconstruct the original control flow. In dynamic scenarios, this process is even easier, since it is often sufficient to know the code relevant for dispatching. Since the dispatcher is called after each state transition, we can observe the interaction between the states without any efforts. As a matter of fact, control-flow flattening can be automatically removed in practice; this excellent blog post by Quarkslab as well as Stadeo are only two examples that automate this for specific flattening implementations via symbolic execution.

Similarly, it is easy to automatically detect generic instances of control-flow flattening based on its underlying structure. Before we go into the details, we will first cover a bit of graph theory to better understand the subsequent sections.

Control-flow Analysis

Even though we use control-flow graphs all the time, we may have never thought about what they really represent: Intuitively, a control-flow graph represents all paths in a function that can be traversed during program execution. Such a graph has some nice mathematical properties that are valuable for any kind of code analysis, be it for compiler optimizations, for decompilation or for reverse engineering in general. In the following, we will work towards the goal of automatically detecting loops in control-flow graphs. For this, we first discuss some basic terminology; then, we introduce the concept of dominance relations. For a better understanding, let’s keep it simple and demonstrate the concepts by means of the following graph:

We see a graph with seven nodes and nine edges; one node (a) has no incoming edges, another node (f) no outgoing edges. We call a an entry and f an exit node. Furthermore, a chain of transitions between nodes is called a path. If we are looking at the chain of transitions a -> b -> d -> f, it is a path between a and f. We further say that a loop is a subgraph in which each node can be reached from each other node in the subgraph. In our example, c, e and g form a loop, since we can walk from c to c via c -> e -> g -> c; analogously, we can walk from e to e and from g to g. In the following, we introduce the concept of dominance relations that helps us to identify loops automatically.

Dominance Relations

Dominance relations describe the hierarchical relations between nodes. In particular, they describe which basic blocks are executed before others. More formally, a dominator x guarantees that a basic block x is always executed before a basic block y. In this case, we say that a node x dominates a node y if all paths between the entry node and y go through x. In other words, if there is a path from entry to y that does not go through x, then x is no dominator of y. By design, each node dominates itself.

Given that, we can determine the following dominance relations in our graph. For each node x, we list all nodes that are dominated by x:

node x: nodes dominated by x

a:      a,b,c,d,e,f
b:      b
c:      c, e, g
d:      d
e:      e, g
f:      f
g:      g

Since a is the graph’s entry, it dominates all the other nodes. c dominates c, e and g, since all paths from a to e go through c and all paths from a to g go through g. Consequently, e also dominates g. However, the nodes b and d, f and g only dominate themselves. b dominates only itself since d can also be reached through c; the same holds for d, since f is also reachable through e. Finally, f and g only dominate themselves since they are exit nodes.

Okay, what now? What does all of this mean? Intuitively, this shows us that a will always be executed first and f last. Since c dominates g, we also know that c is always executed before g. Going a step further, we can say that there is an execution hierarchy: a is executed before c, c is executed before e and e is executed before g. In this hierarchy, we can introduce an important concept: immediate dominators.

Immediate Dominator

We say that the closest node x that dominates a node y—where x and y are distinct—is the immediate dominator of y. In our graph, we know that c dominates c, e and g. Since c is the closest dominator to e, it is its immediate dominator. Similarly, since e dominates g and it is the closest dominator to g, we say e is the immediate dominator of g. Applying this to the whole graph, we end up with the following relations:

node x: x is immediate dominator of nodes
a:      b, c, d, f
b:      -
c:      e
d:      -
e:      g
f:      -
g:      -

Basically, we see that a is the immediate dominator of b, c, d and f, while c is the immediate dominator of e and e of g. Now, we can introduce a compact representation of these dominance relations, the dominator tree.

Dominator Tree

A dominator tree is a compact representation of dominance relations; it represents them in a tree-like graph. To build the dominator tree, we simply consider all immediate dominators as edges. Therefore, we can construct the tree by using the following edges for a:

• a -> b
• a -> c
• a -> d
• a -> f

Further, we add the edges for c and e:

• c -> e
• e -> g

Then, the final dominator tree looks as follows:

This tree reveals the dominance relations and hierarchy in a compact form. Especially, we see that a dominates the whole graph, while c dominates e and g. With this representation, we now can easily detect loops in the graph.

Loop Detection

Based on the dominator tree in the control-flow graph, we can automatically identify natural loops by looking for back edges: If we find an edge from a node back to one of its dominators, we identified a loop. We say that the dominator controls the loop since it dominates all blocks in it. In our graph, we see an edge from g to c. Since c dominates g, we have a loop between c, e and g, where c controls it. More intuitively, we can say that the dominance relations guarantee us that c is executed before g; in other words, we can reach g from c. Since we also can reach c from g, we have a loop.

To sum up, we can identify loops by looking for back edges to dominators. Now, we make use of this concept and build a heuristic to automatically identify control-flow flattening.

Control-flow Flattening Detection

Before we talked about control-flow analysis, we said that control-flow flattening removes structure from a control-flow graph by introducing a state machine; this is realized by dispatching a variable that keeps track of the control flow in an endless loop. The following graph is a representation of such a state machine on the binary level:

While this dispatcher directs the control flow to the next basic block via indirect control-flow transfers (using jump tables), other dispatchers are implemented as as a binary search tree. In most cases, the individual states merge in a single block branching to the dispatcher.

To automatically identify these constructs, we design a heuristic that tells us how likely a graph’s structure is similar to those of a flattened graph. To achieve that, we want to know

1. if there is a back edge to a dominator …

2. … that dominates most of the basic blocks in the function

In other words, we are looking for a block that controls a loop and also dominates large parts of the function. We express this as a score between 0 and 1 by looking for a back edge to a dominator x and divide the number of basic blocks that are dominated by x by the total number of basic blocks in the graph. In short, we define the flattening score as follows:

# basic blocks dominated by x
------------------------------
# basic blocks in the function

To apply the heuristic to the graph above, we divide the number of basic blocks dominated by the dispatcher (9) by the total number of basic blocks in the function (10) and obtain a score of 0.9.

Now that we know how the heuristic works, we can implement this in a few lines of Python on top of Binary Ninja:

def calc_flattening_score(function):
    score = 0.0
    # 1: walk over all basic blocks
    for block in function.basic_blocks: 
        # 2: get all blocks that are dominated by the current block
        dominated = get_dominated_by(block)
        # 3: check for a back edge
        if not any([edge.source in dominated for edge in block.incoming_edges]):
            continue
        # 4: calculate relation of dominated blocks to the blocks in the graph
        score = max(score, len(dominated)/len(function.basic_blocks))
    return score

The implementation works as follows:

1. Walk over all basic blocks in the current function.

2. Get all basic blocks dominated by the current block using get_dominated_by(block).

3. Check if the dominator has an incoming edge from any of the blocks that it dominates.

4. Calculate the flattening score by dividing the number of dominated blocks by the number of basic blocks in the function.

To obtain all blocks that are dominated by a given block, we make use of Binary Ninja’s dominator API and perform a depth-first search on the dominator tree:

def get_dominated_by(dominator):
    # 1: initialize worklist
    result = set()
    worklist = [dominator]
    # 2: perform a depth-first search on the dominator tree
    while worklist:
        # get next block
        block = worklist.pop(0)
        # add to result
        result.add(block)
        # add children from dominator tree to worklist
        for child in block.dominator_tree_children:
            worklist.append(child)      
    return result

After initializing the worklist, we iteratively visit all children in the dominator tree. In the end, we return the set of all descendants in the dominator tree for a given block.

Finally, we apply this heuristic to all functions. For this, we walk over all functions, calculate their flattening scores and filter those whose score is higher or equal to 0.9.

def find_flattened_functions():
    # walk over all functions
    for function in bv.functions:
        # calculate flattening score
        score = calc_flattening_score(function)
        # skip if score is too low
        if score < 0.9:
            continue
        # print function and score
        print(f"Function {hex(function.start)} has a flattening score of {score}.")

To sum up, we designed a simple but powerful heuristic for the detection of control-flow flattening that can be implemented within a few lines of Python. Now, we analyze how this heuristic performs on obfuscated and non-obfuscated code.

Evaluation

To get a feeling of how good the heuristic works, we evaluate it on malware samples as well as on common non-obfuscated programs. Intuitively, we expect to pinpoint functions that implement control-flow flattening or other complex state machines which share a similar graph structure. So let’s check out how many functions we spot (relative to the amount of functions in the binary) and, in particular, what kind of state machines we identify.

Malware

Starting with malware, we use three samples in total: two of them are obfuscated with control-flow flattening, the third sample is not obfuscated at all. One sample, FinSpy, uses the control-flow flattening from Obfuscator-LLVM, the other, Emotet, ships a custom flattening implementation. The third, unobfuscated sample is a version of PlugX.

FinSpy is a remote access trojan being used for espionage in various countries. For this sample, the heuristic identifies 10 out of 57 functions in the binary (17.54%). While most of the functions are flattened, the only non-flattened function is of particular interest: in an endless loop, it performs anti-debug checks and terminates the application if any of the checks fail. All other functions share a similar flattening structure; the following snippet is created with Binary Ninja’s decompiler and serves as a example for the others. In this snippet, var_18 is used as a state variable.

uint64_t sub_405270(int64_t* arg1) {

  int32_t var_14 = 1
  int32_t var_18 = 0x16a0bd81
  while (true) {
      int32_t var_20_1 = var_18 - 0xe07f7fa9
      if (var_18 == 0xe07f7fa9) {
          int32_t rcx_7 = 0x74c24cdf
          if (sub_405150(arg1) != 0) {
              rcx_7 = 0x16a0bd81
          }
          var_18 = rcx_7
      } else {
          int32_t var_24_1 = var_18 - 0x16a0bd81
          if (var_18 == 0x16a0bd81) {
              var_14 = (var_14 << 1) - 0x55aabee5 + sub_405150(arg1) + 0x55aabee5
              var_18 = 0xe07f7fa9
          } else {
              int32_t var_28_1 = var_18 - 0x74c24cdf
              if (var_18 == 0x74c24cdf) {
                  break
              }
          }
      }
  }
  return zx.q(var_14)
}

For the Emotet sample, the heuristic identifies 57 out of 138 functions in the binary (41.30%). While a handful of non-flattened functions decode memory areas, the vast majority share a flattening structure similar to the following snippet, in which eax is the state variable:

int32_t __fastcall sub_40b370(int32_t* arg1, int32_t* arg2) {

  int32_t* var_10
  int32_t* esi = var_10
  void* var_18
  void* edi = var_18
  int32_t eax = 0x3b2a39f6
  do {
      bool cond:0_1 = eax == 0x2637511e
      if (eax s<= 0x2637511e) {
          if (cond:0_1) {
              void* ecx_1 = esi + 4
              if (ecx_1 u<= edi) {
                  void* eax_3 = *esi
                  *(arg2 + 8) = eax_3
                  if (eax_3 + ecx_1 u<= edi) {
                      *(arg2 + 4) = ecx_1
                      return 1
                  }
              }
              break
          }
          if (eax == 0x110d5577) {
              int32_t* ecx = esi + 4
              if (ecx u> edi) {
                  break
              }
              int32_t eax_1 = *esi
              esi = ecx
              *arg2 = eax_1
              eax = 0x2637511e
              continue
          } else if (eax == 0x14df8275) {
              eax = 0x110d5577
              esi = *arg1
              edi = *(arg1 + 4) + esi
              continue
          }
      } else if (eax == 0x3b2a39f6) {
          eax = 0x14df8275
          continue
      }
  } while (eax != 0x188efd81)
  return 0
}

So far, the heuristic successfully identified control-flow flattening. What happens if the malware sample is not obfuscated? In order to find this out, we try our heuristic on PlugX, a remote access trojan. In total, we identify 26 out of 504 functions in the binary (5.16%). While most of these functions do not look very promising, some of them are very interesting: they implement command dispatching routines (which backdoor functionality will be executed) as well as the communication with the command-and-control server, two of the most interesting state machines in the binary.

Common Programs

Up until now, we already noticed that the heuristic also detects non-flattened functions that implement state machines. How does this perform on common non-obfuscated programs? To answer this, let’s have a quick look at ls (140 KiB), gcc (1.2 MiB) and gdb (8.5 MiB). The heuristic identifies:

• 8 out of 237 functions (3.38%) for ls

• 67 out of 1,456 functions (4.60%) for gcc

• 374 out of 15,603 functions (2.46%) for gdb

While a considerable fraction of functions is not that interesting, others implement complex state machines that perform recursive directory traversals, parse file formats or encode/decode data streams. Often, these state machines are implemented as loops with large dispatching routines based on switch statements.

To conclude the experiments, we see that the heuristic identifies control-flow flattening as well as other state machines; some of them are very interesting for reverse engineers, since they implement complex dispatching routines that are relevant for understanding file formats, communication protocols or other important program logic. Although the heuristic identifies many functions that do not seem to be relevant for various reversing scenarios, they only constitute a small percentage of the whole code base, as these non-representative experiments show.

Setting the Scene

To automatically break a given code obfuscation scheme (as we do in my code deobfuscation training classes), we must often first analyze its characteristics, generalize the patterns and automatically identify them. Only in the second stage, we are able to implement a strategy that removes the obfuscation, regardless whether we patch the binary, clean up the decompilation or work on a simplified intermediate language. If we want to automatically identify obfuscated code, the approach is similar: We have to understand the obfuscation’s underlying structure, generalize its pattern and design a heuristic which pinpoints interesting code locations. In the second stage, we can have a closer look at the identified code parts.

In our case, we designed a simple but effective heuristic for discovering control-flow flattening by looking for functions which have a basic block that dominates large parts of the function and controls a loop. Personally, I use this heuristic quite a lot in early analysis stages to catch a glimpse of a complex code base. In cases where no control-flow flattening is present, it often finds complex program logic which is interesting for reverse engineers. To better filter the results, we can add additional data points to pinpoint a function’s complexity, such as its number of basic blocks or its cyclomatic complexity: #edges - #blocks + 2. We could also think about refining the flattening heuristic to include checks for the state variable at the costs of requiring a code analysis.

For more advanced implementations of control-flow flattening which use indirect or call-based dispatching routines, the heuristic will not work at all, since the graph structure is inherently different. Similarly, the heuristic won’t detect other obfuscation schemes such as opaque predicates, range dividers, mixed Boolean-Arithmetic or virtual machines, which have fundamentally different characteristics. For these schemes, we have to develop custom heuristics tailored to their underlying structures. Perhaps, this is a topic for a future post.

I hope you enjoyed this blog entry! :)