This is a guest entry written by David Catalán from Outpost24. His views and opinions are his own and not those of Hex-Rays. Any technical or maintenance issues regarding the code herein should be directed to the author.
Software reverse engineering involves working with a wide variety of processor architectures, both real and virtual. Thus, having the capability to extend existing tooling to face new challenges is crucial.
Due to its feature-rich kernel, convenient interface, and wide number of existing plugins, IDA Pro provides a perfect environment to develop new tooling. This plugin is yet another example of how IDA Pro can be extended to support different types of binaries.
The Q3VM plugin includes the loader and processor modules to help analyze binaries built with the Quake III virtual machine obfuscator. Despite being initially developed for the Quake III Arena videogame, this project is independent of the game and has been used to obfuscate code fragments within the Rhadamanthys malware.
Dealing with virtual machine obfuscators
To prevent code from being disassembled by the state-of-the-art tools used by reverse engineers, virtual machine obfuscators implement code interpreters that translate a new custom set of instructions to native assembly. As a result, a disassembler is only able to process the interpreter's code, leaving the bytes belonging to the protected code intact.
The solution to the problem VM obfuscators present is neither trivial nor new for reverse engineers. Every time we need to work with a new architecture, we study it and adapt our existing tooling to support it.
IDA's SDK is a very convenient platform for these tasks as it allows the user to focus strictly on understanding the target architecture, forgetting about things like programming a disassembly algorithm, creating a proper user interface to work on, etc.
The Q3VM loader and processor modules
Installing them is fairly easy: build both modules following the instructions provided by the SDK and place them on the corresponding directories (%IDADIR%/loaders and %IDADIR%/procs).
QVM binaries can be standalone or embedded into other binaries. To spot and extract QVM files, we must start looking for the QVM's header magic constant, defined in vm.h. Once the start of the file has been located, it is possible to calculate the total size of the file using the fields of the header, which follows this structure.
If the headers have been manipulated, some extra creativity is needed. Luckily, the QVM bytecode contains plenty of repetitive patterns like a CONST instruction followed by a STORE4, which corresponds to a value being assigned to a local variable. This kind of repetitive pattern can be very useful to locate the code section of a QVM binary.
To disassemble a previously obtained QVM binary, simply open it with IDA Pro. The loading screen should automatically detect which processor to use.
The processor module supports stack variables renaming and cross-referencing calls.
Closing
As previously mentioned, the Q3VM plugins enable reverse engineers to analyze QVM binaries easily within IDA Pro, benefiting from most of its features.
