It is not a surprise to hear the IDA and Decompiler cannot handle all possible cases and eventually fail to recognize a construct, optimize an expression and represent it in its simplest form. It is perfectly understandable — nobody has resources to handle everything. This is why we publish a rich API that can be used to enhance the analysis and add your own optimization rules.
Let us consider the decompiler. Currently, we focus on the compiler-generated code because hand-crafted code can use literally infinite number of tricks to confuse an automatic tool, not to mention seasoned reverses. Nevertheless, some users still want our decompiler to handle some assembler constructs that are usually not used by a compiler. To be concrete, let us take the following example:
x | ~x
This expression always evaluates to -1 (all bits set) but the decompiler does not know about this fact and does not have a dedicated optimization rule for it. However, it is pretty easy to implement a very short plugin for that. The core of the plugin will take just a few lines to carry out the optimization:
if ins.opcode == m_or and ins.r.is_insn(m_bnot) and ins.l == ins.r.d.l: if not ins.l.has_side_effects(): # avoid destroying side effects ins.opcode = m_mov ins.l.make_number(-1, ins.r.size) ins.r = mop_t() self.cnt = self.cnt + 1 # number of changes we made
We check if the current instruction is an OR of an operand with its binary negation. If so, we check for the side effects because we do not want to remove function calls, for example. If everything is fine, we replace x|~x with -1 and increment the number of changes that we made to the microcode.
The rest is just boilerplate code. The full text of the plugin and a sample database are available below.
We hope this very short post and the tiny plugin will inspire you to write more complex optimization rules.