Igor’s tip of the week #79: Handling variable reuse – Hex Rays

Written by Igor Skochinsky | Mar 3, 2022

Previously we’ve discussed how to reduce the number of variables used in pseudocode by mapping copies of a variable to one. However, sometimes you may run into an opposite problem: a single variable can be used for different purposes.

Reused stack slots

One common situation is when the compiler reuses a stack location of either a local variable or even an incoming stack argument for a different purpose. For example, in a snippet like this:

vtbl = DiaSymbol->vtbl;
vtbl->get_symTag(DiaSymbol, (int *)&DiaSymbol);
Symbol->Tag = (int)DiaSymbol;

The second argument of the call is clearly an output argument and has a different meaning and type from DiaSymbol before the call. In such case, you can use the “Force new variable” command (shortcut Shift–F). Due to implementation details, sometimes the option is not displayed if you right-click on the variable itself; in that case try right-clicking on the start of the pseudocode line.

The decompiler creates a new variable at the same stack location, initially with the same type:

IDiaSymbol *v14; // [esp+30h] [ebp+8h] FORCED BYREF

vtbl = DiaSymbol->vtbl;
vtbl->get_symTag(DiaSymbol, (int *)&v14);
Symbol->Tag = (int)v14;

Naturally, you can change its type and name to a better one:

int tag; // [esp+30h] [ebp+8h] FORCED BYREF

vtbl = DiaSymbol->vtbl;
vtbl->get_symTag(DiaSymbol, &tag);
Symbol->Tag = tag;

Using a union to represent a polymorphic variable

Unfortunately, “Force new variable” is not available for register variables (as of IDA 7.7). In such case, using a union may work. For example, consider this snippet from ntdll.dll‘s LdrRelocateImage function:

  int v6; // esi
  int v7; // eax
  int v8; // edi
  int v9; // eax
  
  v6 = 0;
  v20 = 0;
  v7 = RtlImageNtHeader(a1);
  v8 = v7;
  if ( !v7 )
    return -1073741701;
  v9 = *(unsigned __int16 *)(v7 + 24);
  if ( v9 == IMAGE_NT_OPTIONAL_HDR32_MAGIC )
  {
    v18 = *(_DWORD *)(v8 + 52);
    v16 = 0;
  }
  else
  {
    if ( v9 != IMAGE_NT_OPTIONAL_HDR64_MAGIC )
      return -1073741701;
    v18 = *(_DWORD *)(v8 + 48);
    v16 = *(_DWORD *)(v8 + 52);
  }

The function RtlImageNtHeader returns a pointer to the IMAGE_NT_HEADERS structure of the PE image at the given address. After changing its prototype and types of the variables, the code becomes a little more readable:

  int v6; // esi
  PIMAGE_NT_HEADERS v7; // eax
  PIMAGE_NT_HEADERS v8; // edi
  int Magic; // eax
  int v10; // edx
  int v11; // eax
  unsigned int v12; // ecx
  int v13; // ecx
  int v15; // [esp+Ch] [ebp-10h]
  unsigned int v16; // [esp+10h] [ebp-Ch]
  int v17; // [esp+10h] [ebp-Ch]
  unsigned int v18; // [esp+14h] [ebp-8h]
  char *v19; // [esp+14h] [ebp-8h]
  int v20; // [esp+18h] [ebp-4h] BYREF

  v6 = 0;
  v20 = 0;
  v7 = RtlImageNtHeader(a1);
  v8 = v7;
  if ( !v7 )
    return -1073741701;
  Magic = v7->OptionalHeader.Magic;
  if ( Magic == IMAGE_NT_OPTIONAL_HDR32_MAGIC )
  {
    v18 = v8->OptionalHeader.ImageBase;
    v16 = 0;
  }
  else
  {
    if ( Magic != IMAGE_NT_OPTIONAL_HDR64_MAGIC )
      return -1073741701;
    v18 = v8->OptionalHeader.BaseOfData;
    v16 = v8->OptionalHeader.ImageBase;
  }

However, there is a small problem. Judging by the checks of the magic value, the code can handle both 32-bit and 64-bit images, however the current PIMAGE_NT_HEADERS type is 32-bit (PIMAGE_NT_HEADERS32) so the code in the else clause is likely incorrect. If we change v8 to PIMAGE_NT_HEADERS64, then the if clause becomes incorrect:

  if ( Magic == IMAGE_NT_OPTIONAL_HDR32_MAGIC )
  {
    ImageBase = HIDWORD(v8->OptionalHeader.ImageBase);
    v16 = 0;
  }
  else
  {
    if ( Magic != IMAGE_NT_OPTIONAL_HDR64_MAGIC )
      return -1073741701;
    ImageBase = v8->OptionalHeader.ImageBase;
    v16 = HIDWORD(v8->OptionalHeader.ImageBase);
  }

We can’t force a new variable because v8 is allocated in a register and not on stack. Can we still use both types at once?

The answer is yes: we can use a union which combines both types. Here’s how it can be done in this example:

Open Local Types (Shift-F1);
Add a new type (Ins);

Enter this definition:

union nt_headers
{
 PIMAGE_NT_HEADERS32 hdr32;
 PIMAGE_NT_HEADERS64 hdr64;
};

change type of v8 to nt_headers and use “Select Union Field” to pick the correct field in each branch of the if:

  Magic = v7->OptionalHeader.Magic;
  if ( Magic == IMAGE_NT_OPTIONAL_HDR32_MAGIC )
  {
    ImageBase = v8.hdr32->OptionalHeader.ImageBase;
    ImageBaseHigh = 0;
  }
  else
  {
    if ( Magic != IMAGE_NT_OPTIONAL_HDR64_MAGIC )
      return -1073741701;
    ImageBase = v8.hdr64->OptionalHeader.ImageBase;
    ImageBaseHigh = HIDWORD(v8.hdr64->OptionalHeader.ImageBase);
  }

In this specific example the difference is minor and you could probably get by with some comments, but there may be situations where it makes a real difference. Note that this approach can be used for stack variables too.

View full post