Igor’s tip of the week #54: Shifted pointers

Previously we briefly mentioned shifted pointers but without details. What are they?

Shifted pointers is another custom extension to the C syntax. They are used by IDA and decompiler to represent a pointer to an object with some offset or adjustment (positive or negative). Let’s see how they work and several situations where they can be useful.

Shifted pointer description and syntax

A shifted pointer is a regular pointer with additional information about the name of the parent structure and the offset from its beginning. For example, consider this structure:

struct mystruct
{
 char buf[16];
 int dummy;
 int value;            // <- myptr points here
 double fval;
};

And this pointer declaration:

 int *__shifted(mystruct,20) myptr;

It means that myptr is a pointer to int and if we decrement it by 20 bytes, we end up with  mystruct*

In fact, the offset value is not limited to the containing structure and can even be negative. Also, the “parent” type does not have to be a structure but can be any type except void. This can be useful in some situations.

Whenever a shifted pointer is used with an adjustment, it will be displayed using the ADJ helper, a pseudo-operator which returns the pointer to the parent type (in our case mystruct). For example, if the pointer is dereferenced after adding 4 bytes, it can be represented like this:

        ADJ(myptr)->fval

Optimized loop on array of structures

When compiling code which is processing an array of structures, a compiler may optimize the loop so that the “current item” pointer points into a middle of the structure instead of the beginning. This is especially common when only a small subset of fields are being accessed. Consider this example:

struct mydata
{
  int a, b, c;
  void *pad[2];
  int d, e, f;
  char path[260];
};

int sum_c_d(struct mydata *arr, int count)
{
    int sum=0;
    for (int i=0; i< count; i++)
    {
        sum+=arr[i].d+arr[i].c*43;
    }
    return sum;
}

When compiled with Microsoft Visual C++ x86, it can produce the following code:

?sum_c_d@@YAHPAUmydata@@H@Z proc near

arg_0 = dword ptr  4
arg_4 = dword ptr  8

      mov     edx, [esp+arg_4]
      push    esi
      xor     esi, esi
      test    edx, edx
      jle     short loc_25
      mov     eax, [esp+4+arg_0]
      add     eax, 14h

loc_12:                                 ; CODE XREF: sum_c_d(mydata *,int)+23↓j
      imul    ecx, [eax-0Ch], 2Bh ; '+'
      add     ecx, [eax]
      lea     eax, [eax+124h]
      add     esi, ecx
      sub     edx, 1
      jnz     short loc_12

loc_25:                                 ; CODE XREF: sum_c_d(mydata *,int)+9↑j
      mov     eax, esi
      pop     esi
      retn

And initial decompilation looks quite strange even after adding and specifying the correct types:

int __cdecl sum_c_d(struct mydata *arr, int count)
{
  int v2; // edx
  int v3; // esi
  int *p_d; // eax
  int v5; // ecx

  v2 = count;
  v3 = 0;
  if ( count <= 0 )
    return v3;
  p_d = &arr->d;
  do
  {
    v5 = *p_d + 43 * *(p_d - 3);
    p_d += 73;
    v3 += v5;
    --v2;
  }
  while ( v2 );
  return v3;
}

Apparently the compiler decided to use the pointer to the d field and accesses c relative to it.  How can we make this look nicer?

We can find out the offset at which d is situated in the structure via manual calculation, by inspecting disassembly, or by hovering the mouse over it in pseudocode.

Thus, we can change the type of p_d to int * __shifted(mydata, 0x14) to get improved pseudocode:

int __cdecl sum_c_d(struct mydata *arr, int count)
{
  int v2; // edx
  int v3; // esi
  int *__shifted(mydata,0x14) p_d; // eax
  int v5; // ecx

  v2 = count;
  v3 = 0;
  if ( count <= 0 )
    return v3;
  p_d = &arr->d;
  do
  {
    v5 = ADJ(p_d)->d + 43 * ADJ(p_d)->c;
    p_d += 73;
    v3 += v5;
    --v2;
  }
  while ( v2 );
  return v3;
}

Prepended metadata

This technique is used in situations where a raw block of memory needs to have some management info attached to it, i.e. heap allocators, managed strings and so on.

As a specific example, let’s consider the classic MFC 4.x CString class. It uses a structure placed before the actual character array:

struct CStringData
{
    long  nRefs;    // reference count
    int   nDataLength;    // length of data (including terminator)
    int   nAllocLength;   // length of allocation
    // TCHAR data[nAllocLength]

    TCHAR* data()         // TCHAR* to managed data
    {
        return (TCHAR*)(this+1);
    }
};

The CStringclass itself has just one data member:

class CString
{
public:
// Constructors
[...skipped]
private:
    LPTSTR   m_pchData;        // pointer to ref counted string data

    // implementation helpers
    CStringData* GetData() const;
[...skipped]
};
inline
CStringData*
CString::GetData(
    ) const
{
    ASSERT(m_pchData != NULL);
    return ((CStringData*)m_pchData)-1;
}

Here’s how it looks in memory:

               ┌───────────────┐
               │   nRefs       │
               ├───────────────┤
 CStringData   │ nDataLength   │
               ├───────────────┤
               │ nAllocLength  │
               ├───────────────┴─────┐
           ┌──►│'H','e','l','l','o',0│
           │   └─────────────────────┘
           │
           │
         ┌─┴────────┐
CString  │m_pchData │
         └──────────┘

Here’s how the CString’s destructor looks like in initial decompilation:

void __thiscall CString::~CString(CString *this)
{
  if ( *(_DWORD *)this - (_DWORD)off_4635E0 != 12 && InterlockedDecrement((volatile LONG *)(*(_DWORD *)this - 12)) <= 0 )
    operator delete((void *)(*(_DWORD *)this - 12));
}

Even after  creating a CString structure with a single memberchar *m_pszDatait’s still somewhat confusing:

void __thiscall CString::~CString(CString *this)
{
  if ( this->m_pszData - (char *)off_4635E0 != 12 && InterlockedDecrement((volatile LONG *)this->m_pszData - 3) <= 0 )
    operator delete(this->m_pszData - 12);
}

Finally, if we create theCStringDatastruct as described above and change the type of  the CString member to: char *__shifted(CStringData,0xC) m_pszData:

void __thiscall CString::~CString(CString *this)
{
  if ( ADJ(this->m_pszData)->data - (char *)off_4635E0 != 12 && InterlockedDecrement(&ADJ(this->m_pszData)->nRefs) <= 0 )
    operator delete(ADJ(this->m_pszData));
}

Now the code is more understandable: if the decremented reference count becomes zero, the CStringDatainstance is deleted.

More info: IDA Help: Shifted pointers