The Mystery of Anti-Debug by HeapAlloc

first of all, a big thank you to my friend moti who actually provided the final hints to solve that mystery and saved me A LOT of time googling heap structures. i wish everyone of you would have a moti when getting stuck in RE questions :)

to the story: meanwhile, in a zeus trojan. last week i peeked into a zeus just to really quickly 
stumble over an anti-debug trick. SURPRISE! kidding. 

that anti-debug is really easy to pass by, but wasn’t that trivial to explain; or at least that is what it seems like because i couldn’t find any suitable documentation. and this while the interwebz is full of malware reverser’s write ups.. kidding again.

so here we go...

in a nutshell: the malware would allocate heap memory and use the header of the heap entry as an indicator for an attached debugger. simple, after all. is that frequently used? i don’t know.
but, lets start at the end. the debugger would crash with an access violation when executing invalid arguments. those invalid arguments were produced by an unpacking routine, and it took some runs for brute forcing with IDA Stealth to find a possible root cause: NtGlobalFlag. well known you might think now, and indeed i had some people smiling at me with a yes my dear, thats not tricky at all. but i went on to at least spot the check for this flag, which as for example Mr. Yason described very nicely https://www.blackhat.com/presentations/bh-usa-07/Yason/Whitepaper/bh-usa-07-yason-WP.pdf indicates an attached debugger. guess what, didn’t find that check. 
 
in the first illustration you see the call that causes the exception: EnumWindows, that should call into a handler function, which actually is the unpacked code and turned out to be invalid when NtGlobalFlag is set.

Exception happening in EnumWindows handler function

so i held on to that exception and discovered that by turning IDA Stealth on and off the decryption key in the unpacking routine would change. gotcha, challenge accepted. so, investigating that key i eventually ended up in a piece of memory that preceded an allocated heap block. i provided my walkway up the unpacking routine in a screenshot doku, for people who like pictures just as much as me.

in the unpacking loop: bl being used to XOR the future code

up: ebx initialized with key_init

up: modifying key_init with esi

up: tweaking esi

up: grabbing esi initially from allocated_heap-8

root of all evil: HeapAlloc

so i ended up clueless inside of the RtlAllocateHeap function of ntdll.dll. in fact the value that the malware would grab from memory was initialized by RtlAllocateHeap and i admit it took some staring at the memory to accept the fact that it must be part of the header of the allocated heap block. the memory happened to always be allocated at 9A0688h; the value requested in the code therefor was 9A0680h. i could wonderfully watch the value at this offset turn from 03 to 01 when turning IDA Stealth, specifically NtGlobalFlag protection on or off. and this very value was the cause of crash.

Value in Question in the Memory Block Header

here is where moti jumped in and pointed me to the _HEAP_ENTRY structure, which assigns names to the values preceding my heap block:

The Memory Block Header

with data_offset-8 one hits exactly the size variable of that structure, which actually contains the size divided by 4 PLUS.. size of management information - like the header for example, which would explain a +1. or additional debug information, which could explain a +3. overall, the allocated buffer size in that particular case is always 14000h (it is dedicated to contain unpacked code later). the size value in _HEAP_ENTRY therefor is 2803h when a debugger is attached, or 2801h without a debugger or IDA Stealth activated.

Magic happened - Value changed!

so finalizing, when NtGlobalFlag indicates an attached debugger the heap manager understands this as a higher need for debug information so the allocated heap blocks are slightly bigger than without that flag set. this fact is used by the malware, as it uses the lower byte of the size value for calculation of the unpacking XOR key. 

for more information on heap structures check out this article which i found very informative
http://www.informit.com/articles/article.aspx?p=1081496
OR ask your own moti!