Caught somewhere in time: Hunting for timer-queue timers

Hiding injected code is an enduring problem for attackers. Staying resident in memory increases the risk of detection through techniques like memory scanning and so it is desirable for an attacker to hide their implant when not actively performing a task.  
 
More recent techniques have leveraged the Windows thread pool API to line up a sequence of callbacks (via timer-queue timers) to obfuscate injected code, sleep for a period, and then restore its execution. A good open-source example of this technique is Ekko Sleep Obfuscation which uses the CreateTimerQueueTimer API. This technique in turn appears to be inspired by Nighthawk / FOLIAGE.  
 
Furthermore, Windows does not appear to provide any native telemetry sources for observing malicious timer-queue timers, making it extremely difficult for blue teamers to catch adversaries using these techniques. Therefore, it provides a promising TTP for threat actors to utilize in order to stay hidden. This blog attempts to address this gap and will demonstrate a PoC tool to enumerate timer-queue timers in memory.

The aim of this research was to identify a potential signature which could be used to enumerate timer-queue timers in memory. The approach taken was to understand how the Ekko PoC works, understand how the functions it uses work under the hood (e.g., CreateTimerQueueTimer), and attempt to identify some “signaturable” behaviour. 
 
At a high level, Ekko sets up a sequence of callbacks to fire one after the other and which perform a basic obfuscation – sleep – deobfuscation pattern. Hence, it marks itself as read writable, uses SystemFunction032 to encrypt its contents, and then sleeps for a set duration. Once the sleep timer expires, it will decrypt itself, mark itself as executable and continue execution. 
 
From the Ekko PoC, it is clear you need to first call CreateTimerQueue to obtain a “handle” to a timer queue. If we look at this function, it is apparent that the handle is not actually a fully-fledged handle (and hence does not correspond to a kernel object) but rather a pointer to a block of memory allocated on the heap (i.e., the value returned by RtlAllocateHeap below):

Therefore, timer-queue timers are actually fairly “lightweight objects” and appear to be predominantly implemented in user mode. In fact, they are managed by a worked thread in ntdll as part of the thread pool implementation. This has an important implication in that the sleep/wait initiated by Ekko is performed by a worker thread. Furthermore, the callbacks (e.g. VirtualProtect) will also be executed by a worker thread, as shown in the call stack below:

This gives the technique a minimal footprint from a detection perspective as it avoids typical indicators of injected code (e.g., there are no threads created which point to unbacked regions). 
 
Additionally, it is important to stress that timer-queue timers are also distinct entities to fully fledged kernel timer objects that are accessed via NtCreateTimer, etc. For a version of Ekko which uses waitable timers (e.g., kernel timers) see: https://github.com/Idov31/Cronos. 
 
Once a handle has been acquired, an attacker can now call CreateTimerQueueTimer multiple times to queue up a series of callbacks which will perform the familiar obfuscate-sleep-deobfuscate pattern. The function definition for CreateTimerQueueTimer is shown below: 

The key parameters to note are the Callback and Parameter arguments, which are the function and argument to be executed when the timer expires. Note the “handle” acquired via CreateTimerQueue previously is also passed (TimerQueue). 
 
If we look at the implementation of CreateTimerQueueTimer, we can see that it is essentially a wrapper around RtlCreateTimer. This is also demonstrated by setting a breakpoint in windbg on ntdll!RtlCreateTimer as shown in the call stack below:

Therefore, the next step is to dig into the implementation of RtlCreateTimer, of which a key section is shown below: 

This small code stub enables us to draw two initial conclusions: 
 
1) RtlCreateTimer calls RtlAllocateHeap itself on line 45 with an allocation size of 0x60 bytes.

2) As part of the timer initialisation, RtlCreateTimer writes the arguments passed in the call to CreateTimerQueueTimer to this block of memory (e.g. the callback is written on line 57, the parameter on line 58 etc.). 
 
Therefore, we know that the timer callback and parameter can be found in heap memory; however, so far there is not much to go on in terms of developing a signature that we can use to actively hunt for timer-queue timers in memory. 
 
If we continue to examine RtlCreateTimer, we can observe that later on in its execution it calls the function TpAllocTimer, to which it passes a pointer to the previous heap allocation of 0x60 bytes. TpAllocTimer in turn allocates more heap memory (0x168 bytes) as shown in the image below:

This new block of heap memory is initialised in a similar fashion via calling a chain of successive functions (TpAllocTimer -> TppInitializeTimer -> TppCleanupGroupAddMember). At this point, following the exact implementation of each of these functions is complicated, but the TL;DR is that during the call to TpAllocTimer, a repetitive and predictable sequence of pointers are written to this second block of heap memory.  
 
Crucially, this second block also contains a pointer to the initial allocation performed by RtlCreateTimer that contains the callback and parameter. Therefore, from a single call of CreateTimerQueueTimer we can expect to find two ”linked” heap allocations. 
 
This can be demonstrated via running the Ekko PoC under windbg. We can start by enumerating the process heaps and examining the objects within them:

We are interested in objects allocated on the heap with a size of either 0x60 (first allocation) or 0x168 bytes (second allocation). For brevity, two such objects are shown in the output above, which correspond to two linked heap allocations performed by a single CreateTimerQueueTimer call. 
 
If we examine the memory at the latter heap entry (0x00000177a8c75f40), we will discover the second heap allocation performed by TpAllocTimer:

The crucial point to stress here is the pattern of repeatable/predictable pointers, which starts with ntdll!TppTimerpCleanupGroupMemberVFuncs and ends with ntdll!TppTimerpTaskVFuncs.  
 
Furthermore, I mentioned previously that a pointer to the first (“linked”) heap allocation was also written to this block of memory. This can be found at 0x00000177a8c75fa8 (so immediately after ntdll!RtlpTpTimerCallback in memory). To confirm this, we can compare it to the output of “!heap -a” above and see that 0x00000177a8c73340 is located within the first heap entry of 0x60 bytes (which starts at 0x00000177a8c73330). 
 
We can therefore proceed to dump the memory at this location:

Voila! We have found the callback (which points to ntdll!NtContinue). As NtContinue expects a CONTEXT structure, we can assume the next pointer (0x0000006223cfd6e0) is a valid context structure:

In this case, we can see that the parameter contains a CONTEXT structure in which Rip is pointing at VirtualProtect and is setting the memory region located at 0x00007ff7`0e2c0000 (First arg is Rcx) to PAGE_READWRITE (e.g., it is this timer in memory: https://github.com/Cracked5pider/Ekko/blob/main/Src/Ekko.c#L97). 

1. Obtaining a handle to every process and locating all heap memory regions via the PEB.
2. Scanning heap memory for our specific pattern of thread pool pointers (ntdll!TppTimerpCleanupGroupMemberVFuncs, ntdll!RtlpTpTimerFinalizationCallback, ntdll!RtlpTpTimerCallback and ntdll!TppTimerpTaskVFuncs).
3. If we find the pattern, retrieve the initial allocation performed by RtlCreateTimer and obtain the target callback and parameter.

Firstly, I did initially explore enumerating heaps via CreateToolhelp32Snapshot, however, I found it quite flakey as it would often fail to retrieve a snapshot (especially when the target process was slow to break-in to under a debugger as Ekko often is). Hence, I decided to manually locate heap memory in the target process. 
 
Secondly, an obvious race condition exists in that the heap is constantly in flux while we are reading and scanning memory. This could result in failing to spot a malicious timer which is partially scrubbed. A potential fix to this would be to suspend all threads in the process before scanning, but this is obviously more intrusive. In any case, one thing to bear in mind is that a few timers must always be present while the implant is sleeping, as they are needed to fire when the sleep duration ends. Hence, while the beacon may only be visible in memory for a brief period, some timers must always be present for the whole sleep duration and hence our chances of spotting them are considerably higher. 
 
Thirdly, there may be a better and more reliable/performant way to do this. Other sensible approaches would be to approach the problem from the other side; that is to trace the TppWorkerThread which will execute the timer callbacks. I did start investigating this and could see for instance that the PEB contains a linked list which is referenced by TppWorkerThread (TppWorkerpListLock // TppWorkerpList; https://www.geoffchappell.com/studies/windows/km/ntoskrnl/inc/api/pebteb/peb/index.htm). However, this felt more complicated from a quick analysis and, having found something that worked, it was not pursued any further. However, it may yet provide an easier way of enumerating timer-queue timers. 
 
Lastly, as this technique relies on undocumented functionality that is liable to change in future versions of Windows, it may break due to future Windows releases. At the time of publishing, it has been tested on Windows 10 1607 and Windows 10 21h2.

The PoC memory scanner can be found here: https://github.com/WithSecureLabs/TickTock 
 
Below is a demonstration of the results of scanning for timer-queue timers while Ekko is running:

Additionally, in terms of false positives, on a basic build of Windows 21h2 there appeared to be only a few common timer-queues lurking in memory. These belonged to winlogon, dllhost (WORK_QUEUE::OnThreadSentinelTimer), taskhost (COSTimerQueueEntry::Completion_). Hence, finding multiple (unique) timer-queue timers in memory already seems highly anomalous. 
 
As a final note, the PoC released with this blog makes no effort to remove false positives. As with all detection logic, the real test is how a specific detection approach is tailored to an environment and how common FPs are removed. As such, I have made no attempt to address this issue within the scope of this research.