Analysis of CVE-2021-1810 Gatekeeper bypass

By Rasmus Sten on 1 October, 2021

Rasmus Sten

1 October, 2021

Introduction

In my previous blog post, I wrote about how I found a Gatekeeper bypass vulnerability in how archive files are unpacked with Archive Utility. This post analyses the issue in more detail.

Finding the root cause

When double-clicking on an archive file in Finder, Archive Utility is invoked to handle the file. Archive Utility, in turn, delegates the actual unarchiving work to a process called AchiveService. I launched the built-in DTrace script (newproc.d) and then opened a zip file in Finder; this revealed that the following processes are launched:

2021 Jan 12 15:23:50 71676 <1> 64b xpcproxy com.apple.xpc.launchd.oneshot.0x10000083.Archive Utility
2021 Jan 12 15:23:51 71677 <1> 64b xpcproxy com.apple.XprotectFramework.AnalysisService 71143
2021 Jan 12 15:23:51 71677 <1> 64b /System/Library/PrivateFrameworks/XprotectFramework.framework/Versions/A/XPCServices/XprotectService.xpc/Contents/MacOS/Xprotect (...)
2021 Jan 12 15:23:51 71676 <1> 64b /System/Library/CoreServices/Applications/Archive Utility.app/Contents/MacOS/Archive Utility -psn_0_5776770
2021 Jan 12 15:23:51 71678 <71676> 64b /usr/bin/macbinary probe --verbose /Users/user/Downloads/archive-8.zip
2021 Jan 12 15:23:51 71679 <71676> 64b /usr/bin/file -b /Users/user/Downloads/archive-8.zip
2021 Jan 12 15:23:51 71680 <1> 64b xpcproxy com.apple.archiveutility.auhelperservice 71676
2021 Jan 12 15:23:51 71680 <1> 64b /System/Library/CoreServices/Applications/Archive Utility.app/Contents/XPCServices/AUHelperService.xpc/Contents/MacOS/AUHelperSe (...)
2021 Jan 12 15:23:52 71681 <1> 64b xpcproxy com.apple.FileProvider.ArchiveService 71676
2021 Jan 12 15:23:52 71681 <1> 64b /System/Library/Frameworks/FileProvider.framework/XPCServices/ArchiveService.xpc/Contents/MacOS/ArchiveService
2021 Jan 12 15:23:52 71682 <1> 64b xpcproxy com.apple.appkit.xpc.sandboxedServiceRunner 71676
2021 Jan 12 15:23:52 71682 <1> 64b /System/Library/Frameworks/AppKit.framework/Versions/C/XPCServices/SandboxedServiceRunner.xpc/Contents/MacOS/SandboxedServiceRun (...)

So excluding the xpcproxy bootstrapping process, XProtect and file type determination (macprobe, file) there are three key processes involved here:

  • Archive Utility
  • AUHelperService
  • ArchiveService

Coupled with fs_usage, it is evident that the actual writing of files from the zip archive is done by ArchiveService. Furthermore, fs_usage reveals that ArchiveService writes all files extracted from the zip file into a temporary location, for example:

/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)

This is a standard temporary folder as returned by - [NSFileManager URLForDirectory:inDomain:appropriateForURL:create:error] which in this case ends up generating a path length of 152 characters on Catalina.

This determines the length of the absolute path of the extracted items. From the previous testing, we can see that when the full path length (the temporary path prefix concatenated with the path extracted from the zip file) exceeds 1024 bytes, the com.apple.quarantine attribute is lost.
On Big Sur, the path length is 137 characters:

/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/NSIRD_ArchiveService_w86Nam

For completeness: on 10.14 Archive Utility (not ArchiveService) extracts into:

/private/var/folders/tq/06xccl452c735h1sfkqb85_r0000gn/T/Cleanup At Startup/.BAH.qJNqN

This is 86 characters. However, 10.14 does not appear to have this bug; Archive Utility will simply abort with a "File name too long" message if any path is over PATH_MAX. It is possible that this was considered a bug, and when that bug was fixed, it exposed this vulnerability.

It is clear that nuances of the operating system version need to be taken into account; for example, exploiting Catalina and Big Sur will require different path lengths to be taken into account.

At this point we can form a hypothesis that can be tested with some simple experimentation:

  • ArchiveService creates a temporary directory
  • As ArchiveService writes extracted files to that temporary directory, it should set the com.apple.quarantine attribute on each of them because the source zip file has this attribute
  • Setting the quarantine attribute fails when the total path length of the temporary extracted file exceeds PATH_MAX
  • ArchiveService doesn't seem too bothered by this, and proceeds with extractions, resulting in extracted files that are missing the quarantine attribute


(It subsequently transpired that this hypothesis was not entirely correct, but helped guide us in the direction of finding the actual bug and how it was fixed.)

Because the path prefix used for temporary files is longer than the typical target destination (e.g. /Users/username/Downloads/subfolder), when they are moved to their target destination there will be some path names which are longer than PATH_MAX when residing in the temporary folder, but shorter than PATH_MAX (and thus executable) in the final destination folder.

If we want to deploy an innocent-looking app we need to take into account the standard macOS folder structure and the app name. For example, if a standard macOS app called "FakeApp.app" is desired in our zip file, this would require (in Catalina's case):

  • 152 bytes for temporary folder path name
  • 11 bytes for the "FakeApp.app" folder
  • 1024 - 11 - 152 bytes of "padding" = 861 bytes

Each folder name can only be 255 bytes long so 861/255 = 3 plus a remainder of 96

In the examples below I'm testing on macOS 10.15.7 build 19H512 - mostly because I've had some issues disabling SIP in Big Sur in VMWare, but Big Sur appears to work largely the same.

Dtruss was used to analyse the behaviour of the ArchiveService process when it is extracting the proof of concept zip file. As ArchiveScanner is an on-demand launchd job, the command used was:

dtruss -s -W ArchiveService

This caused it to attach to the ArchiveScanner process that starts when the zip file is double-clicked.

Doing so does reveal some interesting artifacts, including a call to the system call setattrlistat() which fails with error ENAMETOOLONG:

setattrlistat(0xFFFFFFFFFFFFFFFE, 0x7F91B085E600, 0x700005F24530) = -1 Err#63

libsystem_kernel.dylib`setattrlistat+0xa
...
ArchiveService`0x0000000106d0d055+0x454
Foundation`-[NSFileCoordinator _invokeAccessor:thenCompletionHandler:]+0x8f
...

In order to obtain evidence to prove or disprove the hypothesis, a debugger was attached and a breakpoint activated in setattrlist(). As this is an on-demand XPC service, we need to wait for it to launch; one way to do this is lldb --wait-for --attach-name ArchiveService, and then set a breakpoint at setattrlistat:

rasmus@catalina-beta-vm ~ % lldb --wait-for --attach-name ArchiveService
(lldb) process attach --name "ArchiveService" --waitfor
Process 1359 stopped
* thread #3, stop reason = signal SIGSTOP
frame #0: 0x00007fff727da502 libsystem_kernel.dylib`__sigsuspend_nocancel + 10
libsystem_kernel.dylib`__sigsuspend_nocancel:
-> 0x7fff727da502 <+10>: jae 0x7fff727da50c ; <+20>
0x7fff727da504 <+12>: movq %rax, %rdi
0x7fff727da507 <+15>: jmp 0x7fff727d5629 ; cerror_nocancel
0x7fff727da50c <+20>: retq
Target 0: (ArchiveService) stopped.

Executable module set to "/System/Library/Frameworks/FileProvider.framework/XPCServices/ArchiveService.xpc/Contents/MacOS/ArchiveService".
Architecture set to: x86_64h-apple-macosx-.
(lldb) break set -n setattrlistat
Breakpoint 1: where = libsystem_kernel.dylib`setattrlistat, address = 0x00007fff727f5b5c
(lldb)

Meanwhile, in a separate Terminal window, we can start an fs_usage instance to track the filesystem activity of ArchiveService. This gives us a convenient way to identify the temporary directory which is used by ArchiveService and monitor it for changes as ArchiveService continues until it hits the next breakpoint:

Process 1412 resuming
Process 1412 stopped
* thread #4, queue = 'NSOperationQueue 0x7fdcc640fbe0 (QOS: UNSPECIFIED)', stop reason = breakpoint 1.1
frame #0: 0x00007fff727f5b5c libsystem_kernel.dylib`setattrlistat
libsystem_kernel.dylib`setattrlistat:
-> 0x7fff727f5b5c <+0>: movl $0x200020c, %eax ; imm = 0x200020C
0x7fff727f5b61 <+5>: movq %rcx, %r10
0x7fff727f5b64 <+8>: syscall
0x7fff727f5b66 <+10>: jae 0x7fff727f5b70 ; <+20>
Target 0: (ArchiveService) stopped.

This revealed that setattrlistat() has nothing to do with extended attributes; in fact, when the ArchiveService process exits, there are no quarantine attributes in the temporary folder. AUHelperService was investigated using the same process, with a lack of any activity related to setting extended attributes present.


In order to identify the function responsible for the extended attributes, I initially wrote a small program that uses Apple's EndpointSecurity API to monitor for all extended attribute changes by listening to ES_EVENT_TYPE_NOTIFY_SETEXTATTR events, but it turns out EndpointSecurity does not expose events involving com.apple.quarantine extended attribute. This may be to prevent vendors (intentionally or accidentally) introduce Gatekeeper bypasses.

As a different approach was needed, I listed all Dtrace probes mentioning "xattr", as in the system call to set extended attributes, setxattr().

"dtrace -l | grep xattr" listed 199 probes, which was reduced to 33 probes when filtering on setxattr only.

root@catalina-beta-vm ~ # dtrace -l | grep setxattr
630 syscall setxattr entry
631 syscall setxattr return
632 syscall fsetxattr entry
633 syscall fsetxattr return
2201 fsinfo mach_kernel VNOP_SETXATTR setxattr
105788 fbt com.apple.filesystems.apfs apfs_vnop_setxattr entry
105789 fbt com.apple.filesystems.apfs apfs_vnop_setxattr return
105830 fbt com.apple.filesystems.apfs apfs_setxattr_as_namedstream entry
105831 fbt com.apple.filesystems.apfs apfs_setxattr_as_namedstream return
105832 fbt com.apple.filesystems.apfs apfs_setxattr_internal entry
105833 fbt com.apple.filesystems.apfs apfs_setxattr_internal return
106923 fbt com.apple.filesystems.apfs apfs_fake_vnop_setxattr entry
106924 fbt com.apple.filesystems.apfs apfs_fake_vnop_setxattr return
120856 fbt com.apple.filesystems.hfs.kext hfs_exchangedata_setxattr entry
120857 fbt com.apple.filesystems.hfs.kext hfs_exchangedata_setxattr return
121056 fbt com.apple.filesystems.hfs.kext hfs_vnop_setxattr entry
121057 fbt com.apple.filesystems.hfs.kext hfs_vnop_setxattr return
121058 fbt com.apple.filesystems.hfs.kext hfs_setxattr_internal entry
121059 fbt com.apple.filesystems.hfs.kext hfs_setxattr_internal return
165623 fbt mach_kernel nfs4_vnop_setxattr entry
165624 fbt mach_kernel nfs4_vnop_setxattr return
166451 fbt mach_kernel fpnfs_vnop_setxattr entry
166452 fbt mach_kernel fpnfs_vnop_setxattr return
167900 fbt mach_kernel setxattr entry
167901 fbt mach_kernel setxattr return
167902 fbt mach_kernel fsetxattr entry
167903 fbt mach_kernel fsetxattr return
168122 fbt mach_kernel vn_setxattr entry
168123 fbt mach_kernel vn_setxattr return
187600 fbt mach_kernel mac_vnop_setxattr entry
187601 fbt mach_kernel mac_vnop_setxattr return
187828 fbt mach_kernel mac_file_setxattr entry
187829 fbt mach_kernel mac_file_setxattr return

A DTrace script to match all of the above probes and print out the stack straces (kernel and user space) for those calls is shown below:

::*setxattr*:entry {
ustack();
stack();
}

This revealed a few interesting calls, from the Archive Utility process:

1 105832 apfs_setxattr_internal:entry
libsystem_kernel.dylib`__mac_syscall+0xa
libquarantine.dylib`_qtn_file_apply_to_path+0x68
Archive Utility`0x0000000101c1988f+0xc2
Archive Utility`0x0000000101c181ef+0x244
Foundation`__NSThread__start__+0x428
libsystem_pthread.dylib`_pthread_start+0x94
libsystem_pthread.dylib`thread_start+0xf

apfs`apfs_vnop_setxattr+0x189
kernel`vn_setxattr+0x2b2
kernel`mac_vnop_setxattr+0x105
Quarantine`quarantine_set_ea+0x5d
Quarantine`syscall_quarantine_setinfo_common+0x5e2
Quarantine`syscall_quarantine_setinfo_path+0xd4
kernel`__mac_syscall+0xee
kernel`unix_syscall64+0x287
kernel`hndl_unix_scall64+0x16

The function _qtn_file_apply_to_path in libquarantine.dylib was a possible candidate. Another interesting piece of information is that it seems that there is a special syscall just for setting the com.apple.quarantine attribute, implemented by the Quarantine kernel extension. Focusing on userspace for now, I decided to set a breakpoint on __qtn_syscall_quarantine_setinfo_path.

(lldb) break set -n _qtn_file_apply_to_path
Breakpoint 1: where = libquarantine.dylib`_qtn_file_apply_to_path, address = 0x00007fff726c613b
(lldb) cont
Process 27579 resuming
Process 27579 stopped
* thread #8, stop reason = breakpoint 1.1
frame #0: 0x00007fff726c613b libquarantine.dylib`_qtn_file_apply_to_path
libquarantine.dylib`_qtn_file_apply_to_path:
-> 0x7fff726c613b <+0>: pushq %rbp
0x7fff726c613c <+1>: movq %rsp, %rbp
0x7fff726c613f <+4>: pushq %r14
0x7fff726c6141 <+6>: pushq %rbx
Target 0: (Archive Utility) stopped.
(lldb) break set -a 0x7fff726c619e
Breakpoint 2: where = libquarantine.dylib`_qtn_file_apply_to_path + 99, address = 0x00007fff726c619e
(lldb) cont
Process 27579 resuming
Process 27579 stopped
* thread #8, stop reason = breakpoint 2.1
frame #0: 0x00007fff726c619e libquarantine.dylib`_qtn_file_apply_to_path + 99
libquarantine.dylib`_qtn_file_apply_to_path:
-> 0x7fff726c619e <+99>: callq 0x7fff726c63f7 ; __qtn_syscall_quarantine_setinfo_path
0x7fff726c61a3 <+104>: testl %eax, %eax
0x7fff726c61a5 <+106>: je 0x7fff726c61c9 ; <+142>
0x7fff726c61a7 <+108>: callq 0x7fff726c7bd6 ; symbol stub for: __error
Target 0: (Archive Utility) stopped.
(lldb) x -f s $rdi
0x7f9a3a904200: "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)"
(lldb) x -f s $rsi
error: failed to read memory from 0x3c.
(lldb) x -f s $rdx
0x7f9a39c05c90: "q/0083;60bca5e1;Safari;ED038CA1-1FD3-4A6A-B3DD-EF64B565C027"

The first argument (in the rdi register) contains a pointer to a C string containing the temporary location. The second argument, in rsi, is 0x3c (60), which appeared to be the length of the extended attribute contents. The third is another C string, that looks exactly like the contents you'd expect to see in a com.apple.quarantine extended attribute (plus a prefix "q/", the purpose of which was unclear). I then displayed the contents of $rdi and $rsi on each execution: 

(lldb) br com add 2.1
Enter your debugger command(s). Type 'DONE' to end.
> x -f s $rdi
> x -f s $rdx
…
(lldb) cont
Process 27579 resuming
(lldb) x -f s $rdi
0x7f9a3a904200: "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

(lldb) x -f s $rdx
0x7f9a39c05c90: "q/0083;60bca5e1;Safari;ED038CA1-1FD3-4A6A-B3DD-EF64B565C027"
Process 27579 stopped
* thread #8, stop reason = breakpoint 2.1
frame #0: 0x00007fff726c619e libquarantine.dylib`_qtn_file_apply_to_path + 99

Looking at that path now, no quarantine attribute is present:

root@catalina-beta-vm ~ # ls -ld@ "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
drwxr-xr-x 3 rasmus staff 96 Jun 5 17:23 /private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:

Now let's cont in the debugger and check again:

root@catalina-beta-vm ~ # ls -ld@ "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
drwxr-xr-x@ 3 rasmus staff 96 Jun 5 17:23 /private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
com.apple.quarantine 57

This is where the quarantine attribute is set:

(lldb) cont
Process 27579 resuming
(lldb) x -f s $rdi
0x7f9a3a904200: "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

(lldb) x -f s $rdx
0x7f9a39c05c90: "q/0083;60bca5e1;Safari;ED038CA1-1FD3-4A6A-B3DD-EF64B565C027"

Process 27579 stopped
* thread #8, stop reason = breakpoint 2.1
frame #0: 0x00007fff726c619e libquarantine.dylib`_qtn_file_apply_to_path + 99
libquarantine.dylib`_qtn_file_apply_to_path:
-> 0x7fff726c619e <+99>: callq 0x7fff726c63f7 ; __qtn_syscall_quarantine_setinfo_path
0x7fff726c61a3 <+104>: testl %eax, %eax
0x7fff726c61a5 <+106>: je 0x7fff726c61c9 ; <+142>
0x7fff726c61a7 <+108>: callq 0x7fff726c7bd6 ; symbol stub for: __error
Target 0: (Archive Utility) stopped.
(lldb) cont
Process 27579 resuming
(lldb) x -f s $rdi
0x7f9a3a904200: "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

(lldb) x -f s $rdx
0x7f9a39c05c90: "q/0083;60bca5e1;Safari;ED038CA1-1FD3-4A6A-B3DD-EF64B565C027"

Process 27579 stopped
* thread #8, stop reason = breakpoint 2.1
frame #0: 0x00007fff726c619e libquarantine.dylib`_qtn_file_apply_to_path + 99
libquarantine.dylib`_qtn_file_apply_to_path:
-> 0x7fff726c619e <+99>: callq 0x7fff726c63f7 ; __qtn_syscall_quarantine_setinfo_path
0x7fff726c61a3 <+104>: testl %eax, %eax
0x7fff726c61a5 <+106>: je 0x7fff726c61c9 ; <+142>
0x7fff726c61a7 <+108>: callq 0x7fff726c7bd6 ; symbol stub for: __error
Target 0: (Archive Utility) stopped.

The process exited without attempting to set the quarantine attribute on all the extracted contents, at least not with this function call. It appears that the iterator exits prematurely when hitting a too long filename, but it's not checking the return code of _qtn_file_apply_to_path. This doesn't affect our exploit, but it hints at a possibility to make an exploit that is slightly simpler than mine; maybe one excessive path length would be sufficient to cause this behaviour. I have not yet looked into it, but would encourage others to.

In order to determine the caller of _qtn_file_apply_to_path to gather more information, the process was repeated:

(lldb) bt
* thread #9, stop reason = breakpoint 1.1
* frame #0: 0x00007fff726c613b libquarantine.dylib`_qtn_file_apply_to_path
frame #1: 0x0000000101734951 Archive Utility`___lldb_unnamed_symbol314$$Archive Utility + 194
frame #2: 0x0000000101733433 Archive Utility`___lldb_unnamed_symbol305$$Archive Utility + 580
frame #3: 0x00007fff3ae1e7b2 Foundation`__NSThread__start__ + 1064
frame #4: 0x00007fff72898109 libsystem_pthread.dylib`_pthread_start + 148
frame #5: 0x00007fff72893b8b libsystem_pthread.dylib`thread_start + 15
(lldb) image list
[ 0] 27E91CD7-37B0-3E51-B1A1-D79B6EC9A961 0x0000000101721000 /System/Library/CoreServices/Applications/Archive Utility.app/Contents/MacOS/Archive Utilit

Hopper was used to check the contents of offset 0x0000000101734951-0x0000000101721000 = 0x13951:

loc_100013951:
0000000100013951 mov rdi, rbx ; CODE XREF=-[BAHDecompressor _propagateQuarantineInformation]+173, -[BAHDecompressor _propagateQuarantineInformati

This was decompiled:

/* @class BAHDecompressor */
-(void)_propagateQuarantineInformation {
rdi = self;
r12 = *ivar_offset(_qtInfo);
COND = *(rdi + r12) == 0x0;
if (!COND) {
r14 = rdi;
rax = [rdi copyTarget];
rax = [rax retain];
rax = objc_retainAutorelease(rax);
var_40 = [rax fileSystemRepresentation];
*(&var_40 + 0x8) = 0x0;
[rax release];
rax = fts_open$INODE64(&var_40, 0x1c, 0x0);
if (rax != 0x0) {
rbx = rax;
rax = fts_read$INODE64(rax, 0x1c, 0x0);
if (rax != 0x0) {
do {
if (((*(int16_t *)(rax + 0x58) & 0xffff) <= 0xd) && (!COND)) {
_qtn_file_apply_to_path(*(r14 + r12), *(rax + 0x28), 0x0);
}
rax = fts_read$INODE64(rbx);
} while (rax != 0x0);
}
fts_close$INODE64(rbx);
}
}
if (**___stack_chk_guard != **___stack_chk_guard) {
__stack_chk_fail();
}
return;
}

This function is apparently using the fts(3) family of functions to traverse the directory hierarchy. Importantly here, it's using (a variant of) fts_read() and passes the result of that on to _qtn_file_apply_to_path(). fts_read returns an FTSENT structure which looks like this:

typedef struct _ftsent {
struct _ftsent *fts_cycle; /* cycle node (8 bytes) */
struct _ftsent *fts_parent; /* parent directory (8 bytes, total 16) */
struct _ftsent *fts_link; /* next file in directory (8 bytes, total 24) */
long fts_number; /* local numeric value (8 bytes, 32) */
void *fts_pointer; /* local address value (8 bytes, 40) */
char *fts_accpath; /* access path (8, 48)*/
char *fts_path; /* root path (8, 56) */
int fts_errno; /* errno for this node (4, 60) */
int fts_symfd; /* fd for symlink (4, 64) */
unsigned short fts_pathlen; /* strlen(fts_path) (2, 66) */
unsigned short fts_namelen; /* strlen(fts_name) (2, 68) */
ino_t fts_ino; /* inode (8, 76) */
dev_t fts_dev; /* device (4, 80) */
nlink_t fts_nlink; /* link count (2, 82) */
short fts_level; /* depth (-1 to N) (2, 84) */
unsigned short fts_instr; /* fts_set() instructions (2, 86) */
struct stat *fts_statp; /* stat(2) information (8, 94) */
char fts_name[1]; /* file name */
} FTSENT;

As the second argument, it's passing offset 0x28 of this struct, which correspond to the fts_accpath struct member above. We can peek inside:

(lldb) x -f A $rax+40
0x6000018c0ca8: 0x00007f8fb813fa00
...
(lldb) x -f s 0x00007f8fb813fa00
0x7f8fb813fa00: "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/FakeA

This function passes the fts_accpath to _qtn_file_apply_to_path(), and the behaviour changes when this path is >1024 bytes. In order to verify that the fts_accpath member actually contains the full path even when it's over PATH_MAX, this was repeated until a long path was identified:

(lldb) x -f A $rax+0x28
0x600000fd6868: 0x00007f8fb813fa00
0x600000fd6870: 0x00007f8fb813fa00
0x600000fd6878: 0x0000000000000000
0x600000fd6880: 0x00000000006c0405
0x600000fd6888: 0x00000003000bca23
0x600000fd6890: 0x0004000401000004
0x600000fd6898: 0x0000000300000001
0x600000fd68a0: 0x0000000000000000
(lldb) x -f s 0x00007f8fb813fa00
0x7f8fb813fa00: "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
warning: unable to find a NULL terminated string at 0x7f8fb813fa00.Consider increasing the maximum read length.

lldb has a built-in string size limit of 1024 characters which helpfully indicates to us that this string is longer than that. Changing the limit revealed the full string: 

(lldb) setting set target.max-string-summary-length 2000
(lldb) x -f s 0x00007f8fb813fa00
0x7f8fb813fa00: "/private/var/folders/3d/4gzq8yw97cz35_55yqpv3pyh0000gn/T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
(lldb) expr (size_t) strlen(0x00007f8fb813fa00)
(size_t) $25 = 1029

It is now clear that the vulnerability is in the specific call to _qtn_file_apply_to_path. A more recent version of macOS was inspected to identify any differences, starting with checking [BAHDecompressor _propagateQuarantineInformation] in the most recent macOS Catalina at the time of writing, 10.15.7 build 19H1217.

Opening this build of Archive Utility in Hopper revealed a slightly different version of that method. Instead of directly using fts_open() to iterate it uses the Cocoa enumeratorAtURL NSFileManager API to enumerate over the file:

var_1C8 = *__NSConcreteStackBlock;
*(&var_1C8 + 0x10) = sub_100014422;
...
rax = [r14 enumeratorAtURL:r15 includingPropertiesForKeys:var_E0 options:0x0 errorHandler:&var_1C8];

It has an interesting errorHandler block containing the code below:

int sub_100014422(int arg0, int arg1, int arg2) {
...
if (r12 != 0x0) {
if ([r15 isEqualToString:**_NSPOSIXErrorDomain] != 0x0) {
if (rbx == 0x3f) {
*(int8_t *)(*(*(r14 + 0x20) + 0x8) + 0x18) = 0x1;
r13 = 0x0;
}
}
else {
r13 = 0x1;
[r15 release];
}
}
...
return rax;
}

There's an interesting constant comparison here: 0x3f = 63 = ENAMETOOLONG. This sets a variable if during enumeration the error ENAMETOOLONG is encountered, and returns NO indicating that the iteration should stop. Further down in _propagateQuarantineInformation, a check is present which, if a variable is set, a separate process via XPC is called into:

if (*(int8_t *)(var_158 + 0x18) != 0x0) {
NSLog(@"-_propagateQuarantineInformation: applying quarantine via helper service");
[var_F0 _propagateQuarantineInformationInService];
}

My belief at this point is that this occured when ENAMETOOLONG was encountered during iteration. This does not appear to be really important (as far as I can tell), except for the fact that it pointed me to the fact that in the fixed version of macOS, there is a new helper service, that is called via _propagateQuarantineInformationInService. On further inspection, AUQuarantineService was part of Archive Utility.

I attached lldb to Archive Utility in a VM with a patched macOS (10.15.7 19H1208). I set a breakpoint at _propagateQuarantineInformation, but it seemingly doesn't trigger at all. However, the helper XPC service, AUQuarantineService, was launched, and dtruss revealed that it is calling _qtn_file_apply_to_path in libquarantine.dylib. I attached to AUQuarantineService instead and set a breakpoint in _qtn_file_apply_to_path:

(lldb) bt
* thread #2, queue = 'com.apple.archiveutility.auquarantineservice.propogationQueue', stop reason = breakpoint 1.1
* frame #0: 0x00007fff6e8fa13b libquarantine.dylib`_qtn_file_apply_to_path
frame #1: 0x000000010247c9e5 AUQuarantineService`___lldb_unnamed_symbol5$$AUQuarantineService + 138
frame #2: 0x00007fff6e86e658 libdispatch.dylib`_dispatch_client_callout + 8
frame #3: 0x00007fff6e87a6ec libdispatch.dylib`_dispatch_lane_barrier_sync_invoke_and_complete + 60
frame #4: 0x000000010247c906 AUQuarantineService`___lldb_unnamed_symbol4$$AUQuarantineService + 285
frame #5: 0x00007fff370c8657 Foundation`__NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT_S3__ + 10
...
frame #9: 0x00007fff6eb0b13b libxpc.dylib`_xpc_connection_mach_event + 934
...
frame #18: 0x00007fff6eac7b77 libsystem_pthread.dylib`start_wqthread + 15
(lldb) image list
[ 0] ABAA25AA-8184-30A8-A6B2-D196C068DB29 0x000000010247b000 /System/Library/CoreServices/Applications/Archive Utility.app/Contents/XPCServices/AUQuarantineService.xpc/Contents/MacOS/AUQuarantine

We can set a base offset in Hopper or just calculate the offset of frame #4 manually: 0x000000010247c906 - 0x000000010247b000 = 0x1906. Opening that address in Hopper reveals an Objective-C method that looks like this:

/* @class AUQuarantineService */
-(void)propagateQuarantineInfo:(void *)arg2 targetURLWrapper:(void *)arg3 withReply:(void *)arg4 {
... // initialization code
r13 = _qtn_file_alloc();
if (r15 != 0x0) {
rax = objc_retainAutorelease(r15);
if (_qtn_file_init_with_data(r13, [rax bytes], [rax length]) != 0x0) {
_qtn_file_free(r13);
}
else {
if (r13 != 0x0) {
r14 = *qword_100003838;
var_60 = *__NSConcreteStackBlock;
*(&var_60 + 0x8) = 0xffffffffc2000000;
*(&var_60 + 0x10) = sub_10000195b; // <- block function pointer
*(&var_60 + 0x18) = 0x107052070;
*(&var_60 + 0x20) = [r12 retain];
*(&var_60 + 0x28) = r13;
dispatch_sync(r14, &var_60);
_qtn_file_free(r13);
[*(&var_60 + 0x20) release];
}
}
}
...
return;
}

So this function calls _qtn_file_init_with_data(r13, [rax bytes], [rax length]), and then it calls a stack-allocated block (via a dispatch_sync() call) that looks very similar to the _propagateQuarantineInformation method we saw earlier in the vulnerable version of macOS:

int sub_10000195b(int arg0) {
r14 = arg0;
var_30 = [objc_retainAutorelease(*(arg0 + 0x20)) fileSystemRepresentation];
*(&var_30 + 0x8) = 0x0;
rax = fts_open$INODE64(&var_30, 0x18, 0x0);
if (rax != 0x0) {
rbx = rax;
rax = fts_read$INODE64(rax, 0x18, 0x0);
if (rax != 0x0) {
do {
if (((*(int16_t *)(rax + 0x58) & 0xffff) <= 0xd) && (!COND)) {
_qtn_file_apply_to_path(*(r14 + 0x28), *(rax + 0x28), 0x0);
}
rax = fts_read$INODE64(rbx);
} while (rax != 0x0);
}
fts_close$INODE64(rbx);
}
var_20 = **___stack_chk_guard;
rax = *___stack_chk_guard;
rax = *rax;
if (rax != var_20) {
rax = __stack_chk_fail();
}
return rax;
}

The effect of the fts_open() and fts_read() code looked virtually identical. In order to verify that this was where the quarantine attribute is set also in the patched macOS version, I set a breakpoint on _qtn_file_apply_to_path and, in the debugger, I was inspecting the second argument to _qtn_file_apply_to_path() every time it was hit:

* thread #2, queue = 'com.apple.archiveutility.auquarantineservice.propogationQueue', stop reason = breakpoint 2.1
frame #0: 0x00007fff6e8fa13b libquarantine.dylib`_qtn_file_apply_to_path
...
Target 0: (AUQuarantineService) stopped.
(lldb) x -f s $rsi
0x7f9231c0a8d8: "FakeApp.app"
(lldb) cont
...
* thread #2, queue = 'com.apple.archiveutility.auquarantineservice.propogationQueue', stop reason = breakpoint 2.1
frame #0: 0x00007fff6e8fa13b libquarantine.dylib`_qtn_file_apply_to_path
...
(lldb) x -f s $rsi
0x7f9231d09538: "Contents"
(lldb) cont
...
* thread #2, queue = 'com.apple.archiveutility.auquarantineservice.propogationQueue', stop reason = breakpoint 2.1
frame #0: 0x00007fff6e8fa13b libquarantine.dylib`_qtn_file_apply_to_path
...
Target 0: (AUQuarantineService) stopped.
(lldb) x -f s $rsi
0x7f9231c0ad28: "_CodeSignature"
...
...
* thread #2, queue = 'com.apple.archiveutility.auquarantineservice.propogationQueue', stop reason = breakpoint 2.1
frame #0: 0x00007fff6e8fa13b libquarantine.dylib`_qtn_file_apply_to_path
libquarantine.dylib`_qtn_file_apply_to_path:
-> 0x7fff6e8fa13b <+0>: pushq %rbp
0x7fff6e8fa13c <+1>: movq %rsp, %rbp
0x7fff6e8fa13f <+4>: pushq %r14
0x7fff6e8fa141 <+6>: pushq %rbx
Target 0: (AUQuarantineService) stopped.
(lldb) x -f s $rsi 0x7f9231d099b8: "Main.story

Comparing that to the debugging session on the vulnerable version on macOS, the key difference is that it previously was passing an absolute pathname to _qtn_file_apply_to_path(), and it is now passing a relative path.

If the file path is relative, the path that it is relative to needs to come from somewhere. In this case, there's no previous opendir() or open directory file descriptors used, which leaves us with the current working directory. We can inspect the current working directory with lsof like this:

root@catalina-beta-vm . # lsof -p 1140
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
AUQuarant 1140 rasmus cwd DIR 1,4 96 12885832512 /T/com.apple.fileprovider.ArchiveService/TemporaryItems/(A Document Being Saved By ArchiveService)/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/FakeApp.app/Contents/Resources/Base.lproj
(...)

The current working directory matches up with the expected value based on the pointer contained in the $rsi register; "Main.storyboardc" is in the folder ".../FakeApp.app/Contents/Resources/Base.lproj". In order to ascertain how the current working directory is being changed, breakpoints were set in fchdir() and chdir():

(lldb) break set -n chdir
Breakpoint 3: where = libsystem_kernel.dylib`chdir, address = 0x00007fff6ea0a314
(lldb) break set -n fchdir
Breakpoint 4: where = libsystem_kernel.dylib`fchdir, address = 0x00007fff6ea294fc
(lldb) cont
Process 1140 resuming
Process 1140 stopped
* thread #2, queue = 'com.apple.archiveutility.auquarantineservice.propogationQueue', stop reason = breakpoint 4.1
frame #0: 0x00007fff6ea294fc libsystem_kernel.dylib`fchdir
libsystem_kernel.dylib`fchdir:
-> 0x7fff6ea294fc <+0>: movl $0x200000d, %eax ; imm = 0x200000D
0x7fff6ea29501 <+5>: movq %rcx, %r10
0x7fff6ea29504 <+8>: syscall
0x7fff6ea29506 <+10>: jae 0x7fff6ea29510 ; <+20>
Target 0: (AUQuarantineService) stopped.
(lldb) bt
* thread #2, queue = 'com.apple.archiveutility.auquarantineservice.propogationQueue', stop reason = breakpoint 4.1
* frame #0: 0x00007fff6ea294fc libsystem_kernel.dylib`fchdir
frame #1: 0x00007fff6e91c35d libsystem_c.dylib`fts_safe_changedir + 90
frame #2: 0x00007fff6e91c572 libsystem_c.dylib`fts_build + 477
frame #3: 0x00007fff6e91c1dd libsystem_c.dylib`fts_read$INODE64 + 893
frame #4: 0x00000001070519ed AUQuarantineService`___lldb_unnamed_symbol5$$AUQuarantineService + 146
...

So with each directory entry iteration with fts_read(), fchdir() will be called, setting the process current working directory. The subsequent call to _qtn_file_apply_to_path can then take only the relative filename as an argument, and will thus never risk exceeding the PATH_MAX limit.

Comparison of the directory traversal code explains the difference in behaviour. In the vulnerable version:

rax = fts_open$INODE64(&var_40, 0x1c, 0x0);

In the modern version:

rax = fts_open$INODE64(&var_30, 0x18, 0x0);

If we look at the man page to fts_open(), the second argument is a bitmask with options. The old code passed the options 0x1c and the new one 0x18. The third bit - 0x4 - has been unset. Looking at the fts.h header file we can see what 0x4 means:

#define FTS_NOCHDIR 0x004 /* don't change directories */

This effectively fixed the issue; there will be no paths larger than PATH_MAX because it will only pass the filename itself, not the full pathname. FTS(3) will ensure safe traversal of the directory hierarchy no matter how deep.

Conclusion

Through some basic system introspection using DTrace and a debugger, coupled with reverse engineering, it was possible to confirm that the issue occurs because an absolute path name was passed to qtn_file_apply_to_path() in the system library libquarantine.dylib, which fails when the total path length exceeds PATH_MAX. The fixed version avoids the issue by calling chdir() as it traverses the directory hierarchy, and only passing the filename, not the full path, to qtn_file_apply_to_path() instead (a file name alone can never exceed PATH_MAX because macOS filenames are limited to at most 255 bytes - NAME_MAX).

As I had a limited amount of time available for this, I have left several avenues unexplored. For example, I did not check where the PATH_MAX limit is actually enforced, but I suspect it's in syscall_quarantine_setinfo_path (or its callees) implemented by the kernel extension Quarantine.kext. It is also clear that Archive Utility and the way it propagates quarantine information has been significantly refactored; there now appears to be two distinct directory traversal routines, one in Archive Utility itself and one in its new helper XPC service AUQuarantineService and the use case for each is not clear. I did not explore whether Archive Utility on Catalina and Big Sur behaves differently after the fix.

Finally, I'd like to thank my fellow Mac developers at WithSecure that supported me and helped me navigate what for me was a relatively novel journey.