During a recent engagement, the Accenture Security Cyber Investigation and Forensics Response (CIFR) team was asked to investigate a potential exploit of a Citrix NetScaler appliance by a vulnerability called CVE-2019-19781. The client provided an export of the virtual machine (VM) — which, in this case, was limited to a copy of the Virtual Hard Disk (VHD) image — for dead box review and analysis.
After completing basic forensics best practices (for example, validating the file hash of the image and generating a working copy of the VHD), we dug in, bringing the VHD into an Ubuntu Linux analysis system and checking initial filesystem observables. Because NetScaler VHD was originally created as a dynamically expandable virtual disk, we knew standard Linux filesystem tools might not correctly report the true size of the disk. It was also clear that the dynamic VHD format could affect the ability to recover deleted files.
Often, we can use standard partition tools, like fdisk and mmls, to view a filesystem container, such as EWF (E01), dd, or VHD. With this VHD, however, these tools did not provide much information about partitions.
From experience, we know that Berkeley Software Distribution (BSD)-based systems label filesystems in a different way and, if we had been on a FreeBSD system, we might have tried the command “disklabel” to show the disk layout.
On Linux, we had installed a similar tool, “disktype,” to help identify information about the VHD image.
With some information on the partitions, we needed to understand where each partition mapped to in the NetScaler system. Using “disktype,” we saw a 20GB disk with a DOS/MBR label and, under that partition, a BSD “disklabel” containing five partitions labeled “a” through “e.” While the basic information about the mount points was useful, we needed more to expose the data for analysis.
Partition 1 is 20GB and starts at sector 63.
- Detail: Boot loader at sector 0 relative to where partition 1 starts.
- Partition a contains a UFS2 filesystem of size 1.6GB with boot code last mounted as /flash.
- Mount as /flash.
- Partition b a 4.1GB swap.
- Cannot mount, but this device is accessible for swap forensics.
- Partition c a 20GB and represent the entire disk.
- We can ignore this partition, legacy way *NIX referenced the whole drive.
- Partition d is 2MB.
- Not likely a valid or useful filesystem.
- Partition e is 14.3GB.
- Undeterminable at this time.
Next, we used the tool “vhdimount” to present the VHD to the analysis system.
At that point we ran the “vhdiinfo” tool on the VHD file to gather more information about the file.
This gave us a “raw” disk image, representing a bit-for-bit copy of the disk, to start analyzing. With a “raw” disk image, the native tools, like “mmls” should now work. First, we started with “mmls” on the raw disk by itself. Next, we used “mmls” to start at offset 63, which we obtained as a start sector from the “mmls” output for the BSD partition. It was here that we saw the difference.
Note: To unmount the VHD, we used the command “fusermount -u /mnt/vhd”
We can attempt to mount each and view the contents to determine the original mountpoint. To help, we loaded a newly downloaded NetScaler VHD image from Citrix to review what a clean system would have mounted and then mounted the partitions to the matching configuration.
It looked like the system had the following mounts:
- a memory disk (md0) mount a /
- partition a is mounted as /flash
- partition b is used as swap
- partition c is the “whole disk”
- partition d is a 2MB slice not mounted
- partition e is mounted as /var
Now, some math
To mount the partitions, we needed to use the offsets with the mount command. The offset value above indicated how far from the start of the disk in sectors the partition began. We based the math using the results from the “disktype” output above. To mount the partition we needed the value in bytes instead of sectors and, from the “mmls” output, we saw that there were 512 bytes in a sector for this machine and did some quick calculations on the partitions starting location with the base offset of 63 sectors.
- Partition a = (63 + 0) * 512
- Partition b = (63 + 3354624) * 512 (SWAP, cannot mount)
- Partition c – represents the raw disk
- Partition d = (63 + 11952128) * 512
- Partition e = (63 + 11956224) * 512
We used the following syntax for mounting a UFS disk read-only:
mount -t ufs -o ufstype=ufs2,offset=<OFFSET> -r /mnt/vhd/vhdi1 /mnt/<MOUNTPT>
Then we ran the following commands:
mkdir -p /mnt/boot /mnt/var
mount -t ufs -o ufstype=ufs2,offset=$(((63 + 0) * 512)) -r /mnt/vhd/vhdi1 /mnt/flash
mount -t ufs -o ufstype=ufs2,offset=$(((63 + 11956224) * 512)) -r /mnt/vhd/vhdi1 /mnt/var
To make the swap device easy to access later for strings analysis, we used “losetup” to map it as a device.
losetup -f /mnt/vhd/vhdi1 -o $(((63 + 3354624) * 512))
Note: “-f” finds the next available loopback device starting at /dev/loop0
We ran “losetup -a” and “df -h|grep loop” and noted that /dev/loop2 was the swap partition. With access to run tools, we looked around the compromised file systems.
We ran a quick check on partition “d” on a clean image to see what it might contain and later validated on the collected image that the “d” partition was similarly empty.
We conducted our analysis of the remaining file systems, noting information about new file creation, events in logs, and other artifacts related to the exploit. We noticed that with the VHD file, we had very limited results in recovering deleted items from the disk. Of particular note, we identified commands in the shell history logs that showed the threat actor’s use of “find” command pipelined to “rm” to files and items of interest in a likely attempt at evading detection. Even with the anti-forensics measures and the nature of the evidence being a dynamic VHD, we were able to verify indicators of compromise (IOCs) specific to the NetScaler exploit and contextualize the incident.
Swap analysis for additional IOCs
With analysis of the visible filesystem complete, we performed a strings analysis on the swap file. We used the command “strings --bytes=8 /dev/loop2 > swapdump.txt” to extract strings with a minimum of eight characters versus the four-character default.
We used “grep” to search swapdump.txt with published IOCs and unique self-discovered items of interest. After reviewing the results, we re-ran the search to pivot around our items of interest. We expanded our review contextually to include what was located near each result. We used grep's "-C" option to return lines before and after the matching item.
Manual reviews of the results showed the following section of contiguous strings in the swap space:
Monero Pool Details
Another item of note was a list of hosts within the monero pool included in swap.
We used a couple of the extracted keywords to validate that we hadn’t missed any hits in the mounted file systems related to the items found in the swap file.
We also leveraged unique incident-related strings found in the swap partition and searched the full 14GB /var raw loop device and from that, pulled out bits of extra information not in a file or carve-able file.
For completeness, we decided to string out the whole /dev/loop1 (var block device) in two different ways to get a feel of the content around the results.
This analysis generated a new hit from the search of the mount /var block device, pointing us to the possible use of cryptonight, a CPU-based miner.
Good-old dead box analysis
Recently, much incident response (IR) work has been focused on triage artifact collection and endpoint detection and response (EDR) tooling, but there are many instances where dead box analysis still plays a key role. To that end, we wanted to share the extra steps we took to perform analysis on the NetScaler VHD disk format and its swap file system — especially as we were able to gain significant insight into the scope and impact of this incident by using these methods.
Accenture Security helps organizations build resilience from the inside out, so they can confidently focus on innovation and growth. Leveraging its global network of cybersecurity labs, deep industry understanding across client value chains and services that span the security lifecycle, Accenture protects helps organizations' protect their valuable assets, end-to-end. With services that include strategy and risk management, cyber defense, digital identity, application security and managed security, Accenture enables businesses around the world to defend against known sophisticated threats, and the unknown. Follow us @AccentureSecure on Twitter or visit us at www.accenture.com/security.
Accenture, the Accenture logo, and other trademarks, service marks, and designs are registered or unregistered trademarks of Accenture and its subsidiaries in the United States and in foreign countries. All trademarks are properties of their respective owners. All materials are intended for the original recipient only. The reproduction and distribution of this material is forbidden without express written permission from Accenture. The opinions, statements, and assessments in this report are solely those of the individual author(s) and do not constitute legal advice, nor do they necessarily reflect the views of Accenture, its subsidiaries, or affiliates. Given the inherent nature of threat intelligence, the content contained in this report is based on information gathered and understood at the time of its creation. It is subject to change. Accenture provides the information on an “as-is” basis without representation or warranty and accepts no liability for any action or failure to act taken in response to the information contained or referenced in this report.
Copyright © 2020 Accenture. All rights reserved. Accenture, and its logo are trademarks of Accenture.