Introduction

In a prior blog entry, I presented Volatility 3 and discussed the procedure for examining Windows 11 memory.

In the current post, I shall address memory forensics within the context of the Linux ecosystem.

Linux Memory Dump Acquisition

Extracting a memory dump from a running Linux system can be a valuable process for forensic investigations, incident response, or troubleshooting purposes.

Historically, the Lime(https://github.com/504ensicsLabs/LiME) tool was frequently utilized; however, AVML(Acquire Volatile Memory for Linux) now offers a significantly faster memory dumping process. Consequently, it is advisable to employ AVML for such purposes.

Microsoft's AVML is a powerful open-source tool designed to collect memory from Linux systems. AVML is developed using the Rust programming language. For those interested in obtaining the source code and compiling it independently, please consult the associated GitHub repository at https://github.com/microsoft/avml.

For this demonstration, I will opt to download the precompiled release version. Download the appropriate AVML binary for your target system architecture (e.g., avml-x86_64) from the GitHub repository: https://github.com/microsoft/avml/releases.

Transfer the downloaded AVML binary to the target system(ex: Ubuntu 20.04).

The procedure for utilizing AVML can be outlined as follows:

$ wget https://github.com/microsoft/avml/releases/download/v0.11.2/avml
$ chmod +x avml
$ ./avml --help
A portable volatile memory acquisition tool

Usage: avml [OPTIONS] <FILENAME>

Arguments:
  <FILENAME>
          name of the file to write to on local system

Additionally, AVML offers functionality for acquiring memory dumps and transmitting them to a designated URL or Microsoft Azure cloud storage. Nonetheless, in this instance, I will solely focus on the procedure for employing AVML on a local computer.

$ sudo ./avml memory.lime

When executing AVML without any specific options and providing only the target file name, a memory dump is extracted.

Furthermore, AVML incorporates a compression feature, which can reduce the size of an 8GB memory extraction result to approximately 3GB.

$ sudo avml --compress compressed.lime

To decompress the compressed file, employ the avml-convert executable file.

$ avml-convert ./compressed.lime ./uncompressed.lime

Linux ISF

Previously, Volatility 2 employed the notion of a profile. However, with the advent of Volatility 3, a symbol table has been introduced as a novel method to supersede the profile. In essence, this approach interprets the symbol information of the Linux kernel based on registered tables, enabling the discovery of the values' meanings within memory. Volatility conducts its analysis of the kernel's symbol table and stores the data in a unique format, known as the ISF (Intermediate Symbol Format) file format. Consequently, the analysis of a Linux memory dump can be conducted using an 'ISF' file rather than the traditional 'profile'.

There are two approaches to acquiring an ISF file. One may either utilize a pre-existing file or generate one independently.

Obtain pre-existing files from an ISF Server

Initially, employ the command below to ascertain the operating system and kernel version associated with the provided memory dump:

$ python3 vol.py -f ./ubuntu20_memory.lime banners.Banners
Volatility 3 Framework 2.4.2
Progress:  100.00               PDB scanning finished
Offset  Banner

0x2b8001a0      Linux version 5.4.0-26-generic (buildd@lcy01-amd64-029) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 (Ubuntu 5.4.0-26.30-generic 5.4.30)
0x2c392e54      Linux version 5.4.0-26-generic (buildd@lcy01-amd64-029) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 (Ubuntu 5.4.0-26.30-generic 5.4.30)
0x6fafd1e0      Linux version 5.4.0-26-generic (buildd@lcy01-amd64-029) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 (Ubuntu 5.4.0-26.30-generic 5.4.30)
0xc4f1f220      Linux version 5.4.0-26-generic (buildd@lcy01-amd64-029) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 (Ubuntu 5.4.0-26.30-generic 5.4.30)
0x10e33bc88     Linux version 5.4.0-26-generic (buildd@lcy01-amd64-029) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 (Ubuntu 5.4.0-26.30-generic 5.4.30)

The output will display information pertaining to the Linux kernel identifiable within the given dump file.

The output yields the subsequent information:

Linux version 5.4.0-26-generic (buildd@lcy01-amd64-029) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 (Ubuntu 5.4.0-26.30-generic 5.4.30)

If an ISF file has already been generated through the efforts of others, there is no further cause for concern. Fortunately, several diligent researchers maintain services that provide such files.

Volatility3 Linux ISF Server

According to the description on this particular site, it retains symbol information for the Linux kernel about various Linux distributions, including Ubuntu and Debian, and currently offers approximately 1,327 symbols.

The search can be conducted using Kernel or Banner information. However, the Banner information may not yield accurate search results, potentially due to the length of the string. Instead, input the kernel version number (e.g., 5.4.0-26-generic) to identify the appropriate file. The Banner information displayed also corresponds to the testing environment in question. If a pre-existing ISF file is available, the process becomes significantly more straightforward.

The ISF file can be downloaded for one hour, after which a new search must be conducted to obtain a download link. The downloaded file can be stored in the /volatility3/volatility3/symbols directory within the Volatility3 path.

$ cd volatility3/volatility3/symbols/
$ wget [download link]
$ ls -l
total 1944
-rwxr-xr-x 1 cpuu cpuu 1979312 May 11 11:19 5.4.0-26-generic_5.4.0-26.30_amd64.json.xz

Create a custom file using the dwarf2json tool

In the event that the required files are not available on the ISF server, it becomes necessary to generate them independently. For this purpose, a tool named dwarf2json is provided. For a comprehensive explanation, consult the Creating New Symbol Tables page (https://volatility3.readthedocs.io/en/latest/symbol-tables.html#mac-or-linux-symbol-tables).

dwarf2json is a Go(1.14+ required) utility that processes files containing symbol and type information to generate Volatility3 Intermediate Symbol File (ISF) JSON output suitable for Linux and macOS analysis.

$ git clone https://github.com/volatilityfoundation/dwarf2json.git
$ cd dwarf2json/
$ go build
$ ./dwarf2json --help
Usage: ./dwarf2json COMMAND

A tool for generating intermediate symbol file (ISF)

Commands:
  linux  generate ISF for Linux analysis
  mac    generate ISF for macOS analysis

Now, with this program, you can create an ISF file format. There are two components required for creating an ISF file:

1. Linux kernel DWARF file (vmlinux)

2. System.Map

In simple terms, a vmlinux file is required, and if there is an additional system.map file, more accurate results can be obtained. If there are conflicts between the provided information, it is said that the system.map information takes precedence. However, based on my execution, it seems that the analysis itself can be done even if only the vmlinux file is provided (analysis is not possible with only the system.map file).

So, how can the vmlinux file be obtained? First, if you go to the /boot directory, you can see a file named vmlinuz-5.4.0-26-generic by default. However, this is a compressed form of the vmlinux file. What we need is the original vmlinux file. One might think that simply decompressing it would suffice, but that is not the case. When vmlinux is compressed to vmlinuz, a considerable amount of kernel debugging information is deleted (stripped) and lost. The information we want to find is included in such debugging data, so simply decompressing it will not be satisfactory.

Therefore, the corresponding version of the vmlinux file must be obtained. First, check the exact version information of the Linux system (if it is an arbitrary memory image, check the banner plugin information):

$ uname -r
5.4.0-26-generic

To install debug symbols, refer to the following page based on Ubuntu (Debug Symbol Packages:

$ echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

$ sudo apt install ubuntu-dbgsym-keyring

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F2EDC64DC5AEE1F6B9C621F0C8CAB6595FDFF622

$ sudo apt-get update

Now, you can input the dbgsym information of the desired kernel image version:

$ sudo apt install linux-image-5.4.0-26-generic-dbgsym
$ cd /usr/lib/debug/boot
$ ls -lh 
total 742M
-rw-r--r-- 1 root root 742M Apr 20  2020 vmlinux-5.4.0-26-generic

The vmlinux file containing kernel debug symbols is located in the /usr/lib/debug/boot directory, as shown above. You can use dwarf2json to extract the ISF output.json file based on vmlinux file.

$ ./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-26-generic > output.json

If you have also obtained the system.map file (check if it exists under the /boot directory), you can combine it:

$ sudo ./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-26-generic --system-map /boot/System.map-5.4.0-26-generic > output2.json

Save the created output.json file to the ~/volatility3/volatility3/symbols/ directory. At this point, the name of the JSON file is not important, as searches are based on the contents of the file. However, for convenient management, it is recommended to use the kernel name and specify a filename like [kernel name].json. For example, you can name it something like linux-image-4.9.0-3-amd64-dbg_4.9.30-2+deb9u2_amd64.json.

Plugins for Linux memory forensics

Now that you have secured the ISF file (either downloaded or created yourself), let's start using plugins to perform a forensic analysis on the memory image.

The full list of plugins for Linux is as follows:

Indeed, the number of plugins offered is still limited compared to Windows. Moreover, some even produce less comprehensive results compared to Volatility 2.

In the early stages, the information provided by the pslist plugin was insufficient.

That's why I contributed to the improvement of the linux.pslist plugin. In the current version of Volatility 3, information is provided as follows:

If you are interested in developing plugins for Volatility 3, I highly recommend referring to this document (https://volatility3.readthedocs.io/en/latest/development.html) and actively giving it a try. There is also an annual plugin contest held.

Conclusion

In summary, this discussion has focused on the usage of Volatility 3, a powerful and versatile memory forensics framework, specifically for Linux memory analysis. I explored the process of obtaining memory dumps using the AVML tool, transitioning from the traditional profile concept to the newer symbol table approach using ISF files, and obtaining or creating these ISF files as needed. Additionally, we examined various plugins available for Linux memory forensics and discussed improvements made to the pslist plugin.

In conclusion, the field of memory forensics is continually evolving, and Volatility 3 plays a significant role in advancing research and development. By learning how to use this tool effectively, practitioners can enhance their ability to analyze and understand memory dumps, leading to more accurate and efficient investigations. As the framework continues to grow, so does the potential for further improvements and additions to its capabilities, making it essential for those in the field to stay up to date with the latest developments and actively contribute to the community.