Network Traffic Data

The network data is everywhere. It is all around us. Even now in this very task.

Network communication and traffic are the natural behaviours of today's interconnected computing world. These behaviours represent a constant data flow of daily activities, including personal interactions and business transactions. The data flow offers invaluable network management, troubleshooting, incident response, and threat-hunting insights. The table below highlights the importance and key benefits of each of these aspects:

| Network Management | Monitoring network performance, identifying bandwidth bottlenecks, and ensuring resource allocation and quality of service. | | --- | --- | | Troubleshooting | Identifying network issues (latency and connectivity issues), validating configuration implementations and changes, and setting performance baselines. | | Incident Response | Incident scope, root cause analysis, and assessment of the compliance aspects of incidents and day-to-day operations. | | Threat Hunting | Proactive analysis for signs of suspicious and malicious patterns, potential threats, anomalies, and IoCs.
Also, behavioural analysis is used to detect intrusions and insider threats. |

Network traffic comes in various data types and formats. Packet capture (PCAP) format (also known as full packet captures) is the first thing that comes to mind. It provides a granular, raw, and comprehensive view of the network traffic. This format provides all possible data represented in packets in a ready-to-investigate format (this approach is also known as deep packet inspection). Therefore, it is an invaluable artefact for network-level operations.

However, this intensive resource needs storage, processing, and analysis capacities to provide comprehensive insight into network traffic. In other words, while PCAPs are very useful for detailed analysis, they are not practical for fast analysis situations as they enclose the actual payload. This situation becomes a pain point when large amounts of data need to be analysed.

The data richness and level of detail provided by the PCAP format come from the payload it carries. At this point, it will be possible to speed up the process considerably by running the analysis process on a data format that doesn't enclose the payload data. As a result, it will be possible to process more data in a shorter time with fewer resources, leaving more time for analysis and decision-making.

Network flow data is a lightweight alternative to PCAPs. It's commonly used in NetFlow format, a telemetry protocol developed by Cisco that focuses on the metadata part of the traffic. In other words, it provides only the "summary" of the traffic; the details appear similarly to how call details appear on your phone bill. Once again, there are no packet content details with this format. This is why storing, processing, and analysing this data format is easier than it is with PCAPs.

It looks like this data format will help the team accomplish the task McSkidy assigned to them!

A Closer Look at PCAPs and Flows

Let's take a closer look at these two formats to see how they differ and understand what to expect from each one.

Note: If you're still unfamiliar with networking terminology and the basics of this task, you can always get help from the rooms listed in the Network Fundamentals module.

Feature	PCAP	Network Flow
Model	Packet capture	Protocol flow records
Depth of Information	Detailed granular data Contains the packet details and payload	Summary data Doesn't contain the packet details and payload
Main Purpose	Deep packet analytics	Summary of the traffic flow
Pros	Provides high visibility of packet details	Provides a high-level summary of the big picture Encryption is not an obstacle (the flows don't use the packet payload)
Cons	Hard to process, and requires time and resources to store and analyse Encryption is an obstacle	Summary only; no payload
Available Fields	Layer headers and payload data	Packet metadata

The table above highlights the conceptual differences between PCAP and network flow at a high level. Now, let's get into a more technical comparison of these two formats and understand why McSkidy chose this approach as a quick solution for understanding overall network traffic activities.

Let's continue with McSkidy's suggestion: explore and use SiLK to help SSOC in this task.

Getting Started With the SiLK Suite

The SiLK suite has two parts: the packing system and the analysis suite. The packing system supports the collection of multiple network flow types (IPFIX, NetFlow v9, and NetFlow v5) and stores them in binary files. The analysis suite contains the tools needed to carry out various operations (list, sort, count, and statistics) on network flow records. The analysis tools also support Linux CLI pipes, allowing you to create sophisticated queries.

The VM contains a binary flow file (suspicious-flows.silk) in the /home/ubuntu/Desktop directory. You can verify this by clicking the Terminal icon on the desktop and executing the following commands:

Changing directory: cd Desktop
Listing directory items: ll

Given artefacts

user@tryhackme:~$ cd Desktop
user@tryhackme:~/Desktop$ ll
drwxr-xr-x  4 ubuntu ubuntu   4096 Nov 20 06:28 ./
-rw-r--r--  1 ubuntu ubuntu 227776 Nov 17 21:41 suspicious-flows.silk

The next step is discovering the details of the pre-installed SiLK instance in the VM. Use the commands provided to verify the SiLK suite's installation. Use the following command to verify and view the installation details:

silk_config -v

SiLK Suite

user@tryhackme:~/Desktop$  silk_config -v

silk_config: part of SiLK [REDACTED].........; configuration settings:
    * Root of packed data tree:         /var/silk/data
    * Packing logic:                    Run-time plug-in
    * Timezone support:                 UTC
    * Available compression methods:    lzo1x [default], none, zlib
    * IPv6 network connections:         yes
    * IPv6 flow record support:         yes
    * IPset record compatibility:       3.14.0
    * IPFIX/NetFlow9/sFlow collection:  ipfix,netflow9,sflow
[REDACTED]..

SiLK mainly works on a data repository, but it can also process data sources not in the base data repository. By default, the data repository resides under the /var/silk/data directory, which can be changed by updating the SiLK's main configuration file. Note that this task's primary focus is using the SiLK suite for analysis. Therefore, we will only use the network flows given by the SSOC team.

Quick win that will help you answer the questions: You now know which SiLK version you are using.

Flow File Properties with SilK Suite: rwfileinfo

One of the top five actions in packet and flow analysis is overviewing the file info. SiLK suite has a tool rwfileinfo that makes this possible. Now, let's start working with the artefacts provided. We'll need to view the details of binary flow files using the command below:

rwfileinfo FILENAME

File info

user@tryhackme:~/Desktop$ rwfileinfo suspicious-flows.silk
suspicious-flows.silk:
  format(id)          FT_RWIPV6ROUTING(0x0c)
  version             16
  byte-order          littleEndian
  compression(id)     lzo1x(2)
  header-length       88
  record-length       88
  record-version      1
  silk-version        [REDACTED]...
  count-records       [REDACTED]...
  file-size           152366

This tool helps you discover the file's high-level details. Now you should see the SiLK version, header length, the total number of flow records, and file size.

Quick win that will help you answer the questions: You now know how to view the sample size in terms of count records.

Reading Flow Files: rwcut

Rwcut reads binary flow records and prints those selected by the user in text format. It works like a reading and filtering tool. For instance, you can open and print all the records without any filter or parameter, as shown in the command and terminal below:

rwcut FILENAME

Note that this command will print all records in your console and stop at the last record line. Investigating all these records at once can be overwhelming, especially when working with large flows. Therefore, you need to manage the rwcut tool's output size using the following command:

rwcut FILENAME --num-recs=5
This command limits the output to show only the first five record lines and helps the analysis process.
NOTE: You can also view the bottom of the list with --tail-rec=5

rwcut

user@tryhackme:~/Desktop$ rwcut suspicious-flows.silk --num-recs=5

            sIP|           dIP|sPort|dPort|pro|pks|byts|flgs|               sTime| dur  |.                  eTime|
175.215.235.223|175.215.236.223| 80| 3222| 6| 1| 44| S A |2023/12/05T09:33:07.719| 0.000| 2023/12/05T09:33:07.719|
175.215.235.223|175.215.236.223| 80| 3220| 6| 1| 44| S A |2023/12/05T09:33:07.725| 0.000| 2023/12/05T09:33:07.725|
175.215.235.223|175.215.236.223| 80| 3219| 6| 1| 44| S A |2023/12/05T09:33:07.738| 0.000| 2023/12/05T09:33:07.738|
175.215.235.223|175.215.236.223| 80| 3218| 6| 1| 44| S A |2023/12/05T09:33:07.741| 0.000| 2023/12/05T09:33:07.741|
175.215.235.223|175.215.236.223| 80| 3221| 6| 1| 44| S A |2023/12/05T09:33:07.743| 0.000| 2023/12/05T09:33:07.743|

Up to this point, we read flows with rwcut. Now, let's discover the filtering options offered by this tool. Re-check the output; it's designed by column categories, meaning there's a chance to filter some. Rwcut has great filtering parameters that will help you do this. At this point, the --fields parameter will help you extract particular columns from the output and make it easier to read.

rwcut FILENAME --fields=protocol,sIP,sPort,dIP,dPort --num-recs=5
This command shows the first five records' protocol type, source and destination IPs, and source and destination ports.

rwcut filters

user@tryhackme:~/Desktop$ rwcut suspicious-flows.silk --fields=protocol,sIP,sPort,dIP,dPort --num-recs=5

pro|              sIP|sPort|             dIP|dPort|
  6|  175.215.235.223|   80| 175.215.236.223| 3222|
  6|  175.215.235.223|   80| 175.215.236.223| 3220|
  6|  175.215.235.223|   80| 175.215.236.223| 3219|
  6|  175.215.235.223|   80| 175.215.236.223| 3218|
  6|  175.215.235.223|   80| 175.215.236.223| 3221|

Note that IANA assigns internet protocol numbers. Examples: ICMP = 1, IPv4 = 4, TCP = 6, and UDP = 17.

Filtering the Event of Interest: rwfilter

We've covered how to read and filter particular columns with rwcut, but we'll need to implement conditional filters to extract specific records from the flow. rwfilter will help us implement conditional and logical filters to extract records for the event of interest.

rwfilter is an essential part of the SiLK suite. It comes with multiple filters for each column in the sample you're working on and is vital for conducting impactful flow analysis. However, even though rwfilter is essential and powerful, it has a tricky detail: it requires its output to be post-processed. This means that it doesn't display the result on the terminal, and as such, it's most commonly used with rwcut to view the output. View the examples below:

rwfilter FILENAME
This command reads the flows with rwfilter and retrieves an output error as the output option is not specified.

rwfilter output error

user@tryhackme:~/Desktop$ rwfilter suspicious-flows.silk 
rwfilter: No output(s) specified
Use 'rwfilter --help' for usage

The command is missing filtering and passing output options, which is why it didn't provide any result in return. Let's explore the essential filtering options and then pass the results to rwcut to view the output.

Remember Elf Forensic McBlue's hints on protocols and decimal representations. Let's start by filtering all UDP records using the protocol filter and output-processing options.

rwfilter FILENAME --proto=17 --pass=stdout | rwcut --num-recs=5
This command filters all UDP records with rwfilter, passes the output to rwcut and displays the first five records with rwcut.
NOTE: The --pass=stdout | section must be set to process the output with pipe and rwcut.

rwfilter and output-processing with rwcut

user@tryhackme:~/Desktop$ rwfilter suspicious-flows.silk --proto=17 --pass=stdout | rwcut --fields=protocol,sIP,sPort,dIP,dPort --num-recs=5

pro|              sIP| sPort|             dIP| dPort|
 17|  175.175.173.221| 59580| 175.219.238.243|    53|
 17|  175.219.238.243|    53| 175.175.173.221| 59580|
 17|  175.175.173.221| 47888| 175.219.238.243|    53|
 17|  175.219.238.243|    53| 175.175.173.221| 47888|
 17|  175.175.173.221| 49950| 175.219.238.243|    53|

Protocols: --proto
- Possible values are 0-255.
Port filters:
- Any port: --aport
- Source port: --sport
- Destination port: --dport
- Possbile values are 0-65535.
IP filters: Any IP address: --any-address
- Source address: --saddress
- Destination address: --daddress
Volume filters: Number of the packets --packets number of the bytes --bytes

Quick Statistics: rwstats

Up to this point, we have covered fundamental tools that help provide some statistics on traffic records. It's now time to speed things up for a quicker and more automated overview of the events.

Before you start to work with rwstats, you need to remember how to use the --fields parameters we covered in the rwfilter section to fire alternative filtering commands for the event of interest. If you need help using these parameters, return to the rwfilter section and practice using the parameters provided. If you are comfortable with the previous tools, let's move on and discover the power of statistics!

rwstats FILENAME --fields=dPort --values=records,packets,bytes,sIP-Distinct,dIP-Distinct --count=10
- --count: Limits the number of records printed on the console
- --values=records,packets,bytes: Shows the measurement in flows, packets, and bytes
- --values=sIP-Distinct,dIP-Distinct: Shows the number of unique IP addresses that used the filtered field
This command shows the top five destination ports, which will help you understand where the outgoing traffic is going.

rwstats and top 5 destination ports

user@tryhackme:~/Desktop$ rwstats suspicious-flows.silk --fields=dPort --values=records,packets,bytes,sIP-Distinct,dIP-Distinct --count=10

dPort| Records| Packets| Bytes|sIP-Distinct| dIP-Distinct|  %Records| cumul_%|
   53|    4160|    4333|460579|           1|            1|[REDACTED]|35.33208|
   80|    1658|    1658| 66320|           1|            1| 14.081875|49.41396|
40557|       4|       4|   720|           1|            1|  0.033973|49.44793|
53176|       3|       3|   465|           1|            1|  0.025480|49.47341|
[REDACTED]...