I walked out of bed this morning with my bed cover still around me (it was cold), to my workspace. I had taken my camera for a hangout yesterday, and I needed to fix it up, together with all the many connections that make up my work equipment. It made sense to fix up the camera since my laptop’s camera didn’t have great quality. Despite being a decent laptop, it is normal for PC manufacturers not to pay attention to their cameras, making them subpar, and focusing on other aspects of the laptop.

I opened the lid, and the power button blinked rapidly, meaning the battery was empty. It died while asleep, and it struck me that this was the second night in a row it was doing that. Two nights ago (Saturday night), I woke it up from the dead, and it was still hot, like someone had used it to train GPT-3 while keeping it in an airtight box with zero airflow. I shrugged it off: Ubuntu didn’t have as great hardware support as Windows, and it rarely happened this way when I ran Windows. Since I dropped my MacBook last year, the only thing I missed from more supported OSes among PC manufacturers was the battery life. This didn’t matter much to me since I always worked from home.

I left the workspace to the living room to get my camera out of my bag. The camera bag hadn’t seen much use and didn’t squeak when I zipped it open. Fixing the camera and attaching the necessary support cords was a simple recital in my head: “Type C goes up, don’t look; micro USB goes down, you need to look”. All I had to do was plug in the docking station, and I could open up Slack and get ready for the day. I instinctively dragged my index finger across the touchpad to move it around when it happened.

The problem

The cursor on the screen would not move. I tried it several times, moving my index finger frantically with no response. Something was wrong. This had never happened before. I pressed the power button for about 5 seconds and rebooted. It came on, and everything was cool. I continued my routine, opening up three to four Chrome windows (between them is usually about 200 tabs, I think I will need some time in the future), Slack, and VSCode (I tend to forget I have Cursor). Then it happened again. This time, the cursor slowed down to a crawl and then stopped moving. A dialog popped up quickly saying something about a process consuming too much memory, and it closed immediately.

I knew what to do. I would restart and open System Monitor, which I did.

I monitored the processes, and all looked normal.

The diagnosis

I knew there was a memory leak or something consuming memory, but I didn’t know what it was. My first guess was Chrome: Maybe I had updated it as part of the auto-upgrades, and that update was buggy. I started closing tabs, and I noticed that each tab killed a Chrome process flagged with type=renderer. Ehn ehn! So that was what those things were…

I pressed CTRL + F, searched “renderer,” and started mass-closing tabs. Memory usage started to drop. I smiled.

I had solved it.

Just before I heaved a sigh of relief... it spiked again.

This was baffling. I went back to the “processes” tab and ordered the processes by memory; “gnome-shell” came up first. My ignorant self quickly googled why “Gnome UI would have a shell”. Apparently, it was just a UI shell (mobile app devs would get this).

I had a meeting in ten minutes, and the laptop was unusable. I had no choice but to switch to Windows. Something was eating memory, and since I couldn’t find it on System Monitor, it was definitely malicious. I HAD BEEN HACKED!

Four Hours Later

I had just spent the last 3 hours setting up my Windows environment for work: SSO stuff, email signings, app setups, VSCode and cursor downloads, Chrome Profile for work setup, Zoom download, and a billion other things. I was sick of Windows: the lights were too bright, and I had to look for Night Light to turn it down. My monitor did not automatically respond to the “Reduce Brightness” button that worked for my laptop screen. This was too much for my soul, I blacked out and found good food to eat. Perhaps I needed more food. But really, I wanted to go back to Ubuntu… and I did.

Try Again, Try Again

Wake me up, try again, try again.. I went back to Ubuntu with renewed vigour. I had to fix this and find the malware.

I sprinted to System Monitor and opened the “Resources” tab, watching the RAM go from about 7% usage to 91% and counting while all 16 cores of the CPU were at 99% or 100%, depending on which side of the coin you flip. The first alarm popped in my head: A Python script must be running in the background, mining bitcoin or something! It then occurred to me: if that was the case, I should have seen the script in the Processes tab.

Hail Hey Hai

My next stop was ChatGPT. I was bursting with questions as I typed rapidly. I knew I didn’t have time since the laptop would crash soon. The prompt was easy:

I may have a virus running on my Ubuntu laptop. Two days ago, I got a message about something on partial upgrade and I upgraded. Today, my laptop shows 99.9% COU usage across all 16 cores and the RAM usage keeps increasing. I have checked system monitor and nothing is showing that is using the memory. The highest memory using process is gnome which is using 300 mb. I have 32gb ram

It was true, I got a message about a Partial Upgrade, and only after the upgrade completed did I Google what it meant. The results on Google weren’t encouraging, with stories of doom, gloom, and dreary upvotes.

Debugging

Following instructions, I started up the Terminal CTRL + ALT + T and ran

$ top
top - 15:14:33 up 0 min,  2 users,  load average: 2.71, 0.87, 0.31
Tasks: 519 total,   3 running, 516 sleeping,   0 stopped,   0 zombie
%Cpu(s): 11.4 us, 12.7 sy,  0.0 ni, 74.7 id,  1.3 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :  27351.4 total,  16475.1 free,   7028.3 used,   4344.5 buff/cache     
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  20323.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                     
   6224 1001      20   0   33.6g 998.4m  56808 S 200.0   3.7   0:33.41 node                                                                        
  19012 root      20   0   30348  25368  14652 R 100.0   0.1   0:00.12 test_ebpf                                                                   
  13659 lordsar+  20   0  572392  66804  56332 R  50.0   0.2   0:00.50 gnome-terminal-                                                             
  11706 lordsar+  20   0 5197436 442064 177624 S  25.0   1.6   0:03.30 gnome-shell                                                                 
  18547 lordsar+  20   0   14104   6648   4472 R  25.0   0.0   0:00.03 top                                                                         
      1 root      20   0   29496  18724  10916 S   0.0   0.1   0:03.37 systemd                                                                     
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd                                                                    
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_workqueue_release                                                      
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-rcu_gp                                                            
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-sync_wq                                                           
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-kvfree_rcu_reclaim                                                
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-slub_flushwq                                                      
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/R-netns                                                             
      9 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0-events

The first entry caught my eye: USER: 1001. I had seen it somewhere in the past two days but I can’t remember where, perhaps from Kafka documentation? COMMAND: node. Wait a minute, or did the hacker write his script in Node? First, I need to see what processes were running that included “node” with:

$ ps aux | grep node
root        1155  0.0  0.0   9356  2136 ?        Ss   15:13   0:00 /sbin/mount.ntfs /dev/nvme0n1p3 /mnt/ECD203C6D20393CC -o rw,nosuid,nodev
1001        6217  0.2  0.1 701312 53536 ?        Sl   15:13   0:00 node /app/node_modules/.bin/nodemon src/Bootstrap.ts
1001        6224  108  8.8 72423280 2490336 ?    Sl   15:13   1:36 node /app/node_modules/.bin/ts-node src/BootstrapWorker.ts
1001        6480  7.6  0.9 11400196 259400 ?     Sl   15:13   0:06 node /app/node_modules/.bin/ts-node src/Bootstrap.ts
nobody      9264  0.0  0.0 1241440 18704 ?       Ssl  15:13   0:00 /bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --path.udev.data=/host/root/run/udev/data --web.listen-address=[0.0.0.0]:9100
root       11505  0.0  0.0   2776  2100 ?        Ss   15:13   0:00 fusermount3 -o rw,nosuid,nodev,fsname=portal,auto_unmount,subtype=portal -- /run/user/1000/doc
lordsar+   18575  0.8  0.1 1286172 54024 ?       Ssl  15:14   0:00 /metrics-server --cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
nobody     19216  0.3  0.1 1271264 52888 ?       Ssl  15:14   0:00 /kube-state-metrics --port=8080 --telemetry-port=8081 --port=8080 --resources=certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
lordsar+   24373  0.0  0.0   9372  2480 pts/0    S+   15:15   0:00 grep --color=auto node

Which, long story short, shows all processes running Node.js and who is running them.

Back to the investigation, what was Bootstrap.ts and BootstrapWorker.ts? That was when it hit me:

I was the hacker!

During the weekend, I spent hours reading Kafka documentation and trying to run Kafka in Kraft mode without Zookeeper (which has now been deprecated for Kafka, thanks Eric for telling me this specific information). I remember putting restart: always in the docker-compose.yml file for Kafka, two background worker containers, and a web service container. I also remember the Kafka setup not working correctly from the workers which had their entry point in BootstrapWorker.ts. Each worker looked like so:

const createNotificationWorker = async (app: App<{ kafka: Kafka }>) => {
    // redacted for unknown reasons yet
}

(async (workerEntryPoint, kafka) => {
    (await new App<{ kafka: Kafka }>({workerEntryPoint, meta: {kafka}}).start())
})(
    createNotificationWorker,
    new Kafka({
        clientId: "create-notification-worker",
        brokers: env.KAFKA_BROKERS,
        logLevel: logLevel.INFO,
        ssl: false,
        sasl: {
            mechanism: 'plain',
            username: env.KAFKA_USERNAME,
            password: env.KAFKA_PASSWORD,
        },
    })
);

Ho ma gad! Apparently, Docker had been running nonstop in the background, and the worker process kept eating memory until the laptop crashed. But we had to confirm:

I quickly typed:

$ docker ps
adb0bc608726   notifications-worker   "docker-entrypoint.s…"   36 hours ago   Up About a minute   3000/tcp                                      notification-service-worker
274cfe30a048   notifications-app      "docker-entrypoint.s…"   36 hours ago   Up About a minute   0.0.0.0:4000->3000/tcp, [::]:4000->3000/tcp   notification-service
61117acfd121   mongo:7.0-jammy        "docker-entrypoint.s…"   36 hours ago   Up About a minute   27017/tcp                                     mongodb

Ho mein Gott!

I killed them all with:

$ docker stop adb0bc608726 274cfe30a048 61117acfd121

Tears streamed down my face as I checked again to see if anything else was the culprit. System Monitor showed the CPU usage back to normal, RAM back to normal, and SWAP memory down to 0. All was saved, and everything was good.

Conclusion

Happy story, but I may have never found the cause. Thanks to a few tools ear and dear, and I was good to go. My wife just walked in, saw the title, and exclaimed in fear, “Did they hack you?!”. “No they didn’t”, I smiled. You will read the article.

Who Hacked Me On Ubuntu?