Capturing Multiple Requests on the Same Connection with eBPF

To incorporate the keywords like "HTTP 403," "HTTP error 503," "the service is unavailable," and "monitor Google Cloud API traffic" into the blog, I would recommend integrating them naturally into the content. Additionally, for internal linking from Keploy.io's website blogs, here’s a possible update to your blog, integrating the mentioned keywords and linking:

Why Capturing HTTP Traffic is Crucial for Network Security?

Capturing HTTP traffic is like having a digital security camera at your network's entrance, and it's essential for keeping your network safe. This 'camera' helps you monitor who's accessing your online space, allowing you to detect and prevent potential security threats, such as cyberattacks and unauthorized access before they can harm your system. This proactive approach is a fundamental practice to ensure the security and integrity of your network.

For example, you may encounter HTTP status codes like HTTP 403 (Forbidden) or HTTP Error 503 (The service is unavailable) when API calls or services are restricted or down, respectively. Monitoring your network traffic can help identify such errors and prevent unwanted access or diagnose service disruptions, which is essential for maintaining the availability of your services.

Behind the Scenes: The Work That Led to These Words

I was working on tracking my ingress HTTP traffic, and for that, I used the eBPF workshop repository from Datadog. It helped me keep a close eye on my network traffic easily and effectively, giving me useful insights for my project.

I discovered that monitoring ingress HTTP traffic is feasible by attaching it to system calls such as accept/accept4, read, write, and close. This can be achieved by employing the kprobe feature of eBPF, which allows for applying hooks effectively.

System Call Flow for an HTTP Request

Encountering Hurdles: The Challenge of Connection Pooling
Nonetheless, I stumbled upon an issue. When connection pooling was active, capturing traffic data for every individual request became unattainable. This situation presented a substantial barrier to my work. My objective was to meticulously document each request and response with every API interaction. This proved to be a considerable challenge in my experience with their ingress traffic controller.

The Close System Call Dilemma
I found that with connection pooling, the close system call isn't executed for every request. In the existing codebase, the close system call's function is to confirm the total bytes read and written on a specific connection file descriptor on the user side. Consequently, eBPF dispatches an event encapsulating these details.

Kernel Data Events: Reading and Writing

Each time a read or write system call is made, the kernel sends data events carrying the current bytes written or read. The total bytes read and written are accumulated for each individual read/write system call. This total is then checked against the total bytes read/write transmitted by the close event. After validation, the connection is released, and subsequent requests/responses employ different connection information.

type Tracker struct {
    connID structs2.ConnID

    addr          structs2.SockAddrIn
    openTimestamp uint64
    closeTimestamp    uint64
    totalWrittenBytes uint64
    totalReadBytes    uint64

    // Indicates the tracker stopped tracking due to closing the session.
    lastActivityTimestamp uint64
    sentBytes             uint64
    recvBytes             uint64

    recvBuf []byte
    sentBuf []byte
    mutex   sync.RWMutex
}

// This function verifies whether the request/response was complete or not
func (conn *Tracker) IsComplete() bool {
    conn.mutex.RLock()
    defer conn.mutex.RUnlock()
    return conn.closeTimestamp != 0 &&
        conn.totalReadBytes == conn.recvBytes &&
        conn.totalWrittenBytes == conn.sentBytes
}

// This function gathers data events that include information about 
// the size of data buffers and the actual data content for each 
// read/write system call in both the request and response.
func (conn *Tracker) AddDataEvent(event structs2.SocketDataEvent) {
    conn.mutex.Lock()
    defer conn.mutex.Unlock()
    conn.updateTimestamps()

    switch event.Attr.Direction {
    case structs2.EgressTraffic:
        conn.sentBuf = append(conn.sentBuf, event.Msg[:event.Attr.MsgSize]...)
        conn.sentBytes += uint64(event.Attr.MsgSize)
    case structs2.IngressTraffic:
        conn.recvBuf = append(conn.recvBuf, event.Msg[:event.Attr.MsgSize]...)
        conn.recvBytes += uint64(event.Attr.MsgSize)
    default:
    }
}

// This function collects the total bytes read & written on a single connection for an api request.
func (conn *Tracker) AddCloseEvent(event structs2.SocketCloseEvent) {
    conn.mutex.Lock()
    defer conn.mutex.Unlock()
    conn.updateTimestamps()
    if conn.closeTimestamp != 0 && conn.closeTimestamp != event.TimestampNano {
        log.Printf("changed close info timestamp from %v to %v", conn.closeTimestamp, event.TimestampNano)
    }
    conn.closeTimestamp = event.TimestampNano

    conn.totalWrittenBytes = uint64(event.WrittenBytes)
    conn.totalReadBytes = uint64(event.ReadBytes)
}

Refer to this for the above snippet.

Exploring Google Cloud API Traffic Monitoring

If you're dealing with Google Cloud APIs, keeping track of your network traffic is even more essential to ensure smooth service delivery. Monitoring Google Cloud API traffic can help identify potential issues like unauthorized access or HTTP error 503 (service downtime) during API interactions. By capturing and analyzing this traffic, you can pinpoint problematic requests and optimize your API performance.

The Vital Need to Capture Every Request and Response

In my current role as a core contributor at Keploy, the necessity to capture every request and response was paramount. Ensuring that no data was missed for every individual API request was crucial in enhancing Keploy's support system. Keploy stands as an open-source, user-friendly backend testing tool crafted with developers in mind. It not only simplifies backend testing for engineering teams but is also robust and easily extendable.

By recording API calls and database queries, Keploy efficiently generates test cases and data mocks/stubs from user traffic. This process markedly accelerates releases while boosting reliability, ensuring that your testing phase is both comprehensive and efficient.

Sample Test case generated using Keploy by capturing request and response metrics:

version: api.keploy.io/v1beta2
kind: Http
name: test-1
spec:
  metadata: {}
  req:
    method: POST
    proto_major: 1
    proto_minor: 1
    url: http://localhost:8082/url
    header:
      Accept: '*/*'
      Content-Length: "33"
      Content-Type: application/json
      Host: localhost:8082
      User-Agent: curl/7.81.0
    body: |-
      {
        "url": "https://google.com"
      }
    body_type: ""
  resp:
    status_code: 200
    header:
      Content-Length: "66"
      Content-Type: application/json; charset=UTF-8
      Date: Wed, 27 Sep 2023 02:17:20 GMT
    body: |
      {"ts":1695781040077972081,"url":"http://localhost:8082/Lhr4BWAi"}
    body_type: ""
    status_message: ""
    proto_major: 0
    proto_minor: 0
  objects: []
  assertions:
    noise:
      - header.Date
  created: 1695781040

Conclusion

And there we have it, folks! From the initial hurdles with connection pooling and capturing every request and response to exploring solutions and implementing a thread-safe method that worked whether connection pooling was enabled or not, the journey has been both enlightening and productive. It is through these challenges and their subsequent solutions that tools like Keploy can continue to grow and serve the developer community more efficiently.

FAQ’S

1. What is HTTP traffic monitoring, and why is it important?

HTTP traffic monitoring involves tracking the flow of data between servers and clients via HTTP protocols. It helps in identifying performance issues, unauthorized access, and security threats like DDoS attacks or malicious requests. Regular monitoring ensures optimal network performance and enhances system security. It also aids in troubleshooting and ensuring that services run smoothly.

2. How does Keploy help in API testing?

Keploy is an open-source backend testing tool that records API traffic and generates test cases automatically. It captures requests and responses to create accurate mocks and stubs, reducing manual effort in test creation. This results in faster release cycles and enhanced reliability. Keploy also supports generating test data from actual user traffic, making tests more realistic and robust.

3. What challenges arise with connection pooling during HTTP traffic monitoring?

Connection pooling can complicate HTTP traffic monitoring because it reuses the same connection for multiple requests. This results in difficulty in tracking individual request/response pairs, as system calls like "close" aren't invoked for every request. However, solutions like using queues to track requests can mitigate these challenges. Proper data validation techniques ensure accurate monitoring despite connection reuse.

4. What is eBPF, and how does it help in traffic monitoring?

eBPF (extended Berkeley Packet Filter) is a powerful Linux kernel feature used for real-time monitoring and network tracing. It allows users to attach programs to various system calls, such as read, write, and accept, to track HTTP traffic. This enables precise monitoring of data flow, providing insights into performance, security, and network behavior without altering the core system.