In recent years, AI and Generative AI have made significant advancements, but their practical use in the field of Penetration Testing is still met with doubt. In this article, we’ll share how we used Generative AI to automate the process of handling multiple HTTP requests during penetration testing.

The Challenge and Context

During penetration testing, pentester often deal with HTTP-based targets like web applications and APIs. Many functions require sending multiple HTTP requests, where each request’s parameters depend on the previous one, forming a chain to make the workflow valid. For example, authentication tokens change constantly, or actions like CSRF and banking transactions require specific token values. This often causes vulnerability scanning tools to work incorrectly, as they cannot recognize which requests need to be chained together or which parameters require updated values. Manual testing faces similar challenges, to execute the function correctly, a pentester must send multiple requests in sequence and manually copy-paste to modify values. This process is boring and can lead to mistakes. Some tools offer support, like Burp Suite’s Macro configuration and AppScan’s Multi-Step feature, which can help address these issues. However, they still require manual setup for each request and specifying which values need to be updated.

Overall, this is a repetitive but necessary task to improve accuracy. However, in real projects, configuration takes significant time regardless of the solution used. That’s why we aimed to find a way to automate multi-request flows with a single click and minimal setup.

Nuclei and Its Applicability to the challenge

Nuclei is a popular open-source tool in the pentesting community. It allows users to create templates that automatically send HTTP requests to scan applications. One key feature of Nuclei is its flexible customization, particularly its ability to extract values from responses (extractors), making it a great solution for handling multiple HTTP requests and updating parameters automatically.

The example above shows a Nuclei template that sends the first HTTP request to retrieve a CSRF token and then automatically replaces the value in subsequent requests to maintain the correct flow.

Using Generative AI to create Nuclei templates with multiple HTTP requests

After choosing Nuclei as the core of our solution, we set out to incorporate AI into the process. The goal was to automatically generate templates, detect parameters that need to be extracted, and update their values.

The idea we used is as follows: We start with a list of HTTP requests from Burp Suite history. The tester indexes the sequence, and then the HTTP requests and responses are processed by AI to generate a Nuclei template that automatically extracts and updates parameters.

From Concept to Product

We developed a Burp Suite extension with the following features:

Group HTTP requests in sequence
Send the raw requests and response to an AI model to generate a Nuclei template with auto-updated parameters
Store the generated Nuclei template
Execute the Nuclei template

Thanks to the open-source Nuclei Burp Plugin by forgedhallpass, we were able to easily build a clean and user-friendly interface for our solution.

To demonstrate the product, we used a functional flow consisting of: Login → View Account Information → Change Email. To complete this flow successfully, several steps are required:

Retrieve a CSRF token before logging in
Obtain a session containing the authenticated login state
Retrieve a CSRF token before changing the email

Figure 1: Group the requests and load them into the extension

Figure 2: After the Nuclei template is generated, run a test through a proxy to verify the effectiveness of parameter value substitution

Figure 3: The results show that after executing the template, it followed the correct sequence and successfully updated the required parameters with valid values

How We Did It

To generate the desired template, we used the GPT-4o model and prompt engineering. We began by providing examples of raw HTTP requests and responses along with a Nuclei template. We explained the syntax and pointed out which values should be updated to help the model understand the input and the structure of the template. Then, we designed detailed prompts to ensure the model knew exactly what to generate.

Some key points for effective prompt engineering include:

Explaining the Nuclei template syntax and attributes in detail to avoid generating invalid YAML files
Clearly specifying when and how parameters should be extracted to prevent unnecessary or incorrect extractions
Using a variety of examples covering different scenarios, like login, CSRF, transaction creation, etc
HTTP requests and responses can be very long, so it's important to optimize the content by removing unnecessary data before feeding it into the model to avoid performance issues
Ensuring AI safety by masking or removing sensitive data like credentials

Integration into Daily Penetration Testing

After proving the effectiveness of our solution, we started using this Burp Suite Extension within our team:

Creating templates to scan for vulnerabilities in functions that require multiple chained requests.
Automating re-testing: Instead of manually logging in and navigating to the vulnerable function each time, the pentester generates a full workflow template right after discovering the vulnerability. When a re-test is requested, simply executing Nuclei with that template is enough to verify whether the issue still exists.

Conclusion

Through this project, we have moved past our previous doubts about using AI in pentesting and are now open to exploring and developing other tools that can assist and reduce manual effort in this process. Nuclei Multi-Step Template Gen, powered by AI, has streamlined multi-step function pentesting tasks, ensuring accuracy and efficiency. As AI continues to improve, we are excited about its potential to further optimize and transform the way pentests are conducted.

References

https://portswigger.net/burp/documentation/desktop/settings/sessions/macros
https://help.hcl-software.com/appscan/Standard/10.7.0/r_ExploreMultiStepOperations063.html
https://docs.projectdiscovery.io/tools/nuclei/overview
https://docs.projectdiscovery.io/templates/reference/extractors
https://github.com/projectdiscovery/nuclei-burp-plugin

Nuclei Multi-Step Template Gen: AI-Powered Automation for Chain-Based Functions in Penetration Testing

Table of contents