Vibe Coding with Amazon Q

Background

So at my current assignment, I am helping an "ESB" team with migrating from on-premise to the AWS. Their landscape is mainly containers on EKS Fargate and CDK Java as Infrastructure as Code (IaC). As my current Java skills are a bit rusty, last time I touched Java was during college 22 years ago. I am used to write CDK in TypeScript or Python.
Within the company, there are a couple of AWS accounts available; a Sandbox, an Internal and a Public account. Because the ESB application lives in an Internal account, due to data classification, external parties have to jump some hoops to get to the Fargate containers on the EKS cluster.

Architecture

Here is where my job comes in, I have to develop the ingress flow for the ESB application.
At the moment there is a third party DDOS protection service where the traffic is coming from. From there the traffic is forwarded to a special AWS account, called shared-ingress. In this account a Gateway Load Balancer will forward all traffic to an security-ingress account to be validated and inspected. If traffic is approved, it will be send back to the shared-ingress account to land in a private subnet on an Application Load Balancer. This load balancer will forward the traffic to the ESB internal account. To forward traffic from the load balancer to the internal account traffic, everything is hooked up via a Transit Gateway.

So looking at the IaC for the application, it is quite straightforward. Deploy an Application Load Balancer, create a target group, some listeners and certificates. So this is not the part I want to focus on.

Problem

Where I want to focus on is the security and compliance tool cdk-nag. Cdk-nag is tool which you can use to check your CDK applications on certain rule sets like HIPPA Security, NIST800 and PCI. Where it comes down to is, if you create your infrastructure without any warnings or errors from cdk-nag, your security hub won’t light up as a Christmas tree and you will be on the safe side. Who doesn’t want to create infrastructure as code following best practices.

So I’ve implemented cdk-nag in the applications and stacks. So in the part where you define the Infrastructure as Code. This was according to a colleague Java developer not the right place. His argument was that cdk-nag is more administrative tasks which should end up in the testing phase and not in the declaration phase. Also looking at blog on AWS, I saw that it was possible to do so, so challenge accepted. Unfortunately here my rusty Java skill came to the surface.

Writing the test in the test directory looked fine:

@Test
    public void testCdkNag() {
        // GIVEN
        App app = new App();

        // WHEN
        var stack = new IngressLoadbalancerStack(app, "CdkNagTestStack", StackProps.builder()
                .env(Environment.builder()
                        .account(“1234567890”)
                        .region("eu-central-1")
                        .build())
                .build());

        // Add CDK NAG to check for AWS Solutions best practices
        Aspects.of(stack).add(new AwsSolutionsChecks());

        // Suppress CDK NAG warnings for security group rules
        // NagSuppressions.addResourceSuppressionsByPath(stack, 
        //     "/CdkNagTestStack/ALBSecurityGroup/Resource",
        //     Arrays.asList(
        //         NagPackSuppression.builder()
        //             .id("AwsSolutions-EC23")
        //             .reason("Open ingress rules are required for this ALB use case, but should be restricted in production")
        //             .build()
        //     ));

        // Synthesize the stack to ensure all validations run
        app.synth();
    }

Even the test result showed that it was all good:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running nl.customer.aws.esb.shared.IngressLoadbalancerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 s -- in nl.customer.aws.esb.shared.IngressLoadbalancerTest
[INFO] nl.customer.aws.esb.shared.IngressLoadbalancerTest.testStack -- Time elapsed: 0.753 s
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Because earlier I used the cdk-nag in the declaration phase, I knew that at least 1 error should pop up to the surface, also commented out in the above snippit, AwsSolutions-EC23. This error nagged about that port 80 and 443 are configured to be opened for everyone, and should be limited. So where I would expect the error to pop up, it didn’t. So the test looked fine, but isn’t doing the job correctly.

So let’s dive in. Because I lack the correct Java knowledge, who better to ask than Amazon Q. Time for some prompt engineering. Looking at the blog, I want what was written there, but then for Java. So Q, do your magic here:

“It seems that my cdknag isn’t working in my unit test. I expect an error on AwsSolutions-EC23 as outcome, but my tests say everything ran successfully. What I want is this TypeScript code, see snippit, but then for Java.”

import { Annotations, Match } from 'aws-cdk-lib/assertions';
import { App, Aspects, Stack } from 'aws-cdk-lib';
import { AwsSolutionsChecks } from 'cdk-nag';
import { CdkTestStack } from '../lib/cdk_test-stack';

describe('cdk-nag AwsSolutions Pack', () => {
  let stack: Stack;
  let app: App;
  // In this case we can use beforeAll() over beforeEach() since our tests 
  // do not modify the state of the application 
  beforeAll(() => {
    // GIVEN
    app = new App();
    stack = new CdkTestStack(app, 'test');

    // WHEN
    Aspects.of(stack).add(new AwsSolutionsChecks());
  });

  // THEN
  test('No unsuppressed Warnings', () => {
    const warnings = Annotations.fromStack(stack).findWarning(
      '*',
      Match.stringLikeRegexp('AwsSolutions-.*')
    );
    expect(warnings).toHaveLength(0);
  });

  test('No unsuppressed Errors', () => {
    const errors = Annotations.fromStack(stack).findError(
      '*',
      Match.stringLikeRegexp('AwsSolutions-.*')
    );
    expect(errors).toHaveLength(0);
  });
});

With the new agentic approach of Amazon Q, Q guides you through all the steps taken:

I'll help you implement the advanced unit testing for your SecurityComplianceTest using the CDK Assertions library, similar to the TypeScript example you provided but adapted for Java.
Let me first check what dependencies you might need for this implementation:

1 directory listed

Let me check the pom.xml to understand the dependencies available:

1 file read

Now I'll update your SecurityComplianceTest.java file to include the advanced testing functionality similar to the TypeScript example:

I've updated your SecurityComplianceTest.java file with the advanced unit testing approach for cdk-nag. Here's what I've implemented:
1. Added the necessary imports for AWS CDK Assertions library:
    * software.amazon.awscdk.assertions.Annotations
    * software.amazon.awscdk.assertions.Match
2. Restructured the test class to follow the pattern from your example:
    * Added @TestInstance(TestInstance.Lifecycle.PER_CLASS) to allow using @BeforeAll with non-static methods
    * Created a setUp() method with @BeforeAll annotation to initialize the stack once for all tests
    * Split the testing into separate test methods for warnings and errors
3. Implemented two main test methods:
    * testNoUnsuppressedWarnings(): Checks that there are no unsuppressed AwsSolutions warnings
    * testNoUnsuppressedErrors(): Checks that there are no unsuppressed AwsSolutions errors
4. Kept a version of your original test as 
testVerboseComplianceCheck() which generates the detailed report
The implementation follows the same pattern as your TypeScript example:
* Set up the stack in a @BeforeAll method (equivalent to beforeAll() in Jest)
* Use the Annotations API to find warnings and errors
* Assert that there are no unsuppressed warnings or errors
This approach is more robust than the original implementation because:
1. It will actually fail the tests if there are unsuppressed warnings or errors
2. It uses the CDK Assertions library to check for issues programmatically
3. It separates the test setup from the assertions for better readability

Again, due to my lack in Java knowledge, the code looked fine. But when I tried to run the test, if failed. So I supplied the error to Q again.

[ERROR] /Users/yvanzee/Code/esb/internet-shared-infra/src/test/java/nl/Customer/aws/esb/shared/SecurityComplianceTest.java:[31,9] cannot assign a value to final variable app
[ERROR] /Users/yvanzee/Code/esb/internet-shared-infra/src/test/java/nl/Customer/aws/esb/shared/SecurityComplianceTest.java:[32,9] cannot assign a value to final variable stack
[ERROR] /Users/yvanzee/Code/esb/internet-shared-infra/src/test/java/nl/Customer/aws/esb/shared/SecurityComplianceTest.java:[46,73] incompatible types: java.util.List<software.amazon.awscdk.cxapi.SynthesisMessage> cannot be converted to java.util.List<java.lang.String>

Amazon Q:

I've fixed the compilation errors in your SecurityComplianceTest class. Here are the changes I made:
1. Fixed the type error:
    * Changed 
List<String> to List<SynthesisMessage> which is the correct return type for the Annotations.findWarning() and Annotations.findError() methods
2. Fixed the "cannot assign to final variable" errors:
    * Removed the 
@BeforeAll setup method and the class fields
    * Moved the stack and app creation into each test method
    * This approach is simpler and avoids the issues with final variables
3. Simplified the imports:
    * Removed unused imports like 
NagSuppressions and NagPackSuppression
    * Added the import for 
SynthesisMessage
4. Removed the 
@TestInstance annotation since it's no longer needed
5. Kept both test methods:
    * 
testNoUnsuppressedWarnings() to check for warnings
    * 
testNoUnsuppressedErrors() to check for errors
The code now follows the same pattern as your TypeScript example but is adapted to work correctly in Java. Each test creates its own stack, applies the AwsSolutionsChecks, and then verifies there are no unsuppressed warnings or errors.
This implementation is minimal but complete, focusing only on the core functionality needed to test for CDK Nag compliance.

Running the test finally gave me the Failure I was looking for. But I was missing the “correct” output. It only showed me it failed. So Q: how can I show in the test output the error of unsuppressed errors?

I'll modify your test to show the details of any unsuppressed errors in the test output.

<DOING Q Magic>

I've updated both test methods to show detailed information about any unsuppressed warnings or errors in the test output. Here's what I changed:
1. For both test methods, I replaced the simple 
assertEquals assertion with a more detailed approach:
    * If there are any warnings/errors, the test builds a detailed error message
    * The message includes the ID, message content, and source location for each issue
    * The test then fails with this detailed message using 
Assertions.fail()
2. The error message format is:
Found unsuppressed warnings/errors:
3.  - [ID]: [Message] at [Source]
4.  - [ID]: [Message] at [Source]
5. 
Copy
This approach gives you much more useful information when a test fails:
* You'll see exactly which CDK Nag rules are being violated
* You'll see where in your infrastructure code the violations occur
* You'll get the detailed message explaining what needs to be fixed
This makes it much easier to identify and fix compliance issues in your infrastructure code. When the tests fail, you'll have all the information you need to address the problems directly in the test output.

Well that all sounds nice. Let’s run the test to check if it is working. Unfortunately an error again… Q can you help fixing the error? after two more iterations it even asked me to run some code:

find /Users/yvanzee/Code/ESB/internet-shared-infra -name "*.jar" | grep -i cdk | xargs -I {} jar -tf {} | grep -i SynthesisMessage

After another fix by Q, and another error message posted by me we came to a working version:

I see the issue. The test is now showing more information, but we're still missing the actual error message content. Let's try to access the Entry object to get more details about the error.

Improving error reporting to extract more useful information from the Entry object
SecurityComplianceTest.java
+30
-14
Undo
Improving warning reporting to extract more useful information from the Entry object
SecurityComplianceTest.java
+30
-14
Undo
Undo all changes
I've improved the error and warning reporting in your SecurityComplianceTest class to extract more useful information from the CDK Nag messages. The key changes are:
1. For both errors and warnings:
    * First, I get the resource ID directly from the SynthesisMessage object
    * Then, I try to access the Entry object which contains the detailed error/warning information
    * From the Entry, I attempt to get the Data field which should contain the actual message
    * If that fails, I fall back to extracting any useful information from the Entry object using reflection
    * I filter out any proxy object references that aren't helpful (like Jsii$Proxy)
2. The output format is now more focused:
    * Resource=<resource-id> - Shows which resource has the issue
    * Message=<message> - Shows the actual error/warning message if available
    * Additional properties from the Entry object as needed
This approach should provide much more useful information about the CDK Nag issues in your test output, making it easier to identify and fix compliance problems in your infrastructure code.
The implementation is minimal but effective, focusing only on extracting the most important information while avoiding unnecessary complexity. It should work well with the AWS CDK library's internal structure and provide meaningful error messages when tests fail.

Because I think you should celebrate successes, I gave the feedback to Q as well:

Finally it is working! do you see optimalizations for the SecurityComplianceTest?

Finally it is working! do you see optimalizations for the SecurityComplianceTest?

I'm glad it's working! Here are some optimizations for your SecurityComplianceTest:
1. The code for handling warnings and errors is identical - let's extract that to a helper method to avoid duplication
2. The stack creation is also duplicated in both tests - we can refactor that
3. Remove unused imports
Here's how I would optimize it:

package nl.anwb.aws.icc.shared;

import software.amazon.awscdk.App;
import software.amazon.awscdk.Stack;
import software.amazon.awscdk.StackProps;
import software.amazon.awscdk.Environment;
import io.github.cdklabs.cdknag.AwsSolutionsChecks;
import io.github.cdklabs.cdknag.NagSuppressions;
import io.github.cdklabs.cdknag.NagPackSuppression;
import software.amazon.awscdk.Aspects;
import software.amazon.awscdk.assertions.Annotations;
import software.amazon.awscdk.assertions.Match;
import software.amazon.awscdk.cxapi.SynthesisMessage;
import org.junit.jupiter.api.Test;

import java.util.List;
import java.util.Arrays;

/**
 * Tests for security and compliance checks using CDK Nag.
 * This class verifies that all AWS resources in the stack comply with
 * AWS Solutions security best practices.
 */
public class SecurityComplianceTest {

    /**
     * Creates a test stack with CDK Nag checks applied.
     *
     * @return A CDK Stack with security checks enabled
     */
    private Stack createTestStack() {
        App app = new App();
        Stack stack = new IngressLoadbalancerStack(app, "TestStack", StackProps.builder()
                .env(Environment.builder()
                        .account("866588538139")
                        .region("eu-central-1")
                        .build())
                .build());

        // Apply CDK Nag checks
        Aspects.of(stack).add(new AwsSolutionsChecks());

        return stack;
    }

    /**
     * Validates that there are no unsuppressed messages of the specified type.
     * If any messages are found, the test will fail with detailed information
     * about each message.
     *
     * @param messages The list of SynthesisMessage objects to validate
     * @param type The type of message ("warning" or "error")
     */
    private void validateMessages(List<SynthesisMessage> messages, String type) {
        if (!messages.isEmpty()) {
            StringBuilder errorMessage = new StringBuilder("Found " + messages.size() + " unsuppressed " + type + ":\n");
            for (int i = 0; i < messages.size(); i++) {
                SynthesisMessage message = messages.get(i);
                errorMessage.append(" - ").append(type).append(" #").append(i + 1).append(": ");

                // Get the data directly from the message object
                errorMessage.append("Resource=").append(message.getId());

                // Try to get the Entry data which contains the actual message
                try {
                    Object entry = message.getClass().getMethod("getEntry").invoke(message);
                    if (entry != null) {
                        try {
                            Object data = entry.getClass().getMethod("getData").invoke(entry);
                            if (data != null) {
                                errorMessage.append(", Message=").append(data);
                            }
                        } catch (Exception e) {
                            // Try to get any useful information from the entry
                            java.lang.reflect.Method[] entryMethods = entry.getClass().getMethods();
                            for (java.lang.reflect.Method method : entryMethods) {
                                if (method.getName().startsWith("get") && 
                                    method.getParameterCount() == 0 && 
                                    !method.getName().equals("getClass")) {
                                    try {
                                        Object result = method.invoke(entry);
                                        if (result != null && !result.toString().contains("Jsii$Proxy")) {
                                            errorMessage.append(", ").append(method.getName().substring(3))
                                                      .append("=").append(result);
                                        }
                                    } catch (Exception ex) {
                                        // Skip methods that throw exceptions
                                    }
                                }
                            }
                        }
                    }
                } catch (Exception e) {
                    // Fall back to basic information
                    errorMessage.append(", Type=").append(type);
                }

                errorMessage.append("\n");
            }
            org.junit.jupiter.api.Assertions.fail(errorMessage.toString());
        }
    }

    /**
     * Tests that there are no unsuppressed warnings from AWS Solutions checks.
     * The test will fail if any CDK Nag warnings are found in the stack.
     */
    @Test
    public void testNoUnsuppressedWarnings() {
        // Create and check the stack
        Stack stack = createTestStack();

        // Find warnings
        List<SynthesisMessage> warnings = Annotations.fromStack(stack).findWarning(
                "*", 
                Match.stringLikeRegexp("AwsSolutions-.*")
        );

        // Validate warnings
        validateMessages(warnings, "warning");
    }

    /**
     * Tests that there are no unsuppressed errors from AWS Solutions checks.
     * Includes suppressions for known exceptions that are acceptable for this use case.
     * The test will fail if any unsuppressed CDK Nag errors are found in the stack.
     */
    @Test
    public void testNoUnsuppressedErrors() {
        // Create and check the stack
        Stack stack = createTestStack();

        // Add suppression for known issues
        // NagSuppressions.addStackSuppressions(stack,
        //     Arrays.asList(
        //         NagPackSuppression.builder()
        //             .id("AwsSolutions-EC23")
        //             .reason("Open ingress rules are required for this ALB use case, but should be restricted in production")
        //             .build()
        //     ),
        //     true);

        // Find errors
        List<SynthesisMessage> errors = Annotations.fromStack(stack).findError(
                "*", 
                Match.stringLikeRegexp("AwsSolutions-.*")
        );

        // Validate errors
        validateMessages(errors, "error");
    }
}

This optimized version:
1. Extracts common code into helper methods
2. Reduces duplication
3. Makes the test methods shorter and more focused
4. Maintains all the functionality of your original implementation

When testing with commenting the suppression and in code removed a logging bucket entry for the load balancer, I received a failed test with a correct output.

➜  internet-shared-infra git:(feat/cdk-nag_to_test) ✗ mvn test -Dtest=SecurityComplianceTest

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running nl.customer.aws.esb.shared.SecurityComplianceTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.332 s <<< FAILURE! -- in nl.customer.aws.esb.shared.SecurityComplianceTest
[ERROR] nl.customer.aws.esb.shared.SecurityComplianceTest.testNoUnsuppressedErrors -- Time elapsed: 0.033 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 
Found 2 unsuppressed error:
 - error #1: Resource=/TestStack/ALBSecurityGroup/Resource, Message=AwsSolutions-EC23: The Security Group allows for 0.0.0.0/0 or ::/0 inbound access.

 - error #2: Resource=/TestStack/CrossAccountALB/Resource, Message=AwsSolutions-ELB2: The ELB does not have access logs enabled.

<..SNIPPIT..>

[ERROR] Failures: 
[ERROR]   SecurityComplianceTest.testNoUnsuppressedErrors:150->validateMessages:100 Found 2 unsuppressed error:
 - error #1: Resource=/TestStack/ALBSecurityGroup/Resource, Message=AwsSolutions-EC23: The Security Group allows for 0.0.0.0/0 or ::/0 inbound access.

 - error #2: Resource=/TestStack/CrossAccountALB/Resource, Message=AwsSolutions-ELB2: The ELB does not have access logs enabled.


[INFO] 
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.702 s
[INFO] Finished at: 2025-05-23T10:14:17+02:00
[INFO] ------------------------------------------------------------------------
```

Lessons learned

Amazon Q helped me with writing my Unit test based upon cdk-nag. With Vibe Coding being a bit of a buzz word, I do think coding agents like Amazon Q can help you in your day to day job. What my lessons learned are:

1. Know your problem. In my case, I knew that the output in the test was wrong as I had cdk-nag working before with output. So by moving the code to the Unit tests, it seemed allright, but it wasn’t.

2. Always verify the output, don’t take it for granted. This is where knowledge comes in to play as well. By understanding your problem and knowing your output, the tools to get there are more scoped and can help you by steering the agent.

3. You need to iterate. Like in normal day to day work, you need to polish, scrape, work it again, start over and all the other technics to come to the end solution. With a coding agents it isn’t different.

And to quote Spiderman: With great powers, comes great responsibilities!