Let's set the stage

Bob is a Developer at a company. Bob has a new project on setting up IPv6 Networking which he's not familiar with. He wants to use the new Alice AI which he's heard so much about. However, he's not sure how much to share with Alice and he's heard of Mallory who's always trying to hack the company and hold them to ransom.

We'll add a lead indicator of risk percentage based on what is sent and how likely that is to occur as well as how bad the security issues are by smell (risk + likelihood). More of an example of what risk level we are at and some likely threats.

Where to start?

Query 1: "What can I do to implement IPv6?"

At this point, the fact you might be implementing IPv6 is apparent to anyone that can see the traffic.

Risk level: low

Likely hood of attack: low

Possible issues: OSINT

Smell: Light

Side note: Standard prompt mistake is not indicating to answer as the expert. Either the tool is the issue and should have known or the user needs to adjust their prompts. Let's assume this is handled by the tool rather than polluting our queries.

Getting into the details

Query 2: "How do I set up the Brand name routers of Version number to handle IPv6 traffic in the quickest manner in production?"

At this point, you've given away a key number of things we could not have inferred from outside the network including:
1. Type of router (brand/version)

2. Your intent to implement it with this system.

3. You're lazy enough to use the quick setup guide in production.

As this probably isn't a two door decision in production that we can roll back if there's a breach we're getting into the high-risk area.

Risk level: medium

Likely hood of attack: medium

Smell: An unpleasant odor

Possible issues: OSINT, 0-day attack, least privileged access

Implementation details

Query 3: "Fix my Infrastructure as code Code block to handle IPv6 traffic for IIS Windows 2016 servers and SQL Server 2012."

At this point, we have a compounding effect of issues. We have all of the previous issues and the revelation of the runtimes/OS server. This is also revealing the company's code which is extremely likely to be a company or individual's IP. There's also the fact that Alice records the sets of queries in order. So seemingly innocent queries that are separate can compound into an issue.

Risk level: high

Likely hood of attack: high

Smell: The Bog of Eternal Stench

Possible issues: OSINT, 0-day attack, least privileged access, IP lost, malicious code injection, OWASP top 10, malicious code across multiple files, weakening of infrastructure as code.

The possible attacker can now look up existing vulnerabilities against OS/DB that are probably out of support. As well as intercept and give back malicious code that may make it into production. Even if you're query did not include code, there could be prompt injection attacks. Even if the particular code you got was good, there could be malicious code spread out across responses. The fact that this also is Infrastructure as code could weaken the overall security.

Let the cat-and-mouse games begin!

Treat the code it produces like someone else's dirty sock. If you're org didn't prescribe this tool then you can't necessarily use it. Let alone wear it (use it in production) as you may get infected.

Build your own sock. Remember these models have limitations like they are a couple of years old, and you don't control the data or security measures around it.

Final thoughts

The cat's out of the bag. You should assume your employees are using this. Training would go a long way to avoid this type of issue. Simulations of attacks are better as you can test if they are actively learning the content. Scanning of AI-generated content would also help. And taking ownership of those generative AI models for the Crown Jewels.

Disclaimer

These opinions are mine and not those of my employer. This is not advice, please seek out guidance such as in the links below for concepts like zero trust.