What could possibly go wrong? AI, GPT, and structured output via JSON Schema (part 1)

Before I explain why I've been compelled to do this experiment and write this article, let's ground with context.

I'll be upfront: I'm conflicted about AI. It is a very powerful tool, there's no question, but just like a hammer can do many things, you don't use a hammer to tighten a screw. We need to be our own fact checkers, just like we do with physical reading material and the internet at large. I GET LLM AI is impressive, and I'll use it occasionally, however I'm still very apprehensive about the prospect of letting it take over business critical operations through the use of "agentic AI", which seems to be a new hype wave.

What's new-ish, but been around for a while now, is structured output from GPTs. Sure, text is lovely, and maybe it's appropriate when you want to have a "discussion" with your AI chat. But, if you need to process the result, or pass parts of data onto another service, then structured data, in the format you want, might be a good idea. Plus, you can make the AI conform to what YOU provide… or at least, that's the promise of the API documentation.

You can use common language specific validation libraries, specifically Pydantic and Zod, but that isn't going to meet everyone's needs. JSON Schema is the language agnostic approach to validate data… or to define what you'll accept.

Hey, that's a trigger right there!

I really, truly, accept that not everyone has the time to invest in reading technical specifications on the internet. Not only do many find them boring and dry, but they can be very complex, and there's often no one to ask for help other than colleagues and friends you know. (Luckily for JSON Schema, we have a slack server, and you're all welcome!)

I understand that AI, as the new shiny, must be added to many things as soon as possible. I do. But. When your company is providing API access to the AI functions, I feel like it's not unreasonable to expect you might read the standards. Those providing access to a proxied version of the API, maybe with modifications, are left high and dry when the underlying API they are calling doesn't provide all the information that IMHO it should.

When the largest / most popular code repository in the world (Yeah, Github) publishes an article on their new improved support for structured data from AI, and I open the article to take a look, I straight away notice what I'd consider a likely problem. I know the problem is not their fault, and comes from OpenAI. It comes from years of people seeing JSON Schema used within other documents where the version is pre-defined.

I'm triggered.

You wouldn't do this with your application code!

I suspect there's a problem. A problem that maybe not many people will face, as it's possible many will stick to defining really simple schemas. But it's potentially MUCH worse than I'd initially thought. The existing documentation, which is duplicated in part across all those who proxy requests through to OpenAI APIs (hence being triggered by GitHub's article), is the fact that while it SAYS you use JSON Schema, it doesn't specify WHICH VERSION of JSON Schema is being used.

"Surley, if this was a problem, they wouldn't do it!" I hear you cry. *heavy sigh*. And this is what makes it so frustrating. JSON Schema is a language. It has versions. Some versions introduce breaking changes.

Listen. You wouldn't push your code to a server where you didn't know what version of the programming language is installed. So why is this OK for JSON Schema?

OK, you might, but with polyfills or something, back to a specific minimum version. But I think that's only viable because of the tooling and occasional requirement to do so. In that case, you're making an explicit decision, and probably have some form of guardrails in place to make sure the version you get is within the version range of support defined.

Maybe it was a business or marketing decision to avoid having the word "draft" in a URL. I don't know. I probably won't ever know. But I do know it's not good.

Breaking changes between versions is bad enough, but we have another problem. Because of how JSON Schema has been developed, unknown keywords are ignored. The implication of this fact is that if you're using a keyword that was added in a newer version of JSON Schema than the one being used, it's just going to be ignored. You won't get an error. It makes no noise. Silence. (We've decided, for multiple reasons, to "fix" that "issue". It was by design. You can read about that in one of the latest articles on the JSON Schema blog.)

Why was it designed that way? That's a whole other series of articles. Let's refocus. Is this a real problem, or just hypothetical? And is it worse than it seems on the surface?

I'm going to find out and report back. Stay tuned.

Photo by Erik Mclean on Unsplash