AI Assisted SwiftUI

I wanted to get a feel for the current capabilities of AI code assistance with respect to building a functioning SwiftUI app from the ground up (for free). This wasn’t an exhaustive survey of what’s available—for this test I just tried ChatGPT and Grok 3. No direct Xcode integration involved here, just good old fashioned copy/paste.

TLDR

Both can make a simple app, but things fall apart pretty quickly once you start adding any complexity.

Demo App MVP Requirements

I wanted a SwiftUI app targeted at iOS 18.x that had these basic requirements:

• Apple Vision framework for face detection

• Uses the returned bounding boxes to overlay a curated handful of emoji over the faces

• Face tracking to try to keep the same emoji pinned to a face if it moves across the screen

FWIW, I’m using Xcode 16.3.

ChatGPT Thoughts

Initial prompt to get starting feedback.

Me:

Can you help make a funny iOS Swift app? I'd like to do on device face detection. If we could put dog emojis over cat faces and cat emojis over dog faces in the video feed that'd be cool, but otherwise let's do a list of funny emoji over human faces (I'll specify a list) using face tracking

ChatGPT:

That sounds like a perfect mix of absurd and awesome! Let’s break this down into something doable and funny. We’ll build an iOS app in Swift using:

• Vision and AVFoundation for real-time face detection via camera

• Core Image or UIKit overlays to place emojis on top of faces

• A way to identify cats/dogs (optional) — this gets more complex, but we can stub it for now

• A customizable emoji list you provide for human faces

Since detecting animal faces reliably on device in real time is tricky without Core ML models trained for cats/dogs, I suggest we start with human face tracking and let you specify your own list of emojis. Then later, we can extend it with a simple classifier for cats/dogs if needed.

After some back and forth it presented a nice looking plan

I’m easy to please, but I was surprised that I could copy the generated Swift classes

and a right-click in Xcode would offer to create the swift file from clipboard.

Now we get to the fun part. Iterating over build failures like the initial code generated:

CameraView.swift:29:38 Cannot find 'EmojiOverlayManager' in scope

After a lot of volleys it would generate a working build, but the app functionality never got quite up to snuff (e.g., no face detection, no video). We also seemed to lose context at some point, so new code would lose something important from earlier. Between the build failures, and the inability to get the functionality working I decided it was time to give Grok a shot.

Grok 3 Thoughts

Initial prompt

Let's make an iOS app with SwiftUI. It will run on iOS 18+. It will use the phone's camera to do face detection. I will provide a list of emojis to overlay over the faces. We'll scale the emoji to cover each face completely, and do face tracking so a given emoji will stick to a given face as it moves around the screen. The app's name is EmojiMasker. The emoji list is “🤬”, “🐱”, “😐”, “🙈”, “🙊”, “🙉”, “🤨”, “🤓”, “🤪”

It provided a breakdown of the project setup (including adding NSCameraUsageDescription to the Info.plist for the app to use the camera), along with the key components involved before it generated the code. The architecture info, etc from ChatGPT definitely looked a bit slicker here.

Example code

The Xcode copy/paste magic still in effect here. Now on to a few rounds of build errors.

e.g.,

CameraManager.swift:86:51 Cannot call value of non-function type 'Dictionary<VNDetectedObjectObservation, String>.Element?' (aka 'Optional<(key: VNDetectedObjectObservation, value: String)>')

After a little patience though, we got a working app!

Time to push our luck with more features. We added a button to switch between front and rear camera. Then changed the button from a text label to using an SF Symbol icon.

After putting the working version in GitHub, Grok was able to get another cool feature added. If you tap the screen it makes an animated explosion with emoji. The explosions only seemed to work around the area where faces are detected, but I was happy enough it kind of worked.

Closing Thoughts

I decided to let ChatGPT contribute something to the app, so I had it generate the app icon (along with instructions on how to include them in the project).

If you’re curious, here’s the repo https://github.com/ThatOrJohn/EmojiMasker