Create a Drawing App for visionOS
Amos Gyamfi
visionOS’s unique multimodal interactions, such as using the eyes, hands, and voice to interact with objects, make it an exciting platform to experiment with. The platform allows developers to mix and place 2D and 3D content and display them in Windows and Volumes. Using the Immersive Space Render, you can also place digital content in users’ physical space.
In this article, you will create your first SwiftUI drawing app for visionOS using PencilKit. You will also discover how to bring an existing iOS or iPadOS app into visionOS and display it unmodified, with the vision simulator. At the end of this tutorial, you will have a simple but fully featured and functional window-based drawing board with tool sets supporting different pencil types.
Before You Start
To follow along with this tutorial, run and test the sample drawing app. You will need Xcode 15.2 or any of its later versions. During the installation, you will get a prompt to download iOS and the visionOS simulators. Make sure to install them all. You may use an Intel Mac to work along with the tutorial. However, I recommend using an M-series Mac.
Learning Objectives
This tutorial has four main objectives:
Build your first freeform and hand-drawing app for Apple Vision Pro
Understand and apply visionOS’s concepts, such as ornaments and glass background effects.
Familiarize yourself with Apple’s freeform drawing framework, called PencilKit.
Bring your existing iOS or iPadOS app to visionOS.
Project’s Source Code
The tutorial consists of two separate accompanying SwiftUI projects. Download on GitHub and explore each of them.
- EyeDraw: visionOS drawing app.
- FaceBoard: iOS drawing app that runs on visionOS unmodified.
Explore Gestural Interactions in visionOS
visionOS has several interaction styles and gestures that differ from those on tvOS, macOS, and iOS. A drawing app is an excellent example for exploring these gestural interaction techniques. Although we will test the app with a simulated visionOS device, the gestures explained in this section will work the same when you run the app on Vision Pro. To simulate the visionOS gestures, use your Mac’s keyboard and mouse.
Look and tap: On an actual Vision Pro device, people can look at drawing tools to bring them into focus. Tapping the thumb and index finger will select the specific drawing tool.
Pinch and drag: After selecting a drawing, you can pinch and drag with the thumb and index to reposition it on the drawing canvas.
Pinch and flick: The default drawing controls from PencilKit are displayed on the canvas on a draggable control bar. With the help of your wrist, thumb, and index finger, you can smoothly move the control bar by dragging and throwing it to the corners of the drawing window.
Pinch and hold: The drawing canvas has a pencil symbol that reveals drawing tools with pinch and hold. In the visionOS simulator, tapping and holding with the Interact tool lets you see and pick different drawing tools. With an actual Apple Vision Pro, this works by tapping and holding with the thumb and index finger together.
Drawing UI: Overview
The project is a window-based visionOS and SwiftUI app consisting of a whiteboard, drawing and erasing/undoing tools for making freeform sketches and writing by hand. The drawing window has collaboration buttons on its top center. This tutorial will not implement these collaboration buttons for chat, calling, screen sharing, and recording. However, you can explore the implementations in the compatible iOS app by downloading it from GitHub and running it with the visionOS simulator.
Introduction To PencilKit
PencilKit is a drawing framework by Apple that allows developers to add hand-drawing and free-form creative writing support to their apps. PencilKit has seamless support for macOS, iOS, and visionOS. To integrate PencilKit support to a SwiftUI project, add ‘import PencilKit’.
Note: SwiftUI does not support PencilKit natively. So, creating a drawing view should conform to UIViewRepresentable. Also, visionOS only supports some of the drawing tools PencilKit provides. For example, placing a ruler tool on the drawing window does not work yet at the time of this article's publication.
Please read our blog’s PencilKit Overview section of the SwiftUI and PencilKit tutorial to learn more about this drawing framework. You can also watch our full PencilKit YouTube tutorial.
Available Drawing Tools
A drawing app with only one ink tool would be boring. In the app you build in this tutorial, users can draw by selecting from seven tools with varying strokes, as demonstrated in the video above. The available sketching tools are:
Pencil: A uniform stroke for freeform sketching and making thin lines.
Pen: For creating bold lines. Excellent for making outlines and handwriting.
Monoline: Similar to the pencil tool and ideal for making light sketches.
Fountain Pen: Has a variable stroke for making elegant drawings and signatures.
Marker: A thicker stroke, ideal for making writing and sketching vibrant.
Crayon: Produces textural marks, which are excellent for making creative art.
Water Color: Has a transparent paint style suitable for blending purposes.
Integrate the Drawing Canvas With PencilKit
Let’s start with a new visionOS project by launching Xcode. Select visionOS as the template and click next.
Choose your preferred project name and create it. In this example, we use EyeDraw as the project name.
Find ContentView.swift in the Xcode project navigator, rename it to FreeFormDrawingView.swift, and let's start coding. Let's follow these steps to support drawing on the visionOS window.
Make PencilKit accessible by importing it.
Define the drawing properties.
Create the drawing view structure.
Define binding properties for drawing.
Declare ink and eraser tools.
Create a function to make the drawing view.
Create a function to update the drawing view.
4. Display the drawing canvas and tools.
5. Define a function to save freeform writing and artwork.
Make PencilKit accessible by importing it
Add import PencilKit above or below the SwiftUI import to make PencilKit available in this Swift file.
Define the drawing properties
In the FreeFormDrawingView struct, add the following properties.
@State private var canvas = PKCanvasView()
@State private var isDrawing = true
@State private var color: Color = .black
@State private var pencilType: PKInkingTool.InkType = .pencil
@State private var colorPicker = false
@Environment(\.undoManager) private var undoManager
In summary, we add the property canvas of the type PKCanvasView() to help capture users' fingers and Apple Pencil inputs as drawings. We also define the pencil type, color picker, and another to manage undo and redo.
Create the drawing view structure
Create a new struct called DrawingView and use the sample code below for its content.
struct DrawingView: UIViewRepresentable {
// Capture drawings for saving in the photos library
@Binding var canvas: PKCanvasView
@Binding var isDrawing: Bool
// Ability to switch a pencil
@Binding var pencilType: PKInkingTool.InkType
// Ability to change a pencil color
@Binding var color: Color
//let ink = PKInkingTool(.pencil, color: .black)
// Update ink type
var ink: PKInkingTool {
PKInkingTool(pencilType, color: UIColor(color))
}
let eraser = PKEraserTool(.bitmap)
func makeUIView(context: Context) -> PKCanvasView {
// Allow finger and pencil drawing
canvas.drawingPolicy = .anyInput
canvas.tool = isDrawing ? ink : eraser
canvas.isRulerActive = true
canvas.backgroundColor = .init(red: 1, green: 1, blue: 0, alpha: 0.1)
// From Brian Advent: Show the default toolpicker
canvas.alwaysBounceVertical = true
let toolPicker = PKToolPicker.init()
toolPicker.setVisible(true, forFirstResponder: canvas)
toolPicker.addObserver(canvas) // Notify when the picker configuration changes
canvas.becomeFirstResponder()
return canvas
}
func updateUIView(_ uiView: PKCanvasView, context: Context) {
// Update tool whenever the main view updates
uiView.tool = isDrawing ? ink : eraser
}
}
You have noticed the DrawingView above conforms to UIViewRepresentable. It conforms to UIViewRepresentable because SwiftUI does not provide native support for PencilKit. In the struct, we define binding properties for drawing, ink and eraser tools, and functions for making and updating the drawing view.
In the makeUIView function, we define our drawing policy as any input canvas.drawingPolicy = .anyInput. We are setting the drawing policy as any input to allow the canvas to take both finger and Apple Pencil inputs as user drawings.
Whenever the main canvas view updates or when a user changes a drawing tool, we call the function updateUIView to respond to the changes.
Display the Drawing Canvas and Tools
In the body computed property, we add a NavigationStack and call the DrawingView to display it.
DrawingView(canvas: $canvas, isDrawing: $isDrawing, pencilType: $pencilType, color: $color).
Note: We will use ornaments in visionOS to display the drawing tools. Let’s head to the next section to learn what ornaments are.
Pinning Drawing Tools To Ornaments
In this app, the primary purpose of the window view (canvas) is to afford drawing. Erasing, undoing, selecting, and picking colors are related to the canvas drawing. We can house these related contents in ornaments. In visionOS, Ornaments are attached to a window view for displaying related information. When the window moves, ornaments move with it. However, ornaments remain fixed when the content in the window moves or scrolls while the window itself remains at rest.
We can add ornaments in two ways. The first option is to use the ornament modifier and specify the window location where you want the ornament to appear. Secondly, you can use the standard Toolbars and TabViews to make content automatically appear in ornaments.
Let’s add the following code as a toolbar to the DrawingView we called previously in the NavigationStack. In the .toolbar, closure, we create a ToolbarItemGroup and set its placement parameter as the bottom ornament to display the main drawing tools at the bottom part of the window ToolbarItemGroup(placement: .bottomOrnament).
.toolbar {
// Bottom Ornament
ToolbarItemGroup(placement: .bottomOrnament) {
HStack { // Drawing Tools
Button {
// Pencil
isDrawing = true
pencilType = .pencil
} label: {
VStack(spacing: 8) {
Image(systemName: "pencil.and.scribble")
Text("Pencil")
.foregroundStyle(.white)
}
}
.buttonStyle(.plain)
Button {
// Pen
isDrawing = true
pencilType = .pen
} label: {
VStack(spacing: 8) {
Image(systemName: "applepencil.tip")
Text("Pen")
.foregroundStyle(.white)
}
}
Button {
// Monoline
isDrawing = true
pencilType = .monoline
} label: {
VStack(spacing: 8) {
Image(systemName: "pencil.line")
Text("Monoline")
.foregroundStyle(.white)
}
}
Button {
// Fountain: Variable scribbling
isDrawing = true
pencilType = .fountainPen
} label: {
VStack(spacing: 8) {
Image(systemName: "scribble.variable")
Text("Fountain")
.foregroundStyle(.white)
}
}
Button {
// Marker
isDrawing = true
pencilType = .marker
} label: {
VStack(spacing: 8) {
Image(systemName: "paintbrush.pointed")
Text("Marker")
.foregroundStyle(.white)
}
}
Button {
// Crayon
isDrawing = true
pencilType = .crayon
} label: {
VStack(spacing: 8) {
Image(systemName: "paintbrush")
Text("Crayon")
.foregroundStyle(.white)
}
}
Button {
// Water Color
isDrawing = true
pencilType = .watercolor
} label: {
VStack(spacing: 8) {
Image(systemName: "eyedropper.halffull")
Text("Watercolor")
.foregroundStyle(.white)
}
}
// Color picker
Button {
// Pick a color
colorPicker.toggle()
} label: {
VStack(spacing: 8) {
Image(systemName: "paintpalette")
Text("Colorpicker")
.foregroundStyle(.white)
}
}
} // Drawing Tools
.padding(.horizontal)
.foregroundStyle(
LinearGradient(gradient: Gradient(colors: [.green, .yellow]), startPoint: .leading, endPoint: .bottom)
)
} // Bottom Ornament
}
The leading (left) ornament contains tools for modifying drawings. Let’s attach it to the window’s left side by adding the .ornament modifier to the DrawingView inside the NavigationStack.
.ornament(attachmentAnchor: .scene(.leading)) {
// Modify Tools
VStack(spacing: 32) {
Button {
// Clear the canvas. Reset the drawing
canvas.drawing = PKDrawing()
} label: {
Image(systemName: "scissors")
}
Button {
// Undo drawing
undoManager?.undo()
} label: {
Image(systemName: "arrow.uturn.backward")
}
Button {
// Redo drawing
undoManager?.redo()
} label: {
Image(systemName: "arrow.uturn.forward")
}
Button {
// Erase tool
isDrawing = false
} label: {
Image(systemName: "eraser.line.dashed")
}
.foregroundStyle(
LinearGradient(gradient: Gradient(colors: [.white, .yellow]), startPoint: .leading, endPoint: .top)
)
} // Modify tools
.padding(12)
.buttonStyle(.plain)
.glassBackgroundEffect(in: RoundedRectangle(cornerRadius: 32))
}
You can repeat the same procedure you used to attach the left ornament to add the right or trailing ornament. Here, you should set the attachment anchor to the trailing.
.ornament(attachmentAnchor: .scene(.trailing)) {
VStack(spacing: 32) {
Button {
// Set ruler as active
canvas.isRulerActive.toggle()
} label: {
Image(systemName: "pencil.and.ruler.fill")
}
Button {
// Tool picker
//let toolPicker = PKToolPicker.init()
@State var toolPicker = PKToolPicker()
toolPicker.setVisible(true, forFirstResponder: canvas)
toolPicker.addObserver(canvas)
canvas.becomeFirstResponder()
} label: {
Image(systemName: "pencil.tip.crop.circle.badge.plus")
}
// Menu for pencil types and color
Menu {
Button {
// Menu: Pick a color
colorPicker.toggle()
} label: {
Label("Color", systemImage: "paintpalette")
}
Button {
// Menu: Pencil
isDrawing = true
pencilType = .pencil
} label: {
Label("Pencil", systemImage: "pencil")
}
Button {
// Menu: pen
isDrawing = true
pencilType = .pen
} label: {
Label("Pen", systemImage: "pencil.tip")
}
Button {
// Menu: Marker
isDrawing = true
pencilType = .marker
} label: {
Label("Marker", systemImage: "paintbrush.pointed")
}
Button {
// Menu: Monoline
isDrawing = true
pencilType = .monoline
} label: {
Label("Monoline", systemImage: "pencil.line")
}
Button {
// Menu: pen
isDrawing = true
pencilType = .fountainPen
} label: {
Label("Fountain", systemImage: "paintbrush.pointed.fill")
}
Button {
// Menu: Watercolor
isDrawing = true
pencilType = .watercolor
} label: {
Label("Watercolor", systemImage: "eyedropper.halffull")
}
Button {
// Menu: Crayon
isDrawing = true
pencilType = .crayon
} label: {
Label("Crayon", systemImage: "pencil.tip")
}
} label: {
Image(systemName: "hand.draw")
}
.sheet(isPresented: $colorPicker) {
ColorPicker("Pick color", selection: $color)
.padding()
}
}.padding(12)
.buttonStyle(.plain)
.glassBackgroundEffect(in: RoundedRectangle(cornerRadius: 32))
}
Define a function to save freeform writing and artwork
You noticed we have the following function in the FreeFormDrawingView struct. We call this function to save drawings to the users' Photos Library.
func saveDrawing() {
// Get the drawing image from the canvas
let drawingImage = canvas.drawing.image(from: canvas.drawing.bounds, scale: 1.0)
// Save drawings to the Photos Album
UIImageWriteToSavedPhotosAlbum(drawingImage, nil, nil, nil)
}
To combine all the code snippets above, refer to FreeFormDrawingView.swift in the Xcode project.
Congratulations 🎉 🥳. You have now created your first drawing app for visionOS. You should see the following result when you run the app with the visionOS simulator.
If you have an existing iOS or iPadOS app, you can run it in visionOS. Let’s dive into this aspect in the next section.
Bring Your Existing iOS or iPadOS App to visionOS
In a previous iOS article, we created the same drawing app in this tutorial, with additional features for integrating chat and video calling for collaboration. In the above video preview, the app is running on an iPad. Download the iOS/SwiftUI sample app from GitHub to test it with the vision simulator.
To run the app in visionOS, you should add the simulated visionOS device as a supported destination as highlighted in the image below
Launch the app in Xcode.
Open the devices dropdown menu in the Xcode toolbar and select Apple Vision Pro (Designed for iPad).
Run the app.
You will see the following video preview after running the app against the visionOS SDK.
Note: We did not cover the messaging and calling features in this section. Visit the iOS Chat and Video documentation to learn more.
Wrap-Up
Congratulations! You have created your first visionOS drawing app using PencilKit and SwiftUI. You learned about ornaments in visionOS and how to apply them to present related content in different locations of a window-based app. Eventually, we brought an iOS app and ran it against the visionOS SDK.
Originally published at https://getstream.io.
Subscribe to my newsletter
Read articles from Amos Gyamfi directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by