We've been hard at work getting Secret Chest to suppor the new Apple Vision Pro. It works but the experience could be better (e.g. some swiftui elements that look great on iPhone or iPad aren't easy to select on the Vision Pro). Along the way, we've learned a good bit about it and some of what we've learned might help others.
The Apple Vision Pro is the most advanced AR headset I’ve gotten my grimy mits on! However, even the most advanced tech can encounter bumps in the road, no matter how good a job Apple did at making it simple and intuitive to enter the new world of pinching in space as opposed to tapping. The Vision Pro feels more like a face mounted iPad in many ways.
Now, I’m hard on tech. I compiled an app that used the LiDAR camera and the device never bounced back. Along the way to getting my Vision Pro working again, I tried all the things that Apple recommends and then ended up as the first one to be worked on at the Genius Bar closest to my house - which gave me some interesting insights into how the teams at the stores are taught to troubleshoot. And ultimately, as with the iOS variants, there aren’t a ton of things that can be done to troubleshoot, but let’s look at what’s possible!
Troubleshoot the Intent
What will you use the Vision Pro for: There are some apps, but the App Store isn’t yet full of games that embrace the AR uses. Apple has done a great job with the way the center of gravity works, but It’s a bit heavy on the neck after prolongued wearing. So if you’re planning to watch full movies or use it as a monitor for a Mac, think about that. I am writing code for it, so it’s more put it on to test my crappy code and then take it back off. Easy on my neck. There’s a really cool Alicia Keys studio spatial video on Apple +. Check that out, or some of the early Disney stuff, to see why people might wanna’ use it as a massive, awesome television or monitor.
It’s AR, not VR: It generates sound and video, not smells (I literally heard someone say that it should). It isn't just a VR headset; it's a spatial computer aiming to redefine personal computing. But as of the time of this writing, it’s been shipping for less than a month and was only unveiled in June 2023 (after of plenty of time on the rumor mills).
Cost is a thing: It’s far more expensive than many of the other devices on the market. Having said that, spec to spec it should be about the same, and there are some great features. Like instead of it snapping things back to a center automatically, you press the digital crown (like what the Apple Watch uses) in order to reset the view. This might be the single best thing for me personally compared to other VR/AR headsets I’ve used, ‘cause it doesn’t make my stomach hurt like most of them have!
Nasea, dizziness, and eye strain: Anyone who already suffers from these from televisions, phones, or computers is probably going to have even more issues. Apple did build in some great “Accessibility” options, so ymmv.
Content creation: I create .stl files. Vision Pro uses the .usdz format, with tools like Reality Composure used to capture 3D models. You can also drop an .obj file exported from a tool like Zbrush or Blender into Reality Converter (still in Beta I think, but a quick and easy download) to batch convert files. Going from .stl to .usdz doesn’t include much along the lines of color - it’s a format for monocrome triangles. However, it’s easy to go from a .stl to a .obj and then to .usdz. I think I’ll write a batch converter…
Developers: The above is one of many examples of a fresh, new space and marketplace. This first wave of devices is likely to be the people building the first generation of more immersive apps that can do things like guide artists, electricians, plumbers, or people that do things like brick laying. That cool Home Depot DIY book you got when you got your first place of your own. Imagine that or anything else that blends the real and virtual worlds, together at last.
Not for younger kids: Apple is explicit that the device is currently being manufactured for humans (and maybe other primates) who are 13 and up.
OK, enough pontificating about the correct intent of a device. Next, it helps to understand what’s in the thing, ‘cause yes it’s more than just an iPad stuck to a face:
Lots of chips: There’s an M2 with an 8-core CPU, a 10-core GPU, a 16-core neural engine, and 16 gigs of unified memory. This loads visionOS and powers many of the functions of apps. There’s a Secure Enclave, which powers Optic ID. There’s a R1, the Real-Time Vision Processing Specialist, which powers the 12 cameras, 5 sensors, and 6 microphones on the device, does the spatial awareness, so analyzes camera data and spatial maps, and does the sensor fusion, which combines data from sensors like the LiDAR and gyroscopes to compute the movements and interactions, so handles figuring out that an index finger and thumb were tapped together to select whatever the eye is highlighting.
Micro-OLED displays: Offering stunning visuals with vibrant colors and deep blacks.
Input devices: The 12 cameras, 5 sensors, and 6 microphones were mentioned, but they’re inside to track the eyes and outside to track the world. This powers things like FaceTime with Persona: Life-size video calls where your digital avatar reflects your real-time movements. In other words, it’s less bandwidth intensive and more private to send movement data via the Persona than it is to stream actual video of a person wearing the thing.
Basic Troubleshooting
Doesn’t feel good or light comes in from the edges: Check out this page from Apple on adjusting the fit of the Vision Pro: https://support.apple.com/guide/apple-vision-pro/adjust-the-fit-dev1d800e084/visionos - Apple usually scans your face when you buy one and ships the correct strap and even fabs new lenses in stores for those who wear glasses (SEISS Optical Inserts they call them)
Frozen or unresponsive device: Force restart your Apple Vision Pro by holding both the top button and the Digital Crown until the displays go black. After it shuts down, press and hold the top button for 3 seconds to turn it back on.
Battery issues: Ensure the battery is properly connected and charged. If not, connect it to a power outlet using the provided charger. It should have a green light that flashes for several seconds if it’s charged, and amber light if it has enough, but will pulse slowly as it starts to run out of juice. You can always swap out batteries using the funny little cable that links it, but that’s one of my least favorite Apple industrial designs (although to be clear, I coudln’t design anything that’s any better - but I suspect there will be a better design on an upcoming version).
Display Issues: If the display appears blurry or distorted, make sure your Apple Vision Pro is positioned correctly on your face. Adjust the fit to ensure the displays are close enough to your eyes without light leakage. If the display looks funny and the article above was used to troubleshoot, try Settings > Eyes & Hands > Realign Displays. It’s not exactly a multi-user device and I’ve had to do that when letting someone else use it.
Cleaning: Apple has an article for this at https://support.apple.com/guide/apple-vision-pro/clean-your-apple-vision-pro-tan6200165e8/visionos but keep in mind that a lot of these cameras can’t see through lotion that gets smeared on the plastic housing for the cameras, makeup, drops of water (don’t get this thing wet at all), stickers (srsly though), or whatever else can block things. Use the article above when cleaning, as you really don’t wanna’ scratch this thing or it’ll be a really expensive brick. Consider getting the case.
Move around less: The video just shot off to the left and I can’t see it any more! Yup, I moved. Hold the Digital Crown down real quick to reset the position. Also according to how big an app is and where it is on the screen, you might bump into stuff. Not saying you need to sit all the time, but if you pace while on a call you’re probably gonna’ have bruises.
Run a Software Update: Sure, this should happen automagically, but hey… Settings > General > Software Update and install an update if there is one.
Surroundings are totally black: Take off the cover! I did this. Not smaht.
Eye Setup and Sensors/Tracking
Eye setup difficulties: Open your eyes as wide as possible during the setup process. Remove any objects near your eyes like glasses, eyelashes (don’t pull out real eyelashes, but be weary of fake ones), extra lotion that’s shiny, or makeup that might interfere. Dry or watery eyes can also affect setup.
Poor eye tracking: Check for environmental factors like bright lights or reflective surfaces that might disrupt tracking. Ensure no hair or other objects block the sensors near your temples.
Be aware of hand positions: You can pinch from anywhere, not just grabbing at stuff. If your fingers get too high, the LiDAR camera can't see them!
Know thy gestures: Touch the wrong fingers together and it’s not gonna’ select something. More on supported gestures later, but keep in mind those are OS-level, not app-level. Some apps will support their own unique set of gestures, like Apple's sample dev app for making a heart with two hands!
Get a good fit: Covered this earlier, but if Optic ID asks you to reposition the device every time, the Light Seal or Light Seal Cushion is off or something. If trading back and forth with others, expect to be prompted to press and hold the Digital Crown, which aligns the positions of the displays to the eyes. If that doesn’t work, do the eye tracking setup mentioned earlier. If you can see that looking at objects moves the position of the cursor but gestures aren’t working, redo the hand setup (on the same screen as the eye tracking setup earlier) and try again. It’s quick. If you can’t tap on anything, use accessibility (mentioned later) to scroll through menus with the Digital Crown (like you do on an Apple Watch) to re-run these steps. If it never bounces back, restore the device, which we’ll get to in a bit.
Turn the volume up or down: I’m gonna’ lump sound in here. Look at the volume icon and Use the Digital Crown to do so, or open Control Center and do it there. Also consider AirPods if you’re in a room with others like I am when at work. They can totally hear it when I’m goofing off instead of working!
Software and Connectivity
Can’t find an app: Scroll to the right from the home screen (pinch while not holding an app and swipe over). Seems intuitive but took me about 2 seconds. Or search. Or ask Siri. If it’s still missing, open the App Store from the icon on the left when at the home screen, tap Account (or the photo) and tap PUrchased. Find it and install it again. If there’s an open button, use that. If you’re a developer and it’s in Test Flight, do it from within the Test Flight app. This is basically the same for content, just using the location that was purchased to do the thing instead.
App or system crashes: Force restart your Apple Vision Pro as mentioned earlier. Update to the latest software version to benefit from bug fixes and performance improvements.
Force Quit Apps: Force quit an app on the Apple Vision Pro just like on other Apple products. To do so, press and hold the Digital Crown and top-left button until you see Force Quit Applications. Provided the LiDAR camera is working, you can then just select each to force quit.
Connectivity issues: Make sure your Apple Vision Pro is connected to a stable Wi-Fi network. If using Bluetooth, ensure your paired device is within range and has Bluetooth enabled. Beyond that, use Control Center to swap Wi-Fi networks, and consider using an iPhone as a hot spot to check if an issue (I actually had one of these on one of my Cisco APs at home) is with the network or a task being performed. In my case I had something blocked and it kept my iCloud account from installing.
Optic ID isn’t working: Settings > Optic ID & Passcode > Erase Data and start over. It’s quick, and what Optic ID is protecting should be in iCloud Keychain, so it’ll make new keybags when you do the thing.
Sending bug stuffs to developers: Crash reports from a tool like Test Flight work as they do on iPhone, iPod, or Mac. Also consider using a screenshot by pressing the Digital Crown and the left button at the same time. I do this a lot. You can also ask Siri to do it by saying “Siri, take a screenshot” but I hate asking her to do things I’m perfectly capable of! Further, to record a view (also great when sending how a feature is working to coworkers), go to Control Center and tap the handy-dandy record button, or if it isn’t there, add it in the Control Center options screen.
Advanced Troubleshooting
Turn on Voiceover: Triple-click the Digital Crown.
Turn on Accessibility options: Triple-click the top button.
Calibration issues: If hand gestures or gaze-based interactions seem inaccurate, recalibrate your Apple Vision Pro by following the instructions in the user guide.
Specific app problems: If only one app misbehaves, try restarting the app, updating it, or reinstalling it. Contact the app developer for further assistance.
Diagnostics Mode: Before we put it into diagnostics mode, there aren’t any publicly available tools to do anything while there, just yet. Expect them in the future for sure, given how much power Apple wants to put into the hands of customers to self-service troubleshoot devices. A device can be put into diagnostics mode by pressing and holding the Digital Crown until it goes black, then attaching the battery cable while continuing to hold the Digital Crown. – Continue holding the Digital Crown for 8 to 10 seconds until the Apple logo appears. You’ll see the front glow a little. It’s likely going to require a “Developer Strap” do much with it, but that’s currently for doing things like loading a fresh ipsw like with Apple Configurator.
Reboot: Disconnect the battery, wait a bit, connect and then hit that fancy Digital Crown to turn it back on. Only thing here is that it will re-establish the state of the device as it was prior to the reboot. To force restart the OS, press and hold the top button and the Digital Crown concurrently until the Force Quit Applications menu appears and disappears, and then the displays turn black.
Factory Restore (aka Erase and Quick Start): Backup (see below) then Sign out of iCloud (Settings > Your Name > iCloud), and then from Settings > General select “Transfer or Reset This Apple Vision Pro”
Restore from a backup: Reset and then either use the Set Up Manually option or use the Quick Start process (which works like the Apple Watch Qucik Start) to restore from an iCloud backup, if one is reachable. To run a backup, go to Settings > Your Name > iCloud > iCloud Backup and tap Back Up Now.
If none of the above solutions work, contact Apple Support for further assistance.
Gestures
Finally, we mentioned early to know the gestures. Consider these (and be pedantic when we say index finger):
Tap index finger and thumb together (aka pinch): Selects an option or button, and opens apps. You look at something to select it. Think of this as a mouse-click on a computer or a tap on an iPhone.
Hold the fingers closed (aka pinch and hold): Shows additional options, like contextual menus. Also works with two hands to zoom. Or look down at the bottom of a screen, pinch and hold it to zoom it.
Swipe: Pinch and hold while gesturing to the side (or up and down if there’s a big old app that can do that).
Tap: Some things like a virtual keyboard allow for a classic tap option. Ymmv with this one as the pinch action usually works better tbh.
Press the Digital Crown button: Goes home.
Hold the Digital Crown: Recenters a View.
Double-click the Digital Crown: Shows surroundings.
Triple-Click the Digital Crown: Enables Accessibility.Press Digital Crown and top left button: Screenshot capture.
Press top left button: Enables capture.
Turn the Digital Crown: turns volume up or down, or alternates between menu items when in a contextual menu with Accessibility enabled.
I keep learning something new about this thing every day. So we'll try and keep the document as more of a living and breathing thing. Especially when we get to load up profiles and proxies and all that fun stuff in the coming days!