Navi brings subtitles and live translation to FaceTime via SharePlay but faces the threat of Apple copying its ingenious idea

Navi, an universal app that brings subtitles and live translation to FaceTime calls, is arguably the most nifty implementation using Apple’s SharePlay feature. At WWDC 2021 in June, Apple announced iOS 15 with a focus on features that should have been introduced right at the start of the pandemic.

Apple SharePlay: What you need to know

Last year, when the COVID-19 pandemic started to spread and countries began imposing lockdowns, tech companies raced to bring shared experiences from the offline world to our online-first world. One of the best examples is Netflix Party, a Chrome extension that allows you and your friends to watch Netflix shows and movies together right from the comfort of your own home, all while maintaining social distancing.

Netflix Party is akin to movie night with your friend with the only difference being you and your friends are not at the same place. These shared digital experiences really took off and Apple did take notice. It introduced SharePlay as a feature that allows users to bring content from other apps like Apple TV+ or Apple Music to their FaceTime calls.

At WWDC 2021, Apple also opened an API for developers to integrate their apps with SharePlay. The idea was simple. With the SharePlay API, developers will be able to allow their apps to run as an experience within FaceTime calls. If you ever wondered about video and music sharing right from FaceTime then SharePlay made that possible. All media and entertainment apps jumped in despite the delay in its roll out but the best use yet for SharePlay could have arrived in the form of Navi.

Navi wants to make FaceTime inclusive for people with hearing impairments

One of the biggest selling points of FaceTime is its seamless integration with the Apple ecosystem and its ability to bring people together. However, not everyone can hop on a FaceTime call and the service is definitely not designed for people with hearing impairments and other disabilities. Navi, designed by Amsterdam-based developer Jordi Bruin, wants to fix this gap in user experience.

Navi is a simple app that tries to do two things within FaceTime. The first is to enable subtitles or instant closed captioning for all your FaceTime calls. The second is to offer live translation allowing you to interact with another person even if you don’t speak their language. Bruin is also mindful of the fact that FaceTime call is the important entity here while subtitles and live translation are improving the call.

So, he has built the app specifically with Picture in Picture functionality. Navi always runs in the background while your FaceTime call takes the centrestage. Last week, Apple released macOS 12.1 with SharePlay support and if you use Navi on a Mac, the subtitles are overlaid directly on the video window.

Bruin tells Silicon Canals that the idea of Navi began earlier in the year when he successfully built an app to help mute people speak on the phone using text-to-speech. After text-to-speech, he tried to do the reverse, but was unable to make it work due to technical limitations. But when Apple announced SharePlay, Bruin tried to enable subtitles using the feature.

“I love to build things that make people’s lives easier, and the SharePlay framework was also a lot of fun to learn along the way. [Within] 24 hours, I had the first prototype ready and I knew I would be able to build a full app out of it,” he says.

Subtitles, Live Translation and Privacy

Bruin says that he built Navi for people with hearing impairments who have difficulty with video calls. The feature allowing people to communicate with those who speak another language is essentially an add-on here. Because of its targeted user base, Bruin is offering Navi as a free app on the App Store but to unlock the full experience, Navi users can opt for in-app purchase.

Navi users can try the subtitles at no additional cost until the end of this year. In 2022, Navi will give five free calls with subtitles each month but to unlock the full feature, you will need to make a one-time payment of €3.99. To use the Live Translation feature, users will need to buy a pack to make calls, which starts at €5.99 and offers 100,000 characters or about an hour of translation.

“Users can try the subtitles on a few calls for free. If they feel the app has value for them they can purchase a one time payment to be able to use Navi for all their calls. If users want to translate the incoming subtitles, they can purchase translation credits, which gives them a certain number of characters or minutes of translation,” Bruin says, explaining the model.

With Apple championing itself as the major advocate for user privacy, Navi also adheres to all the guidelines laid out by the Cupertino giant. “To ensure the privacy of conversations, [the app] performs on-device speech recognition on both devices, which then gets transmitted over the encrypted SharePlay channel to the other devices on the call. On the other side, the user can then let the app translate the incoming messages if they choose to,” Bruin explains.

While the regular or non-translated subtitles never leave the device, Bruin is still relying on a cloud translation service for its live translation service. He is planning to make it on device as well to eliminate any privacy concerns that users might have. “I plan to bring this on device as soon as the APIs are available, which I’m hoping will be somewhere in the next year,” he adds.

Apple’s walled garden and in-house challenge

One of the biggest challenges for an App Store developer right now is adhering to Apple’s strict guidelines and working within the confines of its walled garden. Apple is in a unique position where it builds the platform (iOS, macOS, watchOS, etc.) and also runs the storefront on those platforms (App Store). As a result, developers often find themselves at the thick end of the bargain.

While these challenges are real, Bruin says he did not run into any issues related to limitations that “Apple puts on the development process on their systems”. “Over the years, I have aligned my goals and roadmaps with those of Apple. Now, it’s easy for me to steer clear from grey areas that might cause problems in the future,” he explains his development process.

While it is a clear example of a developer aligning their goals with that of the creator of a major platform, there is always a looming threat of Apple just pulling the plug on its API or simply turning a successful third-party app into native feature of its operating system. The misfortune of Apple copying an idea even has an industry term now – “Getting Sherlocked”.

Bruin, however, is not worried about getting sherlocked by Apple. In fact, he fully expects Apple to build something like this into FaceTime eventually. “It’s a logical next step for the app, so I’m not bothered by it,” he tells Silicon Canals.

“They will probably be able to make a better experience, and I think it’s a very valuable thing for hearing impaired users to have so I hope they build it at some point. In the meantime I hope Navi can already help people out. And I will just start building something else afterwards,” he added with a wink emoji at the end.

This realisation of Apple turning his brilliant idea into a native feature also highlights the friction that Navi has to navigate. To really use Navi, you have to first start a voice or video call on FaceTime and then return to your home screen and open Navi while staying on the call. Now, tap on Subtitles and wait for other people to join the call to get a SharePlay experience with subtitles and live translation feature.

It might sound like a few steps but for those in the Apple ecosystem, it would sound like two steps too many. An ideal experience will be one where you open FaceTime, call your friends, relatives and simply click on a toggle to enable or disable Subtitles without ever leaving that window. Bruin agrees with this notion and says he cannot speak for Apple’s future plans, but he does see such an experience coming to iOS eventually.

“I expect that they will use this first year to discover how apps are using SharePlay and what kind of features can be added to make the experience easier. Having the app integrated directly within FaceTime would make the experience easier so I would like to see a feature like that added at some point,” he says.

Navi is a great example of a simple idea being used to completely transform user experience for a set of people who were always left behind by technology. However, Navi’s biggest challenge could be the platform on which it is available right now. The success of this app or lack thereof will decide not only Bruin’s own future as an app developer but also how Apple treads the murky waters of copying a brilliant third-party app idea without putting that developer out of business.