Vuforia Fusion - Part 1

This week for R&D Friday we worked on Vuforia Fusion and it's Unity integration. Our findings will be split across two blog posts, the first (this one) covering the results of our investigation and the second a write up of what we hope to see next.

What is Vuforia Fusion?

I'll let Vuforia explain this for you:

"Vuforia Fusion is a new capability, introduced with Vuforia 7, that is designed to provide the best possible AR experience on a wide range of devices.

Fusion solves the problem of fragmentation in AR-enabling technologies, including cameras, sensors, chipsets, and software frameworks such as ARKit and ARCore. It senses the capabilities of the underlying device and fuses them with Vuforia features, allowing developers to rely on a single Vuforia API for an optimal AR experience. Vuforia Fusion brings advanced Vuforia features to ARKit-enabled devices, in addition to over 100 Android and iOS device models."



Sounds fantastic, and something that has been a hot topic within the Heavy Projects since ARKit was announced. A lot of our work to date has been focused around trigger based augmentations. The biggest drawback to trigger based applications is the inability to let users walk / look away from the trigger and carry on a seamless experience. When Vuforia Fusion was announced our wish had been granted... or so we thought.

Using Vuforia Fusion

To begin we need to start with what Vuforia Fusion is not. It's not an SDK, its not a set of API's we can interface with. It's terminology. Vuforia Fusion is what the PTC owned outfit are describing their underlying "integrations" with other AR-enabling technologies. Another point to cover is ARCore support. You would assume that it is supported based on the description Vuforia provides above, however in the post on the Vuforia developer site regarding Vuforia 7.2, ARCore will be added when that is released. I feel the current description is a little deceiving, even though it states it "can" offer support for ARCore, it does not. And will not until 7.2 is released (7.1 is currently only available in the beta version of Unity 2018.1).

Now we have that out of the way, how exactly do we use it? Well the short answer is you do not. Simply by developing with the Vuforia SDK as you usually would to build your AR enabled projects enables them with Fusions support. And here is where our first issue lies. How do we know if ARKit or ARCore is being used for a specific feature, and even worse, how is it being used? Vuforia has released its own environmental tracking solution called Ground Plane. It tracks horizontal surfaces and allows content to be placed on those via a Vuforia API (similar to what ARKit and ARCore offers). Using this feature is how you leverage the power of ARKit and ARCore's features, however you have no idea if its being used or not. Technically on ARKit / ARCore enabled devices those are the SDK's that should be providing the plane mapping and not Vuforia's integrations.

Issues aside, it needed to be tested. We created a simple application targeting the Android platform which would allow content to be placed on the ground (horizontal plane tracking). Simple example of what ARKit and ARCore offers, however the results were poor. The tracking quality was low and drifting affected the experience a lot.

Now I am not expecting Ground Plane to be as good as ARKit or ARCore (due to the fact it's actually based off of their Smart Terrain feature which from experience was pretty poor), but this is not a usable solution. From this experience alone, we knew we could not trust Fusion to provide a consistent experience across both iOS and Android devices, even if they had ARKit or ARCore capabilities. Having a quick look through the Vuforia forums we found others with the same issue.

Moving on we wanted to tackle what we had wanted to use Vuforia Fusion for; A seamless experience where users could initiate using a trigger and then freely move around their environment without the requirement to keep the trigger in view. It did not take long to create an experience here to demonstrate this. In fact, this is one of the reasons I like Vuforia over other AR SDK's. They put the developers first and ensure their SDK integrations are simple to use and understand. Now to test it. We built to Android and triggered some content via an image, then hid the trigger so the content would switch over to "Fusion" to handle the device and environment tracking. Again, results were poor. Content was offset (by quite a large amount) and drifting was again a huge issue.

For now we are putting Vuforia Fusion on hold within our research studies. Released mid December 2017, after 4 months it still feels like this is not ready for developers to use. It needs to be labelled a "beta" at best right now. It's not stable and Vuforia need to improve their documentation on these new features. As part of this they need to be transparent about how their "Fusion" system will handle use cases, so developers can know what to expect when users start using their projects.

I am quite disappointed, Vuforia has always been a go to solution for Heavy Projects due to its robust and industry leading image tracking, however this release seems far too rushed. Capitalizing on the release of ARKit and ARCore is not a bad thing, and fusing all of the technologies together is something that the developer community needs to enable consistent creation of AR enabled apps. But it needs to be done right, and unfortunately so far, it has not. They have already offered an integrated solution with HoloLens which provides robust image tracking from Vuforia, then hands off to the HoloLens services to track the environment. It works fantastic, do the same for ARKit and ARCore.

Please check part two of this blog post to understand where we hope this technology will go within the next year, and other threats that may stand against Vuforia.

Note: You may have noticed we only talked about Android devices when testing, and we even mentioned that ARCore is not "supported". This is because on iOS 11.3 using ARKit 1.5 and Xcode 9.3, Vuforia does not render the camera view. It's a known bug that we are awaiting a patch from Vuforia for, and when that comes we shall test our demo experience to see if we get any better results.

Adam Goodchild

CTO, Heavy Projects

Adam Goodchild