Mobile device talks with browser

There might be some situations that you wish you could operate your browser screen with your mobile device(s) or vice versa. Let's say you want to extend your desktop to a browser screen for instance dropping an article for later reading without having to browse to an article on your mobile screen. Or would you like as a game developer use the browser screen as an extended display for a mobile game, to display statistics or items or things like that?

Well this and more can be done using QR codes, WebSockets and Javascript, but first we must connect our device with the browser in use. You could do this in several ways. You could make the user to punch in a random code from browser screen to mobile device or vice versa, but the user don't want to do that, right? You could use NFC tags, but then you'd be stuck with static codes and physical NFC stickers attached on your computer. That is no good.

In this article, we use QR codes to connect mobile device with web service on browser. The QR codes are cool. They can contain a few hundred alphanumeric characters within one tag depending on the size and complexity of the tag. Usually they are used statically in posters or ads. The QR codes contain mostly a link to a website related with the subject at hand. BTW you can create your custom QR-codes here https://www.the-qrcode-generator.com.

We pass a dynamic QR code from web page optically to our application within our device. The QR code that we use, contains PHP session ID of the web service. The session ID is created  when the web service is fired the first time and it will remain the same until the end of the client web session, unless the client empties his or her cache. You could use some other identifier like IP address, but session ID is better identifier because it is inevitable unique and anonymous for the user.

Now, when the user loads our web page, session ID is generated. Then we connect the web page with our WebSocket service. WebSockets handle real time, bi-directional communications between clients and servers. WebSocket usage is not within the limits of this article, but there are a lot of open source solutions available such as Socket.IO and PHP based Ratchet.

Next, the user fires up mobile application and points her or his device to browser screen. The application reads the QR code from the browser screen. In our application, we use the Unity library called ZXing.Net to read the code. Then application connects to WebSocket using the acquired QR code and voilà, you now shoud have a WebSocket connection from your device to your browser screen!

What happens next is completely up to you as a application / game developer. We use this technique to trigger Javascript events in our upcoming fighting game. Events such as changing the fight  arena background screen, to start background music in browser, to display score and fighter energy bar, to make screen flicker when people are being punched and some other crazy shit like this ;)

To receive our development blog, sign up out Facebook, Twitter and Google+! 

 Here is a QR Code, open it at your own risk...

Experimenting Vuforia. Check out demo videos

At the very core of augmented reality based applications there is this thing called image recognition. Basically it means that when an image is viewed through the camera of a device, this image is recognized and it's orientation and distance can be determined within application.

When we know the distance and orientation of the image against our mobile device, we can use this information to create 3D space in our application that appears to be a part of the image that is viewed with camera.

Image recognition is hard. Building algorithms that calculate the position of tracking image in real time is a task that requires an expert of the matter -or rather an army of them. Luckily for us independent game developers, there are free SDKs that everyone can utilize in their projects such as Qualcomm Vuforia.

Using Vuforia with Unity is easy. Just download the Vuforia AR Extension from Vuforia site or directly from the Unity Asset Store. Import it into your Unity project. Create your image targets in Vuforia Developer website, download them, import them to yor project. Add AR Camera component to your scene and define Image Target objects. Remember to switch on "Activate dataset" at AR Camera GameObject and that's it!

In order to demonstrate this image recognition based augmented reality 3D space, we have prepared three videos for you. First there is the demo with Spark the Spider



And then there is Shaft the Shark demo. It jumps when being tapped!



Finally we experimented to recognize image from computer screen and it worked just as well as with the printed images. Say hello to Herman the Hellraiser!


We at the Frak are currently using this feature to construct a fighting action game for iOS and Android. In addition to AR, we have some other pretty cool features as well. Stand by for next time as we discuss QR code based optical tracking.

Sign up for our mailing list and like us on facebook, Twitter and Google+ to receive updates!