We are currently seeing a growth in AR and VR applications, and the area is expected to see a significant growth in revenue the months and years to come. The challenge lies in finding relevant business cases and good uses for the technology, not because the technology is fancy but to find actual needs. With Room for Game, we wanted to explore AR and find a use case for a game in a museum setting. The game was supposed to combine virtual and physical components and inspire user’s to start programming.
HoloCity is a virtual city, a very unlucky one at that. The city is suffering from several environmental issues such as fires and earthquakes. A professor in the town has built a very capable robot, but unfortunately not a very smart one. The robot has the functions needed but it does not know how to use them. It is up to the users to teach the robot how to use its functions. This is being done by basic programming where the focus is on getting into the proper mindset rather than specific programming functions.
Apart from social challenges such as large user groups with different needs, the main technical challenge of this project was to track a moving target and project a hologram on top of it at all times.
Our solution is based on the computer vision library OpenCV. We place a small colored object on top of the robot and then use OpenCV to track it in every image sent from the camera. By knowing the position of the object in the image we could calculate a position in the 3D world. We did this by combining the X and Y coordinates of the ball in the image, the head position and the distance of the ball from the camera.
The tracking solution has allowed us to create a game that combines a physical robot with virtual pieces projected on it. The user is only controlling the physical robot, but is able to choose what virtual pieces are being projected through basic programming similar to Scratch.