Introduction

Three dimensional graphs illustrated in two dimensions can often be difficult to interpret. The product of this augmented reality application will allow people to view and interact with three dimensional versions of these flat graphs.

Description of application

When learning mathematics (and other disciplines) graphs are commonly used to better understand equations and to gain a visual interpretation of mathematical models. Mathematics is often taught in a 2D space, either a physical handout, whiteboard or on a projected display. While 2-dimensional graphs can be easily conveyed using this teaching style, 3-dimensional graphs cannot. Representations of three-dimensional graphs can be shown in 2D (flat 3D graphs), but these can be difficult to understand and don’t show all angles of the graph, limiting the information that can be conveyed. This can be seen below.

To improve on this, I developed an interface that could display these graphs in 3D space. It needed to be accessible to everyone, minimal setup and work within the already existing 2D teaching methods. AR interface technology met all these requirements. It is easily accessible and contained within a phone, no external tracking or projecting devices are needed.

The phones camera is pointed at a flat 3D-dimensional graph, either on a handout or during a lecture, and the phones display will show an augmented reality with the 3-dimensional graph sitting atop the flat version. This interface also allows the user to change aspects of graphs in real time. Being able to interact with these graphs in a 3D space increases the information bandwidth to the student, making it much easier for them to interpret the graph and helps them develop a better understanding of mathematics being taught.

Currently the application only works with a few pre-set graphs. Ideally the application would be able to use artificial intelligence to interpret any flat 3D graph (possibly even hand drawn) and compute its own 3D model. The current and more viable use case would be to have teachers provide a collection of pre-set graphs, corresponding to certain lecture notes, that can be loaded into the app.  

Interaction design

Data annotation is the form of AR that is being used in this application. The user will have control over a tangible viewpoint (phone) to view virtual graphs that have been placed in the real environment. Using video tracking, the graphs are situated above flat versions of itself which can be translated and rotated to provide a different viewing angle of the graph. A simple user interface was also implemented to assist the user in specific use cases. This user interface consists of virtual buttons and sliders placed below the graph that can be interacted with through the viewpoint. This ensure that the buttons don’t clutter up the viewpoint and it is clear which UI corresponds to which graph.

The UI allows the user to rotate the graphs in cases when they cannot, or do not want to rotate the tracked image, such as in a lecture. There is a scale slider to allow the user to resize the graph which is useful if the graph is very detailed. The scale slider is also useful when comparing multiple graphs, they can be made smaller (or even turned off) to allow for more graphs to be seen through the viewpoint. Similarly, the UI can be minimised to reduce clutter of the virtual objects. There are also two other sliders that can be used to change the shape of the graph. Being able to interact with the graph in this way allows a student to gain a much better understanding of the equations behind the graphs.

There are many other interactions that this application could benefit from. When different graphs are moved close together, they should combine, becoming part of the same axis. This would make great use of the augmented reality, allowing for users to easily compare graphs. Being able to change the colour and transparency of a surface when there are multiple on one axis would be needed to ensure the graph could be understood clearly. An interaction which allows the user to press on the surface that they want to alter would also be needed. Numbered axis and text displaying the equation of the graph with the current ‘a’ and ‘b’ values would greatly increase the amount of information that could be communicated to the user, allowing them to better understand the math behind the graphs.

Technical development

This application was designed in Unity using Vuforia, an augmented reality software development kit. Vuforia makes use of video-based tracking to situate virtual objects in the real environment. The unity project is based around a prefab called ARGraph. This prefab can be placed in the scene and will work with pre-set or custom image or surface functions. The prefab consists of an empty game object containing a script name Graph and a separate UI canvas. The Graph script is used to generate the axis and meshes for a graph and has a few public functions to update, scale and rotate the graph. It contains input variables to adjust the initial scale and appearance of the graph and provides a choice between a few pre-set surface functions. Custom functions can be added into the Graph script. When adding in a new prefab it is important to add the Graph game object to its own layer, otherwise other graphs will be affected by the light associated with the newly added graph.

A big challenge with this application was to create flat 3D graphs that can be easily tracked. Generally, graphs are not that detailed which doesn’t allow for many tracking points. Trying to keep the images “graph like”, attempts were made to improve the tracking potential (an attempt can be seen below). This did help, but most images in the tracking process have a very low tracking score, leading to some strange behaviors. To overcome this a trackable outline could be developed and put in place around each graph, or if the graphs are being generated using a computer, the surface of the graph could be a detailed pattern.

Currently the application works best when the tracking images are place lying down. Unfortunately, when using the application to track images standing up (such as on a screen) the UI jumps forward. It is still usable but not ideal.

Description of 3D models

The only 3D models used in this application was the graphs. These where generated using a script, line rendering objects were used as the axes and a mesh for the graphs surface. Icons for the UI were designed using Inscape to enhance the users experience.

References

Line Rendering - For this project I used line rendering to create the axis of the graph. To learn about line rendering I used the tutorial by Isamail Camonu[1].

Mesh creation - I chose to create my own mesh to illustrate the surface of the graph. This mainly consisted of creating a simple grid of vertices and defining the triangles. The tutorial refenced below by Jasper Flick stepped through this process[2].

[1]        Ç. İSMAIL, “How to draw lines, circles or anything else using Line Renderer,” codinBlack, 2019. https://www.codinblack.com/how-to-draw-lines-circles-or-anything-else-using-line... (accessed Aug. 05, 2022).

[2]        J. Flick, “Procedural Grid,” Catlike Coding. https://catlikecoding.com/unity/tutorials/procedural-grid/ (accessed Aug. 06, 2022).