Fun. Play

Inspiration

Children in general engage and learn more effectively with technology; this is especially true with children with autism. AR’s ability to input amusing, interactive toys and objects provide higher interactivity with the real world. This overlaying element paves the way to facilitate learning and develop communication skills which will enhance the relationship between children, caretakers and therapists.

What it does

FunPlay is a program that will assist therapists conducting ABA therapy. Currently the program is designed to teach language skills. The way that FunPlay works is that it provides a series of interactive objects on a screen. The therapist will cue the child to pick an object. We have designed the program to work by recognizing grasping motions by the child. The initial “level” of the program will start with muted background and only one object so that the child is not overly stimulated, there is more control over the environment and the child is able to associate the word that the therapist is saying with the object (ABA therapy is conducted in this way to establish a baseline and begin word association). As trials are conducted by the therapist and FunPlay, FunPlay will keep track of trial data by keeping a record of the responses by the child and input by the therapist on a companion app—which we did not have time to develop but did design screens for. When the therapist deems the child has mastered a skill, they will increase the level of the game so that there are more objects on the screen. The therapist can also increase the difficulty of the game by reducing the filter of the background to make it less muted; this increases the amount of stimulus that the child will receive. Increasing the stimulus allows us to make the skills more transferable to the child’s everyday life.

Parents of children with autism have a particularly difficult time communicating with their children, and this can be extremely frustrating for both the child and caretakers because their needs are not met. This can lead to dangerous tantrums and cause anxiety and depression for families. We developed FunPlay— short for Functional Play—with the idea that we would be able to engage children to learn language skills through an interactive interface. Children with autism have shown incredible improvements in social skills when taught through technology. Over the past few decades, several thousand peer-reviewed studies have documented the effectiveness of Applied Behavioral Analysis (ABA). One of the important aspects of ABA is to put a child in a natural environment. ABA functions best when conducted in a environment free of distractions. With these two factors; AR is a ideal platform to enhance ABA therapy by merging effective methods with technological environments.

How we built it

We worked Unity dividing the whole project into three tasks.

1. The core of the behavior of the application- what events triggers the animation of the objects and display of objects
2. Know how to interact with objects, adding the therapy process. For example: setting the environment for the child and how the child will trigger the interactions set from the first block.
3. With App running on Unity- how to load unto HoloLens

To lay out assets we used Google Poly and for sound Soundbible. This created a more personalized experience than using generic objects. We used objects that resembled toys that would be appealing to the children.

We tested the app in two stages: one: that the main unit would work, and two: the HoloLens environment

Challenges we ran into

We haven’t played with HoloLens before or used AR in a development process.

Also we needed to figure a natural environment for the children to be placed in.

We chose AR over VR for the potential future use especially for Autistic children. The natural world is key and with AR we can place the children in another world while also blending the reality of their natural SAFE world called home.

Accomplishments that we're proud of

From the pitch process to the launch of the app we were a team. All of us bonded over the topic and concept of the app early on which helped us find a focus and direction quickly.

We also pitched the idea to caretakers of people we know with Autism and they are excited not only to try it but also that people are showing interest in topics that seem overlooked.

Additionally we are able to use our strengths to create a stronger final presentation with individuals showcasing their fields while also lending to other elements of creativity.

What we learned

How to use:

Microsoft HoloLens Gesture Tracking Unity GitHub Google Blocks

What's next for Fun. Play

In the future: how we conceived this whole project is for children at home: through machine learning the app recognizes objects in the kid’s home and incorporates it into the game to enhance the learning process. When with a therapist, the app helps the child pick the correct object and make the appropriate gesture related to it. Also to utilize the appropriate word associated with the object and finally the child will be able to say I want this etc.

Built With

• github
• google-blocks
• google-poly
• microsoft-hololens
• unity