Augmented Reality

Augmented reality (AR) is a technology that renders computer-generated images, sounds, or other sensory enhancements onto a user's view of the real world. It is often achieved through the use of a device such as a smartphone, tablet, or specialized headset that displays the augmented reality content.

AR can be used in a variety of ways, such as to provide information or assistance in real-time, to enhance entertainment experiences, or to facilitate communication and collaboration. For example, a person using an AR app on their smartphone might see virtual labels or graphics overlaid on physical objects in the real world, or might be able to interact with virtual objects that appear to be in the same physical space as the user.

AR technology has the potential to revolutionize many industries, including education, gaming, and healthcare, by making it possible to access and interact with digital content in a more natural and immersive way.

Stack AR

Augmented reality (AR) allows the rendering of digital content on top of the physical space, which is a promising medium for tangible interaction. Marker-based AR is widely used thanks to its low costand ease of integration, but the gameful aspect of manipulating the physical AR markers remains understudied. In this paper, we explore the stacking mechanics of transparent AR markers and create an AR music game called Stackable Music. Stackable Music can be developed, assembled, and set up at the home or office with a printer using several sheets of transparent film and a PC or mobile device with a camera.

Flow Chemistry

  • Microsoft Hololens 2
  • MRTK 2.4.7
  • Research In Progress
  • Project Website
  • Claire Li, Noelle Rakotondravony, Hilson Shrestha, Karen Royer, Laurie Mazza, Shano Liang, Varun Bhat, Prof. Lane Harrison, Prof. Robert Dempski

Worked with Professor Robert Dempski in summer 2020, we developed a HoloLens AR application for training students to assemble components in order. HoloLens AR enables the user to watch the tutorial while working on the real-life components at the same time. We worked as a team of 6, spending 2 months to make this application.

Using mixed reality in chemistry laboratory settings is a perfect fit because you can get access to both virtual content and real-life at the same time. Head-mounted AR serves as an additional information source to the researcher, which doesn’t affect his/her ability in observing real-life events. This is crucial especially for chemistry laboratories because if the researcher’s attention is fully attracted by an electronic device, it might cause safety issues.

Brick Block in AR

  • Microsoft Hololens 2
  • MRTK 2.8.2
  • GitHub
  • Side Project In Progress

An AR remake of Pskar Stalbergs Brick Block. The original brick block can be found Here.

Food Chain AR

  • iOS, Android, Microsoft Hololens 2
  • MRTK 2.8.2, Vuforia v10
  • Research In Progress
  • Project Website
  • Rachel Foye, Dinah Agyemang, Ava Stockton

The overall objective of this research is to develop an Augmented Reality (AR) assisted learning tool with educators and students involved in the design. Balancing between informative and engaging, the AR tool is aiming to become both engaging and helpful in terms of enhancing high cognitive level learning outcomes. Applying AR in education in an interdisciplinary research area including pedagogy, learning science, interactive media, and psychology. This study follows a participatory research approach and includes educational technology engineers, educators, and students to co-create an AR tool iteratively – engineers come up with a prototype, educators give critical feedback, and students evaluate the tool from both engagement and learning outcomes aspects. The study will bring insights into how to utilize participatory design in AR educational tools and how AR has an impact on students learning outcomes, as well as effective ways of prototyping AR experiences.

Sewn Into Memory

  • Microsoft Hololens 2
  • MRTK 2.8.2, Vuforia v10
  • Major Qualifying Project Completed
  • Shruti Mahajan, Amanda Jones, Megan Letendre, Elise Nerden, Prof. Gillian Smith, Prof. Erin Solovey

Personal memories are associated with much more than just a picture or a video, for example, clothes, T-shirts, or other fabrics. Such tangible/tactile objects are important for memory and nostalgia. Quilts are often made with a loved one's old shirts or for special occasions. These quilts have special cultural importance and interaction that triggers emotional responses by touch. We envision a future of AR where people will be able to augment personal items with visuals and audio. Family or friends co-creating a memory quilt that is a collection of special clothing items pieced together to celebrate important milestones, special events, or the life of a loved one augmented with AR visualizations and audio.

HoloLens Wizard-of-Oz Kit

  • Microsoft Hololens 2
  • MRTK 2.8.2
  • GitHub
  • Paused

Integrating new interfaces in AR experience is challenging because of the uncertainty in the interfacing technology. Wizard of Oz is a HCI technique in which subjects interact with a computer system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human researcher. This project builds the bridge between researcher the "wizard" and HoloLens.

Virtual Reality

Virtual reality (VR) is a simulation of a three-dimensional environment that can be interacted with in a seemingly real or physical way by a person using specialized equipment, such as a headset with a screen or screens inside. VR immerses the user in a simulated experience that can include sight, sound, and other sensations.

In VR, the user is completely immersed in a simulated environment, while in AR, the user can still see and interact with the real world while also seeing and interacting with virtual content.

Delsys's Training Games

  • Vive Pro 3 (Android)
  • Contract
  • Laurie Mazza, Michael Schubert, Mengliu Lu, Prof. Jennifer deWinter, Monty Sharma, Delsys Team

This project is the development of VR games targeted toward children with muscular dystrophy in the 8 to 12-year-old age range. They have a range of movement capabilities but all gameplay is to be seated and have periods of rest built-in. The goal is to encourage the children to participate in clinical recovery training through gameplay interactions instead of direct instructions.

We worked as a team of 4 and spent 4 months on the development. This project is a collaboration among Delsys, Worcester Polytechnic Institute, and Clark University.


  • PC, Oculus Quest 2
  • GitHub
  • Research Article
  • Claire Li, Yihong Xu, Alex Sirois, Risa Kiani, Prof. Yunus Telliel, Prof. Yuko Oda, Prof. Erika Lewis, Prof. Kelilah Wolkowicz

WheelUp is a collaborative research project between Worcester Polytechnic Institute (WPI) and University of Massachusetts Lowell. The overall objective of the study is to co-design a high-fidelity VR wheelchair simulator for diverse body. It uses participatory design to engage local communities with alternate-abilities for the development of a wheelchair training simulator with multimodal input interfaces for wheelchair guidance and control. Simulated and virtual reality (VR) training allows users to practice difficult or stressful tasks in a safe environment. The virtual environment will integrate joystick, muscle signals (EMG), and eye gaze (EOG). Aesthetic attributes of the simulator such as color, texture, and layout will be considered to enhance the overall visual experience, making the user experience more enjoyable and appealing. The simulated environment includes standardized hospital or home; variable environments with changing obstacles such as shopping centers or workplaces. (PI: Prof. kelilah wolkowicz)

360 Lab Tours

We create 360 lab tours for virtual walkthroughs and training sessions.

Novel Interfaces

Novel interfaces have the potential to revolutionize the way we interact with computers and technology.

EMG is a method of measuring the electrical activity of muscles, and it can be used to control prosthetics or other devices through muscle signals. This can be particularly useful for people with physical disabilities, allowing them to perform tasks that would otherwise be difficult or impossible.

BCI, on the other hand, involves using brain activity to control devices or machines. This has the potential to allow people with severe physical impairments to communicate and interact with the world around them in ways that were previously unimaginable. BCI technology is still in the early stages of development, but it holds great promise for the future.

BCI-informed Visuals

  • Experimental, NIRx's fNIRS System
  • Thesis
  • Prof. Erin Solovey, Prof. Gillian Smith

we explore the use of brain-computer interface (BCI)-adapted visual effects to support atmosphere in a walking simulator, and investigate its impact on player-reported immersive experience. While players were using a keyboard or joystick controller to control the basic movement of a character, their mental state was accessed by a non-invasive BCI technique called functional near-infrared spectroscopy (fNIRS) to implicitly adjust the visual effects. Specifically, when less brain activity is detected, the players' in-game vision becomes blurry and distorted, recreating the impression of losing focus. fNIRS measures blood oxygenation levels, are related to brain activation. When a particular area of the brain is activated, it receives an increase in blood flow. With this biological indication, we designed a BCI-controlled game, in which the vision becomes blurry and distorted when less brain activity is detected, recreating the impression of losing focus.To analyze the player’s experience, we conducted a within-subjects study where participants played both a BCI-controlled and non-BCI-controlled game and completed a questionnaire after each session. We then conducted a semi-structured interview to investigate player perceptions of the impact the BCI had on their experiences. The results showed that players had slightly improved immersion in the BCI-adaptive game, with the temporal dissociation score significantly different. Players also reported the BCI-adaptive visual effects are realistic and natural, and they enjoyed using BCI as a supplemental control.

BCI Live Coding

  • TidalCycles, Processing, Muse S
  • Course Project
  • Project Website
  • Samin Shahriar Tokey, Prof. Charlie Roberts, Prof. Gillian Smith

We did a live coding performance using visuals and audio integrated Brain-Computer Interface input. We asked for a volunteer (audience or co-performer) to wear the BCI headset while we perform. With the BCI signal from the audience, we will generate audience-adapted visuals for our performance.

We followed the approach of performance-led research in-the-wild, which falls under the broad umbrella of Research through Design. This means research findings are emerging from reflection on the making of specific interactive artifacts. Being performance-led means the practice is led by artists, who follow an artistic process and deliver a professional artistic product, in our case, a live coding performance. Being in-the-wild means the artistic product is experienced by public audiences, in our case, being under critical judgment by the live coding cohort. The approach involves documenting both the artist’s reflection on producing the work and the audiences’ experience of it.

Immersive Experiences

An immersive experience is an activity or environment that fully engages a person's senses and allows them to fully immerse themselves in the experience. Here, Immersive experiences refer to interactive exhibits or installations, and miscellaneous interactive products.

Holiday Escape: Escape Room

The Escape Room

Holiday Escape is one of our best WPI Annual Escape Room series. Our escape room is a 30-minute experience for groups of people where the goal is to leave or escape the locked room by solving a series of puzzles.

This project is a class project of IMGD 5100 Interactive Media. We worked as a team of 16 and spent 2 months to design, build, and run the escape room.

The Story

You are boarding the train for your holiday tour of Santa’s toy shop, but something is afoot… the conductor can’t seem to get the doors to open. Upon successfully boarding the train, are greeted by a message from The Elves, who have rebelled and hijacked the train after tired of years of unsatisfactory working conditions making toys for the shop. If the train reaches its destination, you will be turned into elves and forced to take their place in the workshops for all eternity!

If players fail, they are “turned into elves” and wear elf hats for final photo. If players succeed, the elves unionize and improve their working conditions without capturing the players

My Role as the Lead Designer ("Puzzlemaster")

I led puzzle design and iteration, built and tested puzzle prototypes collected and analyzed playtest feedback.

Lessons Learned

The narrative in escape room needs to be explicit and straightforward. However, not all players care about the story. Embedding puzzles to storyline is the creative process that matters the overall experience.

Dream Land

  • Theme Park
  • Shanghai, China 2020
  • Jiadi Zuo, Ekings Multimedia

The Show

IKALI Dreamland is the first comprehensive children's amusement park in China under the theme of scenario-based costume play. It integrates the novel concept of role play and edutainment for children, which is put into practice in the form of a nationwide brand park chain.

In December 2020, Ekings Multimedia and IKALI jointly operated Dreamland Shanghai. Daily admission flow reaches 300 at its peak.

My Role

During Jun 2020 to December 2020, I worked as a programmer at Ekings Multimedia, where I worked with clients on designing and developing interactive media applications and games with various needs.

I designed and developed a mini game Air Balloon Pilots, in which player controls their air balloon by balancing its mass.

I assisted in setting up and testing the infrared camera multi-touch screen system. It uses IR camera to track the user's hand gesture and projector to cast the interaction feedbacks, therefore it can turn any flat surface into a touch screen.

Lessons Learned

Immersive experience pulls a viewer into another real or imagined world, enabling them to manipulate and interact with their environment. It has a long history, from Panoroma landscape painting to Cinerama widescreen projection process, to Cave Automatic Virtual Environment where projectors are directed to walls of a room-sized cube. While the comtemporary screen media erode, if not erase, viewer's perceptual differentiations between the actual reality and the virual reality, the gap between real and virtual is where designers perform their the magic touch.

The market of projection mapping, IR camera touch screens, and sensor-driven interactions is open - the entry cost is relatively low comparing to VR/AR hardware research. The future lies in designing a meaningful and aesthetic experience. Level 99 Entertainment combines escape rooms, small "challenges", and arena games with microcontrollers and sensors, making the traditional plays into self-reset "smart" games. TeamLab brings artists, programmers, CG animators, mathematicians and architects together to navigate the confluence of art, science, technology, and the natural world. The list of examples can go on.

Collection: Non-learning Storytelling

  • Craft + Experimental Storytelling
  • Presented at PAX East 2022
  • Shano Liang, Tony Dai

The Show

Collection is an experiment combining non-linear storytelling and traditional origami. It appears as an opened package, with a letter and a treasure box inside. The treasure box contains old pieces of newspaper clips, lab notes, drawings, and segments of a diary, forming a bizarre tragedy about the Werners.

It is up to you to decide how to explore the Werners’ story. There are 16 pockets on the top, but more are hidden underneath them. It is a tangible physical book that you can even tear apart. Search through the pieces in the Collection to discover the truth.

Lessons Learned

This treasure box format is similar to the take-home escape room adventures. Instead of having multiple physical objects, you have only one complex object. However, it also limits the format of story segments to be paper-based.

Origami and tangible interface provide the enjoyment and engagement while exploring the puzzle.

Game Design & Culture

Game can be designed to achieve a specific educational or training goal, rather than purely for entertainment. Serious game is a type of game that is specifically designed to support learning and teaching. These games can be used in a variety of settings, including the classroom, after-school programs, and homeschooling. They can be used to teach a wide range of subjects, including math, science, reading, and social studies. Like serious games, games for education are often designed to be more educational in nature than traditional video games. However, they may still incorporate elements of entertainment and enjoyment to keep learners engaged.

Game culture studies is a field of study that investigates the cultural, social, and psychological aspects of video games and gaming culture. Video games are an increasingly significant part of contemporary culture, and understanding the cultural, social, and psychological impacts of games is important for understanding the ways in which they shape and are shaped by society. Video games have the potential to be used as a tool for learning, education, and social change, and game culture studies can help us understand how games can be used effectively in these contexts. Video games are a rapidly evolving medium, and game culture studies helps us understand the ways in which games are changing and how they might continue to evolve in the future.

Fighting Against Coronavirus: Pandamic Simulator

Fighting Against Coronavirus game is a single-player strategy game. The player takes the role of a government official in “City W” starting from January 1, 2020. The player’s mission is to guide local disease prevention and control work, as well as take care of the people at the epicenter of the ongoing outbreak with minimal loss in human life and economic growth. We worked as a team of 3, spending 4 months to make this game.

I designed the in-game epidemic model by modifying the SEIR model (Susceptible, Exposed, Infectious, and Recovered), which makes me think of the close connection between mathematical modeling, system dynamics and game. Later I learned Machinations framework from the book Game Mechanics Advanced Game Design, and it is similar to the stock and flow diagram, which confirms the close connection between in-game economic system and system dynamic.

Circuit Dungeon

Cirbuit@Dungeon is a web application for learning basic electronics. This project is supported by WPI Academic Research Computing Center.

Chinese Murder Mystery Games - "Jubensha"

  • Research
  • In Progress

The importation of western games and gaming culture have influenced China, while localization and creativity of its derivative games form a unique new trend. Murder mystery games, also known as “script killing”, is an ideal object for studying the adaptation of the games across cultures. Digitally, players tend to use mobile applications to hold murder mystery games virtually, and structured gameplay rules are set within the application and modified as needed by writers. In-person murder mystery games create a new market for renting and hosting the game, and under this circumstance, a combination of escape room and live-action murder mystery game is born. In this study, I focused on what is distinctly “Chinese” in these game, looking specifically at in-game environments, external environments, and users in determining localization.