A screenshot of our flower environment in progress
For our project, we will be making a series of environments that attach to different mind states. We are experimenting with what I describe as ‘mirrored spaces’ , spaces that have similar elements but different themes and feelings. For example, where one environment has a large leafy tree, another has the same tree only dead and bare. At the present we are focussing on only two opposing mind states (and therefore environments): a binary pair of positive and negative, happy and sad. This might change in the future but for now we are focusing on these two rudimentary emotions.
Bringing the environment into Unity as a test revealed several points that need to be addressed. First off, unlike Maya, Unity uses backface culling by default, meaning that polygons can only be seen from one side. Flat objects such as leaves, flowers, and grass will have to be rebuilt to be double sided. The tree will also have to be rebuilt because the built-in tree from Maya has too many vertices to be imported into Unity. To facilitate easy updating of assets and future building of the environment, we believe it would be more efficient to export individual models into Unity and assemble the environment there, rather that import the whole environment as one large asset with multiple copies of models. As such, all the models will be rebuilt or copied as individual files, and the environment reassembled in Unity.
Upon recommendation of Dr. Gromala, we decided to try out some Playstation 3 games that would help us in understanding the ideas of immersion to a greater extent. She pointed us towards three games by Japanese game company “thatgamecompany“, a company well known for their attention to art direction and ease of play. The company desribes their games as “experimental” and rewarding if the player feels something out of playing them.
We first played Flow, a game which places you as a creature in the sea looking to grow by absorbing other creatures. This is done by touching the different creatures (only works if your bigger than them!), with all the different lifeforms being represented abstractly. Some of the things we liked were:
- the game interface is immersive… no menu. never interrupted. simple directions given, even when paused, you are in the game
- dynamic environment… everything is always moving. particles create depth and energy
- it feels like you can play “forever”, gameplay never ends
- different layers can be seen, gives you idea of the “next step”. aided by opaque
- no game over, can’t lose: creates sense of endlessness
- sound adds to the movement, flow
- the different stages have different sound design, ambience, music; creates uniques environments
- light patterns are interesting to watch and keeps your attention
The next game is called Flower. In this game, you are a pedal that has to collect other pedals to continue through the game, always moving and flying through virtual space. Dr Gromala suggests that this game induced motion sickness, which we should aim not to do with our virtual movement, and we wanted to see if we found this to be true. We found this statement congruent to our experience to an extent, but for us the movement was not a deterrent from playing the game. In fact, we found the abstract representation of “pedals flying” to be relaxing. Some of the things we found good:
- motion is very fluent… feels like you’re always moving
- grass reacts in an interesting way, very flowy. adds to the motion of the pedals
- flowers react to movement and the colour makes it feel alive
- zoom in and out are used to add speed, “vertigo effect”; focal length gives impression of depth
- simple to play… very obvious what your have to do
- music reacts to movement, “chinese cords strike”, flower sound add melodic compliments to main soundtrack
- clear progression adds interest; rewarded with sound and colour when completed objectives
- motion graphics introduce different areas, use of mixed media
The final game we tried out was called Journey. It is an adventure game where you are a mysterious person who is exploring a desert landscape, solving puzzles and meeting strange beings along the way. What we liked:
- world feels very empty but full of things you want to explore at the same time: there is a want to explore because of the mystery surrounding the game
- sand looks very good: constantly moving and reacting to your movement. begs you to interact with it.
- lots of mystery surrounding the character and story, makes you want to find out more
- option for multiplayer adds dimension to experience “makes you feel less alone”.
We had a lot of takeaways from these games and we definitely enjoyed playing them in between the other research we were doing. We look forward to looking for more games that relate to our project.
In his TED Talk, Neuroscientist Vilayanur S. Ramachandran explores the topics of the human brain and what it means to study it. He goes over three major “clues”, that of Capgras delusion (where you cannot recognize what things are), phantom limbs (amputees seeing a fake arm), and synesthesia (where you associate sensorial stimuli to other senses).
All three of these he treated using body illusions, finding ways to trick the brain. For example, with the phantom limb example, Dr. Ramachandran used a mirror box with a reflection of the real arm to trick the person’s mind into believing they had another real arm. This worked so well that it helped the patients move on from their visions of a fake arm.
How we found this inspirational to our project was that these researchers used innovative ways to tackle real world problems. Digital technology opens up things even further: using body responsive, biofeedback technology we will be able to take full advantage of the human system and be able to help pain patients and other users understand their body to a greater extent.
The article The Ultimate Display by Ivan Sutherland explores the means in which we have the ability to represent and control data, exploring our physical reality through the realm of mathematics. With mathematics we have the ability to calculate physical phenomena’s and as a computer works off of mathematics, we are able to simulate experiments that couldn’t be conducted in our reality as we abide to certain physical laws. as an example, with a virtual realm that controls friction and how it reacts with objects, we can toy with friction parameters, or even turn it off, simulating an environment not possible in our known world. As our processing power increases we get closer to being capable of re-creating physical phenomena’s to even simulating physics not possible available.
This can relate to our project as we wish to simulate a visual representation of our psychological stimulus, meaning we can create feedback to what we feel as human beings. As the user’s use the application more their feelings might become clearer through relating them to visual and auditory cues. Such a surreal audio/visual experience would only be possible in a virtual environment, hence our reasoning for wanting to work with this medium.
This weekend, we visited a sensory deprivation tank facility in Abbotsford called Cloud 9 Spa. To give you some background, what happens in these tanks is that you lose all senses – upon entering the tank, one floats in salts that give you the feeling that your body is floating in complete darkness, and void of any sound. The tank is also room temperature which makes it easier for your body to adapt to.
All three of us on the team tried it out for an hour, each with a different experience at the end of it. What some of the most prominent sensations were a feeling of loss of time, literally feeling like “floating” in blank empty space, and large amounts of thinking (while also devoting time to being rid of it).
Some takeaways from the experience was wanting to make the subject feel that they are floating, to aid in immersiveness, and preparing the room to feel empty and quiet to help the HMD have total effect. It goes to show that even though a space is empty, it can really change your perceptions of what happens around you. The Pain Lab was gracious enough to provide us with a space to work in and were hoping to put this to full potential. We are still dabbling with the idea of producing a CAVE instead of using a HMD but this is something we will explore more in the future.
We plan on posting as we go along a number of “inspirations” of other researchers, artists that have functioned as precedents to our work and are helping to guide our work. Our first log will focus on the work of “Group Ehrson”.
“Group Ehrsson”, a Brain, Body & Self Laboratory, is based out of the Department of Neuroscience at the Karolinska Institutet in Sweden and is run by Henrik Ehrson. Here, the researchers look into making people feel “out of body”, using mannequins, rubber arms and virtual reality to create body illusions. These illusions find ways to convince people that they have swapped bodies with another person, gained a third arm, or grown to giant proportions.
One of his exhibits we found particularly interesting, was one where the subject felt like they had shrunk into the size of a Barbie doll. This was accomplished by the subject laying down, wearing a head mounted display (HMD), and getting poked using a stick. On the display they are shown live footage of a doll being rubbed. This is enough to trick people’s minds into thinking they are a small doll.
What we found the most interesting about Henrik’s exhibits is that they are able to use the technology in a way that alters’s perceptions of self, and in a fashion that could only be accomplished by using a camera and HMD. In the above example, the “eye of the camera” in collaboration with the HMD changes the person’s vision into another, allowing for controlled immersion. We hope with our project to be able to pull off such radical immersiveness, and be able to give visitors an experience of changed self and greater awareness of body after using our system.
Hi! Welcome to our blog for our upcoming project Sensorium. Here we will be documenting our progress developing the system, and listing some of our inspiration in making it. You might even experience some witty commentary along the way.
If you have any questions for the team, feel free to comment on our posts, or send us an email at firstname.lastname@example.org