Publications | Stefan Marks

2025

Perceptions of liveability and sense of community in five different neighbourhoods in Tāmaki Makaurau Auckland, New Zealand

P Besen, A Burgess, S Vallis, S Marks, R Horne, A Geddes, J Pearson, J Cruz, D Patel, and K Paala

In Proceedings of the International Conference on Smart and Sustainable Built Environment (SASBE 2024), 2025

Abs DOI

The way buildings and neighbourhoods are designed, built and maintained can influence people’s lifestyles, social interactions and overall quality of life. Tāmaki Makaurau/Auckland has diverse urban and suburban environments that have been going through significant transformation in recent years, especially due to the need to build additional housing through densification. Post-occupancy evaluation is rarely applied in Aotearoa New Zealand’s buildings, and even less frequently at the neighbourhood scale. Existing literature has assessed liveability in specific neighbourhoods in New Zealand, with limited ability to make comparisons between different areas due to a lack of available data. Architects, urban designers and planners often make assumptions about design strategies to create medium-density walkable, liveable and well-connected communities, but there are few feedback loops to understand how these perform in practice in each location and how these built environments are perceived by residents. The aim of this paper is to fill these gaps in knowledge by assessing liveability in five different neighbourhoods in North, South, East, West and Central Auckland, through mapping and an online survey with residents. Although there were different perceptions of liveability on various aspects being assessed, the overall results demonstrated higher levels of perceived liveability in the following order: Mount Eden (1st), Saint Heliers (2nd), Northcote (3rd), Takanini (4th), Massey (5th). This study explores potential inequities in current densification strategies and it provides insights to help create more evidence-based policies and design practices for socially sustainable and well-connected future neighbourhoods.
Virtual reality rehabilitation for complex regional pain syndrome: A proof-of-concept trial

D Bean, D Rice, N Shaikh, C Pollard, S Marks, N Magni, N Tuck, H Stephens, L Aiono Le-Tagaloa, E Morunga, D Kumarapeli, S Reay, I Nakarada-Kordic, G Bennett, and G Lewis

Mar 2025

2024

Projects and portfolios — An educator’s reflections on the summative assessments in a game programming course

S Hooper, B Wünsche, S Marks, A Luxton-Reilly, and P Denny

In SA ’24: SIGGRAPH Asia 2024 Educator’s Forum, Mar 2024

DOI
Perceptions of liveability and sense of community in five different neighbourhoods in Tāmaki Makaurau Auckland, New Zealand

P Besen, A Burgess, S Vallis, S Marks, R Horne, A Geddes, J Pearson, J Cruz, D Patel, and K Paala

Nov 2024

Abs

The way buildings and neighbourhoods are designed, built and maintained can influence people’s lifestyles, social interactions and overall quality of life. Tāmaki Makaurau/Auckland has diverse urban and suburban environments that have been going through significant transformation in recent years, especially due to the need to build additional housing through densification. Post-occupancy evaluation is rarely applied in Aotearoa New Zealand’s buildings, and even less frequently at the neighbourhood scale. Existing literature has assessed liveability in specific neighbourhoods in New Zealand, with limited ability to make comparisons between different areas due to a lack of available data. Architects, urban designers and planners often make assumptions about design strategies to create medium-density walkable, liveable and well-connected communities, but there are few feedback loops to understand how these perform in practice in each location and how these built environments are perceived by residents. The aim of this paper is to fill these gaps in knowledge by assessing liveability in five different neighbourhoods in North, South, East, West and Central Auckland, through mapping and an online survey with residents. Although there were different perceptions of liveability on various aspects being assessed, the overall results demonstrated higher levels of perceived liveability in the following order: Mount Eden (1st), Saint Heliers (2nd), Northcote (3rd), Takanini (4th), Massey (5th). This study explores potential inequities in current densification strategies and it provides insights to help create more evidence-based policies and design practices for socially sustainable and well-connected future neighbourhoods.
Mixed interaction: evaluating user interactions for object manipulations in virtual space

Y Lee, AM Connor, and S Marks

Journal on Multimodal User Interfaces, Nov 2024

Abs DOI

This paper presents an evaluation of a potential new interaction mode in virtual reality (VR) to determine whether it provides any positive impact in terms of how users interact with content. We evaluated the user experiences for 3D object manipulation across three modes of interaction. Interaction using controllers and gestures are used as baselines from which to gauge the potential value of the new mode of interaction, where a single controller and gestures are combined. This paper reports on a user study that captures quantitative and qualitative data related to a variety of object manipulation tasks in a Virtual Environment (VE). We investigated the impact of this new interaction mode with 40 participants across a number of interaction tasks, with the quantitative evaluation indicating that generally, the mixed mode of interaction resulted in task completion times consistently faster than gesture-based interaction and, in some cases, faster than with the use of controllers alone. A qualitative evaluation of the user experience indicated potential application areas for the new mode of interaction.
Artist Talk on Ngā Raraunga o te Mākū: the data of moisture

J Randerson, R Shearer, and S Marks

Feb 2024
Ngā Rauranga o te Mākū: the data of moisture

R Bull, S Marks , J Randerson, and R Shearer

Jan 2024

2023

Mākū, te hā o Haupapa: Moisture, the breath of Haupapa (2.2)

S Marks , J Randerson, R Shearer, R Bull, and H Purdie

Dec 2023

Abs

The SIGGRAPH Asia 2023 Art Gallery is a juried exhibition within the Experience Hall. The Art Gallery theme is "In Unruly Times." This theme encompasses a diverse range of works spanning art, science, and technology, including data visualization, 3D printing, XR, robotics, AI, sound, performance, NFTs, and interactive installations. "Unruly" here signifies a wild, experimental, and uncontrollable quality, reflecting turbulent moments in history, such as conflicts, climate crises, pandemics, inequality, surveillance, and financial and biosystem collapses. "Times" refers to various temporal aspects, including deep time, digital time, ancestral time, and finite resources. In these times, there is an increasing need for sustainable, inclusive, and agile practices that embrace diverse voices, experimental processes, and kindness. // A Note on Software and Physical Install Versions of each iteration of this collaborative work. In each of the following works the software was developed and the install setting was varied significantly: 1.0 Haupapa: the chilled breath of Rakamaomao (Online, Te Tuhi and World Weather Network websites, 2022) 2.0 Mākū: te hā o Haupapa (Te Tuhi, Iris Fisher Gallery, Auckland) 2.1 Ngā Raraunga o te Mākū: te ha o Haupapa. Blue Oyster, Dunedin. 2.2 Mākū, te hā o Haupapa: Moisture, the breath of Haupapa (Siggraph, Sydney – Proceedings & Creative work) 2.3 Ngā Raraunga o te Mākū: the data of moisture (Khoj, New Delhi, 2024)
Mākū, te hā o Haupapa: Moisture, the breath of Haupapa

S Marks , J Randerson, R Shearer, R Bull, and H Purdie

In [Proceedings of] SA’23: SIGGRAPH Asia 2023 Art Gallery, Dec 2023

DOI
Computer graphics and extended reality courses for the programmophobic

S Marks, and S Gil Parga

In [Proceedings of] SIGGRAPH Asia 2023 Educator’s Forum, Dec 2023

DOI
Ngā raraunga o te Mākū: te hā o Haupapa

R Bull, S Marks , J Randerson, R Shearer, and H Purdie

Sep 2023
Artist Talk: Ngā raraunga o te Mākū: te hā o Haupapa

J Randerson, S Marks, R Shearer, and R Bull

Sep 2023
Mākū: te hā o Haupapa

J Randerson, S Marks, R Shearer, H Purdie, and R Bull

Jun 2023

Abs

MĀKŪ, te hā o Haupapa: Moisture, the breath of Haupapa Collaborating artists: Ron Bull (voice); Stefan Marks (programming); Janine Randerson (video); Rachel Shearer (sound); Heather Purdie, glaciologist and scientific advisor, University of Canterbury Live data stream courtesy of NIWA | Climate, Freshwater & Ocean Science The cracking and melting Haupapa glacier and lake, Aotearoa’s fastest growing body of water, are present at Te Tuhi through a live cast of mākū, life-giving moisture. Tiny bubbles of ancient breath and atmosphere are pressed inside Haupapa’s glacial ice—including sea breezes, pollens, carbon dioxide and methane, as well as the ash of Australian fires. Ron Bull’s voice, recorded live on the lake, is woven through the sound and images to gift and acknowledge Kāi Tahu matauraka, words and names of the elemental ancestors. Glaciologist Heather Purdie’s research has centred on Haupapa lake and Haupapa glacier for many years. She has discovered that the glacier is melting from within crevasses in the glacier accumulation area that retain the sun’s heat; and that at the end of Haupapa, there are submerged ice ramps in the lake, which cause large icebergs to split off that accelerate glacier recession. Rachel and Janine gathered images and sounds from visits to Haupapa glacier in March and September in 2022 with Ron and Heather as guides. Spring is the most turbulent month when ice calves off, separating from the terminus of the glacier and the water is grey, while in March in late summer the water is clear and bright blue. Sound sequences occur in response to specific weathers and wind directions, constructed from atmospheric field and hydrophone recordings from 30 metres deep in the pro-glacial lake, digitally manipulated to create layered ambient textures. The artists relinquish the ordering and qualities of sound and video to the weather conditions of Aoraki, recorded by NIWA instruments in place near the Haupapa glacier. Stefan created a connection hub gathering the NIWA data stream, which is then used by the reactive installation to subtly alter the brightness, direction, and movement of the images and sounds according to the real-time weather conditions, and wind direction. Depending on the weather, the image changes and the sound and vocal sequence is endlessly variable. On days of high solar radiation, bright, clear ice and sun predominate and also move the images on screen accordingly, on cloudy days, the image darkens.
Pedagogical design in education using augmented reality: A systematic review

S Gil Parga, U Singh, J Gutierrez, and S Marks

Interactive Learning Environments (NILE), Jun 2023

DOI

2022

Humanness is in the eye of the beholder: Role of predictability and theory of mind on anthropomorphism in human-computer interactions

J Ayache, AM Connor, S Marks, A Sumich, and N Heym

Dec 2022
Haupapa: Approaching A Glacier. He mahi toi, he pūtaiao me ngā atua, Art, science and the atua

J Randerson , J Randerson, R Shearer, M Sheehan, S Marks, H Purdie, R Bull, and HTT Himona

Nov 2022

Abs

https://tetuhi.art/event/haupapa-approaching-a-glacier/ He mahi toi, he pūtaiao me ngā atua, Art, science and the atua You are warmly invited to a kōrerorero, round table discussion with the collaborators of ‘Haupapa: The Chilled Breath of Rakamaomao’; an artwork that spans the fields of art, creative technologies, science, live data and Matauraka Māori. In mediating between these realms, in response to the melting Haupapa glacier in Aoraki National park, we sensorially approach a pressure zone where we face water scarcity in some parts of our isles, and vanishing glaciers, flooding, severe storms and coastal erosion that meets rising seas on the other. In the gathering of Ron Bull’s recitation of words and names of the atua of that locale, woven through sound and images collected through audio hydrophones and underwater camera, and triggered by a live data stream from NIWA (National Institute of Air and Water), we open up to more-than-human scales of aural and visual perception. We invite consideration of what ecological imaginaries might evolve through such voicings and data patterns of natural elements at a time of global crisis. In addition, we are interested in negotiating with care the realm of the sacred and the atua, and what is publicly shareable when bringing together parallel realms of knowledge.
Haupapa: The Chilled Breath of Rakamaomao

J Randerson, S Marks, R Shearer, R Bull, and H Purdie

Sep 2022

Abs

See Also: https://worldweathernetwork.org/station/te-tuhi/ Haupapa: The Chilled Breath of Rakamaomao is an online artwork that assembles mountaineer-glaciologist Heather Purdie, sound artist/designer Rachel Shearer (Rongowhakāta Te Aitanga a Mahaki), moving image artist Janine Randerson, orator Ron Bull, of Kai Tahu, Kati Mamoe and Waitaha whakapapa, and programmer Stefan Marks. We create a creative "weather report" from Haupapa/Tasman glacier, Aotearoa’s largest body of wai, water, a glacier formed from a deep exhalation of Aoraki, the ancestor-maunga, as he readied to speak. We respond to the "hau" of Haupapa, translated fluidly as moisture, air, breath, wind, tears and vitality. Within Kai Tahu whakapapa, Rakamaomao is related to Aoraki and is one of the progenitors of wind and weather. Through audio-visual modes of gifting and listening, we approach Haupapa as ancestor, a shape-shifting collaborator. Tiny bubbles of ancient breath and atmosphere are pressed inside Haupapa’s glacial ice – including sea breezes, pollens, carbon dioxide and methane, as well as the ash of Australian fires. We collectively attune to the glacier through Kai Tahu cosmologies, instruments of science, audio hydrophones and underwater camera receivers to more-than-human scales of aural and visual perception.
Evaluation of a multi-agent ‘human-in-the-loop’ game design system

J Kruse, AM Connor, and S Marks

ACM Transactions on Interactive Intelligent Systems, Sep 2022

Abs DOI

Designing games is a complicated and time-consuming process, where developing new levels for existing games can take weeks. Procedural content generation offers the potential to shorten this timeframe, however automated design tools are not adopted widely in the game industry. This paper presents an expert evaluation of a human-in-the-loop generative design approach for commercial game maps that incorporates multiple computational agents. The evaluation aims to gauge the extent to which such an approach could support and be accepted by human game designers, and to determine whether the computational agents improve the overall design. To evaluate the approach, eleven game designers utilized the approach to design game levels with the computational agents both active and inactive. Eye tracking, observational and think aloud data was collected to determine whether designers favored levels suggested by the computational agents. This data was triangulated with qualitative data from semi-structured interviews that were used to gather overall opinions of the approach. The eye tracking data indicates that the participating game level designers showed a clear preference for levels suggested by the computational agents, however expert designers in particular appeared to reject the idea that the computational agents are helpful. The perception of computational tools not being useful needs to be addressed if procedural content generation approaches are to fulfill their potential for the game industry.

2021

Exploring the “dark matter” of social interaction: Systematic review of a decade of research in spontaneous interpersonal coordination

J Ayache, A Connor, S Marks, D Kuss, D Rhodes, A Sumich, and N Heym

Frontiers in Psychology, Oct 2021

DOI
Feeling closer despite the distance: How to cultivate togetherness within digital spaces

J Ayache, N Heym, A Sumich, D Rhodes, AM Connor, and S Marks

In Handbook of research on remote work and worker well-being in the post-COVID-19 era, Oct 2021

Abs DOI

In the framework of “togetherness” as a psychophysiological experience of social presence, the current chapter highlights the importance of work environments to socializing. The absence of such physical collective spaces impacts group-dynamics and team performance in online meetings, which also tend to prioritize task-solving discussions and limit non-verbal exchanges. Interpersonal coordination (or “social glue”), characterized by a spontaneous mutual attunement, both in speech and gestures, is classically observed during collective events where social-bonding and affiliation are promoted. This chapter will review the cognitive, behavioral, and physiological consequences of togetherness and integrate those in the context of recent technological advancements in computer-mediated interaction which have culminated in the advent of virtual and augmented reality. Given the potential of such methods to increase embodied interactions, they have been coined as “empathy machines” and could be seen as a technological solution to restore the experience of togetherness in the workplace.
An interactive multi-agent system for game design

J Kruse, AM Connor, and S Marks

The Computer Games Journal, Oct 2021

DOI

2020

Development of a virtual construction approach for steel structures considering structural and non- structural elements, and installation equipment

T Phan, S Ramhormozian, C Clifton, G MacRae, R Dhakal, L-J Jia, and S Marks

In The 54th International Conference of the Architectural Science Association, Oct 2020
Discrete versus continuous colour pickers impact colour selection in virtual reality art-making

M Alex, D Lottridge, J Lee, S Marks, and B Wünsche

In Proceedings of the 32nd Australian Conference on Human-Computer-Interaction (OzCHI 2020), Oct 2020

Abs DOI

© 2020 ACM. Colour selection is an important task in digital art and 3D modelling applications. Most colour pickers are based on continuous colour spaces or a representative sampling of them, such as the Munsell colour palette. Continuous colour space-based pickers enable users to select from all colours by displaying full saturation hues with options to lower saturation and modify value. The two-step process of colour selection from continuous pickers requires understanding of 3D colour space, e.g., where to find "brown"or "sand". In this research we investigate how continuous versus discrete pickers affect colour selection in virtual art. We compared an HSV picker with a discrete picker in a study with 40 participants aged 16-60. We found that the colour picker impacted the kinds of colours used in artworks, with significant differences in colour distribution characteristics. We discuss implications of colour selection tools for virtual reality art-making.
An evaluation of the effectiveness of virtual reality in air traffic control

Y Lee, S Marks, and AM Connor

In Proceedings of the 4th International Conference on Virtual and Augmented Reality Simulations, Oct 2020

DOI
Multi-device collaboration in virtual environments

S Marks, and D White

In ICVARS 2020: Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations, Oct 2020

Abs DOI

We present a multi-device collaboration principle for virtual environments, using a combination of virtual and augmented reality (VR/AR) technology, used in the context of two educational applications, a virtual nasal cavity, and a visualisation of earthquake data. A head-mounted display (HMD) and a 3D-tracked tablet create two views of a shared virtual space. This allows two users to collaborate, utilising the strengths of each of the two technologies, e.g., intuitive spatial navigation and interaction in VR, and touch control of the visualisation parameters via the AR tablet. Touch gestures on the tablet are translated into a pointer ray in VR, so the users can easily indicate spatial features. The underlying networking infrastructure allows for an extension of this application to more than two users and across different rendering platforms.

2018

Extending a Virtual Reality Nasal Cavity Education Tool with Volume Rendering

M Magdics, D White, and S Marks

In Proceedings of 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Dec 2018

DOI
Evaluation of a Virtual Reality Nasal Cavity Education Tool

S Marks, D White, and M Magdics

In Proceedings of 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Dec 2018

DOI
Active learning and teaching through digital technology and live performance; ‘choreographic thinking’ as art practice in the tertiary sector

J Nikolai, G Bennett, S Marks, and G Maynard

International Journal of Art and Design Education, Sep 2018

Abs DOI

This article is an interdisciplinary reflective response to an intensive studio learning and teaching experience involving artists, academics and postgraduate students. The authors of this article teach, research and practise in coding, digital design, dance, and virtual and live performance. As lecturers and students we reflect upon and propose future approaches to art practice in tertiary education informed by live performance, performance capture and studio-based responses to digital and virtual platforms. We reflect on an innovative contribution to the field of research–teaching nexus as informed by digital and virtual data capture identifying the key element of immediacy in live performance and choreographic improvisation with systems. We reflect on practice-based inquiry via the Choreographic Coding Lab (CCL) model – a dialogical negotiation between capture technology and interdisciplinary artists in industry and academia. How can we encourage potential studio inquiry as an adapted model in tertiary learning and teaching? Our interdisciplinary voices, presented as authors’ reflections, provide suggestions for future studio-based, active learning contexts.
Virtual Reality

S Marks

In The SAGE Encyclopedia of the Internet, Sep 2018

Abs DOI

Popular media and recent consumer products give the impression that virtual reality (VR) is mostly about being tethered to a computer with a head-mounted display (HMD) covering the face, headphones over the ears, and some sort of controller in either hand for interaction. While this is indeed one valid incarnation of a VR system, it is not the only one. VR can be defined fairly broadly as synthetically generated sensory impressions (e.g., video, images, sound, touch) that are intended to immerse the users in an artificial, simulated, and interactive environment. This entry defines VR and discusses its history, hardware, and applications.
From von Neumann architecture and Atanasoffs ABC to neuro-morphic computation and Kasabov’s neuCube: Principles and implementations

N Sengupta, JIE Ramos, E Tu, S Marks, N Scott, J Weclawski, AR Gollahalli, MG Doborjeh, ZG Doborjeh, K Kumarasinghe, V Breen, and A Abbott

In Studies in Computational Intelligence - Learning Systems: From Theory to Practice, Sep 2018

Abs DOI

© Springer International Publishing AG, part of Springer Nature 2018. During the 1940s John Atanasoff with the help of one of his students Clifford E. Berry, at Iowa State College, created the ABC (Atanasoff-Berry Computer) that was the first electronic digital computer. The ABC computer was not a general-purpose one, but still, it was the first to implement three of the most important ideas used in computers nowadays: binary data representation; using electronics instead of mechanical switches and wheels; using a von Neumann architecture, where the memory and the computations are separated. A new computational paradigm, named as Neuromorphic, utilises the above two principles, but instead of the von Neumann principle, it integrates the memory and the computation in a single module a spiking neural network structure. This chapter first reviews the principles of the earlier published work by the team on neuromorphic computational architecture NeuCube. NeuCube is not a general purpose machine but is still the first neuromorphic spatio/spectro-temporal data machine for learning, pattern recognition and understanding of spatio/spectro-temporal data. The chapter further presents the software/hardware implementation of the NeuCube as a development system for efficient applications on temporal or spatio/spectro-temporal across domain areas, including: brain data (EEG, fMRI), brain computer interfaces, robot control, multi-sensory data modelling, seismic stream data modelling and earthquake prediction, financial time series forecasting, climate data modelling and personalised, on-line risk of stroke prediction, and others. A limited version of the NeuCube software implementation is available from http://www.kedri.aut.ac.nz/neucube/.

2017

Drawing on Hope: A Virtual Reality Project

K Marks, and S Marks

In , Dec 2017

Abs

This presentation introduces a Virtual Reality participatory art project, with a focus on social action (Levine & Levine, 2011). ‘Drawing on Hope’ is a collaboration between Kathrin (Arts Therapy) and Stefan (Computer Science). The project is connecting countries through the universal language of art. As people interact with the virtual world by creating their own drawings of symbols and metaphors for hope, a collective artwork evolves, a forest of art that people can wander through, drawing on and with hope. The presentation further focuses on the role of hope within the therapeutic setting (Larsen, Edey, & Lemay, 2007). A specific focus is on the dialectic between hope and despair (O’Hara, 2011) where the therapist often holds hope for the respective client until they feel able to invite it back into their own lives.
Step into my (virtual) world: An (auto)ethnographic exploration of virtual reality drawing applications for arts therapy

K Marks, S Marks, and A Brown

Australian and New Zealand Journal of Arts Therapy (ANZJAT), Dec 2017

Abs

This article explores the feasibility and potential of virtual reality (VR) in the context of arts therapy. Although technology advances at an ever-increasing rate, arts therapists have been slow and hesitant in taking up computers and software. Here the authors provide a brief overview of research to date into reasons for this apparent lack of adoption, and list the requirements of technology used in the context of therapy, followed by the introduction of VR applications for arts therapy. Employing art-based and practice-led research, they document their findings, which emerged in three phases: free exploration of the VR drawing application (open studio approach, transitional objects); use of the narrative therapy framework; and introduction to ANZATA symposium attendees in Christchurch in 2016. Based on these findings, the article highlights the benefits and limitations of using VR drawing applications in arts therapy.
Getting up your nose: A virtual reality education tool for nasal cavity anatomy

S Marks, D White, and M Singh

In SIGGRAPH Asia 2017 Symposium on Education Proceedings, Nov 2017

Abs DOI

This article explores the application of virtual reality (VR) to the area of anatomical education, specifically the shape of and the airflow through the human nasal cavity. We argue the benefits of VR technology in this specific domain, and describe the creation of the VR application which is intended to be used in future courses. Through two preliminary case studies, we describe our experiences, and discuss advantages and disadvantages of the use of VR in this area.
Immersive visualisation of 3-dimensional spiking neural networks

S Marks

Evolving Systems, Sep 2017

Abs DOI

Recent development in artificial neural networks has led to an increase in performance, but also in complexity and size. This poses a significant challenge for the exploration and analysis of the spatial structure and temporal behaviour of such networks. Several projects for the 3D visualisation of neural networks exist, but they focus largely on the exploration of the spatial structure alone, and are using standard 2D screens as output and mouse and keyboard as input devices. In this article, we present NeuVis, a framework for an intuitive and immersive 3D visualisation of spiking neural networks in virtual reality, allowing for a larger variety of input and output devices. We apply NeuVis to NeuCube, a 3-dimensional spiking neural network learning framework, significantly improving the user’s abilities to explore, analyse, and also debug the network. Finally, we discuss further venues of development and alternative render methods that are currently under development and will increase the visual accuracy and realism of the visualisation, as well as further extending its analysis and exploration capabilities.

2016

Step into my (virtual) world

S Marks, and K Marks

In , Dec 2016

Abs

Background: With an increasing number of clients growing up in the “information age”, interest in adopting new computer-based technologies is rising (Kapitan, 2007). Virtual reality (VR) based systems are one option, since recent technological advancements have made them more available to the consumer market. Methods: This study is primarily arts-based, focusing on the process and intuitive development of methods suited for VR-based tools. Participants were asked to "draw a problem", using narrative therapy in an art therapy framework. (Auto)ethnographic data was gathered through before and after reflections as well as during therapeutic conversations facilitated by the therapist. Results: To date, data from four participants has shown that the use of a VR painting tool in the context of art therapy opens new and exciting ways for clients to playfully explore their ‘problem’ in a non-verbal and experimental manner. Due to the intuitiveness of the tool and the mobility and unobtrusiveness of the custom VR hardware, participants were able to freely experiment with large 3D sculptures created within very short timeframes. A “teleport” function expands the available space and invites participants to vary distances to their ’problem’, achieving safety and perspective. Furthermore, the therapist can literally step into a client’s world (McLeod, 1999), deepening the therapeutic conversations. Conclusion: VR-based 3D painting applications are valuable additional tools that are attractive to an increasingly digital clientele. White latest technological developments have reduced the complexity and cost of such systems, they are, however, still too impractical or too expensive for widespread use
Step into my (virtual) world

S Marks, and K Marks

In Festival of Artful Transitions, Nov 2016

Abs

Participants are invited to step into and explore 3D artworks created in therapeutic processes with some of the latest commercially available Virtual Reality (VR) technology. The workshop facilitators encourage discussion about the details of the process, possibilities, and challenges. Available technology permitting, there will also be an opportunity for participants to create, explore, and share their own virtual 3D artwork.
Step into my (virtual) world

S Marks, and K Marks

In Artful Transitions - ANZATA 2016 Symposium Programme, Nov 2016

Abs

Collaborative research and exploration of art therapy within a virtual reality environment, using a tool similar to as well as the Google Tilt Brush. Three particular areas have been researched: 1. Use of an Open Studio approach 2. Possibility for creating/printing of transitional objects 3. Art and Narrative Therapy, using the prompt: “Draw your ‘problem’”, followed by therapeutic conversation and the therapist then literally stepping into the client’s world. These areas will be introduced as well as illustrated by (auto)ethnographic accounts through showing video footage, presenting 3D printed models, and inviting the audience to step into the virtual world.
Problem solving at the edge of disciplines

AM Connor, R Sosa, S Marks, and AG Jackson

In Handbook of Research on Creative Problem-Solving Skill Development in Higher Education, Aug 2016

DOI
A wearable haptic game controller

J Foottit, D Brown, S Marks, and AM Connor

International Journal of Game Theory and Technology, Mar 2016

DOI
Creative Technologies for Multidisciplinary Applications

AM Connor, and S Marks

Mar 2016

DOI
Design of a virtual trainer for exergaming

LA Shaw, R Tourrel, B Wünsche, C Lutteroth, S Marks, and J Buckley

In ACM International Conference Proceeding Series, Feb 2016

Abs DOI

Exergames are becoming increasingly popular as a way of motivating people to exercise. However, merely adding exercise elements to a game may not achieve the desired level of motivation and long term adherence. By designing an exergame which takes into account the user’s personality profile, the user’s level of motivation to play the game and thus exercise may be increased. In this paper, we present an exergame using a virtual trainer system which can be customized for the personality of the user. The trainer system supports two modes: a competitive mode for players who are motivated by pushing themselves to beat an opponent, and a cooperative mode for players who enjoy working with another player to perform well. We conduct a brief pilot study to evaluate our virtual trainers in which participants’ personalities are evaluated using the Sport Orientation Questionnaire. They then play three short sessions of the exergame: a control condition without a trainer system, and one for each of the two trainer system. Our initial results indicate that the training systems are highly motivating when matching the personality of the user, particularly for competitive individuals.
Exposing core competencies for future creative technologists

AM Connor, R Sosa, S Karmokar, S Marks, M Buxton, AM Gribble, AG Jackson, and J Foottit

In Creative technologies for multidisciplinary applications, Feb 2016

DOI
Evolving spatio-temporal data machines based on the NeuCube neuromorphic framework: Design methodology and selected applications

N Kasabov, NM Scott, E Tu, S Marks, N Sengupta, E Capecci, M Othman, MG Doborjeh, N Murli, R Hartono, JI Espinosa-Ramos, L Zhou, FB Alvi, G Wang, D Taylor, V Feigin, S Gulyaev, M Mahmoud, Z-G Hou, and J Yang

Neural Networks, Feb 2016

Abs DOI

The paper describes a new type of evolving connectionist systems (ECOS) called evolving spatio-temporal data machines based on neuromorphic, brain-like information processing principles (eSTDM). These are multi-modular computer systems designed to deal with large and fast spatio/spectro temporal data using spiking neural networks (SNN) as major processing modules. ECOS and eSTDM in particular can learn incrementally from data streams, can include ‘on the fly’ new input variables, new output class labels or regression outputs, can continuously adapt their structure and functionality, can be visualised and interpreted for new knowledge discovery and for a better understanding of the data and the processes that generated it. eSTDM can be used for early event prediction due to the ability of the SNN to spike early, before whole input vectors (they were trained on) are presented. A framework for building eSTDM called NeuCube along with a design methodology for building eSTDM using this is presented. The implementation of this framework in MATLAB, Java, and PyNN (Python) is presented. The latter facilitates the use of neuromorphic hardware platforms to run the eSTDM. Selected examples are given of eSTDM for pattern recognition and early event prediction on EEG data, fMRI data, multisensory seismic data, ecological data, climate data, audio-visual data. Future directions are discussed, including extension of the NeuCube framework for building neurogenetic eSTDM and also new applications of eSTDM.
Development of a wearable haptic game interface

J Foottit, D Brown, S Marks, and AM Connor

EAI Endorsed Transactions on Creative Technologies, Feb 2016

DOI

2015

Creating creative technologists: Playing with(in) education

AM Connor, S Marks, and C Walker

In Creativity in the Digital Age, Apr 2015

DOI
Immersive visualisation of 3-dimensional neural network structures

S Marks, J Estevez, and N Scott

In 13th International Conference on Neuro-Computing and Evolving Intelligence (NCEI) 2015, Feb 2015
Proceedings of the sixteenth Australasian User Interface Conference (AUIC 2015)

S Marks, and Rachel Blagojevic

Jan 2015
Development and Evaluation of an Exercycle Game Using Immersive Technologies

LA Shaw, B Wünsche, C Lutteroth, S Marks, J Buckley, and P Corballis

In Proceedings of the 8th Australasian Workshop on Health Informatics and Knowledge Management (HIKM 2015), Jan 2015
Challenges in virtual reality exergame design

LA Shaw, B Wünsche, C Lutteroth, S Marks, and R Callies

In Conferences in Research and Practice in Information Technology (CRPIT), Jan 2015

Abs

Exercise video games have become increasingly popular due to their potential as tools to increase user motivation to exercise. In recent years we have seen an emergence of consumer level interface devices suitable for use in gaming. While past research has indicated that immersion is a factor in exergame effectiveness, there has been little research investigating the use of immersive interface technologies such as head mounted displays for use in exergames. In this paper we identify and discuss five major design challenges associated with the use of immersive technologies in exergaming: motion sickness caused by sensory disconnect when using a head mounted display, reliable bodily motion tracking controls, the health and safety concerns of exercising when using immersive technologies, the selection of an appropriate player perspective, and physical feedback latency. We demonstrate a prototype exergame utilising several affordable immersive gaming devices as a case study in overcoming these challenges. The results of a user study we conducted found that our prototype game was largely successful in overcoming these challenges, although further work would lead to improvement and we were able to identify further issues associated with the use of a head mounted display during exercise.

2014

An unexpected journey: Experiences of learning through exploration and experimentation

AM Connor, C Berthelsen, S Karmokar, S Marks, B Kenobi, and C Walker

In Action!-Doing Design Education, Dec 2014

DOI
An Intuitive Tangible Game Controller

J Foottit, D Brown, S Marks, and AM Connor

In The 10th Australasian Conference on Interactive Entertainment (IE 2014), Dec 2014

Abs DOI

This paper outlines the development of a sensory feedback device providing a low cost, versatile and intuitive interface for controlling digital environments, in this example a flight simulator. Gesture based input allows for a more immersive experience, so rather than making the user feel like they are controlling an aircraft the intuitive interface allows the user to become the aircraft that is controlled by the movements of the user’s hand. The movements are designed to feel intuitive and allow for a sense of immersion that would be difficult to achieve with an alternative interface. In this example the user’s hand can become the aircraft much the same way that a child would imagine it.
Towards the Holodeck: Fully immersive virtual reality visualisation of scientific and engineering data

S Marks, JE Estevez, and AM Connor

In 29th International Conference on Image and Vision Computing New Zealand (IVCNZ) 2014, Nov 2014

DOI
Proceedings of the Fifteenth Australasian User Interface Conference (AUIC 2014)

B Wünsche, and S Marks

Jan 2014

2013

Experimental study of steer-by-wire ratios and response curves in a simulated high speed vehicle

S Marks, and R Wellington

In Proceedings of the Fourteenth Australasian User Interface Conference (AUIC2013), Jan 2013

Abs

In this poster, we outline a research study of the steering system for a potential land speed record vehicle. We built a cockpit enclosure to simulate the interior space and employed a game engine to create a suitable virtual simulation and appropriate physical behaviour of the vehicle to give a realistic experience that has a suitable level of difficulty to represent the challenge of such a task. With this setup, we conducted experiments on different linear and nonlinear steering response curves to find the most suitable steering configuration. The results suggest that linear steering curves with a high steering ratio are better suited than non-linear curves, regardless of their gradient.
An ethnographic study of a high cognitive load driving environment

R Wellington, and S Marks

In Proceedings of the 14th Australasian User Interface Conference (AUIC 2013), Jan 2013

Abs

This poster outlines Ethnographic research into the design of an environment to study a land speed record vehicle, or more generally, a vehicle posing a high cognitive load for the user. The challenges of empirical research activity in the design of unique artifacts is discussed, where we may not have the artefact available in the real context to study, nor key informants that have direct relevant experience. We also describe findings from the preliminary design studies and the study into the design of the yoke for driving steer-by-wire.

2012

Head Tracking Based Avatar Control for Virtual Environment Teamwork Training

S Marks, JA Windsor, and B Wünsche

Journal of Virtual Reality and Broadcasting, Dec 2012

Abs

Virtual environments (VE) are gaining in popularity and are increasingly used for teamwork training purposes,e.g., for medical teams. One shortcoming of modern VEs is that nonverbal communication channels, essential for teamwork, are not supported well. We address this issue by using an inexpensive webcam to track the user’s head. This tracking information is used to control the head movement of the user’s avatar, thereby conveying head gestures and adding a nonverbal communication channel. We conducted a user study investigating the influence of head tracking based avatar control on the perceived realism of the VE and on the performance of a surgical teamwork training scenario. Our results show that head tracking positively influences the perceived realism of the VE and the communication, but has no major influence on the training outcome.
Using Game Engine Technology for Virtual Environment Teamwork Training

S Marks, JA Windsor, and B Wünsche

In WSCG ’2012 Conference Proceedings - Part 1, Jul 2012

Abs

The use of virtual environments (VE) for teaching and training is increasing rapidly. A particular popular medium for implementing such applications are game engines. However, just changing game content is usually insufficient for creating effective training and teaching scenarios. In this paper, we discuss how the design of a VE can be changed to adapt it to new use cases. We explain how new interaction principles can be added to a game engine by presenting technologies for integrating a webcam for head tracking. This enables head-coupled perspective as an intuitive view control and head gestures that are mapped onto the user’s avatar in the virtual environment. We also explain how the simulation can be connected to behavioural study software in order to simplify user study evaluation. Finally we list problems and solutions when utilising the free Source Engine Software Development Kit to design such a virtual environment. We evaluate our design, present a virtual surgery teamwork training scenario created with it, and summarize user study results demonstrating the usefulness of our extensions.
Design and evaluation of a medical teamwork training simulator using consumer-level equipment

S Marks, JA Windsor, and B Wünsche

In Medicine Meets Virtual Reality 19, Jul 2012

Abs DOI

Virtual environments (VE) are increasingly used for teamwork training purposes, e.g., for medical teams. One shortcoming is lack of support for nonverbal communication channels, essential for teamwork. We address this issue by using an inexpensive webcam to track the user’s head and using that data for controlling avatar head movement, thereby conveying head gestures and adding a nonverbal communication channel. In addition, navigation and orientation within the virtual environment is simplified. We present the design and evaluation of a simulation framework based on a game engine and consumer-level hardware and the results of two user studies showing, among other results, an improvement in the usability of the VE and in the perceived quality of realism and communication within the VE by using head tracking avatar and view control.

2011

A Virtual Environment for Medical Teamwork Training with Support for Non-Verbal Communication using Consumer-Level Hardware and Software

S Marks

Sep 2011

Abs

Simulation is an increasingly important component of the medical curriculum due to procedures becoming more complex and the use of real patients being constrained by ethical restrictions, costs, availability of patients, and logistics. A large number of simulators are available, mostly for training of technical skills, such as suturing or cutting, but increasingly also for non-technical skills, such as teamwork and decision making. The majority of those simulators are very expensive and unaffordable for smaller hospitals, healthcare providers, and education institutes, especially in third world countries. Furthermore the hardware and content are usually proprietary, making it difficult and expensive to add new functionalities. In this thesis, a virtual environment (VE) for medical teamwork training, based on consumer-level hardware and software, is presented and evaluated. Advanced graphics, physics and multi-user capabilities are provided by a game engine, and non-verbal communication, an important aspect of teamwork, is facilitated by using a standard webcam for tracking the movement of the user’s head and mirroring that movement onto the avatar. Different game engines are evaluated with regard to their suitability for medical simulations and guidelines and ideal parameters for stable head tracking using a web-cam are presented. Based on these results, a simulation framework is developed and evaluated for its suitability for medical teamwork training. A user study is presented, demonstrating that participants perceive non-verbal communication cues well and can intuitively control the view perspective with head movement. A multi-user study with a medical simulation scenario shows that the inclusion of view and avatar control by head tracking improves the perceived realism and naturalness of the simulation and the communication. The results demonstrate that medical simulators can be constructed using cheap consumer-level hardware and software. Game engines provide a suitable software foundation and allow users to design their own content. Peripheral devices, such as web-cams in combination with computer vision techniques, can be used to improve realism and immersion. Constraints centre predominantly on the simulation of complex technical skills and complex multi-user interactions.
Virtual environment for and physical simulation of a supersonic land speed record vehicle

S Marks

Jun 2011

Abs

JetBlack is a project striving to achieve a new land speed record. In order to develop the cockpit controls, I have developed a physical simulation of the vehicle to be used in conjunction with the chassis displayed in the HCI lab. This simulation provides a reasonably realistic and challenging simulation of driving the vehicle so that the vehicle controls have to be designed optimally to distract the driver as little as possible while at the same time providing as much information as necessary.
Head tracking based avatar control for virtual environment teamwork training

S Marks, JA Windsor, and B Wünsche

In GRAPP 2011 - Proceedings of the International Conference on Computer Graphics Theory and Applications, May 2011

Abs

Virtual environments (VE) are gaining in popularity and are increasingly used for teamwork training purposes, e.g., for medical teams. One shortcoming of modern VEs is that nonverbal communication channels, essential for teamwork, are not supported well. We address this issue by using an inexpensive webcam to track the user’s head. This tracking information is used to control the head movement of the user’s avatar, thereby conveying head gestures and adding a nonverbal communication channel. We conducted a user study investigating the influence of head tracking based avatar control on the perceived realism of the VE and on the performance of a surgical teamwork training scenario. Our results show that head tracking positively influences the perceived realism of the VE and the communication, but has no major influence on the training outcome.

2010

Evaluation of the Effectiveness of Head Tracking for View and Avatar Control in Virtual Environments

S Marks, JA Windsor, and B Wünsche

In 2010 25th International Conference Image and Vision Computing New Zealand, IVCNZ 2010 - Conference Proceedings, Nov 2010

Abs DOI

Virtual environments (VE) are gaining in popularity and are increasingly used for teamwork training purposes, e.g., for medical teams. We have identified two shortcomings of modern VEs: First, nonverbal communication channels are essential for teamwork but are not supported well. Second, view control in VEs is usually done manually, requiring the user to learn the controls before being able to effectively use them. We address those two shortcomings by using an inexpensive webcam to track the user’s head. The rotational movement is used to control the head movement of the user’s avatar, thereby conveying head gestures and adding a nonverbal communication channel. The translational movement is used to control the view of the VE in an intuitive way. Our paper presents the results of a user study designed to investigate how well users were able to use our system’s advantages.

2009

Optimisation and comparison framework for monocular camera-based face tracking

S Marks, JA Windsor, and B Wünsche

In 2009 24th International Conference Image and Vision Computing New Zealand, IVCNZ 2009 - Conference Proceedings, Nov 2009

Abs DOI

Abstract—Tracking the position and orientation of the human face with respect to a camera has valuable applications in human computer interaction (HCI). Examples are navigating through a virtual environment, controlling objects using head gestures, and enabling avatars in a virtual environment to reflect the user’s behaviour. Tracking performance can be heavily influenced by environmental parameters. Developers and users of face tracking plugins without computer vision experience need guidelines how to optimise face tracking performance in real world set-ups and they need measures how environmental parameters influence the results. In this paper we develop a qualitative framework for determining ideal working conditions of face tracking algorithms. We apply our framework to a commercially available face tracking solution and present the results of this analysis.
The Impact of Non-Verbal Communication in Virtual-Environment-Based Teamwork Training

S Marks, JA Windsor, and B Wünsche

In SimTecT 2009 Health Simulation Conference, Sep 2009

Abs

Aims: To present a simple method to use a webcam to enhance non-verbal communication in virtual-environment-based teamwork training. Background: Over the last decade, healthcare profession training has been enriched by educational technology such as virtual-reality simulators for skills training and mannequins for team training. More recently, there has been the development of several metaverses that are now being used for virtual-environment-based teamwork simulations. This type of simulation lends itself to networking, allowing participants to participate from remote locations. The significant drawback of training in virtual worlds is that the avatars, in-world representatives of the participants, are capable of very limited non-verbal communication. Of note there is no facial expression, gaze control or head movement. Our thesis is that enhanced non-verbal communication will improve training outcomes. Methods: We have developed a model of the aspects of non-verbal communication that a simple webcam (Creative Live! Cam Video IM Pro) can capture. This model has been integrated into a program that monitors and evaluates the webcam input. The data is fed into a simulation based on Valve’s Source Engine that has been modified to mirror the information on the avatar. Results: Initial tests suggest a more realistic and effective communication between users in virtual world simulation. Feedback from medical and other professionals suggests that this approach has significant advantages and potential to enhance team training. Conclusions: Using additional input from a webcam to control the avatar, we have achieved enhanced non-verbal communication. Initial feedback is very positive. The next step is to conduct extended user studies.
Enhancing Virtual-Environment-Based Teamwork Training with Non-Verbal Communication

S Marks, JA Windsor, and B Wünsche

In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2009, Jun 2009

Abs

Virtual reality simulations for individual training of surgical skills are increasingly used in medical education and have been shown to improve patient outcome. Since recent research suggests that a large percentage of mistakes in clinical settings are due to problems with non-technical skills like communication, teamwork training simulators are developed and used to address this problem. Virtual-environment-based teamwork training simulators are very cost-efficient and allow for non-co-located settings, but have their limitations in communication among the participants. We present an inexpensive camera-based system for capturing aspects of non-verbal communication of participating users and projecting these onto the avatars in the simulation. This additional information has the potential of increasing the realism of the simulation and the effectiveness of team communication, resulting in a better training outcome – for all kinds of simulation that involves human communication.
Enhancing Virtual Environment-Based Surgical Teamwork Training with Non-Verbal Communication

S Marks, JA Windsor, and B Wünsche

In New Zealand Computer Science Research Student Conference (NZCSRSC) 2009, Apr 2009
Enhancing Virtual Environment-Based Surgical Teamwork Training With Non-Verbal Communication

S Marks, JA Windsor, and B Wünsche

In GRAPP 2009 - Proceedings of the 4th International Conference on Computer Graphics Theory and Applications, Feb 2009

Abs

Virtual reality simulations for training surgical skills are increasingly used in medical education and have been shown to improve patient outcome. While advances in hardware and simulation techniques have resulted in many commercial applications for training technical skills, most of these simulators are extremely expensive and do not consider non-technical skills like teamwork and communication. This is a major drawback since recent research suggests that a large percentage of mistakes in clinical settings are due to communication problems. In addition, training teamwork can also improve the efficiency of a surgical team and as such reduce costs and workload. We present an inexpensive camera-based system for capturing aspects of non-verbal communication of users participating in virtual environment-based teamwork simulations. This data can be used for the enhancement of virtual-environment-based simulations to increase the realism and effectiveness of team communication.

2008

Camera based face tracking for enhancing surgical teamwork training with non-verbal communication

S Marks, JA Windsor, and B Wünsche

In 23rd International Conference Image and Vision Computing New Zealand, IVCNZ, Nov 2008

Abs DOI

In recent years, the increased use of simulation for the training of surgical skills has improved the medical curriculum and the overall patient outcome. Advances in hardware and simulation techniques have resulted in many commercial applications for training technical skills. However, most of these simulators are extremely expensive and do not consider non-technical skills like teamwork and communication. This is a major drawback since recent research suggests that a a large percentage of mistakes in clinical settings are due to communication problems. In addition training teamwork can also improve the efficiency of a surgical team and as such reduce costs and workload. We present an inexpensive camera-based system for the acquisition of aspects of non-verbal communication of users participating in virtual environment-based teamwork simulations. This data can be used for the enhancement of virtual-environment-based simulations to increase the realism and effectiveness of team communication.
Evaluation of Game Engines for Simulated Clinical Training

S Marks, JA Windsor, and B Wünsche

In , Oct 2008

Abs

The increasing complexity and costs of clinical training and the constant development of new procedures has made virtual reality based training an essential tool in medical education. Unfortunately, commercial training tools are very expensive and have a small support base. Game engines offer unique advantages for the creation of highly interactive and collaborative environments. This paper examines the suitability of currently available game engines for developing applications for clinical education and training. We formally evaluate a list of available game engines for stability, availability, the possibility of custom content creation and the interaction of multiple users via a network. Based on these criteria, three of the highest ranked engines are used for further case studies. We found that in general it is possible to easily create scenarios with custom medical models that can be cooperatively viewed and interacted with, though limitations in physical simulation capabilities make some engines less suitable for fully interactive applications. We show that overall game engines represent a good foundation for low cost clinical training applications and we discuss technologies which can be used to further extend their physical simulation capabilities.

2007

Collaborative Soft Object Manipulation for Game Engine-Based Virtual Reality Surgery Simulators

S Marks, JA Windsor, and B Wünsche

In 22nd International Conference Image and Vision Computing New Zealand, IVCNZ 2007, Dec 2007

Abs

In this paper we analyse and evaluate the capabilities of popular game engines to simulate and interact with soft objects. We discuss how these engines can be used for simulated surgical training applications, determine their shortcomings and make suggestions how game engines can be extended to make them more suitable for such applications.
Evaluation of game engines for simulated surgical training

S Marks, JA Windsor, and B Wünsche

In Proceedings of the 5th International Conference on Computer Graphics and Interactive Techniques in Australia and Southeast Asia, Dec 2007

Abs DOI

The increasing complexity and costs of surgical training and the constant development of new surgical procedures has made virtual surgical training an essential tool in medical education. Unfortunately, commercial tools are very expensive and have a small support base. Game engines offer unique advantages for the creation of highly interactive and collaborative environments. This paper examines the suitability of currently available game engines for developing applications for medical education and simulated surgical training. We formally evaluate a list of available game engines for stability, availability, the possibility of custom content creation and the interaction of multiple users via a network. Based on these criteria, three of the highest ranked engines are used for further case studies. We found that in general it is possible to easily create scenarios with custom medical models that can be cooperatively viewed and interacted with. Limitations in physical simulation capabilities make some engines unsuitable for fully interactive applications, but they can be used in combination with predefined animations. We show that overall game engines represent a good foundation for low cost virtual surgery applications and we discuss technologies which can be used to further extend their physical simulation capabilities.
Don’t Shoot Them - Heal Them

S Marks

Oct 2007

This poster won the 1st prize at the University of Auckland 2007 Exposure poster competition

Abs

Virtual surgery training has become an essential tool in medical education. Unfortunately, commercial simulators are very expensive and mainly focus on individual training instead of teamwork. Game engines offer unique advantages for the inexpensive creation of interactive and collaborative environments. We show that it is possible to easily design multi-user scenarios with custom medical models. Our results will help to develop low cost surgical simulators that will improve health care outcomes for smaller hospitals or in regions which do not have access to advanced training tools.
An investigation of meshless deformation for fast soft tissue simulation in virtual surgery applications

A Henriques, B Wünsche, and S Marks

Computer-Assisted Radiology and Surgery, Oct 2007

DOI

2006

Evolving autonomous locomotion of virtual characters in simulated physical environment via neural networks and evolutionary strategies

S Marks, W Conen, and G Lux

In Proceedings of the ninth 3IA International Conference on Computer Graphics and Artificial Intelligence 3iA2006, May 2006

Abs

In this paper we examine principles to automatise or to support the process of animation of an articulated character by allowing it to “learn” its movements autonomously. For this, it is necessary to model the physical properties of the character, connect virtual sensors and actuators to a neural network, and adjust the weights and thresholds of the network with evolutionary strategies. We conclude with a discussion and showcase of the results and present ways for further improvement.
Evolving autonomous locomotion of virtual characters in a simulated physical environment via neural networks and evolutionary strategies

S Marks

Jan 2006

Abs

The animation of virtual characters is a process that although supported by various software and hardware can be tedious and costly especially when the character to animate is very complicated and/or detailed. The method presented in this thesis tries to automatize or to support this process by letting the character ’learn’ its movements autonomously. This is established by first modelling physical properties (e.g. mass, inertia moments, joints, degrees of freedom) additionally to the optical ones to allow the interaction of the character with a simulated physical environment. In the second step the sensors (e.g. pressure, forces, angles, speed) and actors (e.g. motors, ’muscles’, suspension elements) that the character uses are defined. Third the sensors and actors are connected with the inputs and outputs of a neural network whose bias values and link weights are still uninitialized. These values are then modified by evolutionary strategies to find naturally looking movements of the character in its physical environment.