COST-SID WG4 Report and Minutes of Berlin Sonification Workshop June 2009

Meeting and Sonification Workshop, Friday 12-Monday 15 June 2009 T-Labs, Ernst-Reuter-Platz 7, 10587 Berlin

A meeting and sonification workshop was organised by Thomas Hermann and Matthias Rath at the Deutsch Telekom T-Labs research building in Berlin. It overlapped a weekend and allowed for a meeting of the sonification group as well as a practical workshop focusing on four themes or topics.

The topics investigated were:

• Aesthetics in Sonifications of Elite Rowing Data
  
• Sonification of Spatial Gestures on Mobile Phones
  
• Sonic Spirit Level
  
• Sonification in Support of Shape Feedback
  

Aesthetics in Sonifications of Elite Rowing Data

This thread explored a number of new approaches to the sonification of rowing data.

Sonifications presented or developed include:

• Sin-ification – (Reiner Gerhet, Nina Schaffert) - Used a sine tone is used to sonify the acceleration of the skiff.  YouTube video example
  
• Xylophone MIDI-fication – (Anton Schertenlieb, Nina Schaffert, Stephen Barrass) - This sonification was made by loading the data into the xSonify tool in order to map it to a MIDI Xylophone.  YouTube video example
  
• Ear-cleaning Sonification (Hanna Buhl and Georg Spehr) - This sonification explored the possibility that the rowers could be taught to listen to auditory cues about synchronicity, rowing motion, and acceleration that are natural consequence of rowing.  MP3 recording of rowing sounds
  
• Ecological/Metaphorical? Sonification (John Williamson, Roderick Murray-Smith) - This sonification used computational models of splashing, wooden and metal impacts to communicate points or thresholds of interest. Impacts on these models are triggered by events in the data such as turning points and thresholds in the acceleration and the rate of change of acceleration.  YouTube video example
  
• Techno Music Probe (Stephen Barrass) - The changes in the pitch corresponding with the drive and recovery phases, together with the sync of the percussion, were used to explore the possibility of a musically based sonification of the data.  YouTube video example
  
• Music-ification prototype (Stephen Barrass) - A development of the `Techno Music Probe' where the sonification added a narrative element was introduced by adjusting the mix over the course so that a change in the music conveys the final push to the finish.  YouTube video example
  
• PAF Streaming (Reiner Gerhet and Stephen Barrass) - This sonification is an extension of the PD patch used to produce the original Sin-ification. The oscillator was replaced with a Phase Aligned Formant (PAF) synthesiser. The acceleration maps to the pitch (as before) and the derivative of acceleration maps to both the width and centroid of a formant. This causes moments of rapid change in acceleration (or jerking) to emerge as a distinctive figure against the smoother bass ground.  YouTube video example
  
• Sonic Branding (Hanna Buhl, Georg Spehr, Stephen Barrass) - This work discussed the possibility that the sonification could also include some reference to the national identity of the rowing team.

Stephen Barrass has produced a longer webpage with more details and examples:  Report on Aesthetics in Sonifications of Elite Rowing Data

Sonification of Spatial Gestures on Mobile Phone

This theme explored the sonification of gestures using mobile phone, in particular it looked at using acceleration data sonification from Nokia phones which transmitted over WiFi to a laptop running OSC and SuperCollider to generate the sonification. Four sonifications were explored, the first sonification tied X,Y, and Z information to three different parameter outputs (A sine wave, a resonant filter whose impulse response is like that of a sine wave, and a ping-like sound). The second sonification used audio streams (e.g. filtered noise) for each axis (x,y,z) using leaky integrators. The third sonification linked polar co-ordinates within 2 dimensions to sonify formants or vocal-like sounds. The fourth sonification linked polar co-ordinates within 2 dimensions to sonify musical sounds. Future directions include exploring physical models and glass interaction metaphors. The sonifications had been tied into a game idea where two players would compete. The first player would create a gesture and using the sonifications as an indication the second player would try to replicate the gesture. Two concurrent mobile phones transmitted their data that drove the sonifications. A minor issue arose regarding different implementations of Python with regard to different phones and the lack of the same API for interpreting accelerometer data across different phone models.

Sonic Spirit Level

Exploring how sonification can be used in the context of a spirit level to address issues such as fine accuracy, stability, latency, aesthetics, prevision depending on theta, and tremor around the zero level. The input was the angle (theta) and the outputs were three sonifications. The first used a tick like sonification, the second used THX dissonance (frequency adjustment), and the third used a physical model of a bowed string. Future directions include consolidating the three approaches for sonifying the level, moving the development to an iPhone based platform, to link to WG1 to work on an experimental validation of the prototype. Other possible future implementations include exploring different levels of temporal data (raw, smooth, very smooth, etc. ) ~= 2-level scale of a calliper rule or a goniometer, exploring secondary targets e.g. +45°/-45° with added values with regards to the physical device, and using different timbres depending on negative/positive values around zero e.g. odd/even harmonics distinction.

Sonification in Support of Shape Feedback

---

WG4 Meeting Sunday 14th June 2009

Present WG4 delegates:

• Thomas Hermann (MC, WG4 chair)
• Matthias Rath (WG4)
• Stephen Barrass (WG4)
• Roderick Murray-Smith (MC, WG4)
• John Williamson (WG4)
• Eoin Brazil (MC, WG4)
• Nicolas Misdariis (WG4)
• Yon Visell (MC, WG4)

Absent WG4 delegates:

• Sami Ronkainen (WG4)

T0: Approval of the Agenda and Minutes of previous WG4 meeting (Jyväskylä, 2008-09-16) [10']

T1: Discussion abut next WG meeting [10']

The decision was made to hold the next meeting in conjunction with the MC meeting in Holon, Israel in November.

T2: Outstanding WG issues and work [60']

Web Ambassador - Eoin - update pages to reflect workshop in Berlin and continue working with other web ambassadors to co-ordinate action on main COST SID website.

Examples Ambassador - John & Matthias - TODO collect links of projects and organise, hit fellow WG4 for links. Put the first draft of collected material online.

Bibliography Ambassador - Yon - Decision has been reached to use BibTex format and will re-send email around to solicit more references. The existing references will be placed online as a BibTex file and as a web page with comments.

Evaluation Ambassador - Nicholas - A short discussion was presented on the ATM study results, usability of sounds in interactive situation (cash-machine), will be presented in SMC conference (see this paper for more details), links to WG1 as done together with Patrick Susini. Work has started on a short paper on interaction design and how to do this type of evaluation in an experimental way, continue to link on this with the members of WG1, interactive 3D gestural device.

Chair - Thomas - Collect and email open actions to Ambassadors to deal with.

T3: Discussion on the Research Workshop and comments [15']

Breaks are needed but still is very segmented and the time to work is a bit too short. YV suggested the idea of a Barcamp type atmosphere. Intensive immersion. Size grew a little too big to be manageable. TH and rest of WG4 thanked MR for his excellent hosting and facilities. SB suggested that we should have a particular focus in advance, a research project done in one weekend with a number of approaches. TH to solicit comments and distil.

T4: Discussion on the Related Publications and on Sonification Handbook [15']

TH, AH, JN - then EB, SB and TH had short talk in York. The idea is now to go for an open access book regardless of other publishers. COST will support such open access publications. The new approach will be to use their publishers and the open book format. An open book publisher in Berlin are also an option. Agreement to take this as WG4 related activity.

T5: Any other business [10']

The discussion continued informally in the nearby bier garden.