DANCE EU ICT H2020 Project

Multimodal Dataset of Lightness and Fragility

2018-01-22T14:07:40+00:00

The aim of this work is to collect multimodal data of short dance performances containing full body movements displaying different expressive qualities. In total we collected 150 segments by 13 participants. The data consists of multiple 3D accelerometer data, video channels as well as audio of respiration and physiological signals. The data (video, audio, IMU, EMG) can be freely used for research purposes.

Task

We recorded a dataset of short dance performances containing full-body movements of two expressive qualities: Lightness and Fragility. The description of these two expressive qualities can be found in this paper. At the beginning, dancers were given the definitions of the expressive qualities. Next, the dancers were asked to perform an improvised choreography containing movements that, in the opinion of the dancer, expressed convincingly the quality. The dancers had different dance backgrounds (classic dance, pop, contemporary dance), and different levels of professional experience. They performed five repetitions of each expressive quality, each trial had a duration of 1 minute.

Setup Hardware and Data

We recorded data from the following devices:

5 IMU sensors (x-OSC) placed on the dancer’s body limbs; the data is captured at 50 frames per second; each frame consists of 9 values: (x, y, z) of accelerometer, gyroscope, and magnetometer;
2 video cameras (1280x720, at 50fps);
2 EMG sensors placed on the dancer’s forearms (MYO); the data is captured at 50 frames per second, each frame consists of 8 signals;
one wireless microphone (Mono, 48 kHz) placed close to the dancer’s nose and mouth, recording the sound of breathing;

Data was recorded and synchronized using the EyesWeb XMI platform. Synchronization of data streams is obtained by using SMPTE time-codes, i.e., a standard which is widely used in multimedia content production.

Segmentation

The recorded video streams were evaluated by dance experts and expressive movement analysis experts. For every trial, they identified segments of about 10s each corresponding to a uniform, coherent sequence of movements. For each dancer and each expressive quality, between 5 and 6 segments were chosen.

To download the whole dataset, please click here (warning: 4.6 GB file download).

Quality	No. segments	Mean duration	Total duration
Lightness	77	10.2 s	13 min 6 s
Fragility	73	10.4 s	12 min 41 s
Total	150	10.3 s	25 min 46 s

Dancer 1:
Intended quality: Lightness

Intended quality: Fragility

Dancer 2:
Intended quality: Lightness

Intended quality: Fragility

Dancer 3:
Intended quality: Lightness

Intended quality: Fragility

Dancer 4:
Intended quality: Lightness

Intended quality: Fragility

Dancer 5:
Intended quality: Lightness

Intended quality: Fragility

Dancer 6:
Intended quality: Lightness

Intended quality: Fragility

Dancer 8:
Intended quality: Lightness

Intended quality: Fragility

Dancer 9:
Intended quality: Lightness

Intended quality: Fragility

Dancer 10:
Intended quality: Lightness

Intended quality: Fragility

Dancer 11:
Intended quality: Lightness

Intended quality: Fragility

Dancer 12:
Intended quality: Lightness

Intended quality: Fragility

Dancer 13:
Intended quality: Lightness

Intended quality: Fragility

Dancer 14:
Intended quality: Lightness

Intended quality: Fragility

More about the data formats used can be found here.

Ranking

Five raters watched the 150 segments resulting from the segmentation. They observed each video segment and they were asked to rate the global level of Fragility and Lightness they perceived by using two independent 5-point Likert scales (from 0 to 4).
If you need the results please write to: Radoslaw Niewiadomski

If you have used our dataset in your research please cite the following our works:

Niewiadomski, R., Mancini, M., Piana, S., Alborno, P., Volpe, G., Camurri, A.,
Low-Intrusive Recognition of Expressive Movement Qualities,
Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017), Glasgow, UK, November 13–17, ACM, New York, NY, USA, pages 230-237, 2017, doi: 10.1145/3136755.3136757
Full paper available here.

and

Vaessen, M.J., Abassi, E., Mancini, M., Camurri, A., de Gelder B.,
Computational Feature Analysis of Body Movements Reveals Hierarchical Brain Organization
in Cerebral Cortex, Oxford University Press, v. 28, n. 10, pp. bhy228, Oct, 2018, ISBN/ISSN: 1047-3211
Full paper available here: here

Thank you!

Expressive Vocabulary

2018-01-02T11:08:04+00:00

This dataset of expressive qualities is the result of the collaboration with Virgilio Sieni, director of the Venitian Biennale Danza (2013-2016) and founder of the Accademia sull'arte del gesto. We worked with the choreographer Virgilio Sieni to define a set of expressive qualities that corresponds to his expressive vocabulary. The dataset constitutes the choreographic language of Virgilio Sieni: some of the most identifiable and isolable gestural figures and movement sequences (his “words”) have been chosen to be mutually compared in different interpretations to investigate movement qualities. For the purpose of the creation this dataset, Virgilio Sieni participated in a number of multimodal recording sessions during the first and second year of DANCE project. We were able to collect a dataset of movement fragments for a number of qualities such as Lightness, Fragility, Suspension, Transmission, and Figure.

The motion capture data can be freely used for research purposes.

Setup Hardware and Data

We recorded data from the following devices:

Qualisys motion capture system sampling dancers' movement at 100 Hz,
video recording system (1280x720, 50fps) (unavailable for download).

Data was recorded and synchronized using the EyesWeb XMI platform.

Expressive Qualities

Lightness

Definition: A necessary condition for a Light movement is Fluidity. Further, a fluid movement should include at least one of the following characteristics: (i) to exhibit a low amount of downward vertical acceleration following gravity (in particular on forearms and knees), (ii) each possible downward acceleration must be counterbalanced by an opposite “harmonic” upward movement (simultaneous or consequent); (iii) vertical downward acceleration movements are turned into a finalization on the horizontal plane.

Lightness_example.zip

Fragility

Definition: In order to describe the Fragility, first we will introduce two its components: Upper body crack and Leg release. The Upper body crack is a single synchronized discontinuity of the upper body movement followed by the movement re-planning. Leg release is sudden movement of the hips (and knees) toward the floor due to a sudden synchronized release of knees. Both these components correspond to the abrupt variation of the acceleration of the body movements. While the upper body crack regards upper body, legs release concerns the lower part of the body. Fragility, is defined as a frequent a-rhythmic interruptions and consequent re-planning of movement. Thus, it can be seen as a non-periodic a-rhythmic sequence of Upper body cracks and Leg releases.
Fragility emerges, for example, when moving at the boundary between balance and fall, resulting in a series of short movements with frequent interruptions and restarts. The resulting movements are non-predictable.

Fragility_example.zip

Suspension

Definition: Suspension refers to non-directional holding of energy on one of the body planes. The body or some parts of it may, for example, waving or rippling. The movements of suspension are often highly predictable and repetitive.

Suspension_example.zip

Transmission

Definition: The transmission refers to circulation of body kinetic energy between different body planes. It may include resonances between different body parts, or changes of the body part that is leading movement.

Transmission_example.zip

Figure

Definition: Articulation of alignments and unisons, consolidation of balance.

Figure_example.zip

More about this work can be found in the paper:

Camurri, A., Volpe, G., Piana, S., Mancini, M., Niewiadomski, R., Ferrari, N., Canepa, C., The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in Movement, 3rd International Symposium on Movement and Computing, MOCO 2016, 5-6 July 2016, Thessaloniki, Greece, 2016. doi: 10.1145/2948910.2948927.

Full paper can be found here.

Multimodal dataset of full-body movement qualities: Lightness and Fragility

2018-01-02T10:08:04+00:00

Multimodal synchronized recordings of short dance improvisations, i.e., full body movements aiming to display different expressive qualities. In total we collected 150 segments by 13 participants. The data consists of 3D accelerometers data, video, respiration (audio, microphone), and simple physiological signals. This dataset can be freely used for research purposes, with the only request to cite the following papers describing and using this dataset:

Niewiadomski, R., Mancini, M., Cera, A., Piana, S., Canepa, C., Camurri, A.
Does embodied training improve the recognition of mid-level expressive movement qualities sonification?
in Journal on Multimodal User Interfaces, ISBN/ISSN: 1783-8738, Dec, 2018. doi: 10.1007/s12193-018-0284-0
Full paper available here...

and:

Vaessen, M.J., Abassi, E., Mancini, M., Camurri, A., de Gelder, B.
Computational Feature Analysis of Body Movements Reveals Hierarchical Brain Organization
in Cerebral Cortex, Oxford University Press, v. 28, n. 10, pp. bhy228, ISBN/ISSN: 1047-3211, Oct, 2018
Full paper available here...

The dataset has been recorded by University of Genoa at the Casa Paganini-InfoMus research centre of DIBRIS, in the EU Project ICT DANCE n 645553 (dance.dibris.unige.it).

Task

We recorded a dataset of short dance performances containing full-body movements of two expressive qualities: Lightness and Fragility (see the above mentioned papers).

Protocol: first, dancers were instructed with the definitions of the two expressive qualities. Next, the dancers were asked to perform an improvised choreography containing movements that, in the opinion of the dancer, expressed convincingly the requested quality. The dancers had different dance backgrounds (classic, dance, pop, contemporary dance), and are at an amateur level of dance experience. They performed five repetitions of each expressive quality, each trial had a duration of 1 minute. From each 1min trial, we extracted segments in which the quality was best expressed, with the help of dance experts.

Setup Hardware and Data

We recorded (synchronized) data from the following devices:

5 IMU sensors (x-OSC) placed on the dancer’s body limbs; the data is captured at 50 frames per second; each frame consists of 9 values: (x, y, z) of accelerometer, gyroscope, and magnetometer;
2 video cameras (1280x720, at 50fps);
2 EMG sensors placed on the dancer’s forearms (MYO); data captured at 50 frames per second, each frame consists of 8 signals;
1 wireless microphone (Mono, 48 kHz) placed close to the dancer’s nose and mouth, to record the audio of respiration.

Data were recorded and synchronized using the EyesWeb XMI platform. Synchronization of data streams is obtained by SMPTE time-codes.

The Data Segments

The recorded video streams were evaluated by dance experts and expressive movement analysis experts. For every trial, they identified segments of about 10s each corresponding to a uniform, coherent sequence of movements expressing the requested quality. For each dancer and each expressive quality, between 5 and 6 segments were chosen.

Quality	No. segments	Mean duration	Total duration
Lightness	77	10.2 s	13 min 6 s
Fragility	73	10.4 s	12 min 41 s
Total	150	10.3 s	25 min 46 s

Below we present some examples of the videos and data. The whole dataset can been seen and download from here.

Dancer 1:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 2:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 3:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 4:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 5:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 6:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 8:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 9:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 10:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 11:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 12:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 13:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

Dancer 14:

Quality	Lightness	Fragility
Example
Video Download	video	video
Data	data	data

More about the data formats used can be found here.

Ranking

Video fragments with SMPTE code

We recorded video fragments data with SMPTE code in the audio track, for synchrionization with the other sensor channels:

Date of recording	Link to download page	Description	Quality video files count
2017-01-13		Total video files: N. 20	Lightness quality video files: N. 10 Fragility quality video files: N. 10
2017-01-14		Total video files: N. 21	Lightness quality video files: N. 11 Fragility quality video files: N. 10
2017-01-18		Total video files: N. 36	Lightness quality video files: N. 21 Fragility quality video files: N. 15
2017-01-25		Total video files: N. 51	Lightness quality video files: N. 18 Fragility quality video files: N. 20
2017-02-01		Total video files: N. 24	Lightness quality video files: N. 12 Fragility quality video files: N. 12

Video fragments with SMPTE

2018-01-02T10:08:04+00:00

Video fragments with SMPTE code

We recorded video fragments data with SMPTE code in the audio track, for synchrionization with the other sensor channels.

If you have used our dataset in your research please cite the following our works:

Niewiadomski, R., Mancini, M., Cera, A., Piana, S., Canepa, C., Camurri, A.
Does embodied training improve the recognition of mid-level expressive movement qualities sonification?
in Journal on Multimodal User Interfaces, ISBN/ISSN: 1783-8738, Dec, 2018. doi: 10.1007/s12193-018-0284-0
Full paper available here...

and:

Vaessen, M.J., Abassi, E., Mancini, M., Camurri, A., de Gelder, B.
Computational Feature Analysis of Body Movements Reveals Hierarchical Brain Organization
in Cerebral Cortex, Oxford University Press, v. 28, n. 10, pp. bhy228, ISBN/ISSN: 1047-3211, Oct, 2018
Full paper available here...

CHI 2018 - Workshop

2017-10-13T08:00:59+00:00

This page will host the CHI 2018 Workshop.

DANCE Platform version 2

2017-02-08T11:08:00+00:00

In this webpage you can find instructions necessary to download, install and run the DANCE software platform Version 2. The platform is based on EyesWeb XMI (http://www.infomus.org/eyesweb_eng.php), allowing users to perform synchronized recording, playback, and analysis of a multimodal stream of data. Details about the platform architecture and data stream formats are provided in Deliverable 4.2.

Download and install EyesWeb XMI

The updated version of EyesWeb XMI can be downloaded here: ftp://ftp.infomus.org/Evaluate/EyesWeb/XMI/Version_5.7.x/EyesWeb_XMI_setup_5.7.0.0.exe

Instructions to download and install EyesWeb XMI can be found on the DANCE software platform version 1 page.

2. Download the DANCE example tools and patches

The DANCE example tools and patches are programs, written to be loaded and executed in EyesWeb XMI, that allow the user to record, playback and analyze multimodal data (video, audio, motion capture, sensors). To run tools you will need to download the corresponding installers, launch them and execute the tools as normal Windows applications. To run patches you will need to download and load them into the EyesWeb application (see step 1 on how to download and install EyesWeb). The current version of the DANCE example tools and patches includes applications allowing you to perform different tasks:

a) Tools and patches for recording and playing back multimodal data

Video recorder tool (download installer) [Warning: a Decklink frame grabber is required to roon this tool]

This module, depicted below, is able to record the video stream of multiple video sources. Each video source is stored in a separate video+audio (e.g., AVI, mp4). To guarantee synchronization with the other data streams, the audio channels of the generated files contain the SMPTE time signal encoded as audio. During playback, the SMPTE is decoded from audio to extract timing information and play the video stream in sync.

Kinect recorder tool (download installer) [Warning: a Kinect 2 is needed to run the tool]

The Kinect recorder tool is depicted below:

The tool shows the current framerate in the "Data Status" field (15.01 frames per second in the example), the name used for this trial (trial_000, progressive numbers are automatically assigned to each trial), and the value of the reference clock (HHHH:MM:SS.mmm; 0000:00:49.160 in the above picture).

The option panel, displayed below, has been redesigned in the new version. It allows to choose which Kinect streams to record, the recording mode (standalone or slave) and to assign an ID to the recording module. The ID will be reported in the master interface, as described in the previous section.

Audio recorder tool (download installer) [Warning: an ASIO compatible sound card is needed to run the tool]

The audio recorder tool is depicted in the below:

It has been improved from the previous version by adding the possibility of recording several stereo audio tracks at the same time. The recorder is currently limited, mainly for performance reasons, to 8 stereo tracks, that is, 16 mono channels. The recorder interface shows the input audio signals as well as the number of recorded audio buffers per second.

The options panel allows the experimenter to set the audio recorder ID and the input audio tracks to be recoded.

IMU recorder tool (download installer) [Warning: one or more X-OSC sensors are needed to run the tool]

The IMU recorder tool is depicted below:

The graph shows the values selected by the user (acceleration, gyroscope, or compass) for each of the four IMUs. In the lower left you may read the current framerate of each of the four sensors (49.95 samples per second). Below the graph you may see both the trial name and the reference clock.

The data is saved by the recording tool in txt files, in a format which is easy to be read by external software (e.g., Matlab), and can be of course read by EyesWeb itself for playback or analysis purposes. The options panel allows you to configure the working mode of the recorder.

Myo recorder tool (download installer) [Warning: 2 Myo sensors are needed to run the tool]

This tool has been added during the second year of DANCE. It allows to record data coming from 2 Myo sensors at about 40 fps. The data consists of muscle contraction (1 value between 0 and 1), acceleration, gyroscope and rotation (yaw, pitch, roll).

Playback and quality extraction tools (download installer)

We provide 2 tools along with the DANCE Platform version 2:

a data playback tool
a quality of movement extraction tool

The installer contains sample data consisting in 2 trials recorded in DANCE during the months of January and February 2017. Both trials consists of multimodal data coming from Inertial Movement Units (accelerometers), Electromyographic sensors, video cameras and microphone. Each trial focuses on a single movement quality in the set of movement qualities identified by the UNIGE partner in collaboration with the choreographer V. Sieni. See Deliverable 2.2 for more details.

The 2 trials are:

Trial 84: a dancer performing "fragile" movements (i.e., a sequence of non-rhythmical upper body cracks or leg releases; movements on the boundary between balance and fall, time cuts followed by a movement re-planning; the resulting movement is non-predictable, interrupted, uncertain)
Trial 246: a dancer performing "light" movements (i.e., fluid movement together with a lack of vertical acceleration, mainly toward the floor, in particular of forearms and knees; for each vertical downward movement there is an opposite harmonic counterbalancing upward movement, simultaneous or consequent; there can be a convertion of gravity into movement on the horizontal plane using rotations and a spread of gravity on the horizontal dimension)

For a detailed description of the above and other movement qualities please refer to Deliverable 2.2.

Playback

The tool window is depicted below:

The trial name is reported on top of the window. In the upper part of the window, the video streams are played back. In the lower part of the window, the EMG (on the left) and intertial data (on the right )is displayed in a time window of 100 samples. ON the bottom, there are buttons to move between the trials and start/stop the playback. The tool is automatically installed in the EyesWeb XMI folder, for example in C:\Program Files (x86)\EyesWeb XMI 5.7.0.0\DANCE_Playback_Tool.

The installer installs 2 example trials in the subfolders "2017-01-18" and "2017-02-01". If more trials are found they are automatically added to the playback tool. Once started, the tools also sends the inertial sensors data through OSC messages. This data can be received by other software modules (for example, the movement features extraction tool, see below and Deliverable 3.2) to be analyzed.

Quality extraction

We propose a Multi-Layered Computational Framework of Qualities consisting of several layers, ranging from physical signals to high-level qualities and addresses several aspects of movement analysis with different spatial and temporal scales. The features computed at lower layers, contribute to the computation of the features at higher levels, which are usually related to more abstract concepts. In more details, within the framework, the movement quality analysis organized on four levels:

points: physical data that can be detected by exploiting sensors in real-time (for example, position/orientation of the body planes)
frames: physical and sensorial data, not subject to interpretation, detected uniquely starting from instantaneous physical data on the shortest time span needed for their definition and depending on the characteristics of the human sensorial channels
qualities: perceptual features, interpretable and predictable, with a given error and correction, starting from different constellations of physical and sensory data from level 2, on the time span needed for their perception (for example, lightness is perceived as a delay on the horizontal plane or as a balance between vertical upward/downward pushes)
affects: perceptual and contextual features, interpretable and predictable, with a given error and correction, starting from a narration of different qualities, on a large time span needed for their introjection (for example, tension created by a pressing sequence of movement cracks/releases or by a sudden break of lightness).

The first layer is devoted to capturing and preprocessing data (e.g., computing the kinematics features such as velocity, trajectories) from sensor systems (e.g., video, motion capture, audio, or wearable sensors). The second layer computes a low-level motion features at a small time scale (i.e., observable frame-by-frame) such as kinetic energy or smoothness from such data. The third one segments the flow of movements in a series of single units (or gestures) and computes a set of mid-level features, i.e., complex features that are usually extracted on groups of joints or the whole body, and require significantly longer temporal intervals to be observed (i.e., between 0.5s and 5s). Finally, the fourth layer represents even more abstract concepts such as emotional states of the displayer, social attitudes, user’s engagement in a full-body interaction.

Along with the DANCE Platform version 2 we provide a quality of movement extraction tool capable of analyzing pre-recorded multimodal data and to extract the following qualities:

Suspension: it is a non-directional retention of energy on one of the body planes. The body or some parts of it may waving or rippling, the movement are of high predictability. We detect it by checking whether the maximum of the energy is retained over a period of time on one body plan.
Fragility: it is a sequence of non-rhythmical upper body cracks or leg releases . Movements on the boundary between balance and fall, time cuts followed by a movement re-planning. The resulting movement is non-predictable, interrupted, and uncertain. We detect irregular sequences of upper body cracks or leg releases in 1 a second time window.
Lightness: this quality is related to the Laban’s Weight quality (for details, see: Rudolf Laban and Frederick C. Lawrence. 1947: Effort. Macdonald & Evans.) It is computed by extracting the Energy vertical component normalized to the overall amount of Energy in the movement.
Kinectic Energy of a moving (part of) body, KE = 1/2*mass*velocity
Smoothness: it is defined as the inverse of the third derivative of position

DANCE Platform version 3

2017-02-08T11:08:00+00:00

In this webpage you can find instructions necessary to download, install and run the DANCE software platform Version 3. The platform is based on EyesWeb XMI (http://www.infomus.org/eyesweb_eng.php), allowing users to perform synchronized recording, playback, and analysis of a multimodal stream of data. Details about the platform architecture and data stream formats are provided in Deliverable 4.3.

Download

Installation

To install the platform, please follow these steps:

download and install the EyesWeb XMI 5.7.2.0 application (first link above)
- run the exe file and execute the installer to install EyesWeb XMI 5.7.2.0
download and install the movement quality extraction patch (second link above)
- run the exe file and execute the installer; to execute it, a valid EyesWeb 5.7.2.0 installation must be already present in your system (see the previous step)
- the installer will create a link on your desktop named "Movement Features Extraction"
download and unzip the sonification patch (third link above)
- unzip the archive in a user folder (e.g., Documents or Desktop folder), then you can delete the archive

To run the platform, please follow these steps:

run the sonification patch by opening the user folder in which you have unzipped the sonification patch (e.g., inside the Documents or Desktop folder) and executing the file named "sonification_patch"
run the movement quality extraction patch by double clicking the desktop link previously created on your desktop by the movement quality extraction patch installer, named "Movement Features Extraction"

After a few seconds, you will see the movement quality extraction patch interface, showed in Figure 4. The interfaces is provided with some pre-recorded sensor data of a dancer displaying the Light and Fragile movement qualities. By pressing the "Previous trial" and "Next trial" buttons you can select the movement quality. The "Play" button starts the playback and movement quality extraction process. The movement qualities extracted from the dancer's movement are sent to the sonification patch in real-time and the corresponding sonification is generated.

DANCE Platform version 1

2016-02-15T11:08:04+00:00

In this webpage you can find instructions necessary to download, install and run the DANCE software platform. The platform is based on EyesWeb XMI (http://www.infomus.org/eyesweb_eng.php), allowing users to perform synchronized recording, playback, and analysis of a multimodal stream of data. Details about the platform architecture and data stream formats are provided in Deliverable 4.1.

The platform supports several research activities of the DANCE Project:

creation of a multimodal repository of recordings of movement qualities;
fine-grain synchronization of multimodal data
segmentation of the recordings in fragments, according to the chosen qualities
playback and testing of the repository
extraction of the movement features and qualities
real-time interaction sonification
design and development process of scientific experiments of DANCE
design and development of the prototypes of applications
design and development of artistic projects exploiting the results of the DANCE project (e.g., artistic performances)

To use the current version of the platform you have to:

download and install EyesWeb XMI
download the DANCE example tools and patches (a patch is a program, written and stored in the EysWeb XMI format; see the description below)
download the sample data needed by the example patches
run EyesWeb XMI, load one or more patches and execute them

1. Download and install EyesWeb XMI

EyesWeb XMI is a modular system that allows both expert (e.g., researchers in computer engineering) and non-expert users (e.g., artists) to create multimodal installations in a visual way. EyesWeb provides software modules, called blocks, that can be assembled intuitively (i.e., by operating only with mouse) to create tools and programs, called patches, that exploit system's resources such as multimodal files, webcams, sound cards, multiple displays and so on.

The last version of EyesWeb is the 5.6.1.0. (released in Nov 2016). You can download it from the following link: ftp://ftp.infomus.org/Evaluate/EyesWeb/XMI/Version_5.6.x/EyesWeb_XMI_setup_5.6.1.0.exe

Once you have finished installing the EyesWeb, proceed to step 2 in order to be able to run the DANCE platform patches.

2. Download the DANCE example tools and patches

The DANCE example tools and patches are programs, written to be execute by EyesWeb, that allow the user to record, playback and analyze multimodal data (video, audio, motion capture, sensors). To run tools you will need to download the corresponding installers, launch them and execute the tools as normal Windows applications. To run patches you will need to download and load them into the EyesWeb application (see step 1 on how to download and install EyesWeb). The current version of the DANCE example tools and patches includes applications allowing you to perform different tasks:

a) Tools and patches for recording and playing back multimodal data

Kinect recorder tool (download installer)

The Kinect recorder tool is depicted below:

The tool shows the current framerate (50 frames per second in the shown example), the name used for this trial (trial_000, progressive numbers are automatically assigned to each trial), and the value of the reference clock (HHHH:MM:SS.mmm; 0000:00:07.440 in the above picture).

The recording tool records avi files. The video is encoded in MPEG-4 format, the resolution is 1280x720 and the framerate is 50 fps. Audio is encoded in AAC format at 48000 Hz. Two channels are recorded: the left channel contais audio from the system’s audio input device (e.g., a microphone), whereas the right channel is the reference clock encoded in SMPTE audio format.

Multiple instances of the video recorder tool can be started and can work standalone, or synchronized with the other recorders. The options panel allows you to configure the working mode of the recorder.

Audio recorder tool (download installer)

The audio recorder tool is depicted in the below:

The recorder creates a stereo file in WAVE format, and the SMPTE is added as the right channel. Audio is sampled at 48000 Hz. The user interface is very similar to the video recorder tool. The main difference is the visualization part. In this tool the audio waveform is shown instead of the video stream. The options panel allows you to configure the working mode of the recorder.

IMU recorder tool (download installer)

The IMU recorder tool is depicted below:

The recorder expects input values (sent as OSC packets, see Deliverable D4.1 for details about this file format) from 4 Inertial Movement Units (IMUs). The graph in the figure shows the values selected by the user (one among accelerometer, gyroscope, or compass) for each of the 4 IMUs. In the lower left part of the recorder interface you can read the current streaming framerate related to each sensor (49.95 fps in this example). Below the graph you can read both the trial name and the reference clock.

The data is saved by the recording tool in CSV format. CSV format is commonly used to to provide portability towards by external softwares (e.g., Matlab), and, obviously, can be read by EyesWeb itself for playback or analysis purposes. The options panel allows you to configure the working mode of the recorder.

Recorder tools options panel

All the recording tools share the same options panel depicted below:

The control type section controls the synchronization mode. If set in master/standalone mode then the tool works with its own clock, without synchronizing it to other devices. In slave mode the tool receives the clock time from an external device (the master). The clock time can be received both via Network (OSC protocol) and via audio (SMPTE Sync).

Playback patch

Once you recorded some audio, video and IMU data, you can play it back using the playback patch.

Download the patch and copy the downloaded file to the parent folder of the recorded data. If you did not record any data you can download some sample data from this website.

For example, if the recorded/downloaded data is located in:
C:\Users\my_username\recordings
then the patch path will be:
C:\Users\my_username\recordings\DANCE_platform_reader.zip

Start EyesWeb and load the reader patch. You will see the following screen:

1) Run the patch by pressing the play button on the EyesWeb toolbar

2) Use the slider “Trial Selector” to select a specific trial to be played back
3) To PLAY press the green "Play" button (or press the "s" key)
4) To STOP press the red "Stop" the playback (or press the "t" key)
5) To PAUSE the playback press the (or press the "p" key)

IMPORTANT: When changing from a recording to another you have firstly to stop the currently played segment and then you can start the new one.

During the playback of a file, video, 3D mocap data and IMU signals will be displayed. An example:

Patches for analyzing multimodal data

Now that you recorded or downloaded some multimodal data and you can successfully play it back, you can procced by performing some analysis on it.

In the DANCE project we aim to innovate the state of art on the automated analysis of the expressive movement. We consider movement as a communication channel allowing humans to express and perceive implicit high-level messages, such as emotional states, social bonds, and so on. That is, we are not interested in physical space occupation or movement direction per se, or in “functional” physical movements: our interest is on the implications at the expressive level. For example: a hand movement direction to the left or to the right may be irrelevant, instead the level of fluidity or impulsiveness of such movement might be relevant. Example: let us consider the movement “Knocking at a door”. We do not want to analyze the functional action of “knocking at a door”, but the intention that lies behind it (e.g., the lover that knocks at the door of his beloved). To study it, we focus on the sets of non-verbal expressive features that are described in detail in Deliverable 5.1.

The following expressive features can be extracted on multimodal data using the patches you can download below:

Energy of movement. Kinectic Energy of a moving (part of) body, Energy = 1/2*mass*velocity
Slowness. This feature indicates whether the movement is performed slowly or not.
Smoothness. This feature is based on Energy and Slowness. If movement exhibits high (respectively, low) slowness and no (respectively, many) energy peaks are detected then smoothness is high (respectively, low).
Weight. This feature is related to the Laban’s Weight quality (for details, see: Rudolf Laban and Frederick C. Lawrence. 1947: Effort. Macdonald & Evans.) It is computed by extracting the Energy vertical component normalized to the overall amount of Energy in the movement.
Suddenness. It is computed using alfa-stable distributions.An alpha-stable fit is performed on peaks of accelerations. A movement is sudden when the product between alpha and gamma is high (see Deliverable 2.1 for details). The algorithm takes as input the 3D joint accelerations on a time window on which the suddenness has to be computed, and then it fits it into the alfa-stable distribution. The output of the app gets close to 1 (i.e., very sudden movements) when there are abrupt increases of the joint's velocity in the input signal, and vice-versa.
Impulsivity. An impulsive movement can be performed by a part of the body or by the whole body and is characterized by the following properties: (P1) it is sudden, that is, it presents a high variation of speed (either from low to high or from high to low); (P2) it is executed with no preparation.
Fluidity. A Fluid movement can be performed by a part of the body or by the whole body and is characterized by the following properties: (P1) the movement of each involved joint of the (part of) the body is smooth, following the standard definitions in the literature of biomechanics; (P2): the energy of movement (energy of muscles) is free to propagate along the kinematic chains of (parts of) then body (e.g., from head to trunk, from shoulders to arms) according to a coordinated wave-like propagation. That is, there is an efficient propagation of movement along the kinematic chains, with a minimization of dissipation of energy. Fluidity is computed as the distance between the evolution in time of Humanoid Mass-Spring model (i.e., a model of the human body conceived as a set of masses connected by springs, see Deliverable 2.1 for details) and the actual movement of a user.

Besides the above expressive features, we are interested in extracting analysis primitives: thay are unary, binary, or n-ary operators that summarize with one or more values the temporal development of low-level features in an analysis time unit (a movement unit or a time window). The simplest unary analysis primitives are statistical moments (e.g., average, standard deviation, skewness, and kurtosis). Further examples of unary operators, that are more complex, include shape (e.g., slope, peaks, valleys), entropy, recurrence, and various time-frequency transforms). Models for predictions (e.g., HMM) can also be applied. Binary and n-ary operators can be applied e.g., for measuring relationship between low-level features computed on the movement of different body parts. For example, synchronization can be used to assess coordination between hands. Causality can provide information on whether the movement of a joint leads or follows the movement of another joint.

The following analysis primitive can be extracted on multimodal data using the patches you can download below:

Synchronization. It is an important concept in human-human communication that has been widely addressed by the HCI research community and in movement studies. In the DANCE platform we exploit Event Synchronization, a particular type of synchronization (see http://wwwold.fi.isc.cnr.it/users/thomas.kreuz/Source-Code/Event-Sync.html).

We split the patches for analyzing multimodal data in 2 groups:

Patches computing features and analysis primitives from IMUs

The links reported below summarize the patches for computing features and analysis primitives from IMUs. To use and test the patches:

dowload the playback patch
download the zip file extract the IMU patches
download and extract the sample IMU data and videos of sample dance fragments used by the patches to demonstrate examples of computation of features; the data has to be extracted in the same folder of the patches
run EyesWeb and load the patches indicated in the table for each feature, and run the patches in the indicated order

Patches computing features from IMUs (you need the sample IMU data to run these patches)
FEATURE/ ANALYSIS PRIMITIVE	PATCH FILES (to be executed in EyesWeb in the indicated order)
Energy	1. DANCE_platform_reader.eywx 2. DANCE_library_energy_imu.eywx
Slowness	1. DANCE_platform_reader.eywx 2. DANCE_library_slowness_imu.eywx
Smoothness	1. DANCE_platform_reader.eywx 2. DANCE_library_smoothness_imu.eywx
Weight	1. DANCE_platform_reader.eywx 2. DANCE_library_weight_imu.eywx
Suddenness Impulsivity	1. DANCE_platform_reader.eywx 2. DANCE_library_impulsivity-suddenness_imu.eywx
Event Sync	1. DANCE_platform_reader.eywx 2. DANCE_library_event_synchronization.eywx
SPIKE Sync	1. DANCE_platform_reader.eywx 2. DANCE_library_spike_synchronization.eywx

Patches computing features and analysis primitives from motion capture data

The links reported below summarize the main patches for computing features and analysis primitives from motion capture data. To use and test the patches:

dowload the playback patch
download the zip file and extract the motion capture patches and extract the motion capture patches
download and extract the sample motion capture data and videos of sample dance fragments used by the patches to demonstrate examples of computation of features; the data has to be extracted in the same folder of the patches
run EyesWeb and load the patches indicated in the table in the feature, and run the patches in the indicated order

Patches computing features from motion capture data (you need the sample motion capture data to run these patches)
FEATURE/ ANALYSIS PRIMITIVE	PATCH FILES (to be executed in EyesWeb in the indicated order)
Impulsivity Suddenness	1. DANCE_platform_reader.eywx 2. DANCE_library_impulsivity-suddenness_mocap.eywx
Fluidity	1. DANCE_platform_reader.eywx 2. DANCE_fluidity_mocap.eywx
Event Sync	1. DANCE_platform_reader.eywx 2. DANCE_library_event_synchronization.eywx
SPIKE Sync	1. DANCE_platform_reader.eywx 2. DANCE_library_spike_synchronization.eywx

3. Download the IMU and motion capture sample data

As reported in the above paragraphs, you have to download and extract some sample data in order to run the DANCE example patches. Without some sample data the example patches will not start, or will start but will not provide any output. The sample data is contained in a zip file and it is a collection of 2 trials consisting in data recorded by a motion capture system, a videocamera, and 4 IMU sensors placed on the dancer's limbs (wrists and ankles). The zip archive contains the following folders and files:

folder: imu
- file: 2015-12-15_t010_imu01_acceleration.txt (CSV format, 3D accelerometer data captured by IMU number 1 during trial number 010 on 2015-12-15)
- file: 2015-12-15_t010_imu01_gyro.txt (CSV format, 3D gyroscope data captured by IMU number 1 during trial number 010 on 2015-12-15)
- file: 2015-12-15_t010_imu01_magnetic.txt (CSV format, 3D compass data captured by IMU number 1 during trial number 010 on 2015-12-15)
- .... (the same files are present for IMUs number 2, 3, and 4 and for trial number 011)
folder: mocap
- file: 2015-12-15_t010_mocap.qam (3D model of the dancer captured during trial number 010 on 2015-12-15)
- file: 2015-12-15_t010_mocap.tsv (3D motion captured data of the dancer captured during trial number 010 on 2015-12-15)
- file: 2015-12-15_t011_mocap.qam (3D model of the dancer captured during trial number 011 on 2015-12-15)
- file: 2015-12-15_t011_mocap.tsv (3D motion captured data of the dancer captured during trial number 011 on 2015-12-15)
folder: video
- file: 2015-12-15_t010_video01.avi (video and audio file of the dancer captured during trial number 010 on 2015-12-15)
- file: 2015-12-15_t010_video01_timings.txt (timings of the video frames contained in the corresponding video file 2015-12-15_t010_video01.avi)
- file: 2015-12-15_t011_video01.avi (video and audio file of the dancer captured during trial number 011 on 2015-12-15)
- file: 2015-12-15_t011_video01_timings.txt (timings of the video frames contained in the corresponding video file 2015-12-15_t011_video01.avi)

4. Run EyesWeb XMI, load one or more patches and execute them

If you downloaded EyesWeb, you installed it and you downloaded some example patches plus the needed sample data you are ready to run the patches:

run EyesWeb, by clicking on the corresponding shortcut in the start menu:
load an example patch: select the file->open menu item; browse for the patch file; select "open":
click on the "play" button:
depending on the example patch your executing, different display windows will appear

The DANCE Project

2015-09-09T06:59:48+00:00

“Closing the eyes is needed to see the immeasurable spaces of a dream, to find images that open up to feelings coming from the least explored places of the brain, to experiment with forms of the visible derived by different sensorial configurations”
(J.Svankmajer, Touching and Imagining)

DANCE maps these thoughts into scientific and artistic research for investigating how affective and relational qualities of body movement can be expressed, represented, and analyzed by the auditory channel. DANCE aims at understanding the meaning of “closing the eyes”, the perception of expressiveness and entrainment in dance, the participation to the emotion conveyed by a sequence of movements in space, the understanding of the non-verbal language of bodies that communicate, imagining and questioning concrete ways to listen to a choreography, feel a ballet.

DANCE dimensions, strongly integrated and interacting, are the following:

Inclusion and rehabilitation. Interactive sonification and musicalization of choreutic movement (processing of sound spaces and ways of using affective music content in relation to the quality of gestures) investigates forms of sensory substitution: to see through listening. Testing new perceptual experiences of body movement in the dark implies the possibility of rapprochement and sharing (of spaces and emotions) between visually impaired and sighted people.
Scientific and technological research. Neuroscience studies on brain plasticity in sensory substitution at University of Maastricht will be combined with studies at Casa Paganini-InfoMus on technologies capable of seeing emotional qualities and social indicators of movement, interpreting them as techniques for searching data in digital archives (e.g. sound, music, visual archives), and with research on interactive sonification at KTH.
Artistic research and production. The choreography itself creates the music: dance can be conceived as musical composition (or re-composition, interpretation), changing its traditional dimension into an eminently or exclusively listening experience: Gesture, as an aesthetic object, is experienced as a sounding object.

Europa: Gestures of History, Dancing Science and Art not to forget EU identity

Europa: Gestures of History, Dancing Science and Art not to forget EU identity, is a scientific performance in Rome (March 23rd 2017). This performance took place in occasion of the dinner at the celebration of the Treaty of Rome, at “La Lanterna” in Rome, on invitation by the EU DG Connect. The objective was to tell metaphorically the narration of the birth and evolution of Europe by means of a short choreographic action, enhanced by an interactive sonification amplifying the qualities of dancer’s movement.

Esodi
Public performance at Museo di Palazzo Reale, Genoa - March 25th 2017

(Salone da ballo - Ballroom)

(Galleria degli specchi - Gallery of mirrors)

Di Fronte agli Occhi degli Altri
Public performance of Virgilio Sieni and Giuseppe Comuniello, Casa Paganini, Genoa - March 24th 2017

	Facebook	L'Atlante del Gesto_Genova @atlantedelgestoGenova
		DANCE EU ICT H2020 Project videos

Consortium

2015-03-04T13:04:23+00:00

	Università degli Studi di Genova (UNIGE)
	Established in 1984 at Università degli Studi di Genova (UNIGE), the Casa Paganini - InfoMus lab carries on scientific research and design, development, and experimentation of multimodal interfaces and multimedia systems. Main research addresses computational models of nonverbal multimodal expressive and social communication, with special focus on human movement and gesture (e.g., dance, full-body movements), audio (e.g., music), expressive and tangible multimodal interfaces. Applications domains include cultural heritage, edutainment, rehabilitation, entertainment, creative industry (music, dance), user-centric media, and mobile distributed multimedia systems. The lab is the main developer of the freely available EyesWeb XMI platform for synchronised recording, real-time interactive processing, and rendering of multimodal data. The Università degli Studi di Genova (UNIGE) is the coordinator of the project. It will lead the project by ensuring an effective coordination and collaboration among al the partners, by monitoring the progress and by participating in the R&D activities. It is mainly in charge of realtime analysis and sonification of non-verbal full-body movement, emotion detection, use-case design, development of the DANCE platform, and integration of its components.

	University of Maastricht (UM)
	The Faculty of Psychology and Neuroscience of the University of Maastricht is home to the department of Cognitive Neuroscience and the Maastricht Brain Imaging Centre, comprising of 9 research groups, that are carrying out research on the topics such as auditory perception, emotion and cognition, columnar level fMRI, physiological (f)MRI, visual perception and learning, developmental neuroscience, and fMRI methods. Prof. Beatrice de Gelder is the head of the Emotion and Cognition group which investigates cognitive en affective neuroscience of intersensory perception. The research focuses on how emotion and cognition interact in humans and on the interaction between different sensory systems, primarily between seeing and hearing. UM will mainly investigate superior auditory localisation and movement perception abilities in the blind. Beyond state of the art neuroscientific investigation of structural and functional changes in cortical and subcortical networks and how they relate to superior auditory localisation in the congenitally blind. UM will actively collaborate with the other partners on real-time analysis and sonification of non-verbal full-body movement, emotion detection, use-case design, development of the DANCE platform, and integration of its components.

	Kungliga Tekniska Hoegskolan (KTH)
	KTH accounts for one-third of Sweden’s technical research and engineering education capacity at university level. Education and research cover a broad spectrum, from natural sciences to all the branches of engineering as well as architecture, industrial engineering and management, urban planning, work science and environmental engineering. In addition to the research carried out by the KTH Schools, a large number of both national and local Competence Centres are located at KTH. Various research foundations also finance a number of research programmes. The project will be carried out by researchers at the Dept. of Media Technology and Interaction Design (MID), School of Computer Science and Communication (CSC), one of the 10 schools which constitute KTH. MID has it´s roots in computer- and behavioural science, graphic technique and media technology, and has today faculty that represents e.g., anthropology nd psychology, computer science and media technology, sound and music computing, interaction design, film- and literature science, and media- and communication science. KTH will mainly conduct research on effective mappings between physical gestures and sound parameters to support feedback and understanding. KTH will actively collaborate with the other partners on real-time analysis and sonification of non-verbal full-body movement, emotion detection, use-case design, development of the DANCE platform, and integration of its components.

DANCE EU ICT H2020 Project

Multimodal Dataset of Lightness and Fragility

Task

Setup Hardware and Data

Segmentation

Ranking

Thank you!

Expressive Vocabulary

Setup Hardware and Data

Expressive Qualities

Lightness

Fragility

Suspension

Transmission

Figure

Multimodal dataset of full-body movement qualities: Lightness and Fragility

Task

Setup Hardware and Data

The Data Segments

Ranking

Video fragments with SMPTE

CHI 2018 - Workshop

DANCE Platform version 2

Download and install EyesWeb XMI

2. Download the DANCE example tools and patches

a) Tools and patches for recording and playing back multimodal data

Video recorder tool (download installer) [Warning: a Decklink frame grabber is required to roon this tool]

Kinect recorder tool (download installer) [Warning: a Kinect 2 is needed to run the tool]

Audio recorder tool (download installer) [Warning: an ASIO compatible sound card is needed to run the tool]

IMU recorder tool (download installer) [Warning: one or more X-OSC sensors are needed to run the tool]

Myo recorder tool (download installer) [Warning: 2 Myo sensors are needed to run the tool]

Playback and quality extraction tools (download installer)

Playback

Quality extraction

DANCE Platform version 3

Download

Installation

DANCE Platform version 1

1. Download and install EyesWeb XMI

2. Download the DANCE example tools and patches

a) Tools and patches for recording and playing back multimodal data

Kinect recorder tool (download installer)

Audio recorder tool (download installer)

IMU recorder tool (download installer)

Recorder tools options panel

Playback patch

Patches for analyzing multimodal data

Patches computing features and analysis primitives from IMUs

Patches computing features and analysis primitives from motion capture data

3. Download the IMU and motion capture sample data

4. Run EyesWeb XMI, load one or more patches and execute them

The DANCE Project

Facebook

L'Atlante del Gesto_Genova @atlantedelgestoGenova

DANCE EU ICT H2020 Project videos

Consortium

Università degli Studi di Genova (UNIGE)

University of Maastricht (UM)

Kungliga Tekniska Hoegskolan (KTH)