Other exploitation areas

From MIReS

Latest revision as of 18:23, 20 April 2013

MIR can be used in settings outside of music distribution and creation, for example in musicology, digital libraries, education and eHealth. In computational musicology, MIR tools have become standard "tools of the trade" for a new generation of empirical musicologists. Likewise, MIR technology is used for content navigation, visualisation, and retrieval in digital music libraries. MIR also shows promise for educational applications, including music appreciation, instrument learning, theory and ear training, although many current applications are still at an experimental stage. eHealth (healthcare practice supported by electronic processes) is also starting to benefit from MIR. Thus, the challenge is to better exploit MIR technologies in order to produce useful applications for other fields of research and practice. For this, current practices and needs from the related communities should be carefully studied. The stakeholders include music professionals, musicologists, music students, music teachers, digital librarians, medical doctors and medical doctors and patients who can benefit from music therapy.

Back to → Roadmap:Exploitation perspective

State of the art

We review here the already existing and potential relations between MIR and musicology, digital libraries, education and eHealth, which we identified as particularly relevant for our field of research.

Applications in musicology

The use of technology in music research has a long history (e.g. see [Goebl et al., 2008] for a review of measurement techniques in music performance research). Before MIR tools became available, music analysis was often performed with hardware or software created for other purposes, such as audio editors or speech analysis tools. For example, Repp used software to display the time-domain audio signal, and he read the onset times from this display, using audio playback of short segments to resolve uncertainties [Repp, 1992]. This methodology required a large amount of human intervention in order to obtain sufficiently accurate data for the study of performance interpretation, limiting the size and number of studies that could be undertaken.

For larger scale and quantitative studies, automatic analysis techniques are necessary. An example application of MIR to music analysis is the beat tracking system BeatRoot [Dixon, 2001], which has been used in studies of expressive timing [Widmer et al., 2003; [Grachten et al., 2009; Flossmann et al., 2009]. The SALAMI (Structural Analysis of Large Amounts of Music Information) project is another example of facilitation of large-scale computational musicology through MIR-based tools. A general framework for visualisation and annotation of musical recordings is Sonic Visualiser [Cannam et al., 2010], which has an extensible architecture with analysis algorithms supplied by plug-ins. Such audio analysis systems are becoming part of the standard tools employed by empirical musicologists [Leech-Wilkinson, 2009; Cook, 2004; Cook, 2007], although there are still limitations on the aspects of the music that can be reliably extracted, with details such as tone duration, articulation and the use of the pedals on the piano being considered beyond the scope of current algorithms [McAdams et al., 2004]. Other software such as GRM Acousmographe, IRCAM Audiosculpt [Bogaards et al., 2008], Praat [Boersma and Weenink, 2006] and the MIRtoolbox, which supports the extraction of high-level descriptors suitable for systematic musicology applications, are also commonly used. For analysing musical scores, the Humdrum toolkit [Huron, 1999] has been used extensively. It is based on the UNIX operating system’s model of providing a large set of simple tools which can be combined to produce arbitrarily complex operations. Recently, music21 [Cuthbert and Ariza, 2010] has provided a more contemporary toolkit, based on the Python programming language.

Applications in digital library

A digital library (DL) is a professionally curated collection of digital resources, which might include audio, video, scores and books, usually accessed remotely via a computer network. Digital libraries provide software services for management and access to their content.

Music Digital Librarians were among the instigators of the ISMIR community, and the first ISMIR conference (2000) had a strong DL focus. Likewise the contributions from the MIR community to DL conferences (Joint Conference on Digital Libraries, ACM Conference on Digital Libraries, IEEE-CS Conference on Advances in Digital Libraries) were numerous. This could be due to the fact that at the end of 90s, musical libraries moved to digitisation of recordings and to multi-information access (video, score images, and text documents such as biographies and reviews) to create multimedia libraries [e.g. Fingerhut, 1999; Dunn and Isaacson, 2001; McPherson and Bainbridge, 2001]. In this first trend, the technological aspects of these libraries relied mainly on the server, database, media digitisation, text search, and synchronisation (often manual) between media. Today this trend still exists and is accessible online for a wide audience. Examples of this are the "Live TV" of the Cite de la Musique (large audience) with synchronisation of video concerts with libretto, scores and comments.

A second trend, that appeared in the mid-2000s, reverses the relationship between Libraries and Music Information Research and Technology. Research and technology enable content estimation, visualisation, search and synchronisation, which are then used in the context of Digital Libraries to improve the usability and access of the multi-documents in libraries (online or not). Examples of this are: inclusion of automatic audio summaries in the IRCAM Library [Mislin et al., 2005], the Bachotheque to compare automatically synchronised interpretations of a same piece [Soulez et al., 2003], optical score recognition and audio alignment for the Bavarian State Library [Damm et al., 2011]. Also, thanks to the development of technologies (Flash, html5, Java-Script), the de-serialisation of media becomes a major theme, along with improved browsing and access to the temporal aspect of media. New concepts of interfaces to enhance listening have been developed which make use of time-based musical annotations (Ecoute augmentee/Increased-listening, or today’s SoundCloud).

A third trend concerns the aggregation of content and the use of user-generated annotation. The content of dedicated libraries can be aggregated to form meta-libraries (e.g. www.musiquecontemporaine.fr) using the shared protocol OAI-PMH. Content can be distributed over the web or aggregated to a local collection. Using Semantic Web technologies such as Linked Data and ontologies, web content can be re-purposed (e.g. the BBC’s use of the Music Ontology). This trend is also found in new forms of music access (such as Spotify) which aggregate content related to the music item (AMG reviews, wikipedia artist biography).

Comparing the suggestions of [Bonardi, 2000] and the observations of [Barthet and Dixon, 2011] a decade later, it is clear that much work is still to be done before MIR technology is fully incorporated into traditional libraries. The European ASSETS project, working with the Europeana multi-lingual European cultural collection, aims to improve search and browsing access to the collection, including multimedia objects.

Applications in education

[Dittmar et al., 2012] partitions MIR methods that are utilised in music education into three categories: music transcription, solo and accompaniment track creation, and generation of performance instructions. Systems for music education that exploit MIR technology include video games, music education software (e.g. Songs2See), and music-related apps focused on learning (e.g. Rock Prodigy). Regarding instrument learning scenarios, MIR is also seeing an uptake via provision of feedback to learners in the absence of a teacher [Wang and Zhang, 2008], interactive ear training exercises (e.g. the Karajan iPhone apps), automatic accompaniment [Dannenberg and Raphael, 2006], page turning [Arzt et al., 2008] and enhanced listening (e.g. iNotes: Orchestral Performance Companion). Research projects focused on music education include IMUTUS (Interactive Music Tuition System), i-Maestro (Interactive Multimedia Environment for Technology Enhanced Music Education and Creative Collaborative Composition and Performance) and M4M (Musicology for the Masses) – for the latter, a web-based tool called Yanno, based on automatic chord detection, was proposed for secondary school music classes. Although much work still remains to be done, large-scale experiments have already taken place, such as the IRCAM Music Lab 1 and 2 for the French National Education.

Education is one of the most understudied and yet promising application domains for MIR. While Piaget’s constructivism and Papert's constructionism are classics of pedagogy and interaction design relating to children, mashup, remix and recycling of contents might be considered a much more controversial and radical approach, especially for their social, ethical and legal implications. However, it is undeniable that young people are embracing remix en masse, and it is integral to how they make things and express ideas. The cultural practices of mashup and remix brought to school, will force us to rethink the role of teachers as part of this knowledge-building process (Erstad, 2008). The development of learning strategies that support such models of creation represents an ongoing challenge as it defies the current model of schooling, with students taking a more active role in developing knowledge. The introduction of MIR-powered tools for musical education and creation among younger children, combined with recent developments in portable devices, opens a new line of research for suitable novel interfaces and applications.

Applications in eHealth (healthcare practice supported by electronic processes)

Use of music information research for eHealth is still in its infancy. Its main use to date has been in music therapy, where it has been employed for quantitative analysis of therapy sessions and selection of musical material appropriate to the user’s ability and taste.

Music information technologies have traditionally been used to characterise one’s musical preferences for applications such as music retrieval or recommendation (see for example the Musical Avatar of [Bogdanov et al., 2013]). Moreover, there has been much research on technologies for affective analysis of music, e.g. on music emotion characterisation. These technologies have a great potential for contributing to music therapy, e.g. providing personalised music tools. For instance, according to E. Bigand, advances in cognitive neurosciences of music have revealed the potential importance of music for brain and cognitive stimulation [Bigand, 2012]. At this ISMIR 2012 keynote speech, he referred to some examples of the relationship between music technologies and cognitive stimulation (e.g. "Happy Neuron" project). Systems making use of MIR for music therapy have already been proposed inside the MIR community, e.g. the work by the team led by Ye Wang at the National University of Singapore [Zhao et al., 2010; Li et al., 2010]. In [Zhao et al., 2010], an MIR system is used to automatically recommend music for users according to their sleep quality in the goal of improving their sleep. In [Li et al., 2010] an MIR system that incorporates tempo, cultural, and beat strength features is proposed to help music therapists to provide appropriate music for gait training for Parkinson’s patients. The Mogat system of [Li et al., 2010] is used to help cochlear implant recipients, especially pre-lingually deafened children. In this system, three musical games on mobile devices are used to train their pitch perception and intonation skills, and a cloud-based web service allows music therapists to monitor and design individual training for the children.

References

• [Arzt et al., 2008] A. Arzt, G. Widmer, and S. Dixon. Automatic page turning for musicians via real-time machine listening. In Proceedings of the 18th European Conference on Artificial Intelligence, pp. 241-245, Patras, Greece, 2008.

• [Barthet and Dixon, 2011] M. Barthet and S. Dixon. Ethnographic observations of musicologists at the British library: Implications for music information retrieval. In Proceedings of the 12th International Society for Music Information Retrieval Conference, Miami, Florida, USA, 2011.

• [Bigand, 2012] E. Bigand. Cognitive estimulation with music and new technologies. In Proceedings of the 13th International Society for Music Information Retrieval Conference, Keynote speech, Porto, Portugal, 2012.

• [Boersma and Weenink, 2006] P. Boersma and D. Weenink. Praat: Doing phonetics by computer, 2006.

• [Bogaards et al., 2008] N. Bogaards, C. Yeh, and J. Burred. Introducing ASAnnotation: A Tool for Sound Analysis and Annotation. In Proceedings of the International Computer Music Conference, Belfast, Northern Ireland, 2008.

• [Bogdanov et al., 2013] D. Bogdanov, M. Haro, F. Fuhrmann, A. Xambó, E. Gómez, and P. Herrera. Semantic audio content-based music recommendation and visualization based on user preference examples. Information Processing & Management, 49: 13-33, 2013.

• [Bonardi, 2000] A Bonardi. IR for Contemporary Music: What the Musicologist Needs. In Proceedings of the International Symposium on Music Information Retrieval, Plymouth, Massachusetts, USA, 2000.

• [Cannam et al., 2010] C. Cannam, C. Landone, and M. Sandler. Sonic Visualiser: an open source application for viewing, analysing, and annotating music audio files. In Proceedings of the ACM Multimedia International Conference, pp. 1467-1468, Florence, Italy, 2010.

• [Cook, 2004] N. Cook. Computational and comparative musicology. In: E. Clarke and N. Cook, editors, Empirical Musicology: Aims, Methods, and Prospects, pp. 103-126. Oxford University Press, New York, 2004.

• [Cook, 2007] N. Cook. Performance analysis and Chopin's mazurkas. Musicae Scientae, 11(2): 183-205, 2007.

• [Cuthbert and Ariza, 2010] M.S. Cuthbert and C. Ariza. music21: A toolkit for computer-aided musicology and symbolic music data. In Proceedings of the 11th International Society for Music Information Retrieval Conference, pp. 637-642, Utrecht, Netherlands, 2010.

• [Damm et al., 2011] D Damm, Ch. Fremerey, V. Thomas, and M. Clausen. A Demonstration of the Probado Music System. In Proceedings of the 12th International Society for Music Information Retrieval Conference, Late-breaking session, Miami, Florida, USA, 2011.

• [Dannenberg and Raphael, 2006] R.B. Dannenberg and C. Raphael. Music score alignment and computer accompaniment. Communications of the ACM, 49(8): 38-43, 2006.

• [Dittmar et al., 2012] C. Dittmar, E. Cano, J. Abesser, and S. Grollmisch. Music information retrieval meets music education. In: M. Müller, M. Goto, and M. Schedl, editors, Multimodal Music Processing, volume 3 of Dagstuhl Follow-Ups, pp. 95-120. Dagstuhl Publishing, 2012.

• [Dixon, 2001] S. Dixon. Automatic extraction of tempo and beat from expressive performances. Journal of New Music Research, 30(1): 39-58, 2001.

• [Dunn and Isaacson, 2001] Jon W. Dunn and Eric J. Isaacson. Indiana university digital music library project. In Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries, 2001.

• [Fingerhut, 1999] Michael Fingerhut. The IRCAM Multimedia Library: a Digital Music Library. In IEEE Forum on Research and Technology Advances in Digital Libraries (IEEE ADL'99), Baltimore, USA, 1999.

• [Flossmann et al., 2009] S. Flossmann, W. Goebl, and G. Widmer. Maintaining skill across the life span: Magaloff's entire Chopin at age 77. In Proceedings of the International Symposium on Performance Science, pp. 119-€“124, Auckland, New Zealand, 2009.

• [Goebl et al., 2008] W. Goebl, S. Dixon, G. De Poli, A. Friberg, R. Bresin, and G. Widmer. Sense in expressive music performance: data acquisition, computational studies, and models. In P. Polotti and D. Rocchesso, editors, Sound to Sense - Sense to Sound: A State of the Art in Sound and Music Computing, pp. 195-242. Logos Verlag, Berlin, 2008.

• [Grachten et al., 2009] M. Grachten, W. Goebl, S. Flossmann, and G. Widmer. Phase-€plane representation and visualization of gestural structure in expressive timing. Journal of New Music Research, 38(2): 183-€“195, 2009.

• [Huron, 1999] D. Huron. Music Research Using Humdrum: A User's Guide. Center for Computer Assisted Research in the Humanities. Stanford, California, 1999.

• [Leech-Wilkinson, 2009] D. Leech-Wilkinson. The Changing Sound of Music: Approaches to Studying Recorded Musical Performance. CHARM, London, 2009.

• [Li et al., 2010] Z. Li, Q. Xiang, J. Hockman, J. Yang, Y. Yi, I. Fujinaga, and Y. Wang. A music search engine for therapeutic gait training. In Proceedings of the ACM Multimedia, pp. 627-630, Florence, Italy, 2010.

• [McAdams et al., 2004] S. McAdams, P. Depalle, and E. Clarke. Analyzing musical sound. In: E. Clarke and N. Cook, editors, Empirical Musicology: Aims, Methods, and Prospects, pp. 157-196. Oxford University Press, New York, 2004.

• [McPherson and Bainbridge, 2001] John R. McPherson and David Bainbridge. Usage of the MELDEX digital music library. In Proceedings of the 2nd International Symposium on Music Information Retrieval, Bloomington, Indiana, USA, 2001.

• [Mislin et al., 2005] F. Mislin, Michael Fingerhut, and Geoffroy Peeters. Automatisation de la production et de la mise en ligne de resumes sonores. Master's thesis, ISTY, 2005.

• [Repp, 1992] B.H. Repp. Diversity and commonality in music performance: An analysis of timing microstructure in Schumann's "Träumerei". Journal of the Acoustical Society of America, 95(5): 2546-2568, 1992.

• [Soulez et al., 2003] Ferréol Soulez, Xavier Rodet, and Diemo Schwarz. Improving polyphonic and poly-instrumental music to score alignment. In Proceedings of the 4th International Conference on Music Information Retrieval, Baltimore, Maryland, USA, 2003.

• [Wang and Zhang, 2008] Y. Wang and B. Zhang. Application-specific music transcription for tutoring. IEEE MultiMedia, 15(3): 70-€“74, 2008.

• [Widmer et al., 2003] G. Widmer, S. Dixon, W. Goebl, E. Pampalk, and A. Tobudic. In search of the Horowitz factor. AI Magazine, 24(3): 111-130, 2003.

• [Zhao et al., 2010] Wei Zhao, , Xinxi Wang, and Ye Wang. Automated sleep quality measurement using EEG signal. In Proceedings of the ACM Multimedia, Florence, Italy, 2010.

Challenges

• Produce descriptive content analysis tools based on concepts used by musicologists. Current MIR tools do not fit many of the needs of musicologists, partly due to their limited scope, and partly due to their limited accuracy. To fill the acknowledged gap between the relatively low-level concepts used in MIR and the concepts of higher levels of abstraction central to music theory and musicology, will call for, on the one hand, the development of better algorithms for estimating high level concepts from the signal, and on the other hand, the proper handling of errors and confidence in such estimation.

• Overcome barriers to uptake of technology in music pedagogy. Generic tutoring applications do not engage the user, because they ignore the essential fact that users have widely varying musical tastes and interests, and that the drawing power of music is related to this personal experience. User modelling or personalisation of MIR systems is an open challenge not just for tutoring but for all MIR applications. Another issue is that MIR technology is currently not mature or efficient enough for many educational applications, such as those involving real-time processing of multi-instrument polyphonic music. Further research is required in topics such as polyphonic transcription, instrument identification and source separation, and in the integration of these techniques, in order to develop more elaborate music education tools than currently exist.

• Provide diagnosis, analysis and assessment of music performance at any level of expertise. A further barrier to uptake is that current music education tools have shallow models of music making (e.g. focusing only on playing the correct notes), and fail to give meaningful feedback to learners or assist in the development of real musical skills. More advanced tools will need to aid learners in areas such as phrasing, expressive timing, dynamics, articulation and tone.

• Develop visualisation tools for music appreciation. The listening experience can be enhanced via visualisations, but it is a challenge to provide meaningful visualisations, for example those which elucidate structure, expression and harmony, which inform and stimulate the listener to engage with the music.

• Facilitate seamless access to distributed music data. In order to satisfy information needs and promote the discovery of hidden content in digital music libraries, it is necessary to provide better integration of distributed data (content and meta-data, regardless of location and format) through the use of standards facilitating interoperability, unified portals for data access, and better inter-connections between institutional meta-data repositories, public and private archive collections, and other content. Open source tools for indexing, linking and aggregation of data will be particularly important in achieving this goal.

• Expand the scope of MIR applications in eHealth. Some preliminary work has demonstrated the value of MIR technologies in eHealth, for example to assist health professionals in selecting appropriate music for therapy. However, the full use of MIR in medicine still needs to be deeply explored, and its scope expanded within and beyond music therapy.

Back to → Roadmap:Exploitation perspective