Roadmap
From MIReS
Please cite this document as:
Xavier Serra, Michela Magas, Emmanouil Benetos, Magdalena Chudy, Simon Dixon, Arthur Flexer, Emilia Gómez, Fabien Gouyon, Perfecto Herrera, Sergi Jordà, Oscar Paytuvi, Geoffroy Peeters, Jan Schlüter, Hugues Vinet, Gerhard Widmer, "Roadmap for Music Information ReSearch", Geoffroy Peeters (editor), 2013, Creative Commons BY-NC-ND 3.0 license, ISBN: 978-2-9540351-1-6
Contents |
Introduction
For the purpose of this Roadmap we consider Music Information Research (MIR) as a field that covers all the research topics involved in the understanding and modelling of music and that use information processing methodologies. We view this research to be very much within the field of Information Technologies thus with the major aim of developing methods and technologies with which to process musically relevant data and develop products and services with which to create, distribute and interact with music information.
This Roadmap aims at identifying research challenges of relevance to the MIR community. The document should also be of relevance to the policy makers that need to understand the field of information technologies, identifying the state of the art in music information technology research and the relevant research problems that should be worked on.
The Roadmap has been elaborated by the researchers involved in the MIReS project. They have been responsible to get input and feedback from the different MIR stakeholders. Many events have been organised to gather information from experts throughout the elaboration of the Roadmap. This document is the result of a very participative process in which many people have been involved. The task of the writers of the document has mainly been an editorial one, trying to capture and summarise what the different stakeholders expressed as being relevant for the future of the field.
There have been some European initiatives with similar aims. It is specially relevant to mention the S2S2 Coordination Action, within which a Roadmap was elaborated and published in 2007. That Roadmap covered the whole Sound and Music Computing field, thus it had a broader perspective than MIR and it also went beyond research issues. A more recent and also relevant document has been the discussion paper entitled "Musicology (Re-) Mapped" promoted by the European Science Foundation and published in 2012. This paper covered the field of Musicology from a modern perspective and discussed some of the current research trends, some of which overlap with MIR. It is also relevant to mention the report on the 3rd CHORUS+ Think-Tank that took place at MIDEM 2011 and addressed the future of music search, access and consumption from an industrial perspective.
This Roadmap focuses on presenting and discussing research challenges, thus it does not aim to cover organisational, industrial, or educational aspects. No attempt is made to predict the future of research in MIR; we believe that this is not possible. The challenges have been identified by studying and using the current state of the art in MIR and related disciplines. We are very much aware that many of the great technological and scientific discoveries result from disruptive changes and developments, and these are impossible to predict using this approach.
The challenges have been grouped into four sections, each one reflecting a different emphasis and perspective: technological, user, sociocultural, and exploitation. The technological perspective is the more traditional one used in MIR, reflecting the core scientific and technical challenges. The other three sections aim to examine the field from non-traditional perspectives, thus emphasising important, though often ignored views, which can give us important insights into our research.
Technological perspective
Music Information Research focuses on the processing of digital data related to music. This includes gathering and organisation of machine-readable musical data, development of data representations, and methodologies to process and understand that data, taking into account domain knowledge and bringing expertise from relevant scientific and engineering disciplines.
Musically relevant data
We define "musically relevant data" as any type of machine-readable data that can be analysed by algorithms and that can give us relevant information for the development of musical applications. The main challenge is to gather musically relevant data of sufficient quantity and quality to enable music information research that respects the broad multi-modality of music. After all, music today is an all-encompassing experience that is an important part of videos, computer games, Web applications, mobile apps and services, specialised blogs, artistic applications, etc. Therefore we should be concerned with the identification of all sources of musically relevant data, the proper documentation of the process of data assembly and resolving of all legal and ethical issues concerning the data. Sufficient quantity and quality of data is of course the prerequisite for any kind of music information research. To make progress in this direction it is necessary that the research community works together with the owners of data, be they copyright holders in the form of companies or individual persons sharing their data. Since music information research is by definition a data intensive science, any progress in these directions will have immediate impact on the field. It will enable a fostering and maturing of our research and, with the availability of new kinds of musically relevant data, open up possibilities for new kinds of research and applications.
Music representations
Data representations impact the effectiveness of MIR systems in two ways: algorithms are limited by the types of input data they receive, and the user experience depends on the way that MIR systems present music information to the user. A major challenge is to provide abstractions which enable researchers and industry to develop algorithms that meet user needs and to present music information in a form that accords with users’ understanding of music. The same challenge applies to content providers, who need to select appropriate abstractions for structuring, visualising, and sonifying music information. These abstractions include features describing music information, be it audio, symbolic, textual, or image data; ontologies, taxonomies and folksonomies for structuring music information; graphical representations of music information; and formats for maintaining and sonifying music data. The development of standard representations will advance MIR by increasing algorithm and system interoperability between academia and industry as well as between researchers working on MIR subtasks, and will provide a satisfactory user experience by means of musically and semantically meaningful representations.
Data processing methodologies
Since its origins, the MIR community has used and adapted data processing methodologies from related research fields like speech processing, text information retrieval, and computer vision. A natural consequential challenge is to more systematically identify potentially relevant methodologies from data processing disciplines and stay up-to-date with their latest developments. This exchange of data processing methodologies reduces duplication of research efforts, and exploits synergies between disciplines which are, at a more abstract level, dealing with similar data processing problems. It will become even more relevant as MIR embraces the full multi-modality of music and its full complexity as a cultural phenomenon. This requires a regular involvement and commitment of researchers from diverse fields of science as well as an effort of communication across disciplines, and possibly even the formulation of joint research agendas. Such a more organised form of exchange of methodologies is likely to have a boosting impact on all participating disciplines due to the joining of forces and combined effort.
Knowledge-driven methodologies
For a long time, the MIR community has been focusing on a range of bottom-up approaches, addressing the kinds of data we use and the types of algorithms we apply to it. A major challenge is to complement this focus and explore other methodologies and fields of science which approach music in a more integrated way. After all, music information research is just one of many sciences that centre on and care about music, which include musicology, psychology, sociology and neuroscience. Over decades of research, each of these fields has aggregated knowledge concerning music which can inform the process of music information research. The focus here is on gaining domain knowledge from outside of MIR as opposed to borrowing methodologies or algorithms. This will require that researchers from different disciplines engage in a dialogue on all aspects of music. The potential impact is that all participating disciplines benefit from the diverse and differing views on the phenomenon of music, in all its aspects and forms.
By musical concept extraction we refer to the estimation of the elements of a notation system from the audio signal and the estimation of higher-level semantic information from these elements. These elements belong to a vocabulary and are assembled according to a grammar specific to a culture. The challenge here is to automatically derive musical concepts from audio signals or from commonly available symbolic data, such as MIDI or scores. Extracting musical concepts from audio signals is technically a very difficult task and new ways to perform this still need to be found. More specifically, we need to develop better source separation algorithms; develop methodologies for joint estimation of music content parameters; and use symbolic information plus audio data to extract higher level semantic concepts. For this reason, this challenge does not only involve researchers (in signal processing, machine learning, cognition and perception, and musicology), but also content providers (record companies), who could help by delivering material for research (such as the separate audio tracks of multi-track recordings). Enabling the description of data in terms of musical concepts can help improve the understanding of the content and hence develop better use or new uses of this content. Having access to separate source elements or to accurate score information may have a huge impact on the creative industries (game industry, music e-learning, ...) and the music distribution industry (better access to music), as well as facilitating large-scale musicological analyses.
Evaluation methodologies
It is paramount to MIR that independent researchers build upon previous research, and an overarching challenge in MIR is to define and implement research evaluation methodologies that effectively contribute to creation of knowledge and general improvements in the field. In many scientific disciplines dealing with data processing, significant improvements over the long term have been achieved by empirically defining evaluation methodologies via several iterations of an experimental "loop" including formalisation, implementation, experimentation, and finally validity analysis. In MIR, evaluation initiatives have played an increasing role in the last 10 years, and the community is presently facing the validity analysis issue: that is, finding the most appropriate way to build upon its own legacy and redefine the evaluation methodologies that will better lead to future improvements, the resolution of which will in turn entail further technical challenges down the line (i.e., down the "loop"). This will require above all a deeper involvement of more MIR researchers in the very definition of the evaluation methodologies, as they are the individuals with the best understanding of relevant computational issues. Importantly, this will also require the involvement of the music industry (via e.g. proposing evaluations of relevance to them), and content providers (in order for researchers to have access to data). Effective MIR evaluations will impact in a fundamental manner the very way MIR research is done, it will positively affect the width and depth of the MIR research, and it will increase the relevance of MIR to other research fields.
User perspective
Music Information Research considers the user perspective, both in order to understand the user roles within the music communication chain and to develop technologies for the interaction of these users with music data. MIR aims to capture, process and model the data gathered through user interaction and develop methodologies for the design of new musical devices in order to enable new interaction possibilities between users and these devices.
User behaviour
Music is listened to, performed and created by people. It is therefore essential to consider the user as central to the creation of user scenarios, hence to the development of technologies. Developing user applications involves analysing the user needs in respect of novel scenarios and the user behaviour in respect of existing ones, thus enabling the creation of the user-specification-development loop. Taking into account user needs applies to all stages of the development loop, however the analysis of user behaviour must be carefully conducted by a specialist. Gathering feedback from users is a research field in itself and shouldn’t be done without carefully designed methods. Considering user needs through the analysis of user behaviour will have a great impact on the usability of the developed MIR technologies.
User interaction
The grand challenge of user interaction is how to design MIR systems that put the user at the centre of the system. This applies to the whole interaction loop, including visualisation, input devices, manipulation metaphors, and also system adaptation to user behaviour. This challenge is relevant because it contributes to both the user’s and to the researcher’s (e.g. system designer’s) understanding of the system’s features and components, the overall purpose of the system, and the contribution the system can make to the user’s activities. The benefit to users is more productive workflows and systems which better serve the users’ needs. The researchers stand to benefit from the feedback loop which enables them to fine-tune and develop systems with greater accuracy. Effective user-oriented research will have a major impact on the usability of MIR systems and their wider deployment.
Socio-cultural perspective
Music Information Research involves the understanding and modeling of music-related data in its full contextual complexity. Music is a communication phenomenon that involves people and communities immersed in specific social and cultural context. MIR aims at processing musical data that captures the social and cultural context and at developing data processing methodologies with which to model the whole musical phenomenon.
Music is a social phenomenon, thus its understanding and modelling requires the inclusion of this dimension. Social interaction is a driving force of music listening, categorisation, preference, purchasing behaviour, etc. Additionally, teams or crowds are usually able to achieve feats that go beyond what individuals accomplish, and this is especially relevant for annotation and other collaborative scenarios. Finally, scattered in different virtual places, formats and time-scales, there is much data available that contains implicit information about music-related social factors, which could make possible the understanding and prediction of trends and other collective behaviours related to music. To carry out this research, which would complement the other, more traditional, approaches to music description, we need to involve people working in Social Computing, Sociologists and experts in Dynamic Systems and Complex Networks. Social computing will help to gather massive annotations and obtain knowledge on the key actors and factors of collective-mediated processes in musical choice, interaction and conceptualisation. Human dynamics will make possible massive-scale predictions about trends, ways and moments to listen to music, and provide pointers to the best locations and conditions for commercial and marketing activities (e.g., Buzzdeck). The main obstacle to promising advances is the scarcity of open data and the privacy issues associated with access to data. Contrastingly, there are also issues to be solved when managing and analysing extremely large amounts of data of this kind.
Multiculturality
Most music makes very little sense unless we experience it in its proper cultural context, thus the processing of music information has to take into account this cultural context. Most of MIR has focused on the mainstream popular Western music of the past few decades and thus most research results and technologies have a cultural bias towards that particular cultural context. The challenge is to open up our view on music, to develop technologies that take into account the existing musical diversity and thus the diverse musical cultural contexts. To approach the multicultural aspects of MIR there is a need to involve researchers from both engineering disciplines (Signal Processing, Machine Learning) and humanities (Musicology, Cultural Studies), and to involve people belonging to the specific cultures being studied. This approach will offer the possibility to identify new MIR problems and methodologies that could impact the whole MIR field. At the same time the development of Information Technologies that reflect diversity should help preserve the cultural richness of our world, which is threatened by the globalisation and homogenisation of the IT infrastructures. This is a topic that has started to be addressed by the MIR community but that will require much bigger efforts, not just by the research community but by political and industrial bodies.
Exploitation perspective
Music Information Research is relevant for producing exploitable technologies for organising, discovering, retrieving, delivering, and tracking information related to music. These technologies should enable improved user experience and commercially viable applications and services for digital media stakeholders.
Music distribution applications
MIR is fundamental for developing technologies to be used in the music distribution ecosystem. The stakeholders in the music value chain are music services, record companies, performing rights organisations, music tech companies, music device and equipment manufacturers, and mobile carriers. The main challenge is to develop scalable technologies that are relevant to both the services that organise and distribute the music and also those services that track what is being distributed. These technologies span from music search and recommendation to audio identification both for recordings and compositions among others. By fully addressing the music distribution challenges, the MIR Community will establish closer ties with the industry which will help accessing resources (such as actual music data) and alternative ways of funding. On its side, the Music Distribution industry will have access to technologies more targeted to actual end-user scenarios which will give them an edge in the global market. Incidentally, it will help reducing innovation cycles from research to development and exploitation which, in turn, will have a clear impact on competitiveness and hence music distribution companies’ profitability.
Creative tools
Creative practitioners produce, transform and reuse music materials. The MIR challenge is how to develop tools that process music information in a way that enhances creative production. Tools for automatically extracting relevant information from audio materials could be developed for purposes such as content-based manipulation, generativity, synchronisation with other media, or real-time processing. Moreover, the large volume of available data requires efficient data manipulation systems that enable new methods of manipulation for creative purposes. This challenge requires collaborative research between music information researchers and the actors, including artists, performers and creative industries professionals. The impact of this research is in generating more creative possibilities and enabling production efficiency in a variety of creative contexts including music performance, music and sound production and post-production, sound engineering applications for audiovisual production, art installations, creative marketing, mobile apps, gaming, commercial installations, environmental installations, indoor and outdoor events. Collaborative research between creative practitioners and music information researchers contributes to bridging the gap between the arts and the sciences, and introduces novel practices and methodologies. It extends the paradigm of user-centred research to creative-input research, where the feedback loop between creative practitioner and researcher is an iterative, knowledge-building process, supported by adaptive modelling of research environments and resulting in more versatile creative tools.
Other exploitation areas
MIR can be used in settings outside of music distribution and creation, for example in musicology, digital libraries, education and eHealth. In computational musicology, MIR tools have become standard "tools of the trade" for a new generation of empirical musicologists. Likewise, MIR technology is used for content navigation, visualisation, and retrieval in digital music libraries. MIR also shows promise for educational applications, including music appreciation, instrument learning, theory and ear training, although many current applications are still at an experimental stage. eHealth (healthcare practice supported by electronic processes) is also starting to benefit from MIR. Thus, the challenge is to better exploit MIR technologies in order to produce useful applications for other fields of research and practice. For this, current practices and needs from the related communities should be carefully studied. The stakeholders include music professionals, musicologists, music students, music teachers, digital librarians, medical doctors and medical doctors and patients who can benefit from music therapy.
Conclusions
This document has been conceived in order to identify current opportunities and challenges within the field of Music Information Research. The aim has been to open up the current views that drive the MIR field by taking into account science, industry and society. A review of the state of the art of the field has been conducted and the challenges have been identified by involving a variety of stakeholders. The proposed challenges have great potential for future impact on both academia and industry. In addition to the scientific and engineering points of view, the challenges have focused on social and industrial perspectives, thus aligning the Roadmap with Horizon 2020, the new EU Framework Programme for Research and Innovation.
By involving a variety of experts and points of view we hope to have provided a useful overview of the interest and expanding scope of the MIR field. The open discussions that have been organised in diverse forums have already made a very positive impact upon the MIR community. From here on the success of this initiative will be reflected by the number of students and researchers that read and use this document to make decisions about the direction of their research, especially when deciding which research challenges to address. The proposed research challenges, as well as the knowledge and the network built during the coordination process should also be relevant for policy makers, facilitating future gazing and the establishment of key Music Information Research funding strategies.
