Knowledge-driven methodologies

From MIReS

For a long time, the MIR community has been focusing on a range of bottom-up approaches, addressing the kinds of data we use and the types of algorithms we apply to it. A major challenge is to complement this focus and explore other methodologies and fields of science which approach music in a more integrated way. After all, music information research is just one of many sciences that centre on and care about music, which include musicology, psychology, sociology and neuroscience. Over decades of research, each of these fields has aggregated knowledge concerning music which can inform the process of music information research. The focus here is on gaining domain knowledge from outside of MIR as opposed to borrowing methodologies or algorithms. This will require that researchers from different disciplines engage in a dialogue on all aspects of music. The potential impact is that all participating disciplines benefit from the diverse and differing views on the phenomenon of music, in all its aspects and forms.

Back to → Roadmap:Technological perspective

Contents 1 State of the art 1.1 Musicology 1.2 Psychology of music 1.3 Sociology of music 1.4 Neuroscience 1.5 References 2 Challenges

State of the art

In what follows, we briefly review the already existing and potential relations between MIR and musicology, psychology, sociology and neuroscience, which we identified as particularly relevant for our field of research.

Musicology

Musicology is fundamental to MIR, and building bridges between disciplines is at the very core of research in musicology [Dahlig-Turek et al., 2012]. Musicologists have taken an active role in the ISMIR community, for instance, musicology has always been considered a key topic in the ISMIR call for papers (see, e.g. research areas related to computational musicology, computational ethnomusicology explicitly considered at ISMIR 2012). Moreover, the conference on Interdisciplinary Musicology (CIM) has included papers on computational modelling in the program, and there is a special edition of this conference on the topic of "Technology" that is planned for CIM 2014. There are also some relevant journals in this intersection (e.g. Journal of Mathematics and Music) and the Special Issue on Computational Ethnomusicology in the Journal of New Music Research. An overview on the relationship between MIR and musicology is provided in the Musicology tutorial presented at ISMIR 2011 by [Volk and Wiering 2011], and a guide to the use of MIR technology in musicology is given in [Leech-Wilkinson, 2009].

Although musicological studies in MIR have traditionally focused on the symbolic domain, recent developments in music transcription and feature extraction technologies from audio signals have opened new research paths at the intersection of musicology and signal processing. Key research topics in this area have been, among others, melodic similarity, key estimation and chord tracking. Musicological and MIR research have been contrasted [1] in terms of, among others, data sources, repertoires and methodologies, and some opportunities for future research have been pointed out. MIR technologies can contribute with tools and data that are useful for musicological purposes, and Musicology can provide relevant research problems and use cases that can be addressed through MIR technologies. A mutual influence is starting to take place, although there is still a need for more collaboration between musicologists and technicians to create a truly interdisciplinary research area and contribute with truly music-rooted models and technologies. Only by this collaboration can we address the current gap between feature extractors and expert analyses and make significant contributions to existing application needs, e.g. version identification, plagiarism detection, music recommendation, and to study how the relationship between people and music changes with the use of technology (e.g. "Musicology for the Masses" project).

Psychology of music

Music is created and experienced by humans, and the ultimate goal of MIR is to produce results that are helpful and interesting for humans. Therefore it is only natural to care about how humans perceive and create music. Music psychology tries to explain both musical behavior and musical experience with psychological methods. Its main instrument therefore is careful experimentation involving human subjects engaged in some kind of musical activity. Research areas span the whole spectrum from perception to musical interaction in large groups. Research questions concern the perception of sound or sound patterns, as well as perception of more musically meaningful concepts like harmony, pitch, rhythm, melody and tonality. The emotions associated with personal music experience are a part of music psychology, as are personal musical preferences and how they are influenced through peer groups and family, and musical behaviors from dancing to instrument playing to the most sophisticated interaction within whole orchestras.

Therefore music psychology should be able to provide valuable knowledge for MIR researchers in a whole range of sub-fields. Indeed there already is a certain exchange of knowledge between music psychology and MIR. Just to give a few examples, Carol L. Krumhansl, an eminent figure in music psychology, was an invited speaker at the 11th International Society for Music Information Retrieval Conference (ISMIR 2010), in Utrecht, Netherlands talking about "Music and Cognition: Links at Many Levels". Her monograph on "cognitive foundations of musical pitch" [Krumhansl, 1990] is still seen as one of the standard texts on the subject. Gerhard Widmer, who has been an important contributor to MIR early on, was a keynote speaker at the 12th International Conference on Music Perception and Cognition (ICMPC), which is one of the most important conferences in the field of music psychology. At last year's major conference in the MIR field (ISMIR 2012) there was a joint presentation of an MIR researcher and a psychologist elaborating on the sometimes complicated dialog of the two disciplines [Aucouturier and Bigand, 2012, 2013].

Sociology of music

Social psychology and the sociology of music focus on individuals as members of groups and on how groups and shared cultural codes influence music-related attitudes and activities. This point of view allows one to ask and answer important questions like: How do individuals and groups use music? How is the collective production of music made possible? How does music relate to broader social distinctions, especially class, race, and gender?

Although it is evident that such a sociology of music should be able to provide important insights not only for the field of MIR, many authors have suggested that research over recent decades has largely ignored the social functions of music at the expense of its cognitive and emotional functions (see e.g. [Hargreaves and North, 1997]). [Hargreaves and North, 1999] concluded that music serves three social functions: it is used by individuals to help manage their moods, self-identity [DeNora, 2000], and interpersonal relationships. [North et al., 2000] elaborated this idea, showing that a sample of 13- to 14-year-olds listened to music to portray a social image to others, and to fulfil their emotional needs. Similarly, [Tarrant et al., 2000] showed that American and English adolescents listened to music to satisfy both emotional and social needs, as well as for reasons of self-actualisation. [Lonsdale and North, 2011] remarked that listening to music was "a social activity", which offered an opportunity for participants "to socialise with friends" (e.g., dancing, sharing live music). Music has a stronger social component for teenagers and young people than for seniors but it still keeps some powers to strengthen social bonds and to provide memory aids when brain functions decline. In this respect, life-span and elderly-centred applications are yet to be fully explored and exploited [Magee et al., 2011]. How MIR can benefit from these and other results concerning the sociology of music is still a largely open question which opens up new and promising areas of research.

Neuroscience

All music psychological questions raised above could of course also be examined with neuroscientific methods. Instead of measuring the subject's behavior in music psychological experiments or directly asking subjects about their experiences concerning music it is possible to measure various signals from the human brain during such experiments. Possible signals range from electro-encephalography (EEG) to magneto-encephalography (MEG) or functional magnetic resonance imaging (fMRI). Each of the signals has its own characteristic strengths and weaknesses. E.g. EEG has a very good temporal but poor spatial resolution where fMRI is just the opposite. No matter what brain signals are being used, the fundamental question is always what parts of the brain contribute in what way to a subject's experience or creation of music. It is not immediately clear what MIR could gain from such a knowledge about brain structures involved in perception and production of music that could go beyond knowledge obtained from psychological experiments not utilising neuroscientific methods. The biggest contribution might concern problems where humans have difficulty self-assessing their performance and experience. One example is the experience of emotions when listening to music. Neuroscientific methods might be able to provide a more quantitative and maybe more accurate picture than human self-assessment (see e.g. [Blood and Zatoore, 2001], [Schmidt and Trainor, 2001]). Differences in brain structure and function between skilled musicians and non-musicians is another well researched subject (see e.g. [Gaser and Schlaug, 2003], [Krings et al., 1999]). The same holds for the study of the neuronal processes during performance of music where the sensorimotor interplay is at the center of interest (see [Zatorre et al., 2007] for a recent review).

References

• [Aucouturier and Bigand, 2012] Jean-Julien Aucouturier and Emmanuel Bigand. Mel Cepstrum and Ann Ova: The Difficult Dialog Between MIR and Music Cognition. In: Fabien Gouyon, Perfecto Herrera, Luis Gustavo Martins, and Meinard Müller, editors, ISMIR 2012, pp. 397-402. FEUP Edições, 2012.

• [Aucouturier and Bigand, 2013] Jean-Julien Aucouturier and Emmanuel Bigand. Seven problems that keep MIR from attracting the interest of the natural sciences. Journal of Intelligent Information Systems, in print, 2013.

• [Blood and Zatoore, 2001] A.J. Blood, R.J. Zatorre. Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. In Proceedings of the National Academy of Sciences of the United States of America 98(20): 11818-11823, 2001.

• [Dahlig-Turek et al., 2012] R. Parncutt, F. Wiering, E. Dahlig-Turek, S. Klotz, editor. Musicology (Re-) Mapped. Standing Committee for the Humanities. European Science Foundation, 2012.

• [DeNora, 2000] T. DeNora. Music as a Technology of Self. Poetics, 27: 31-56, Elsevier, 1999.

• [Gaser and Schlaug, 2003] C. Gaser and G. Schlaug. Brain structures differ between musicians and non-musicians. The Journal of Neuroscience, 23(27): 9240-9245, 2003.

• [Hargreaves and North, 1997] D.J. Hargreaves, A.C. North, A. C. The social psychology of music. Oxford University Press, 1997.

• [Hargreaves and North, 1999] D.J. Hargreaves, A.C. North. The functions of music in everyday life: Redefining the social in music psychology. Psychology of Music, 27: 71-83, 1999.

• [Krings et al., 1999] T. Krings, R. Topper, H. Foltys, S. Erberich, R. Sparing, K. Willmes, A. Thron. Cortical activation patterns during complex motor tasks in piano players and control subjects. A functional magnetic resonance imaging study. Neuroscience Letters, 278 (3): 189-193, 1999.

• [Krumhansl, 1990] C.L. Krumhansl. Cognitive Foundations of Musical Pitch. Oxford University Press, USA, 1990.

• [Leech-Wilkinson, 2009] D. Leech-Wilkinson. The changing sound of music: Approaches to studying recorded musical performance, CHARM project, London, 2009.

• [Lonsdale and North, 2011] A.J. Londsdale, A.C. North, Why do we listen to music? A uses and gratifications analysis. British Journal of Psychology, 102: 108-134, 2011.

• [Magee et al., 2011] Wendy L. Magee, Michael Bertolami, Lorrie Kubicek, Marcia LaJoie, Lisa Martino, Adam Sankowski, Jennifer Townsend, Annette M. Whitehead-Pleaux, and Julie Buras Zigo. Using music technology in music therapy with populations across the life span in medical and educational programs. Music and Medicine, 3(3): 146-153, 2011.

• [North et al., 2000] A.C. North, D.J. Hargreaves, S.A. O'Neill. The importance of music to adolescents. British Journal of Educational Psychology, 70(2): 255-272, 2000.

• [Schmidt and Trainor, 2001] L.A. Schmidt, L.J. Trainor. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition and Emotion 15(4): 487-500, 2001.

• [Tarrant et al., 2000] M. Tarrant, A.C. North, D.J. Hargreaves, English and American adolescents' reasons for listening to music. Psychology of Music, 28: 166-173, 2000.

• [Zatorre et al., 2007] R.J. Zatorre, J.L. Chen, V.B. Penhune. When the brain plays music. Auditory-motor interactions in music perception and production. Nature Reviews Neuroscience, 8: 547-558, 2007.

Challenges

• Integrate insights from disciplines relevant to MIR and make them useful for our research. This requires mutual understanding and exchange of results and researchers. The challenge is to integrate research agendas through the formulation of common interests and goals as well as a common vocabulary and dedicated communication paths. This will be important for both MIR and all other disciplines caring about music since there is a mutual benefit to be gained from this.

• Develop richer musical models incorporating musicological knowledge. MIR has been focusing on a limited number of musical concepts, which are modelled at a shallower depth than they are treated by musicologists. Enriching these concepts will help bridge the gap between low-level MIR representations and higher-level semantic concepts.

• Extend and strengthen existing links to music psychology. An example for a joint interest is the clearer formulation and understanding of the notion of "music similarity" with the help of music psychological results and proper experimentation. This requires that music psychologists be informed about MIR models and methods to compute music similarity and that MIR researchers are being educated about how music psychologists access subjective notions and cognitive aspects of music similarity in humans. Expected outcomes are improved models and algorithms to compute music similarity as well as computer aided selection of research stimuli for the psychological experiments.

• Give due attention to the social function of music in our research. This makes it necessary that MIR cares about groups of individuals and their interaction instead of about disconnected individuals. Taste formation, preference and music selection are a combined function of personal and group variables, and we currently do not know how to weight both aspects to achieve good predictive models. Research and technologies that help to understand, modify, increase or make possible group cohesion, improvements on self-image, or strengthen collective bonds could have a strong impact, especially on disfavoured, problem-prone and marginal groups. The final challenge here would be to be able to shift the increasing trend of enjoying music as an individual, isolated, activity, making social ways to search, share, listen to, and re-create the otherwise "personal" collections of music possible.

• Learn, understand and eventually integrate neuro-scientific results concerning music. The question of how music influences emotions of listeners is a good example which is of great interest to MIR and where a growing body of neuro-scientific results on the basics of emotional experience exists. Comprehension of these results could enable better and richer MIR models of emotion in music. On the other hand, education of neuroscience researchers in MIR technology might help design of brain studies on music (e.g. in producing generative musical research stimuli).

Back to → Roadmap:Technological perspective