Machine-Learning Methods for Analysis and Generation of Expressive Performance

Artificial Intelligence and Machine Learning are some of the leading technology trends in the global scene, due to the easy access to big data technologies that store and analyse data, such as Apache Hadoop, Microsoft HDInsight, NoSQUL, among others. The large amount of data keeps raising and accumulating each year, which pave the way for the analysis of massive amounts of collected data in the future.

Today, anyone with basic knowledge can start using these new technological opportunities that offer great advantages lowering the price of storage, using open source solutions (without any license fees) and constantly improving these technologies to be more accessible. In the last years, a storm of hype hit the marked, initially with Cloud Computing, then Big Data, Internet of things (IOT), following many more. But now, we are in the verge of transforming these massive amount of data into valuable information, and even more, not only we humans can get new insights to learn and improve things, but machines can do that for us, in a massive scale.

Digitalizing everything

Thanks to IOT and Big Data technologies, we are able to hold more data than ever before. This has an extraordinary value especially when the volume of data is huge. IOT technologies are capable to obtain sensor data in an extremely easy and affordable way.

Big Data Systems are attracting the interest of a growing number of Data Scientists, producing new research with limitless possibilities. Some of them are finding a whole new greenfield in the analysis of human behaviour through computational algorithms that reveal patterns in order to predict and understand how our society works. An interesting example is the so-called “mathematics of love”. Although it could be thought love to be the last thing on earth that someone could imagine to digitalize and predict, based on data, we could actually find recognizable and quantifiable patterns on love, human feelings and human behaviours. When we analyse a discreet amount of people, we have no guarantees that this might apply to another group, but when we use a massive amount of data, we can start raising up our odds when predicting or understanding, an ethereal thing, such as Love.

Learning from patterns

Now, we have got into a point, that almost any cognitive science could be transposed into quantifiable patterns. The information could be systematically extracted using a machine that would assist researchers in recognizing specific insights that probably would pass unnoticed by humans. And, certainly, this new technology could also be applicable to the field of performing arts.

The analysis of massive amounts of data using computer software not only can save years of laborious human research, but also might produce results that are more accurate and objective than time other consuming traditional approaches. According, this new technology might imply transforming traditional methods used by researchers. [1] For example, developing a software that generates expressive interpretations played in a historically appropriate manner would revolutionize the approach to early music performance.

Currently, the main trends in computational systems for music performance includes the use of data-driven (machine learning) methods for analysis and generation of expressive performance, expressive interactive systems and cognitive models. The KTH model built by the KTH Royal Institute of Technology in Stockholm, is one of the most successful approaches in the generation of piano performances. It is built using a rule-based methodology in which performance patterns are set manually, as opposed to data-driven methods in which musical patterns are detected from large databases of music scores and performances. [2]

Generating Human performances

But, is the machine really able to emulate human expressive music performances? Although, there have been some brilliant attempts to produce piano performances with computer technologies, the outcomes are distant to be satisfactory from an artistic point of view. This is not surprising at all considering that usually research on music performances rendered by the machine is carried by teams formed mainly by computer experts, with only but a little input from music scientists. For this reason, although some renditions might sound convincing to some listeners, they still do not suffice the exigencies of professionally trained musicians. In order to produce research that is meaningful in the field of performing arts, it is essential to apply musicological methods and historical appropriate approaches to understand the music aesthetics of different historical periods. For example, in order to teach the machine to perform like legendary pianists from 19th and 20th-century, it would be ideal to define expressive patterns from a large corpus of early recordings, in particular piano roll digitalisations. Extractions from CD recordings by pianists does not offer the same accuracy as Musical Instrument Digital Interface (MIDI) recordings. Piano rolls can be converted to Performance data MIDI files data using a pneumatic interface such as the pneumatic roll reader that digitalises the human expression captured on the rolls. [3] In order to achieve this goal, it is essential an in-depth investigation on reproducing mechanical systems and methodologies for the extraction and categorisation of music parameters. [4]

Music scientists’ contribution not only adds artistic value to the performance rendering but also musicological knowledge and historical accuracy. It would also contribute to advance research in embodied cognition generating expressive music performance using historically appropriate sources, and embodying performing practices using computational models. This will lead towards the transformation of historical informed performance (HIP), developing and implementing a new methodological practice-led cyclical research, combining musicological research and artificial intelligence. In so doing, it will further expertise in exploring long term musical structures in the production of meaningful interpretations, which is currently understudied due to its complexity.

Learning to play like the masters

Algorithmic calculations of expressive patterns can be developed to generate human performance. Teaching the machines different piano techniques with relative accuracy could also have a significant impact as an educational tool. It would be possible to emulate a specific music style, comparing early performance practices to our modern interpretations. It could be used to compare pianists by nationality, age, pianistic tradition, even it could be used to compare human aspects such as difference in playing according to age or gender. Furthermore, it could be also used as a piano teacher that is able to identify the student’s skill level and musical style, in this way the students could easily assess their own pianistic technique and stylistic accuracy.

Once, we can quantify and identify such human expressions, the resources to learn from it are limitless, and therefore, we will answer questions that we could never had answered before and raise new questions that we never thought of.


References

[1] Performance-based research that develops a new practice-led approach assisted by computational technologies and digitalised sources in the study of performance see Carolina Estrada Bascuñana “New Approaches to Enrique Granados’ Pedagogical Methods and Pianistic Tradition: A Case of Study of Valses poéticos op. 43,” in Diagonal: An Ibero-American Music Review, 2(1), http://dx.doi.org/10.5070/D82135899 Retrieved from https://escholarship.org/uc/item/3z711898 (last accessed 9 December 2019)

[2] For a detailed critical review of current methods see Cancino-Chacón, Carlos Eduardo, Maarten Grachten, Werner Goebl, and Gerhard Widmer “Computational models of expressive music performance: A comprehensive and critical review,” in Frontiers in Digital Humanities 5 (2018): 25, http://frontiersin.org/articles/10.3389/fdigh.2018.00025/full

[3] A discussion of optical and pneumatic technologies and methods for archiving piano rolls is presented in Peter Phillips, “Piano Rolls and Contemporary Player Pianos: The Catalogues, Technologies, Archiving and Accessibility”, (University of Sydney, 2017), https://ses.library.usyd.edu.au/handle/2123/16939 (last accessed 9 December 2019).

[4] Concerning issues and mechanical limitations in the analysis of recordings captured on reproducing systems see Carolina Estrada Bascuñana “Enrique Granados’s Performance Style: Visualising the Audible Evidence,” in: Rund um Beethoven.Interpretationsforschung heute, ed. By Thomas Gartmann and Daniel Allebach, Schliengen: Argus 2019 (Musikforschung der Hochschule der Künste Bern, vol. 14), p. 150–179, https://www.hkb-interpretation.ch/fileadmin/user_upload/documents/Publikationen/Bd.14/HKB14_Rund_um_Beethoven.pdf (last accessed 9 December 2019)

Creative Commons Licence

AUTHOR: Carolina Estrada Bascuñana

Dr. Carolina Estrada Bascuñana is a PhD musicologist and pianist from Barcelona. She conducts research on 20th century piano music. For this she has been awarded numerous prizes such as the George Henderson Award 2016 and the SGAE Composer's Prize 2016. In addition, she is an internationally performing pianist. She currently holds a research fellowship from the Swiss National Science Foundation and will be leading the SNF research project "Nightingale" at BUA from January 2020.

Create PDF

Related Posts

None found

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *