Laske interview Marvin Minsky Sat, May 25, 1991 2/15/93 page 1 UNDERSTANDING MUSICAL ACTIVITIES: Readings in A.I. and Music The AAAI Press, Menlo Park, CA, 1991 Mira Balaban, Kemal Ebcioglu, Otto Laske, Eds. Send to Mike Hamilton, dmh@aaai.org Live Oak Press, 2620 emerson, palo alto 94306 PO 60036 415-843-0197 FOREWORD Interview edited by Otto Laske (617) 449-0781 A Conversation with Marvin Minsky Introduction The following interview with Marvin Minsky took place at his home in Brookline, MA., on January 23rd, 1991. The interview is a conversation about music, its peculiar features as a human activity, the special problems it poses for the scientist, and the suitability of AI methods for clarifying and/or solving some of these problems. The conversation is open-ended, and should be read accordingly, as a discourse to be continued at another time. A Science Fiction Novel OL: I hear you are writing a science fiction novel. Is that your first such work? MM: Well, yes, it is, and it is something I would not have tried to do alone. It is a spy-adventure techno-thriller that I am writing together with my co-author Harry Harrison. Harry did most of the plotting and invention of characters, while I invented new brain science and AI technology for the next century. OL: At what point in time is the novel situated? MM: It's set in the year 2023. OL: I may just be alive to experience it, then ... MM: Certainly. And furthermore, if the ideas of the story come true, then anyone who manages to live until then may have the opportunity to live forevermore... OL: How wonderful ... MM: ... because the book is about ways to read out the contents of a person's brain, and then download those contents into more reliable hardware, free from decay and disease. If you have enough money... OL: That's a very American footnote ... MM: Well, it's also a very Darwinian concept. OL: Yes, of course. MM: There isn't room for every possible being in this finite universe, so, we have to be selective ... OL: And who selects, or what is the selective mechanism? MM: Well, normally one selects by fighting. Perhaps somebody will invent a better way. Otherwise, you have to have a committee ... OL: That's worse than fighting, I think. The Notion of "Formalizing" Musical Knowledge OL: I wish we could read people's brain when they are engaged in music-making. Since that is, alas, beyond the state of the art of AI, we are forced to deal with such cumbersome issues as how to represent knowledge on the basis of what people tell us they do, which in most cases isn't very close to what they are actually doing. And a further problem is that we can't use what they are telling us directly; rather, we have to translate their verbalizations into even more formal code that is far removed from actual musical activities. So, of course, two questions immediately come to mind, viz., how should one formalize musical knowledge, and, can it be done effectively, so as to generate some kind of musical action? MM: And that in turn raises the question of 'is formalizing the right idea?' There are many kinds of reasons for writing descriptions, and here we ought to have a softer concept than 'formalize.' When we write down things we know about other crafts, we don't usually feel any need to imitate mathematicians. Why should we always feel compelled to do that when describing our musical dispositions? OL: The term 'formalizing' is linked to the notion of computation ... MM: OL: Formalization amounts to making a post facto summary of something. One looks back and tries to straighten out things that don't look right, and make them look 'logical.' Problems With Formalization MM: Well, yes, I like how you put that. Indeed we can use logic to make things look more logical but we can also use it to produce illusions. I suppose that the term originally referred to matters involving understanding and knowledge. But in current computer science 'logical' now means mainly to try to express everything in terms of two quantifiers, viz., 'for all X,' and 'for some X,' -- and two values, viz., 'true' and 'false'. But those menus seem just too small to me! We need much richer ideas than that. OL: In the past, we've had other logics, such as, for instance, Hegel's dialectical logic where you do not work with two quantifiers. So, one always knew there is some other way ... MM: Well, I'm not familiar with any applications of dialectic logic, but I certainly would favor going beyond binary logics. However, the extensions that I've seen in the so-called modal logics seem no improvement to me. Yes, they can express some ideas like 'it is possible', or 'it is necessary,' and so forth, but extensions like those seem inconsequential. We much more urgently need ways to express ideas like 'usually" or 'it is often useful to,' or 'for all objects that resemble X in regard to aspect Y'. OL: The Need For a Variety of Methods MM: Yes indeed. I want AI researchers to appreciate that there is no one 'best' way to represent knowledge. Each kind of problem requires appropriate types of thinking and reasoning -- and appropriate kind of representations. For example, logical methods are based on using rigid rules of inference to make deductions. This works well inside formal domains -- those artificial worlds that we imagine for ourselves. But to cope with the unknowns and uncertainties of reality, we must base our actions on experience -- and that requires us to reason by analogy, because not two situations are ever quite the same. In turn, this means that we have to be able to recollect similar experiences and understand which differences are relevant. But in the logical world, ideas like 'similar' and 'relevant' are alien because logic can only answer questions like "what is this an instance of?" or "what is this a generalization of?" The trouble is that concepts like instance and generalization apply only to ideas -- because no actual object or event can be an instance of anything else. However, real things can be seen as related -- at least in an observer's mind -- by apparent similarities of structures, effects, or useful applications. Certainly this is the case in music. OL: Yes, I think reasoning in terms of similarity is very pertinent to music, especially because much of musical knowledge is really knowledge serving action. Music is something we do, not just something we understand, and much of what we try to understand regarding music is meant to lead to the making of it. MM: And it is hard to understand all that making and understanding because they involve so many different mechanisms -- I like to think of them as like many different animals inside each brain. When you hear a piece of music, different parts of your brain do different things with it, but we know too little about those different processes. One obstacle to understanding such matters is that psychologists still strive for a science that resembles physics, where it is usually better to treat different explanations as competitive, instead of looking for ways to combine them. That strategy indeed works well in physical science -- presumably because there actually are only a very few fundamental laws of nature. But our ancestors had to deal with many different kinds of practical problems, and this lead to the evolution of brains that have hundreds of distinct regions with significantly different micro-architectures and different principles of operation. OL: You were saying that in listening to music, a lot of things are happening, and we have to understand the interrelationship between the different structures and processes that are involved in those reactions and understandings. MM: Yes, and we can still only guess what some of them are. Certainly, our musical apprehensions involve quite a few partially separable processes involved with, rhythm, melody, harmony, timbre, texture, and many other local phenomena -- and each of these appear to involve multiprocessing aspects of their own, such as timbral and contrapuntal voice separation. Sometimes it seems that one can sense some of the distinctness of those processes , as when it seems that one part of the mind is annoyed at the monotonously repetitive rhythmic structure of a certain composition -- while other parts of the mind don't mind this at all -- perhaps because they treat those repeating structures as structures not deserving attention themselves but serving as skeletons or scaffoldings, like a branching Christmas tree on which you hang the decorations. In this view, the significant features are the higher level differences between musical portions or segments that are otherwise extremely similar or analogous. It is those higher-level recognitions that let us treat the repetitive aspects of the music not as an irritating monotony but merely as a textural background. OL: Well, in all media of communication we have a lot of redundancy, to get across those few gems. MM: Precisely. But still, perhaps among all the arts, music is distinguished by this sublimely vulgar excess of redundancy, and we should try to understand its possible neurological consequences. I think Lukas Foss once remarked that anything repeated often enough can become interesting. Perhaps this phenomenon can be seen not as a paradox but as evidence that supports the idea of multiple processing levels. The function of the repetition is then to anesthetize the lower levels of cognitive machinery. (We know that this is the usual rule in neurology: decay of response to constant or repetitive signals.) But the result of this could be to suddenly and strangely free the higher levels of the brain from their mundane bondage to reality -- to then be free.to create new things. "Music" or Musics? OL: Would you say that holds for all kinds of music? 'Music' seems to be a notion like 'God' or 'Love,' something everybody can identify with, but which actually covers many different, even opposite, phenomena ... MM: I think calling so many different things by the same name, 'Music', certainly makes it hard to think about those subjects. The modern tendency, to be tolerant and say 'anything is music,' is bad for the mind, both of the listener and of the critic also. (I don't know whether it's bad for the composer, if he can make a living.) A portmanteau word like "music" that is used for so many activities can not be an element of a serious discussion. OL: So, we would have to speak of 'musics,' and would have to define what we mean in any particular instance, which would be very cumbersome. MM: Yes, like having to say that going to talk about certain German music from the l8th century, or Indian music from such a place at a certain time. And then one can ask, to what extent do these engage similar mental activities? OL: How should we proceed, Music being really a universal? MM: For serious analysis, I think we simply must avoid such universal. When I write about some mechanism of intelligence or learning, I try not to use words like 'intelligence' and 'learning' - except in the title or summary. Music as a Label for Societal Acceptance OL: That is already done in studies in AI and Music, where you find systems for doing harmonic analysis, and systems generating compositional material,-- these are all specific kinds of things, and the claim is not that we know what music is. It seems to me that, viewed in the light of such studies, the term "music" expresses rather an acceptance on the side of society that something is o.k. The composer, as composer, doesn't care whether something is music. He is driven to generate something, and then, if an audience finds that what he produced is acceptable, or 'music,' he is happy, and otherwise he is not -- but it's essentially not his doing as much as it is society's. MM: That raises an exciting and interesting question -- that many people are reluctant to consider, which is the question: what is, or ought to be, o.k.? One thing I like to do is to consider the major human activities, and try to get people to ask: "is it ok?" All over the world many people listen to music for hours each day; in this country many spend substantial portions of their incomes on recordings, high-fis, personal earphone devices, rock concerts, and tolerate background music in their workplaces, restaurants, airplanes, and what not. Is that all right? Similarly, we ought to wonder whether it is reasonable to engage in sports. I ask people, "isn't there something funny about grown people gathering in a huge stadium to see other grown people kicking a ball from one end to the other?" Each of those persons is using a multitrillion synapse brain. It would be fun to ask the religious ones to consider whether it is not a sin to waste such wondrous hardware on watching adults kicking balls around? My own view is that this is less a sin than a symptom -- of infection by a parasitic meme (namely, one that carries the idea that such an activity is o.k.) which has self-propagated through our culture like a software virus, a cancer of the intellect so insidious that virtually no one thinks/dares to question it. Now, in the same way we see grown people playing and working in the context of popular music that often repeats a single sentence or melodic phrase over and over and over again, a instills in the mind some harmonic trick that sets at least part of one's brain in a loop. Is it o.k. that we, with our hard earned brains, should welcome and accept this indignity -- or should we resent it as an assault on an evident vulnerability? Music as a Device for Directing Human Activity OL: You have suggested somewhere that music is often used as a tool for stress management, ...for escaping the painfulness of thought. MM: Certainly that is how music seems to be used at times. It seems very much as though a person can exploit music as an external intervention (in contrast to using an internal, perhaps chemical,.regulatory system) to suppress one or another parts of the brain -- e.g., parts that might otherwise be occupied with sexual or social or other types of thinking which that person may presently not want to entertain. Clearly, people use music for directing their mental activities. After all, that's what it means when we speak of music as stimulating, or as soothing, as like an opiate for relieving pain or anxiety. Or encouraging us to march and fight, or to sorrow at a funeral. OL: The idea that music can be used to control inner states is, of course, an old idea. There is music therapy, for example... MM: There are also other way of listening to music that people like you and I use a lot. Sometimes when hearing some music, I react in the ways we just mentioned, but at other times, when my attention is so drawn to I find myself more concerned with how to make music that sounds that way? What's that sequence? How would you finger it? How did the composer get that idea. What would I have to learn to be able to do that myself. Heavens, how appropriate to double the horn and the oboe here. Did the composer figure that out anew or get the idea from some previous piece? Thus music can be the most stressful of all activities -- from the perspective of the potential composer, because each intriguing new idea portends some unmastered aspect of ability, or newly recognized deficiency in one's own musical machinery. From this viewpoint, it can be very stressful to find oneself forced (from somewhere else within oneself) to like a piece of music without understanding the psycho-musical trick that makes it so effective. OL: So, in that case music is not something that gives pleasure. MM: Right. And even when it does give pleasure, there is no reason to take that pleasure at 'face value'. What pleases you can also control you, by causing you to do something other than you would have done otherwise. The way an adolescent can be enslaved by infatuated with a pretty face borne by a person with no other evident merit. I think Paul Goodman once suggested that "Pleasure is not a goal; it is a feeling that accompanies an important seeming activity." Music as a Tool for Emotional Exploitation OL: In other words, music can be used in part as an exploitative device? MM: Yes. To be a popular composer, what is it you must do? Perhaps to learn enough tricks that cause people to have pleasant sensations without too much stress. What are the tricks for making that catches on in a listener's mind, and keeps repeating long after the performance? There is a superb novel,"The Demolished Man," by Alfred Bester that depicts how the tunesmiths of the future develop jingle-composing to such a degree that it can be used, in effect, for mental assassination. A victim infested by such a tune is helpless to do any useful work, and must hire a specialized therapist to help remove the tune from memory. OL: However, wouldn't such a use of music entail knowing the person one is imprisoning very well? MM: That is a profound question: how universal are our musical techniques? And even if there are powerful universals, there must also be powerful non-universals and, so, perhaps in the future when we have better brain reading instruments, people may commission composers to produce works designed not for audiences, but for particular clients? Surely you can write a much better piece of music for a single listener ('better', of course, in that person's view). And then perhaps it will be considered most masterful to write a piece that only that single client will adore, and everyone else will abhor. OL: Now, if such a work were possible, wouldn't we have almost no chance at all to understand music as something that has universality, and addresses itself to some commonality people share? MM: Good question. One could argue that even if we find universal compositional techniques that are widely universal, in the sense that they evoke strong and similar reactions in most listeners, those techniques would by that fact be in some sense rather shallow -- because, in being so nearly universal, then it must be simply be because of filling some niche, or exploiting some mechanism, or taking advantage of some bug that all human brains have. Consider how many people tend to tap the foot to the rhythm of a piece of music without knowing that they're doing it. An alien being might regard that as some sort of mechanical bug, even if humans regard it as natural. The Present AI and Music Scene OL: If we look at what people in AI and Music do today, we find that problems like we have discussed are not at all on the agenda. Mainly, I think what one tries to do is to go one step beyond merely verbalizing music's effect on people, or verbalizing how one composes music, by taking information to a more 'formalized' stage. In this book, for instance, what you'll find is a discussion of the scope of cognitive musicology, of fundamental issues in knowledge representation, and the attempt to rationally reconstruct certain musical activities, such as composing, performing, learning, and analyzing music. Essentially, the presupposition is made <>> that we already know what music is, and that it is just a matter of spelling that knowledge out more clearly. I think the pernicious idea that is very strong in the field, --if today it can be called a field at all--is that you can take AI as it is, and 'apply' it to music, where 'music' is a medium we already know well. As a consequence, what you do is harness the means of present-day AI, rather than trying to reinvent AI on the basis of musical knowledge. MM: Yes, well, that's largely true -- and perhaps it is also because AI isn't advanced enough yet, even to explain how we use language -- for example, to understand stories. That surely would be an obstacle if it turned out that much of musical thinking involves those same mechanisms. This reminds me of how, in my childhood, there were musicians who talked about music theory. But when I asked what music theory was, it turned out that "music theory" was little more than nomenclature -- the classification of chords and sequences. A syntax with no semantics whatever -- with virtually no ideas how the music conveys its meanings, or whatever it is that makes us react. The situation is a good deal better, in AI-based language theories, because we have the beginnings of theories about the nature of "stories" and how they affect us. For example, Wendy Lehnert developed a nice theory about about the most usual sorts of constraints on the structures of acceptable stories. A respectable story must introduce some characters -- typically, a protagonist and an antagonist, and some sort of problem or anomaly that must eventually be resolved. Lehnert shows how such plots could be assembled, recursively, from elementary activities that she calls "plot units". [Get References from Lehnert.] These presumably are what help you construct, elaborate, comprehend, and (perhaps most important of all) remember, a story. For example, according to this model, if character A does something bad to character B then, eventually, A must compensate by doing something good for B -- or else something bad must also befall A, so that B can "get even". Of course there can be exceptions to this -- but not too many, or else the reader will be compelled to ask, "what is the point ; this story rambles; it makes no sense; it is not a story at all. To "understand" a story one needs a well-controlled agenda of concerns about conflicts among its elements. OL: So, you want to look at music as a kind of "pseudo-story?" MM: Exactly, that's just what I was reaching for. At least in some kinds of "music", as in most kinds of stories, there mustn't be too many loose ends, unresolved problems, irrelevant material, and pointless excursions. However, acceptable compositions differs from acceptable stories in permitting far more repetition. In works of music, there is a lot of redundancy, as shown by the fact that, in much of classical music, measures are of the same length, and there is a binary tree structure: two phrases are repeated to make what music theorists call a period, and two periods repeat to make a musical sentence. In my 1980 paper I suggested the obvious thing: that certain parts of the brain might apply to this a hierarchical analysis that, in effect, parses this input into a binary (or, rarely, ternary) tree-like structure; then the meaning can be extracted from the now easily-recognizable differences between the corresponding parts. Why else would you repeat something so many times except either to learn it by repetition, or to point out to the listener certain small differences between sequences. For instance, we notice in a typical 4-line tune that one of the phrases goes up the first time, near the end, and the 2nd one goes down, that's a sort of pair of parentheses; and we get used to that. OL: Yes. MM: In a computation-based treatment of musical expression you would expect to see attempts to describe and explain such sorts of structure. Yet the most "respectable" present-day analyses -- e.g., the well-known Lerdahl & Jackendoff work on generative grammars for tonal music -- seem to me insufficiently concerned with such relationships. The so-called "generative" approach purports to describe all choices open to a speaker or composer -- but it also tries to abstract away the actual procedure, the temporal evolution of the compositional process. Consequently, it cannot even begin to describe the choices composers must actually face -- and we can understand that only by making models of the cognitive constraints that motivate an author or composer. I suspect that when we learn how to do that, many regularities that today are regarded as grammatical will be seen as results of how the composer's motivations interact with the knowledge-representation mechanisms shared by the composer and the listener. In any case it seems to me that, both in music and language, one must understand the semantics of tension-producing elements -- at least in the forms that resemble narrative. Each initial discord, be it melodic, rhythmic, harmonic, or whatever, can be seen as a problem to be later resolved. A lot of what a composer does is setting up expectations, and then figuring out how to frustrate them. That gives the composer some problems to solve. The problems and their solutions are then like elements of a plot, and composition becomes a kind of story telling. OL: To look at composing as a variety of story-telling, and at music as a pseudo-story, wouldn't that help us to arrive at a theory of musical discourse? The Heart of AI MM: Yes, indeed. And if we hope to apply AI ideas to that, we'll want to exploit the best tools that AI can offer -- different sorts of processing, different ways to represent knowledge, and so forth. But the enterprise of making an artificial intelligence involves not only some tools but also a larger-scale outlook. And AI will fail to illuminate music, just as linguistics did (because of trying to isolate meaning apart from syntax) if we use only the parts but reject the heart. For just as linguistics has a heart -- to find how we communicate, so AI too does have a heart -- to find out how machines can be made to solve significant problems. Now in past years we have seen a certain amount of applying AI tools to traditional music analysis. But I would like to see more from the heart of AI, the study of problem solving, applied to issues of how you solve musical problems. Making a Composer OL: So, then, for you to apply AI to music, if one can say apply... MM: ... would be making composers, or at least listeners... OL: By "making," do you mean to produce a robot-like creature that does certain things like, observably composing? MM: Yes, indeed. And in the case of listening it would have to know when to say "oh, this is exciting," or "how very tender," and the like. I haven't seen much of that. OL: In Japan, one has built a robot that is capable of reading music, and play it on the piano. MM: Yes, that fellow at Mazda. OL: Is that something you have in mind here? MM: Not at all. Because I'm more concerned about what happens at larger scales of phrase and plot. Our listening machine would have to understand the music well enough to recognize from each moment to the next which problems have been solved, and which remain open. OL: How would that understanding have to become manifest? MM: Well, for example, an understanding listener can hear a piano concerto and appropriately say things like "that was a good idea here in the cadenza, but he didn't carry it through." I'd want the robot to make similar analyses. OL: To do that, the robot must be able to recognize solutions, good or bad. But then how would it communicate this to others? MM: One way might be to have it write the sorts of sentences that critics write. Or to have work more in the musical realm by performing as a teacher does, explaining differences by demonstration -- "Look how much better it would be to delay a little these notes here, and make those near the end more staccato, like this, and this." And of course if our machine turned out to able to produce interesting enough interpretations, then we might be satisfied by that alone -- if many listeners were to agree that "really, that performer has a lot of good ideas about this music, and brings out stuff that I didn't realize was there." Does Music Need a Body? OL: This brings to mind for me the people, at MIT especially, who have begun to build robotic insects, which seems to be an approach that is in some contrast to the top-down symbolic approach AI has used in the past. --I don't know whether you see the matter that way. The question here is, if you speak of a robot - what kind of an approach do you have in mind? Is your robot programmed on the basis of symbolic representations, ... high level constructs? MM: Yes, surely understanding music requires many levels, and a good deal of cultural knowledge. OL: Do you need a body level also, for the case of performing, say? MM: I don't think that this will be important. OL: So you don't consider this to be a critical problem -- of how emotion, as distinguished from cognition, is in most cases linked to an organic body ... MM: Not really, because I don't expect that emotional behavior will turn out to be singularly hard to simulate. I'm inclined to agree with Niko Tinbergen that the basic emotions come from the activities of various almost separate processes, brain-systems that each have different, and rather clear-cut goals. For example, in the brain stem at least a dozen such systems have evolved and, to a large extent, they behave like different animals. In "The Society of Mind" I called them "Protospecialists.". When you're low on sugar, the protospecialist for 'hunger' gets activated -- and causes you to apply your available knowledge to find a way to get food. If you're too cold, then another specialist gets activated, suppressing the others; it uses the available knowledge to get out of the chill and into the sun, or to cover your body with insulators, or to huddle together with your friends. I think most people assume that emotions are very deep and complicated, because they seem so powerful and hard to understand. But my view is just the opposite: it may be largely because they are basically simple -- but wired up to be powerful -- that they are hard to understand. That is, they seem mysterious simply because they're separate and opaque to your other cognitive processes. However, there is also another thing that makes these emotions hard to understand, at least in their older, more adult forms. It is that although an emotion may be simple by itself -- for example,a brain center that is genetically wired to learn actions that keep one warm -- the end result of what it may learn can become arbitrarily complicated. The problem is that each infantile emotion, such as hunger, defense, nutrition, or sex, can eventually acquire a huge cognitive system for achieving its own goal, perhaps by exploiting the other emotions. And presumably virtually every adult cognitive activity develops in some complex way from infantile activities. So I see as false the commonsense distinction between "intellect" and "emotion". It would be better to emphasize the development over time from simple to complex. Nevertheless, music appears to depend for some of its effects on being able to arouse emotions; therefore, in order to build a competent composing or listening machine, we might need to understand such activities well enough to simulate them. Some of this could surely be done with the sorts of decentralized agents and agencies being used in those insectlike AI's -- but I don't suppose they would work well for this unless we managed to program them with roughly the required emotions and cognitions. I have never heard of any non-artificial insect that could learn to prefer Schubert to Mendelsohn. OL: Right. These insects would have to climb the tree of knowledge first. The Need for An Emotional Culture MM: Yes, indeed. And to use them inside our music machine, they might have to evolve to be quite similar to the ones inside our human brains. Our reactions to the sound of a violin may exploit our reactions to the pitch centers of female voices; the sound of a cello may seem more male. A machine that was really competent to listen to 19th century classical chamber music might well need some knowledge- understanding of human social affairs -- about aggression and conciliation, sorrow and joy, and family, friendship and strangership. And of course there are other constraints on how we use those instruments. Generally the cello will have to play longer and slower notes that emphasize foundational notes of the chord and there's nothing sexual about that -- but nevertheless, our human perceptual experience will tend to associate the higher notes with children's voices, the middle octave with female, and the lower one with male voices. It might even be natural to link yet lower tones with those of large fierce animals. OL: There is historical evidence for what you just said. The violin in Renaissance Italy was developed by people who were very eager to simulate the human voice. For these people, the human was the ideal on the basis of which to refine the violin family of instruments. MM: I have to admit that the first time I encountered a modern synthesizer, the most exciting stop for me was the choral stop, the high female voices. Having been a pianist all my life, it was so astounding to be able to touch a key, and have those soprano voices come out, and basses, tenors, contraltos. Completely entrancing. For an hour or two, like a fairyland. But eventually it seemed wrong that I couldn't make these voices talk, they were always making the same bleating sound. But what a marvelous experience.... Requirements of Being a Music Scientist OL: We were wondering why none of these ideas about cognition and emotion seem to be of relevance in present-day AI and Music research... MM: I think that's because the AI people have suffered from the same misconception that most cognitive psychologists have suffered from, viz., the idea that "well, we'll do the easy things first, like understanding memory and simple reasoning and so forth; but emotions are surely too difficult, so let's put off researching them for now." I once came across a statement by Freud in which he complains along lines like this: --"people think I work on emotions because those are the profound, important things. Not so, What I'd really like to understand is common sense thinking. And it is only because that's so difficult, so incredibly complicated, that I work on emotions instead -- because emotions are so much more simple." This is why I like Douglas Lenat's project to build a commonsense data base. I'm sure that Freud would have liked it too. This is a big project; the CYC data base will involve millions of different items, fragments of our cognitive machinery. But until our machines have access to commonsense knowledge, they'll never be able to understand stories -- or story-like music. One reason that AI has not gone very far in such domains is because researchers have been afraid to say: "I think emotions are simple enough that we make useful models of them." I'm not saying that emotions are trivial; they surely involve some complicated machinery. But you can't make progress unless you're willing to begin with simple theories, to serve as first approximations to the science we'll eventually need to understand musical activities. OL: Could you explain that a little further? MM: Sure. I mean that one should not be daunted by the apparent complexity of emotions, because it may be feasible to get a good start by making what may seem to be oversimplifications. You could begin by saying "maybe there are only three basic emotions"-- happy, sad, and, whatever you like. The "sentics" model of Manfred Clynes stipulates seven. It doesn't matter, so long as you start somewhere. I want to avoid the disaster that has befallen syntactic linguistics which, for all its apparent accomplishments, is dead in the water because of failing to include even the simplest caricature of a theory of what words mean. In my view, syntax has led to some useful discoveries, but to fewer than would have come from building more comprehensive models of language production and understanding. So I would like to see some music-theorists start with models based on simulating a few postulated emotions, each linked to a few procedural rules about how various sorts of rhythmic, harmonic, and melodic elements might work to arouse, suppress and otherwise engage a listener's various feelings and attitudes. Unless you start somewhere you'll go nowhere. OL: Are you saying, that the best way to approach music is to go at it from the basic emotions, rather than from ideas of problem solving, or the idea that music is a kind of chess game? Music as a Pseudo-Story MM: Well, emotion might not be the right word, but we must somehow formulate and engage what we believe are the important musical problems. And clearly, an important aspect of "understanding" music experience is the listener's experience of apprehensions, gratifications, suspense, tensions, anxieties, and reliefs -- feelings very suggestive of pains and joys, insecurities and reassurances, dreads and reverences, and so forth. So, just as a good story confronts us with conflicts and resolutions, so do good musics, which take over the listener's disposition with feelings like: "I'm worried that something bad will happen unless this conflict is resolved." Problems and solutions. Once I was trying to finish a two- part invention but there was something wrong with it. I asked a musical friend for help and was surprised to hear a simple answer. "Look here, you went up into this octave but then you never made anything of it. You can't do that. When let a voice get into a new range, like breaking new ground, you must have a good reason for it and you must also think of a way to get that voice out of there afterwards. But here you just left those notes hanging up there and I ended up wondering what will happen to them." Now I wonder if this doesn't reflect some sense of territorially instinct, in which the listener is disturbed by not knowing who controls which area.... *** place for Broekx??*** OL: Well, it also relates to rules of musical discourse, doesn't it? If you make an argument, you have to follow it up ... MM: Right -- and that reminds me again of the constraint-like operations of Lehnert's Plot Units. Those dangling high notes evidently require follow-up, some resolution, and unless the composer brings them down, the listener is left with some unresolved conflict, concern, or problem. As though one of your emotional protospecialists were left in an active state. After all, what are musical problems if not problems having to do with the resolution of conflict? OL: When you look into a book on counterpoint, it will tell you, "having made a particular step, there are only so many things you can do;" and, "not following up some step you've made is a mistake in terms of the rule system that adhered to in counterpoint." Whereas you were saying, following up some compositional decision is required for being emotionally or story-wise convincing in one's music. Where All Those Rules Go ... MM: Yes, and that raises questions about the numbers of and characters of those "rules" -- and those two dimensions are far from independent. Indeed, this is what I was thinking about in that analogy with syntax. For if you tried to analyze contrapuntal music, you could write rule after rule to constrain its surface structure, but you'd never get quite enough rules to tell composers what to do. However, I suspect that if we managed to build the right sort of model for musical plots and conflicts, we might end up with a more compact theory that solves many more problems. A theory that takes account of both transmission and reception, of composing and listening. Chapters 22 and 26 of my book "The Society of Mind" proposes such a theory of how language works. The idea is that when a speaker explains something to a listener, the goal is to produce in the listener a structure that resembles a certain semantic network in the speaker's brain. Many linguistic tricks are used for controlling the growth of that network. My conjecture is that mechanisms of this sort could lead to good description of utterances are understandable -- and could do this, I suspect, with simpler and fewer rules than the usual kinds for describing sentential grammar -- because many grammatical rules that seem separate and independent might result from these deeper mechanisms. It is OK to begin by looking at surface regularities. Musicological Taboos OL: For the longest time, a taboo has existed in music theory and musicology regarding any attempt of understanding music as an activity. Researchers have typically concentrated on music as an object, an artifact, a product, rather than studying the process by which music is generated, both in the composer and the listener. How do you explain this taboo? Is that something you are aware of in other sciences, too, -- that one simply decides not to look into certain things? MM: I think the taboo made sense in the past simply because before the advent of modern theories of complex information processing there really was no useful to think about how minds might do such things. In my view the theories of philosophers before the era of Freud, Piaget, and Tinbergen were too primitive to provide much insight. Freud recognized that higher forms of human thought involved the pursuit of goals by acquiring and exploiting knowledge. But it was not until the 1960s that workers like Allen Newell and Herbert Simon formulated adequate theories of systems with goals. To be sure, there were earlier attempts to base psychology on simple principles of association -- as in the models of Pavlov and Skinner -- but it was never clear how these could lead to higher levels of cognition. Nevertheless, crude surface- behavioral descriptions became dominant, at least in American Psychology -- and in my view this included both Skinner and Chomsky, despite their famous debate; neither of them seemed comfortable with the idea of making models of the internal processes that underlie behavior. In any case, outside of psychoanalytic circles, making complex theories about mental processes did become taboo. OL: It didn't make sense at the time to ask questions about musical processes. MM: I think so, considering the lack of progress on most other mental activities. But beginning with the 1950s, AI and computer science started to produce an enormous quantity of new concepts. Just look at the lingo--a "stack," a "push down list," "default inheritance." In these fields, there emerged literally a thousand new ideas and terms for describing mental processes, whereas before, maybe there were a few dozen, in language. So, humanity was just not prepared to understand anything as complicated as the process of a thinking machine. Until it had some practice with it, due to the computer. OL: Doesn't it seem to you that the taboo we've talked about is still in force, even in AI and Music today? MM: Yes, especially in music. There, the taboo is often justified by arguing that too much inquiry will spoil it. There's the apprehension that if you understood music, you would lose your interest and destroy the beauty of it. OL: There is also this Western notion of a work of art, as something that is just too good to be drawn into the question of how it was made, and the further notion that it would be best not to speak about that it was made at all ... MM: Right -- and we know what Freud would say about any such notion, viz., that, if there is such a fear, it's because you're worried that there's not so much there, and you're repressing your worries. The more angry you get when it's questioned, the more you are giving yourself away. I once had an extended argument with a chemist friend because I said: "why don't we synthesize good wine? Why be at the mercy of the weather and the vintages?" And he replied: "That simply can't be done; it is impossible; the sense of taste is infinitely complicated." And I replied: "Well, why not analyze some wine; we might find that there only a few important chemicals in each good wine, and then we just figure out the amounts of them." And he said: "No, there must be a million chemicals." I said: "And you can taste them all, and they all matter?" "Yes," he said, "they all stick together to form one indescribable whole." The argument seemed interminable and I wondered, did this outstanding chemist unconsciously fear that chemistry itself was inadequate? Did he fear the thought that he himself might be "merely" chemical? Or did he fear that all his likes and dislikes could be explained -- and then there'd be no mystery to life? Is there a danger if we understand music, or art, or liking food, there will be nothing important left to do? I'm not worried, myself, about running out of mysteries. How Do Works of Music Relate to Their Process? OL: There is another problem worth investigating, viz., the relationship between a work and the process that produces it. For me at least, that problem is the crux of any truly musicological inquiry. It is not enough to make explicit the structure of the work (as is done in music analysis and traditional musicology), and it is not enough either to make explicit the structure of the process that produced the work (as is done in AI and Cognitive Science). The crucial musicological question regards the link between them, the issue of whether one can describe the link, and in what way. That seems to me to be the most crucial question in all of the human sciences, not just musicology, but in all sciences that deal with human artifacts of some kind. MM: Very interesting idea. We're used to seeing completed works -- but it would be interesting to see the steps in between. Suppose you were to watch an artist making a painting. Of course, that doesn't show you the artist's thoughts, but it shows some of the planning and structure. How would that affect your relation to the painting. Similarly when you hear music, the steps are concealed -- but some composers do work at the keyboard, and it would be interesting to have recordings of those activities. Beethoven made many revisions in some of his manuscripts of Opus 111, some bits written over many times, and pages of alternative sketches; but only the final results are performed. We should ask the next Beethoven to provide us with recordings of his improvisatory sketches. But perhaps that won't be necessary if, over another few decades, we find ways to more directly record a composers actual brain activities. This touches on the relation between composing and improvising. In composing you can be out of real time, and make revisions, and cover your tracks. Improvisors cannot be quite so devious -- and therefore, in some cases, they may communicate more successfully. And this reminds me of a different experience when I was a student; reading books on mathematics always seemed peculiarly difficult and always took a long time. But one day, I ran across a book by John von Neumann on "Mathematical Foundations of Quantum Theory" and it was the clearest, most pleasant mathematics book I'd ever read. I remember understanding his explanation of the Metric Density theorem as like a real time experience, like being inside another person's head. (This is a theorem about probability or, more precisely, about measurable sets. It says, for example, that if a certain subset U of a square S has probability larger than zero -- that is, if when you throw a dart at the square there is a non-zero probability of hitting a point in U -- then there must exist smaller squares in S in which the U-points have probabilities arbitrarily close to 1.) I mentioned this to my friend Stanislaw Ulam, who had worked with von Neumann, and Ulam thought he recalled that von Neumann had been in a hurry when he wrote the book, it was a first draft and he never revised it. And so, he was writing down how he thought about the problems. OL: He improvised the book ... MM: Yes, and I found that once I had read it, I never have to go back. Whereas most other math books have been made terribly tight; all extra or "unnecessary": words removed, and all traces of how the author actually thinks about the problem. My point is that it can take much longer to read a shorter book with the same content. So too, in music, it sometimes might be nice to have the composers' first drafts -- that is, from composers we like, and who improvise at the keyboard ... Knowledge Acquisition in Music OL: Of course, making music with computers gives you the possibility of doing knowledge acquisition. I, for instance, have given composers a problem using a particular program for computer-aided composition, and then have generated a protocol (as Herbert Simon would call it) that documents the actions carried out by composers using the program. It is not a verbal protocol, however; it's an action protocol, one might say, that simply documents what operators a composer has chosen in working on a particular task over a period of time... MM: They have to shape a piece by applying some operators to the material given them ... OL: Yes. I have done this experiment with children, too. As a result, you get a kind of "action protocol" that you can then try to understand in its control structure, and relate what you find to what you know about the structure of the "work" composed during the session.... MM: Then you could ask the composers why they did what they did, I suppose ... OL: You could. Maybe not children, but you could certainly ask professional composers or people who are learning to be composers ... MM: They might not be able to tell you anything useful, though. One just doesn't know those processes. It's just like asking somebody why they used a certain clause in their sentence, ... what could they tell you? OL: Not much indeed. I don't presume that this would be conscious knowledge ... MM: But you could look over the protocol, and maybe find things that you never otherwise find ... OL: Yes, that's my point. For the first time in musical history, we are able to produce empirical traces of a musical process, and we can study such a process in terms of the actions it is made up of. I tend to believe that a musical form derives from the process that produced it, and would think that the control structure of the process is intimately linked to the form that emerges. Music As A Bug Between Brain Areas MM: Yes -- and perhaps in some years from now we'll be able to see those processes even more directly by using high-resolution brain activity imaging instruments. I would certainly like to be able to see what's happening in my brain when I improvise, because I have virtually no direct insight into that; I'm thinking about other things when I do it -- sometimes even about some completely different piece of music. I suspect that my music production machinery is mostly in my motor brain regions -- that is, in the front half of the brain -- more than in than the back or sensory half of the brain. (I really hate to hear so many people repeating superstitions about the left and the right sides of the brain; the brain has hundreds of different parts, so you can divide it in two in many ways.) The result of this is really annoying to me; I have great difficulty writing down music on paper until my hands play it on the keyboard so that I can hear it. Now I don't actually mean hearing, literally, because I can play it on an imaginary keyboard in the air. But I can't hear the music nearly so well without moving the fingers. This probably means that in order to close the loop, I have to use some bundles of nerve fibers that tunnel under the central sulcus from the motor regions back to the sensory regions of the cortex. It would be nice to understand this -- and there's no reason why not, with new instruments. OL: If you are right with your hunch that to understand how humans make music we need to know how they use their brains, then, of course, we are right at the question of how music developed in the history of the human brain. MM: Yes, and that should be extremely interesting because music seems to have no evolutionary origin. So I suspect that in each musical person there has been some early incident in which some musical process-knowledge comes to occupy some piece of brain that isn't dedicated to something else, and it probably happens somewhat differently in each person. I don't think anybody has mapped this very much. OL: You said something for me astonishing, viz., "Music has no evolutionary history." MM: Well yes, in the sense that one can't see anything much related to it in animals descended from our ancestors. OL: There are no traces ... MM: ... of musical concerns or abilities in our ancestors or relatives -- so far as I know, no sign of musical interest in any other primates. Nor, so far as I know, in any other animals. I know of no animals that even tap their feet to rhythms or any thing like that. I don't consider bird- songs to be songs; yes they have communicative functions but no reason to relate that to music. Or whale songs. We call them songs but we still have no idea about their functions -- and again, I see no reason to call them musical. (I suspect that they contain information about major ocean currents or coastline features or lists of where various individuals have been seen, or other functional things like that. But no one knows.) Of course we all meet romantic people who maintain that their plants enjoy and thrive in musical environments. But the only careful experiment I've seen only demonstrate that vibrations tend to retard plant growth by injuring rootlets. My best guess is that music became possible because of some anatomical innovation that just happens to facilitate interactions between other, older functions -- for example between some of the brain that does planning for paths in space and some of the parts involved with language, or story-like memory systems. If that were the case, it might explain why hearing certain kinds of sounds might come to give you the feeling that you understood something, or give you the experience of being in some other place. If so, we could even regard music as being related to some sort of "hardware bug" perhaps involving brain areas concerned with visualization, kinetic imagery, language, or whatever -- some combination of structures that could lead to almost autonomous kinds of activity that we label "musical".* OL: Looking back at the ancient Greek culture you find that the term 'mousik' denotes something very special, viz., the linkage between poetry and sound, and that there is really no term for instrumental music as we know it. The kinds of sound making that are not linked to the human voice were considered as being a kind of techn, and thus much more lowly than mousik. I would agree, therefore, that the split between poetry and music, and the coming into being of music as we know it today, is a rather recent event, even in terms of the short history of humanity. MM: Right. OL: Of course, one would have to look at other cultures. MM: Surely. And in any case, however the brain is involved, music clearly interacts with many with memory mechanisms. Not only in music but in other realms, it seems much easier to remember things that are grouped in regular temporal structures that resemble rhythm and meter. The other day I complained to Carl Sagan that we have no good theory of why people are so attracted to regular rhythms. He pointed out that the mother's heartbeat is a prominent context of every baby's development. I'm sure that there is something to that -- except that dogs and cats hear heartbeats too, but do not seem to tap their feet or otherwise show much sign of being affected by music.* A Musical Common Sense Data Base OL: Indeed, music is a human privilege. Well, since our time is almost up, to conclude our conversation, could I ask you what you think has been achieved in the field of AI and Music so far, and beyond that, what you think should be worked on, or might be worked on, in the next decade or so? Is that a fair question to put to you? MM: A vital question. And to guess at an answer, we need to understand our position in history. AI is only 30 or 40 years old, but people are always asking: "What has it done lately?" or: "What are its important achievements?" The trouble is, we can't really say, because what seems important at one moment may not be what turns out to have been important 10 or 20 years later. We can't be certain which are the good ideas yet. Perhaps we need more research on case-based reasoning. Or on structures like Lehnert's plot units, to understand more about the structures of stories -- and of musical compositions. (In Roger Schank's recent theories of learning and reasoning, story- structures play critical roles.) Or, perhaps, most important of all, we need to know more about common sense reasoning, and the data structures that underlie it. Because no computer in the world today yet knows the meaning of enough words to understand a story. OL: Yes. MM: That is the gap that Doug Lenat is trying to fill, viz., by working on a common sense data base. And similarly we need some sort of musical common sense data base. MM: In college I attended one of Walter Piston's courses. He used to complain that no one knew enough what makes good tunes. He said that there were only rules of thumb, like the rules for good manners or good behavior; a good melody must some sense of direction and return; it shouldn't be too jerky -- and it shouldn't be too smooth -- but there were no useful formal theories about such things. And I remember thinking then -- before I ever thought about AI -- "Maybe there actually can't be such rules. Because perhaps it works a different way, by analogy. Perhaps I match each tune I hear with a hundred tunes that I learned when I was a baby. A machine to do this might have to know all those nursery rhymes and folk songs and lullabies and Christmas carols. Perhaps the reason why we like certain tunes is largely because of already liking other similar ones? OL: Most likely. MM: And if that were, then the important thing about a tune would be how well it resembles some of those repertoried tunes -- and what are their interesting differences. In any case those music courses didn't seem to know how to tell me how to compose. They were infuriating ... * OL: They still are ... MM: Now, today, things ought to be better. We have so many new good ideas. Case-based ideas, grammatical ideas, problem-solving ideas. Expert-system ideas. So much has been accomplished that we may be ready the renaissance of many an old inquiry. Perhaps the research community as a whole already has a critical mass of the needed insights -- and is only waiting for someone to see a way to pull them together. Anyway, in my view the most critical thing, in both music research and general AI research is to learn how to build a common music data base. But nobody in music research works on that yet, do they? OL: Not that I know of ... MM: It's mostly the same in AI. Most hope to understand language by using compact theory-tricks, like formal grammars. But that simply can't do enough by itself. You need some sort of data base, of experience. A few little stories about each word. And the same for music. Surely you can't react "properly" unless you possess some "stories" about each chord sequence, or melodic turn, or rhythmic variation. OL: So, that's what you call a common sense musical data base: a collection of stories about important compositional or auditory constructs? MM: Precisely. And consequently, despite all those popular, fancy, formal theories, we're missing all the substance beneath. We need a musical Lenat to start a musical CYC project. END OF THE INTERVIEW ====== [note to Laske: in one of your papers there was a haunting description or quote about music describing a story about important characters involved in important activities -- only there was no clue about who they were or what they were doing. Do you remember this?>