Towards a Science of Emerging Media Barry Smith Uncorrected preprint version of Chapter 1 of Philosophy of Emerging Media: Understanding, Appreciation and Application, edited by J.E. Katz and J. Floyd, Oxford University Press, 2015, 29‐48. Abstract: If media studies are to become established as a genuine science, then it needs to be determined what the subject matter of this science is to be. I propose a specification of this subject matter as consisting in: 1. the new sorts of digital entities that have been added to social reality through the invention of the digital computer, and 2. the new sorts of interactions involving human beings which such entities make possible. I support this proposal by examining examples of some of the ways in which the digital products of emerging media differ from entities of other sorts. I then draw consequences from this examination to demonstrate how these products mark out a new realm within the larger domain of social ontology. 1. Preamble Already in 1991, in his The Future of the Mass Audience, W. Russell Neuman provided a list of predicted effects of (what we here calling) emerging media.1 They will a) alter the meaning of geographic distance, b) increase the volume and speed of communications, c) allow for more channels of information flow, d) enable interactive communications, e) provide more control for individual users, 1 I follow here a widely dissemination definition of 'media' as: communication channels through which news, entertainment, education, data, or promotional messages are disseminated. This definition has its origins in the world of mass media, to which Neuman's attentions were directed. I define 'emerging media' as media 2 f) allow forms of communication previously separate to overlap and interconnect, and g) increasingly displace general‐interest mass‐audience media with more specialized narrow‐audience media.2 Here I want to add the following additional effects: h) create new kinds of digital entities, including entities of internet and near‐ internet scale, i) transform human beings through interaction with such digital entities, and thereby j) allow new forms of human agency, including massive human agency and massive interactivity both between human beings and between human beings and machines. This is not of course an exhaustive list of the effects of emerging media. I list these examples here only because they will allow me to illustrate some of the ways in which emerging media are already changing the social world, especially in ways that bring benefits to human beings. These examples will also allow me to illustrate some of the ways in which philosophy can throw a more positive light on emerging media, where standard (particularly postmodernist and deconstructionist) philosophical approaches in this area have tended to emphasize their negative features.3 2. Eric Whitacre's Virtual Choir An example which illustrates almost all of the aforementioned effects is provided by the series of digital performances created by the American composer/conductor Eric Whitacre through what he calls the Virtual Choir.4 In Fly to Paradise of 2014,5 for example, Whitacre combines into a single digital entity some 6000 separate solo performances contributed by singers from more than 100 countries by merging into a single whole YouTube videos created by these singers in accordance with his instructions. 2 Neuman (1991), pp. 6f., 12f. and as cited in Croteau and Hoynes (2003), p. 322. 3 This is partly under the influence of technophobes such as Heidegger (1949). On the more positive side is Max Scheler – another prominent member of the phenomenological movement founded by Husserl – who refers to man's 'instincts to build, play, put together and experiment, which are the instinctive roots simultaneously of all positive sciences and also of all types of technology' (Technik) (1960, pp. 65–66). 4 http://ericwhitacre.com/the‐virtual‐choir, accessed December 1, 2014. 5 https://www.Youtube.com/watch?v=1hmS3TAuQH8, accessed December 1, 2014. 3 Hitherto the actions of very large numbers of human beings have been coordinated on the basis of direct physical interactions in space and time. With the Virtual Choir Whitacre achieves massive coordinated action on the part of human beings who do not interact physically in any way at all. He does this by taking advantage of the ways in which computers can allow digital entities to be combined and recombined to yield new digital entities in hitherto unanticipated ways. The process starts through the posting on the web of a score and associated YouTube video of Whitacre conducting in accordance with this score. Singers then create videos of themselves performing under Whitacre's direction. They then submit these performances in the form of email attachments which are synchronized and edited by Whitacre to form a unitary work. He thereby brings about a virtual intermeshing of shared intentions and actions of thousands of human beings acting independently and at different locations throughout the world. Whitacre's achievement is not only musical and technical, but also economic and legal. The funding needed to create the successive productions of the Virtual Choir can be secured only with the help of complex legal instruments which guarantee the intellectual property and clarify the other rights and obligations associated with each performance (Smith 2013). Each work produced in problem the Virtual Choir series is thus not merely a digital entity (with corresponding mathematical properties) and an aesthetic entity (with musical and visual properties), but also a legal and economic entity; it has an author, an owner, and a status in copyright law. Like works of art of other sorts it also undergoes changes through time, so that there is a sense in which – even though it may remain fixed as a digital entity – it nonetheless has a 'life' of the sort described by Roman Ingarden in the case of works of literature.6 What Ingarden had in mind can be illustrated by considering the series of changes which Shakespeare's Hamlet has undergone over the centuries, even though the text of the play has remained unchanged. These reflect, for instance, changes in English pronunciation and styles of acting, in stage set technology, in critical (perhaps also psychological and political) sophistication of successive generations of audiences, and so forth, from one century to the next. Like Hamlet, Fly to Paradise is similarly able, when once created, to float free from its author and enjoy a life of its own, to which contributions will be made for example by advances in web technology, or by developments affecting the 6 See Ingarden (1931, chapter 13). Ingarden was another member of the phenomenological movement, a philosopher of art who defended a view works of art as entities whose ontological structures and histories of change reflect the experiences and intentions of their respective audiences and critics. The ideas presented in this chapter can be viewed as an extension of Ingarden's thinking to the digital entities created through the use of emerging media. Ingarden's 1931 focused primarily on works of literature, his (1962) deals with music, pictures, architecture, and film. See also my (1988). 4 musical/visual/digital genre to which it belongs. The life of Fly to Paradise will be marked in addition by the fact that it will itself spawn further works through mash‐ups and re‐mixes, each of which has the potential to enjoy its own a history of changes.7 Even when talking about Fly to Paradise in Whitacre's original conception, however, we need to distinguish two distinct works. Because the score exists, and because this score can be performed by other singers and under the direction of other conductors, there exists first of all a work of music in the familiar sense in which Beethoven's 3rd Symphony is a work of music. Secondly, however, there is the YouTube video created by Whitacre and posted by him on the web. This is not a work of music in the traditional sense, but rather something which stands to the latter in something like the relation in which Laurence Olivier's Hamlet of 1948 stands to Shakespeare's play. The YouTube Fly to Paradise, which incorporates also complex visual elements, is thus best conceived as an instance of an emerging hybrid sub‐genre of internet art within the larger genre of choral music videos. The instances of this sub‐genre are digital artifacts of a second order – they are digital videos created through combination of other digital videos which had themselves been created in accordance with a score. 3. Emerging Realms of Digital Entities Our thesis is that the emerging science of emerging media will have as its subject matter the new sorts of digital entities (and the new kinds of human interactions which such entities make possible) – entities of a sort which have hitherto been neglected by philosophers interested in the different kinds and categories of beings. We can here only make preliminary gestures in the direction of how the needed treatment of such entities might look, beginning with a brief survey of some examples of families of emerging digital entities in different realms. 3.1. Journalism Even while causing serious problems for traditional print journalism, emerging media have also brought significant benefits to journalists, for whom it is now much easier to identify and validate needed data and to find and establish contact with potential sources. Journalists also have many more venues in which to publish the results of their work, and they are able to create new sorts of venues for example to address specific types of niche audiences, and even to create new niche audiences ex nihilo (McArdle, 2013). The decline of traditional investigative journalism has been compensated by the rise of computational journalism, where journalists collaborate with hackers, 7 https://soundcloud.com/groups/fly‐to‐paradise‐remix, last accessed December 22, 2014. See Moore (2013). 5 data engineers and computer scientists as well as with the wider public, sometimes in the service of political ends, as when in 2009 The Guardian posted on the web some half million pages of expense documents filed by British MPs, and asked its readers to help identify suspicious items (Cohen, et al., 2011, p. 2). Other phenomena illustrating the breaking down of walls between previously separate forms of communication include the live‐blogging of political or sports events (live TV content enhanced through text), or inclusion of dynamic graphical content in statistics‐based journalism (text and numerical content enhanced through video). The ways human beings interact with journalistic content are illustrated most conspicuously by phenomena such as the Drudge Report, Google News, LexisNexis, Wikileaks, of the Global Terrorism Database. These have led in turn to new kinds of recombinations and to new tools by which such recombinations can be steered, illustrated by Google Ranking, Google Fusion Tables, and (for example in the field of social media data management) by resources such as Hootsuite, TweetDeck, SproutSocial, SocialEngage, SocialOomph, and many more. And as the web becomes an ever more important vehicle for journalistic content, so it is being called into play as a locus for the sorts of actions which journalists write about – for example by serving both as trigger for terrorist acts and as megaphone for broadcasting the consequences of such acts – leading thereby to new sorts of intermeshings of the activities of terrorists and other political actors and activists not only with journalists but also with corporations, government agencies, hackers and cybersecurity experts, and so forth. At the same time, as terrorist organizations perfect their use of the internet as megaphone, then they have less and less reason to keep alive the traditional journalists they encounter in their midst and they begin to turn them to their ends in other ways. 3.2. Science The gigantic expansion of publicly available online resources is bringing about similar effects also in science, as massive amounts and varieties of data that had hitherto been hidden are brought out into the open. Information hitherto accessible only to those with specific expertise, or with special local knowledge or permissions, now becomes potentially discoverable by more or less anyone armed with a smartphone and the right sorts of key words. In this way the internet is contributing to the advance of knowledge not only by allowing the traditional strategies used in scientific research to be pursued more successfully, but also by promoting the advance of new technologies for the harvesting, integration and analysis of data and thereby make possible new sorts of scientific research. Scientific data mining is perhaps not something new. Now however we have the mining of scientific literature, where repositories such as Web of Science, Scopus, or PubMed are used for example to predict protein function or to 6 identify potential drug‐drug interactions or potential new uses for drugs (Vinod and Hannah (eds.), 2014) or for the detection of plagiarism and fraud (Garner, 2011). In this way the status of the single journal article as the privileged unit of scientific publishing is being challenged. Here the digital entity constituted by the entirety of scientific papers ever published in a given subject‐area becomes the primary object of research. This digital entity may be augmented in its turn through incorporation of supplementary data, semantic annotations, videos, reviews, corrections, retractions, as well as through dependency links connecting scientific claims and hypotheses to supporting or conflicting data. Scientific publishing itself is being transformed through the proliferation of open access journals and open access collections of articles from traditional journals,8 and also through new experiments in semantically enhanced publishing (Shotton, 2009) or in nanopublishing (Sernadela, et al., 2014), where credit for scientific contributions might be awarded at the level of the single comment or review or of single contributions to supplementary data. These and the many related developments are changing science not only by advancing the discoverability of existing data, but also by advanced the speed at which new discoveries are made, and disseminated, and translated into practical applications. 3.3. Finance When Lehman Brothers filed for Chapter 11 bankruptcy protection on September 15, 2008 this exposed the degree to which new sorts of digital entities – for example new derivative financial instruments – were being produced in a manner hardly appreciated even by those involved in creating them. It would conceivably have been possible to avoid the crash by establishing very rapidly a reliable accounting of Lehman's assets and liabilities. This however would have required tracking back through all of the documentation recording how those assets and liabilities had been acquired and, in the case of derivative instruments, estimating on the basis of the relevant chains of records the precise values of the underlying pools of debt. As we shall see, however, the problematic way in which some of these instruments had been produced would likely have made such an accounting impossible. Consider, for purposes of illustration, the case of the Collateralized Debt Obligation (CDO). For an obligation to exist there must be some legal entity which is the obligor, that is: the bearer of the obligation in question. To have something to play this role there was created for each CDO a 'Special Purpose Vehicle' (SPV), a legal entity – typically a limited company – brought into being with the sole task of serving as obligor for a given CDO. In the simplest case the CDO itself is associated with an underlying debt instruments, each instrument 8 http://www.ncbi.nlm.nih.gov/pmc/, last accessed December 22, 2014. 7 being identified through its CUSIP number.9 But there were also more complex cases, including the so‐called CDO2 (CDO3, CDO4, and so on), where the underlying instruments are themselves further CDOs. In the case of what are called synthetic CDOs, there is no pool of underlying debt at all; rather the cash flow that would be generated by such a pool is simulated using a Credit Default Swap (CDS). This last example provides the clearest illustration of the sorts of problems that can be caused through the production of derivatives of new types. For when, at the critical moment, attempts were made to establish who the obligor might be for a given synthetic CDO, and to whom this obligor is obligated and what the nature of this obligation might be, then it proved impossible to unravel the data to reveal this information – the data were opaque as concerns the logical relations between the entities they referred to. In his (2012) Hernando de Soto pointed out that Lehman's problems were not so much a lack of assets as a lack of knowledge and transparency. Here, as in other spheres, our ability to deal cognitively with the new world of digital entities lags behind our ability to create these entities themselves.10 As the authorities have since recognized, if we are to rectify this lack of knowledge in advance of similar crises in the future, then we need to know what financial instrument are being created and with what owners, documentation, ratings, counterparties, due dates, spreads, coupons, accruals, seniority, and governing law they are associated. At the very least we need a common system of 'legal entity identifiers' (LEIs) for corporate entities (SIFMA, n.d.), just as we have systems for registering the instruments created and for recording the liabilities, assets, etc., with which they are associated at any given time. To create such a system, however, we need to know what types of Special Purpose Vehicles are being created, and we need a coherent uniform terminology for referring to all the types of entities relevant to their effective management in order to allow rapid and reliable mustering of needed information during market crises. The Financial Industry Business Ontology (FIBO) is an important step along the road to the creation of the needed typology (EDM Council, 2014). At the same time there are ontological questions which FIBO does not address but which are salient to our purposes here. What, to give just one example, are you buying when you buy a CDO? Not, it would seem, anything physical; and thus also not something able to stand in 9 This is a 9‐digit National Securities Identification Number for financial products issued from both the United States and Canada under the auspices of Committee on Uniform Security Identification Procedures. 10 Hendricks and Pelle (2014) show that there are serious problems of mismatch between our ability to create information streams and our ability to deal cognitively with the information (and misinformation) they contain also in other areas. 8 relations of cause and effect. Yet the CDO is somehow tied to time and change. It is associated with physical actions of human beings (or computerized agents) who created it, buy and sell it, bet against it, and so forth. CDOs can be advertised and marketed, aggregated and partitioned, packaged and re‐ packaged, ramped‐up and amortized. They are, on the one hand, digital entities, analogous to computer software. But they are at the same time normative entities ('O' stands for 'obligation'). Interestingly, the act of issuance of a CDO is referred to in the FIBO Ontology as a 'production' – in the sense in which this term is used for example in the case of a theatrical or musical production. Like the works produced by Whitacre through his Virtual Choir, CDOs are entities of a new genre; they have mathematical properties and properties which turn on the fact that they are created as a result of the actions of human beings; they have a history of changes – a life – reflecting their interactions with further human beings over time.11 3.4. Social Media What type of entity is a Twitter account? your Facebook Wall? my Kred Influence network? a Bitcoin Blockchain? What is it that gets stolen in a case of identity theft? What sort of entity is managed by an online reputation manager? Is there such an entity as a customer service avatar? Or is it rather the case that, when you put questions to the sequence of human‐head‐shaped pixel arrays on your computer screen (or to the female voice emanating from your smartphone), you are falling victim to some sort of Santa‐Claus‐like fantasy of no ontological significance?12 What sorts of things are the virtual artifacts that are bought and sold each year in the 500 million (real) dollars' worth of commercial activity that is generated by Second Life? Is this13 11 This does not mean that the CDO has causal powers; rather its life is a reflection of the intentions of salient market participants, just as for Ingarden the life of a literary work is a reflection of the intentions of its readers, audience, performers, critics, and so forth. Views in some ways parallel to those of Ingarden are propounded by Rossi‐Landi in his Linguistics and Economics (1975). Rossi‐Landi views linguistic production as an extension of the realm of material production. In the same way, we can now see digital production as an extension of the linguistic production made possible through systems of documents and document acts (see section 5. below, and compare Rossi‐Landi's diagram of levels of production on p. 107 of his (1975). 12 For a discussion of the possibilities, here, see Johansson (2005). 13 From: http://www.Youtube.com/watch?v=mhR43Yt9Pcs, last accessed December 22, 2014. 9 Figure 1: Carl Bildt, then the Swedish Minister of Foreign Affairs, officially opening the Swedish Embassy in Second Life on May 30, 2007 a joke? Philosophers who engage in ontological or metaphysical studies of the nature of existence – thus of the being of entities of different types – have as yet had little to say in response to questions such as these. Since Plato the view has dominated amongst philosophers according to which the two realms of what is abstract and what is concrete do not overlap, and that time and change pertains exclusively to the realm of what is concrete. Already Neuman referred to the merging of media forms as a mark of emerging media. Now, however, we can go further and point to the merging of media forms with other forms of human activity. When you are bidding in a web‐based auction, or writing an online restaurant review, or submitting your résumé to an on‐line resume mill, or tweeting your reactions to some politician's speech, or posting a warning of a police speed‐check on Waze, you are in each case interacting, virtually with other human beings through the medium of the web. No less than the singers in the Virtual Choir, you are also helping to create and shape certain digital entities. The value of the goods being auctioned or of the resume service to potential employers or of the Waze‐posted route to potential drivers; or the reputation of the restaurant or politician or Uber driver (or Uber customer) – all of these, too, are entities belonging to the digital realm which result from the merging of (in some cases very large) numbers of contributions from separate web users, thereby influencing how others will act in the future. 10 3.5. Medicine One of the earliest examples of the fusion of existing forms of human activity with activity mediated by digital technology – in this case activity at the borderlines of clinical therapy and entertainment – is the Eliza Doctor script, a digital simulation of a Rogerian psychotherapist using a primitive version of natural language processing, dating back to the 1960s (Waizenbaum, 1966). Today we see more and more attempts to support (real and make‐believe) clinical therapy through computational means, from simple game‐like applications to monitor diet or exercise, all the way up to the tailoring of medical treatment to the individual patient on the basis of molecular signatures (Winslow, et al., 2012). The DNA testing company 23andme, which is pursuing a business model that is a combination of social media and biotechnology, is using the web, together with lab analysis of saliva samples which subscribers send in by mail, to allow individuals to access genetic information without the intervention of a physician. Originally the plan was to provide information to individuals about both ancestry and gene‐associated health risks. Progress on the latter is currently stymied by legal interventions from the Food and Drug Administration, but the personal genomic data collected by 23andme from over 750,000 individuals is already being used to help clinical and pharmacological researchers gain an improved understanding of inherited diseases for example by allowing the identification through DNA analysis of specialized patient cohorts for clinical studies. The goal, for the future, is to accumulate sufficiently large quantities of personal genome data that it will be possibility to identify for example rare variants which can be used to combat disease. 3.6. War(gaming) Perhaps the most radical example of the ways existing human activities are already being changed by emerging media is in the use of video war games in military training. One crucial feature which distinguishes such technologies from traditional training methods is the degree to which they can be programmed to generate a succession of progressively changing scenarios each calling for appropriate responses on the part of their users in such a way as to bring about what we can think of as a neurological re‐engineering of these users themselves. And as soldiers are trained by means of such methods in ways that incorporate also connections to their weapons and helmet‐ or vehicle‐mounted sensor, display and information systems, then they become progressively integrated with these systems themselves. A soldier training on an individual weapons trainer, for example, links his weapon to a computer and views a wall‐sized screen. The scenario has non‐ combatants in various situations such as protesting, and has potentially hostile targets popping up in the crowd or on the tops of buildings, and so on. The soldier's task is to engage hostile targets and avoid shooting non‐combatants. 11 This training enhances the soldier's ability to assess a crowd, reaction time for drawing his weapon, skill at acquiring a proper aiming posture (the latter typically becomes a reflexive action, where the actual physical sights on the weapon may not even be used). Of course traditional military drill, which existed long before there were digital computers, also brings about a type of physical re‐engineering of the solder.14 But the use of computer systems in military training represents a significant expansion of the possibilities of drill‐coordinated human action. Such possibilities are expanded still further when computer‐driven re‐engineering is applied not just to individual troops but also to entire teams of soldiers, including their associated command centers and integrated computer systems, with the latter being themselves progressively re‐engineered to work every more seamlessly with the warfighters they serve. The goal of such training and consequent continuous re‐engineering is the pre‐ coordination of intermeshed plans and actions of human beings, so that soldiers will coordinate spontaneously when executing the sorts of actions for which they have been trained. The same sort of precoordination is of course found also in the musical world. Just as soldiers are trained in the use of specific weapons and information systems, so instrumentalists are trained in the use of specific instruments, and bring the expertise thereby acquired to their performance in the orchestra. And just as it is the job of the conductor to drill the players in separate sections of the orchestra in such a way that their actions will be coordinated with the actions of all the players of the other instruments in the performance of each work, so it is the job of the commander of a military unit to drill the soldiers, engineers, information technologists and other personnel under his command to ensure coordinated action. There are of course many differences between military command and control and its orchestral counterpart. The former is marked above all by a chain of command – which means a hierarchical organization consisting of modules on a succession of levels, each having its own subsidiary commander, who serves as conductor for that module. It is marked also by the fact that there is a gulf between training and actual warfighting which has no counterpart in the relation between rehearsal and performance on the side of the orchestra. One effect of the advancing realism of the simulated environment in which military training occurs, however, is that this gulf, too, is gradually being eliminated. This is not because warfighting is becoming transformed into something that would take place in a merely virtual reality. Wars will never be composed, through the sort of synchronization and editing that we find in the case of the Virtual Choir. 14 Even the neurological re‐engineering of human beings on the basis of repetitive keyboard action goes back as far as the 14th century, when the clavichord and harpsichord were introduced. 12 Rather, it is because wargaming simulations themselves are coming more and more to incorporate the environment of actual warfare. For in order to advance the realism of training exercises, designers of military wargaming systems are learning how to use real data in building their simulations (Bizub and Brandt, 2011). This means not only data pertaining to topography or climate or weapons capabilities, but also data deriving from data streams generated in actual warfare. As Corey Mead expresses it in his War Play: Video Games and the Future of Armed Conflict (2013): The reality is that soldiers are now a form of information technology, responsible for a far broader range of roles, decisions, and systems‐based interactions than in any previous conflict. ... Today soldiers' skills are measured largely in relation to the technological systems the soldiers will be using. (2013, p. 4)15 For many soldiers the re‐engineering of their own brains through video simulations begins already before they join the military. It is in part for this reason that the Army is able to use virtual reality simulations as effective military recruitment tools (Rothbard, 2013). In some cases such re‐engineering continues after the soldier leaves the Army, as when virtual reality exposure therapy is used in treating veterans' post‐traumatic stress disorder (Mead, 2013a). And finally wargaming becomes embedded into military reality also at higher levels, as wargaming is used to test and refine military plans, and as the wargaming industry itself becomes a target of enemy intelligence, which uses it to predict military thinking on tactics and strategy. 4. From Speech Act Theory to Social Ontology What conclusions are we to draw from these examples for our understanding of emerging media studies, and more specifically for our understanding of what the subject‐matter of a future science of emerging media might look like? Our general thesis can now be reformulated as follows: that such a science will study what we can think of as social reality as it is becoming gradually augmented by combinations and re‐combinations of digital entities of various types. Our philosophical task, therefore, starts out from the ontology of social reality, itself an emerging philosophical subfield to which John Searle has made 15 Mead makes an important point. But he also exaggerates. For soldiers' skills were in fact always measured in this way. Before the Lightweight Handheld Mortar Ballistic Computer (LHMBC) they had to use a plotting board, compass, numerous charts, maps, and aiming stakes in order to calculate proper angles and distances to a target. The fact that this technology was replaced by another, computer-based technology, does not mean that warfighters are now more technically savvy then they were before. Rather it means that they are now focused upon the new technological possibilities. 13 especially important contributions (1995, 2010). Searle's approach grows out of the theory of speech acts, and of the ways in which, by using language, we can change the reality around us – for example by issuing commands, apologizing, insulting our neighbor, and so on (Mulligan, 1987). When I order a beer, from this perspective, then the world is changed. I become obligated to pay the barman for my beer. The barman, when he registers my order, in turn becomes obligated to me in turn to bring me my beer. We both – more or less consciously – observe thereby a nested set of rules relating to how to deal with money, receipts, other customers, wait staff, glassware, and so forth, and as a result our social reality is here augmented in a quasi‐legal way: the human being to whom I speak counts as a waiter, I count as a customer, my speaking counts as the making of an order, the piece of paper that is left on my table counts as a bill; and so on. A series of such changes unfolds itself whenever human beings interact, changes that result, in the simplest case, from the speech and gestures and habitus of the persons involved. I speak to the barman in a kindly voice and he becomes ever‐ so‐slightly happier than he had been in the seconds before I spoke. The friends who are with me in the bar are cajoling each other with questions about local restaurants, interrupting each other constantly as suggestions as to where we might eat are supplied, reacting to each other's suggestions with more questions and complaints. As this complex network of intermeshing conversations unfolds, slight changes occur in the moods and appetites and beliefs and expectations and plans and obligations of each of those involved, changes which occur in tandem with the unfolding of speech and gesture. Over longer periods of time some of these changes will have cumulative effects in the form of more profound alterations in the persons involved, as friendships deepen or are broken off, couples are formed, joint plans are hatched, and so forth. 5. From Speech Acts to Document Acts Such behaviors are as old as the earliest human societies. But they have in course of time, and especially with the invention of writing, been gradually augmented by radically new sorts of phenomena – including rituals and theatrical performances, hierarchical political, governmental, legal and military organizations, formal education, professions and careers, science and theology, trading in extended markets. In modern times the repertoire of such phenomena has expanded enormously, including insurance, banking, stocks and shares, accounting, corporations, land title, wills, public administration, urban planning, building regulations, and so forth. Many of the latter are associated with what we might call 'document acts', reflecting the fact that they essentially involve some engagement with one or more documents, whether this be in paying a bill, buying a theater ticket, 14 contesting a will, issuing a driver's license receiving a professional diploma, or being granted a divorce. In (Smith 2012, 2014) I have attempted to show how Searle's approach to social reality might be extended by taking account of the ways in which the sorts of ontological changes vehiculated by speech and gesture have in the course of history become ever more sophisticated precisely through the creation and refinement of document systems of various sorts, allowing ever more complex forms of documents acts. Hernando de Soto's The Mystery of Capital (2000) is a study of the role played by document systems and document acts in the realm of economic activity. He points especially to the differential rates of economic development as between countries with and countries without effective systems for documenting assets. But a similarly vital role is played by document systems and document acts also in our systems of law.16 The phenomenon of marriage, for example, started as a creature of customary law; but it has evolved through time into what we might refer to as a production of the written law (in ways varying from jurisdiction to jurisdiction) and it is tied to multiple different sorts of document acts. The latter are not merely incidental aids to the performance of speech acts. They are indispensable to the very existence of the human activity we call getting married as this activity is realized in all civilized societies. Something similar applies to the activities which form our systems of politics and administration, of construction and urban planning, of science and education, of medicine and public health, citizenship and immigration, and also of organized warfare – in other words to practically all of the forms of human action characteristic of modern civilization in the period leading up to the birth of the digital computer. 6. The Institutions of Emerging Media We are now witnessing a period in which digital technologies, and especially the internet, are taking such augmentations of human reality one step further. The transition from speech acts to document acts (or, more generally, from the world of speaking to a world which includes also writing and printing) is now being extended further as we enter into a world which includes (for instance) cell phones and RFID chips. The law is being augmented17 by e‐law; commerce by e‐ commerce; publishing by e‐publishing; tourism by the virtual tourism (and associated virtual fieldwork) of Grand Theft Auto.18 The market for real estate is being augmented by Zillow and Trulia; the market for taxi services by Uber and Lyft and Sidecar. By turning to the Sim market you can also '[b]uild military empire and economic superpower! You can join your forces with others, create alliances and reinvent worlds map.' 19 16 Boehme‐Nessler, Volker, Pictorial law: from law of words to law of pictures, Berlin ; London : Springer, 2011. 17 Potentially also transformed (Katsh 1989, 1995). 18 Miller, 2012, Part 1. 19 'When you will know all the rules and habits in e‐Sim you can focus on: 15 In almost all of these cases such augmentations retain the document structures with which we are familiar, but make the preparation, manipulation and discovery of documents – for instance filling out your tax forms – much easier. In some cases, however, new sorts of documents are being brought into being, and created and discovered and manipulated in new sorts of ways. A brief encounter in a bar is recorded on someone's cell phone and posted to YouTube; it thereby takes on a life of its own; the video is transformed into part of a preview demo to raise funding for an App that will create a movie of your life based on 1 second YouTube fragments; later on it is used as grounds for divorce.20 Maurizio Ferraris (2014) refers in this connection to the 'total mobilization' that is being brought about through the omnipresence of the mobile phone in our lives and its breaking down of formerly existing barriers between the world of work and the worlds of family and leisure. 7. Conclusion: The Science of Emerging Media We explored how, through emerging media, new kinds of digital entities are being created, whose nature and potential impact on human beings are still hardly understood. We also saw how emerging media are creating new forms of human interaction, for example allowing new forms of massively shared agency through the Intermeshing of plans involving human beings separated in both space and time. Our proposal, in sum, is that the future science of emerging media will have as its subject matter precisely these digital entities and the new sorts of human interactions which they make possible. It will be distinguished from its neighboring disciplines – such as computer science and the science of human‐computer interaction – by the fact that it will treat such entities not from the (hardware or software) perspectives of data entry and data processing and data retrieval but rather precisely from the point of view of how they contribute to shaping and augmenting the social reality of the future. And while I have focused here on the positive effects of such shaping and augmenting, it is worthwhile to note also some of the negative effects, which will also, of course, fall within the domain of the envisaged science. Interestingly, some of these negative effects are listed among the predictions documented by Neuman in his (1991, pp. 6f.). These include, inter alia: − overwhelming the individual with a paralyzing overload of information, - traveling to another countries - starting your own business - setting up a newspaper - political career - military career - business career - and finally conquering other countries!' From http://primera.e‐sim.org/, last accessed December 22, 2014. 20 http://1secondeveryday.com/, last accessed December 22, 2014. 16  blurring the distinction between public and private communication, and  giving governments and security agencies the ability to closely monitor the behavior of citizens, including what they read and see in the media, what they say through electronic media, and every economic transaction they conduct involving a bank or credit card. We could add, now, that enemy governments and security agencies, too, as well as criminal and terrorist hackers, will display similar abilities. It is the task of science not to change reality but to understand it. The task of the science of emerging media as conceived in the foregoing is, correspondingly, to understand the digital entities which will shape our future. Only on the basis of such understanding, I suggest, will we be in a position to mitigate some of these negative consequences. Acknowledgments With thanks to Mike Bennett, Mark Kerin, Bill Mandrick, Peter Morosoff, Kevin Mulligan, and J. K. Nyíri for helpful comments. References Bizub, Warren and Brandt, Julia (2011), 'Transitioning to the Next Generation (Nextgen) Defense Training Environment (DTE)', Proceedings of the 2011 Winter Simulation Conference, S. Jain, R. R. Creasey, J. Himmelspach, K.P. White, and M. Fu (eds)., 2564‐2575. Cohen, Sarah, Chengkai Li, Jun Yang, and Cong Yu (2011), 'Computational Journalism: A Call to Arms to Database Researchers', CIDR 2011, 148‐151. Croteau, David and Hoynes, William (2003) Media Society: Industries, Images and Audiences (third edition), Thousand Oaks: Pine Forge Press. de Soto, Hernando (2000), The Mystery of Capital: Why Capitalism Triumphs in the West and Fails Everywhere Else, New York: Basic Books. de Soto, Hernando (2012), 'Knowledge Lies at the Heart of Western Capitalism', Financial Times, January 29. EDM Council (2014), Financial Industry Business Ontology (FIBO) Semantics Repository, http://www.edmcouncil.org/semanticsrepository. Last accessed, December 22, 2014. 17 Ferraris, Maurizio (2014), 'Total Mobilization', The Monist, 97:2, 201‐222. Garner, H.R. (2011), 'Combating Unethical Publications with Plagiarism Detection Services', Urologic Oncology: Seminars and Original Investigations, 29 (1), 95‐99. Heidegger, Martin (1949) 'Die Frage nach der Technik', in Gesamtausgabe, Volume 7, Vorträge und Aufsätze (1936–53), edited by Friedrich‐Wilhelm von Herrmann, Frankfurt am Main: Vittorio Klostermann, 2000), English translation as 'The Question Concerning Technology' in: Martin Heidegger, Basic Writings: Second Edition, edited by David Farrell Krell (New York: Harper Collins, 1993). Hendricks, Vincent F. and Pelle G. Hansen (2014), Infostorms: How to Take Information Punches and Save Democracy, New York: Copernicus Books / Springer. Ingarden, Roman (1931), Das literarische Kunstwerk. Eine Untersuchung aus dem Grenzgebiet der Ontologie, Logik and Literaturwissenschaft, Halle: Niemeyer; English translation of 3rd edition by G. G. Grabowicz, The Literary Work of Art, Evanston, Ill.: Northwestern University, Press, 1973. Ingarden, Roman (1962), Untersuchungen Zur Ontologie Der Kunst: Musikwerk, Bild, Architektur, Film, Tübingen: Niemeyer. English translation as The Ontology of the Work of Art, by Raymond Meyer with John T. Goldthwait, Athens, Ohio: Ohio University Press, 1989. Johansson, Ingvar (2005), 'Money and fictions' r Kapten Mnemos Kolumbarium: Philosophical Communications, 73‐101. Katsh, M. Ethan (1989) The Electronic Media and the Transformation of Law, Oxford: Oxford University Press. Katsh, M. Ethan (1995), Law in a Digital World, Oxford: Oxford University Press. McArdle, Megan (2013), 'Lazy Journalists Aren't to Blame for Death of Print', December 2013, http://www.bloomberg.com/news/2013‐12‐05/lazy‐journalists‐aren‐t‐to‐blame‐ for‐death‐of‐print.html, last accessed December 22, 2014. Mead, Corey (2013), War Play: Video Games and the Future of Armed Conflict, New York: Houghton‐Mifflin‐Harcourt. Mead, Corey (2013a), 'Virtual‐Reality Videogames May Be the Best Treatment for PTSD', Serious Gaming (The Daily Beat), http://www.thedailybeast.com/the‐ hero‐project/articles/2013/09/26/virtual‐reality‐video‐games‐may‐be‐the‐best‐ treatment‐for‐ptsd.html. 18 Miller, Kiri (2012), Playing Along: Digital Games, YouTube and Virtual Performance, Oxford: Oxford University Press. Moore, Joseph G. (2013), 'Musical Works: A Mash‐Up', in: Art and Abstract Objects, edited by Christy Mag Uidhir, Oxford: Oxford University Press, 284‐306. Mulligan, Kevin (1987), 'Promisings and other Social Acts: Their Constituents and Structure', in: K. Mulligan (ed.), 1987, Speech Act and Sachverhalt: Reinach and the Foundations of Realist Phenomenology, Dordrecht: Nijhoff, 29‐90. Neumann, W. Russell (1991), The Future of the Mass Audience, Cambridge University Press. Ong, Walter J. (2002), Orality and Literacy: The Technologizing of the Word, New York: Routledge. Rossi‐Landi, Ferruccio (1975), Linguistics and Economics, The Hague: Mouton. Rothbard, Sarah (2013), 'The Six‐Point Inspection: The Army's Most Effective Recruiting Tool Is a Video Game', http://www.slate.com/blogs/future_tense/2013/09/04/war_play_countdown_a nd_the_outer_limits_of_reason_reviewed.html. Scheler, Max (1960), Probleme einer Soziologie des Wissens as reprinted in volume 8 of Scheler's Gesammelte Werke, edited by Maria Scheler, Bern: Francke), English translation by Manfred S. Frings as: Problems of a Sociology of Knowledge, London: Routledge and Kegan Paul, 1980. Sernadela, Pedro, Eelke van der Horst, Mark Thompson, Pedro Lopes, Marco Roos, José Luís Oliveira (2014), 'A Nanopublishing Architecture for Biomedical Data', 8th International Conference on Practical Applications of Computational Biology & Bioinformatics (PACBB 2014), (Advances in Intelligent Systems and Computing 294), 277‐284. Searle, John R. (1995), The Construction of Social Reality, New York: Basic Books. Searle, John R. (2010), Making the Social World: The Structure of Human Civilization, New York: Oxford University Press. Shotton, David, Katie Portwin, Graham Klyne, and Alistair Miles (2009), 'Adventures in Semantic Publishing: Exemplar Semantic Enhancements of a Research Article', PLoS Computational Biology, 2009; 5: e1000361. SIFMA (n.d.), Legal Entity Identifier Resource Center, http://www.sifma.org/issues/operations‐and‐technology/legal‐entity‐ identifier/overview/, last accessed December 22, 2014. 19 Smith, Barry (1988), 'Practices of Art', in J. C. Nyíri and Barry Smith (eds.), Practical Knowledge. Outlines of a Theory of Traditions and Skills, London/Sydney/New York: Croom Helm, 172‐209. Smith, Barry (2012), 'How to Do Things with Documents', Rivista di estetica, 52 (50), 179–198. Smith, Barry (2013), 'Diagrams, Documents, and the Meshing of Plans', in: A. Benedek and K. Nyíri (eds.), How To Do Things With Pictures: Skill, Practice, Performance, Frankfurt: Peter Lang, 165‐179. Smith, Barry (2014), 'Document Acts', in Anita Konzelmann‐Ziv, Hans Bernhard Schmid (eds.), 2013. Institutions, Emotions, and Group Agents. Contributions to Social Ontology (Philosophical Studies Series), Dordrecht: Springer, 19‐31. Vinod D. Kumar, Hannah Jane Tipney (editors), (2014), Biomedical Literature Mining (Methods in Molecular Biology, Volume 1159), New York: Springer. Weizenbaum, Joseph (1966), 'ELIZA – A Computer Program for the Study of Natural Language Communication between Man And Machine", Communications of the ACM, 9 (1), 36–45. Winslow, R. L., N. Trayanova, D. Geman, M. I. Miller (2012), 'Computational Medicine: Translating Models to Clinical Care', Science Translational Medicine, 4 (158), 158rv11.