Keynote Speakers
• Clare Painter, University of NSW (read abstract)
• Edward McDonald, University of Auckland (read abstract)
• Jim Martin, University of Sydney
• John Knox, Macquarie University (read abstract)
• Kay O'Halloran, National University of Singapore (read abstract)
• Maree Stenglin, University of Sydney (read abstract)
• Naomi Knight & Chris Cleirigh, University of Sydney (read abstract)
• Radan Martinec, Ohio Northern University (read abstract)
• Sigrid Norris, Auckland University of Technology (read abstract)
• Susan Hood, UTS (read abstract)
• Theo van Leeuwen, UTS (read abstract)
Ambience in picture book images
Clare Painter
Dept of English, Media and Performing Arts, UNSW
Narrative picture books frequently deal with highly affective content. This may be matters of personal and family relationships central to young children's lives or, for older readers, political and historical issues relevant to their broader cultural context. In managing the interpersonal work that is involved in both engaging the reader and developing thematic content, it is the visual semiotic that takes the leading role in these bi-modal texts. While there are a number of visual systems relevant here, this paper will focus specifically on the way an emotional mood or ‘ambience' is created visually by the use of colour in these narrative images. ‘Ambience' was originally proposed by Stenglin (2004) as a relevant dimension of the meaning of 3D space and it would appear equally relevant for the interpretation of printed images. This means that whereas the creation of circumstantial meaning in language is considered to belong within the ideational metafunction, in the semiotics of the image, aspects of the circumstantial setting play as great a role within the interpersonal function. Drawing on collaborative research with J.R. Martin and Len Unsworth into the semiotics of children's picture books, this paper will outline and discuss a proposed system of AMBIENCE, showing how the resources of colour are used interpersonally and in relation to visual choices within the textual metafunction, exemplifying from key examples of the picture book genre. This enterprise raises questions about other visual resources that contribute to the realisation of ambience, about other roles for colour within the visual semiotic and about the distribution of meanings in bi-modal texts such as picture books.
Stenglin, M. 2004 Packaging curiosities: Towards a grammar of three-dimensional space. PhD thesis, University of Sydney.
Multimodality and Multifunctionality in Performance
Edward McDonald
University of Auckland
The penultimate scene of Mozart's Don Giovanni (1787) brings together three of the main characters – the Don himself, his comic servant Leporello, and his nemesis the Commendatore – in a complex interplay of dramatic character and musical motif held together through an inexorable tonal and rhythmic progression. In this supreme example of opera, the musico-verbal genre par excellence in the Western tradition, how does the librettist create the different characters through their words, how does the composer provide them with music fit for their roles and emotions, and finally how does the performer bring all these to life in performance? Starting from a simplified performance of a key section of this scene, the current study goes on to characterise the relevant linguistic and musical systems involved, and to problematise the respective contributions of the verbal, musical, visual and kinetic semiotics involved in performance. Such an analysis raises important questions about the interaction between different semiotic systems and how best a multi-functional semiotic model can capture their similarities and differences without bias towards any particular system.
Visualising the verbal - verbalising the visual: hard news on an online newspaper home page
John Knox
Linguistics Department, Macquarie University
The discourse of newspapers is in a constant process of change. This applies to verbal story-telling practices (e.g. Iedema, 1995; Iedema, Feez & White, 1994), visual story-telling practices (e.g. Hall, 1981; Hartley, 2007), and visual-verbal story-telling practices (e.g. Caple, 2007; Knox, 2007; Kress & van Leeuwen, 1998). Between February, 2002 and November, 2005, the Sydney Morning Herald online , a major Australian newspaper which went online in 1995, showed a remarkable degree of change in the design of its home page. However, over the same time period, the use of images in hard-news stories on its home page remained remarkably consistent, both diachronically and synchronically. These hard-news images are small ‘thumbnails', and are most typically close crops of faces. Their small size, their consistent and limited subject matter, and their positioning in news stories represents a new practice in hard-news reporting, and raises questions about the role they play in the multimodal story-telling practices of the newspaper, and about the discursive practices of online newspapers more generally.
SFL-based studies of the interaction between text and image in newspaper stories have, for the most part, concentrated on ‘multiplicative meanings' (Lemke, 2002) in higher strata (e.g. Caple, 2007; Economou, 2006; Macken-Horarik, 2003a, 2003b). The small thumbnails on the home page of the Sydney Morning Herald online invite a complementary analytical approach, in which graphology also plays an important role. This paper accepts that invitation, and considers the ways in which verbiage and image combine to make meaning in newsbites (the short headline-plus-lead stories which feature on online newspaper home pages - see Knox, 2007) on the home page of the Sydney Morning Herald online . Multimodal discourse analysis continues to explore ways in which to model the multiplicative meanings generated when different semiotic modes interact in text. This paper asks whether the texts under investigation are truly multimodal, or whether, due to their historical and discursive location, they instantiate a unique expression of the visual potential of language.
Caple, H. (2007, August 15-18). Multimodal communication in an Australian broadsheet: A new news story genre. Paper presented at the 4th International Symposium on Genre Studies, Taburao, Brazil. Available online at: http://www3.unisul.br/paginas/ensino/pos/linguagem/cd/English/12i.pdf
Economou, D. (2006). The big picture: The role of the lead image in print feature stories. In I. Lassen, J. Strunck & T. Vestergaard (Eds.), Mediating ideology in text and image: Ten critical studies (pp. 211-233). Amsterdam: John Benjamins.
Hall, S. (1981). The determinations of news photographs. In S. Cohen & J. Young (Eds.), The manufacture of news: Social problems, deviance and the mass media (Revised ed., pp. 226-243). London: Constable.
Hartley, J. (2007). Documenting Kate Moss: Fashion photography and the persistence of photojournalism. Journalism Studies, 8 (4), 555-565.
Iedema, R. (1995). Political newsreporting: The media as 'Secondary Orality'. Social Semiotics, 5 (1), 65-99.
Iedema, R., Feez, S., & White, P. R. R. (1994). Stage two: Media literacy. A report for the Write it Right Literacy in Industry Research Project Sydney: Disadvantaged Schools Program, N.S.W. Department of School Education.
Knox, J. S. (2007). Visual-verbal communication on online newspaper home pages. Visual Communication, 6 (1), 19-53.
Kress, G., & Van Leeuwen, T. (1998). Front pages: (The critical) Analysis of newspaper layout. In A. Bell & P. Garrett (Eds.), Approaches to media discourse (pp. 186-219). Oxford: Blackwell.
Macken-Horarik, M. (2003a). A telling symbiosis in the discourse of hatred: Multimodal news texts about the 'children overboard' affair. Australian Review of Applied Linguistics, 26 (2), 1-16.
Macken-Horarik, M. (2003b). Working the borders in racist discourse: The challenge of the 'Children Overboard Affair' in news media texts. Social Semiotics, 13 (3), 283-303.
Lemke, J. L. (2002). Travels in hypermodality. Visual Communication, 1 (3), 299-325.
Inter-Semiotic Expansion of Meaning: Hierarchical Scales and Metaphor in Mathematics Discourse
Kay L. O'Halloran, National University of Singapore
Abstract
The inter-semiotic construction of experiential meaning across language, visual images and mathematical symbolism as a multi-directional and multi-layered process which transverses the hierarchy of scales in the systemic functional multimodal discourse analysis (SF-MDA) framework for mathematics is explored (O'Halloran, in press) . The SF-MDA approach is based on the concept of hierarchical ranks for grammar and discourse on the content plane, and hierarchical levels on the context plane (Baldry and Thibault, 2006; Djonov, 2007; Halliday, 2004; Martin, 1992) . The inter-semiotic orchestration of meaning is traced across the ranks and levels to demonstrate how experiential relations across language, visual images and mathematical symbolism are reconfigured and realigned solve mathematical problems. The inter-semiotic realignment of process and participant configurations gives rise to metaphorical forms of expression which do not follow the same semantic drift towards the entity (i.e. noun) found in grammatical metaphor in language (O'Halloran, 2005, 2007) . The significance of the mathematical symbolism and its integration with language and mathematical diagrams and graphs are evident in the analysis. However, mathematical symbolism remains on the margins of semiotic research in mathematical and scientific discourse.
The SF-MDA analysis reveals the complexity and the difficulty of tracing, annotating and describing inter-semiotic shifts of meanings across scales, and it highlights the need for a new approach to multimodal discourse analysis which extends beyond static page-based transcription. The lack of an effective practical means for multimodal analysis of texts and events hinders the development of the theoretical and conceptual frameworks necessary to advance our understanding and knowledge of multimodal phenomena. Therefore, the objective of the Multimodal Analysis Lab in the Interactive Digital Media Institute (IDMI) at the National University of Singapore is to undertake a major research programme where social scientists and computer scientists collaborate to develop prototype software to dynamically model and analyse multimodal discourse.
References
Baldry, A. P., and Thibault, P. J. (2006). Multimodal Transcription and Text Analysis . London: Equinox.
Djonov, E. (2007). Website Hierarchy and the Interaction between Content Organization, Webpage and Navigation Design: A Systemic Functional Hypermedia Discourse Analysis Perspective. Information Design Journal 15: 2, 144-62.
Halliday, M. A. K. (2004). An Introduction to Functional Grammar (3rd edition, revised by C. M. I. M Matthiessen ed.). London: Arnold.
Martin, J. R. (1992). English Text: System and Structure . Amsterdam: John Benjamins.
O'Halloran, K. L. (2005). Mathematical Discourse: Language, Symbolism and Visual Images . London and New York: Continuum.
O'Halloran, K. L. (2007). Systemic Functional Multimodal Discourse Analysis (SF-MDA) Approach to Mathematics, Grammar and Literacy. In A. McCabe, M. O'Donnell & R. Whittaker (Eds.), Advances in Language and Education 75-100. London & New York: Continuum.
O'Halloran, K. L. (in press). Inter-Semiotic Expansion of Experiential Meaning: Hierarchical Scales and Metaphor in Mathematics Discourse. In C. Jones & E. Ventola (Eds.), New Developments in the Study of Ideational Meaning: From Language to Multimodality . London: Equinox.
‘Spaced out': An evolving cartography of a visceral semiotic
Maree Stenglin
University of Sydney
[Spaces] . . . ‘their influences are at once the most subconscious and the most powerful, the hardest to verbalise but the easiest to recall' (Falk & Dirking 1995: 31)
From the moment of our conception 3D space envelops all aspects of our lives. In addition to sheltering and protecting us from the elements, it provides the place where we work, relax, sleep, eat, learn and carry out the myriad of complex tasks that constitute our lives. It also encompasses us as we enact our personal and social relationships, and can evoke some raw and visceral responses. Our experience of space, moreover, tends to be dynamic as it often involves movement through a series of interconnected areas that unfold both spatially and temporally as we ‘flow' through them.
This paper aims to explore this ubiquitous and pervasive modality. It begins by briefly examining the work on architecture and space that has already been carried out by theorists working with the philosophical, structural and social traditions of semiotics. It then identifies some of the areas that require further consideration and pursues these in order to chart a metafunctionally diversified account of this omnipresent modality.
Deciphering Laughter: interpreting the manifestations of having a laugh
Naomi Knight and Chris Cleirigh
University of Sydney
How much lies in Laughter: the cipher-key, wherewith we decipher the whole man.
-Thomas Carlyle
Laughter is a popular pastime, a pleasant and sometimes infectious phenomenon.
Scholars have variously described it as an expulsion of built-up energy in maintaining inhibitions (Freud 1905), a tool for gaining and maintaining superiority over others (Bergson 1899), and more recently as a communicative device (cf. Glenn 2003).
From its origins as a physiological reaction to stimuli, the status of laughter is approaching that of speech in recent study, and it is clear that laughter must be examined for its meaning-making potential. The expulsive burst of the body becomes the expression of the social body in an environment of social meanings.
In this paper, we will explore laughter for its biological and semiotic functions as situated in SFL theory, and will argue that laughter is functional and manifests meanings in social life between members of communities. In particular, the interpersonal functions of laughter in the expression of attitude will be exhibited, along with its role in construing and reinforcing affiliation. This includes the different ways that laughter can be realised (its phonological valeur) and its importance as a shared expression.
It will be argued that laughter, in manifesting conscious states, can perhaps be considered not paralinguistic, but proto-semiotic in its development. The status of laughter is a continuation of our evolutionary path to meaning, and lies in between behaviour and semiosis. The notions of language strata and realization will be discussed in relation to the meaning potential of laughter to situate it in a theory of semiosis.
Bergson, H. 1940 [1899]. Le rire: essai sur la signification du comique. Paris: Quadrige.
Freud, S., Strachey, J., & A. Richards. 1976 [1905]. Jokes and their relation to the
unconscious. New York: Penguin.
Glenn, P. 2003. Laughter in interaction. Cambridge: Cambridge University.
Mapping out semiosis: existing and nascent systems and the semiotician's task
Radan Martinec
One of the reasons why various modalities of meaning other than language have been sidelined is that they have not been considered to function systematically. I will argue that, although gestures and some other kinds of embodied action certainly fall short of being ‘languages', aspects of them can be modeled as semiotic systems, in which certain meanings are systematically and regularly realized by certain forms. I will argue that identification of such systems in itself constitutes a kind of emancipation of such modalities that have so far been considered by some to languish on semiotic margins, and brings them closer to the mainstream. Gestures and other kinds of embodied action have evolved over long periods of time - some scholars would in fact argue that they preceded language. If certain areas of them do constitute semiotic systems, the semiotician's work should thus be to simply map them out. The same cannot be said of very recent systems, such as new media design, which are still in a nascent form. In such cases, the semiotician's task is not only to map out their already existing structures but perhaps to also suggest where the current practices seem to make sense and where they do not and why, as well as to suggest possibilities that have not yet been exploited but could be to an advantage. An interesting case is the system of image-text relations, which has been evolving for many years but different parts of which seem to be suited to new and others to old media design. I will discuss and exemplify the general issues raised in the above and then focus on the image-text system and the suitability of its different parts to different kinds of media.
Can we really speak of Semiotic Margins?: a theoretical un-focussing of language.
Sigrid Norris
Auckland University of Technology
When taking the mediated action as our unit of analysis and investigating the multiple interlinking modes of communication that social actors utilize in any higher-level action (such as a conversation) in order to create that action, we take a step away from any particular mode and look at communication in a more holistic way. Certainly, the question of what is part of language remains an important one, but in this concept the question of where does language actually fit into this holistic notion of communication appears to turn into an even greater one. In a way, we have to un-focus language in order to reach some kind of holistic understanding of communication in use, thereby questioning if our language-based ways of looking at the world of communication suffice or if we need to look at new ways of theorizing the complexities that all social actors utilize on a day-to-day basis without ever thinking about it. Language, often viewed as the primary mode of communication, is not always hierarchically superior to other modes in a communicative event. Building on this very notion, I will question whether we can really speak of semiotic margins.
Embodying language: bringing gesture in from the paralinguistic cold
Sue Hood
University of Technology, Sydney
This paper draws on a study of gesture in teacher talk in face-to-face classrooms. The study proceeds from a social semiotic perspective, asking how meanings are realised in bodily movements, and how those bodily movements relate to speech and interact with other visual semiotics (Martinec, 2000; 2001). The analyses explore the ways in which metafunctional meanings are realised in gestures, and have resulted in the construction of system networks for some semantic systems (identification and appraisal). From a multimodal perspective, gestures and speech can be seen to cooperate as co-expressive of meanings or as complementary in terms of metafunctional load. Gestures also play a key role in coordinating the integration of other semiotic systems. Having focused to this point on realisation, I conclude with some reflections on the relationship of gesture and speech from the point of view of instantiation, that is on how meaning potential is distributed and may be more or less committed in one or other mode.
Martinec, R. 2000. Rhythm in multimodal texts. Leonardo 33 (4), 289-297.
Martinec, R. 2001. Interpersonal resources in action. Semiotica . 135 1/4, 117-145.
Theo van Leeuwen
University of Technology, Sydney
Recent studies of voice quality (Van Leeuwen, 1999), colour (Kress and van Leeuwen, 2002) and typography (Van Leeuwen, 2006) have shown that these semiotics cannot readily be modeled as (mostly) binary systems of increasingly delicate choices, and are better modeled as parametric systems of simultaneous choices at a single level of delicacy.
A parallel has been drawn with Jakobson and Halle's distinctive feature theory (1956), with the proviso that parametric systems function not only distinctively, but also semantically.
The paper will explain and illustrate examples of parametric systems and argue three key theoretical points:
• Systems, whether parametric or taxonomical, are the way they are because of the ways in which they are socially regulated, and the social regulation of semiotic systems in a given environment reflects the way social relations are organized in that environment. The move towards parametric forms of semiotic organization reflects the move from hierarchical and binary social relations to distributed ‘lifestyle community' social relations.
• In terms of the social semiotic model of stratification developed in Kress and Van Leeuwen (2001), parametric systems belong to the stratum of ‘production'.
• Meanings which are predominantly realized in the ‘production' stratum differ from those predominantly realized in the ‘design' stratum in four ways: (1) they realize the specific semantic field of identity; (2) they are not arbitrary but based on the principles of experiential metaphor; (3) they can only be modeled parametrically; (4) they evolve, over time, into clusters of parameters that draw their meaning from cultural connotation.
Jakobson, R. and Halle, M. (1956). Fundamentals of Language . The Hague: Mouton
Kress, G. and Van Leeuwen, T. (2001) Multimodal Discourse: The Modes and Media of Contemporary Communication . London: Arnold
Kress, G. and Van Leeuwen, T. (2002) Colour as a semiotic mode: notes for a grammar of colour. Visual Communication 1(3): 343-368
Van Leeuwen, T. (1999) Speech, Music, Sound . London: Macmillan
Van Leeuwen, T. (2006) ‘Towards a semiotics of typography'. Information Design Journal 14(2): 139-155