Visual and Multimodal Literacy

May 2013

Definition of Literacy: Australian Curriculum

Literacy involves students in listening to, reading, viewing, speaking, writing and creating oral, print, visual and digital texts, and using and modifying language for different purposes in a range of contexts.

Definition of Literacy: NSW English K10 English Syllabus

A synthesis of language, thinking and contextual practices through which meaning is shaped. Effective literacy is intrinsically purposeful, flexible and dynamic and involves interactions in a range of modes and through a variety of media.

J. Dawkins 1991, Australias Language: The Australian Literacy and Language Policy, AGPS, Canberra.

Definition: Visual Literacy

The ability to decode, interpret, create, question, challenge and evaluate texts that communicate with visual images as well as, or rather than, words. Visually literate people can read the intended meaning in a visual text, such as an advertisement or a film shot, interpret the purpose and intended meaning, and evaluate the form, structure and features of the text. They can also use images in a creative and appropriate way to express meaning.

Definition: Visual Features

Visual components of a text such as: Placement Salience Framing Representation of action or reaction Shot size Social distance Camera Angle
NSW English K-10 Syllabus: Glossary

Handas Surprise

Definition: Visual Language

Language that contributes to the meaning of an image or the visual components of a multimodal texts and are selected from a range or visual features like: placement salience framing representation of action or reaction shot size social distance camera angle

Visual language can also include elements such as:

symbol colour scene and frame composition setting and landscape lighting use of editing

Georgia OKeeffe Hawaii 1939

Definition: Multimodal
Comprising more than one mode. A multimodal text uses a combination of two or more communication modes, for example, print, image and spoken text as in film or computer presentations.
English K-10 Syllabus: Glossary

Both words and picture are required to make meaning here in Theres a Sea in My Bedroom.

Definition: Multimodal
Australian Curriculum
Multimodal texts combine language with other means of communication such as visual images, sound track or spoken word, as in film or computer presentation media. Texts provide opportunities for learning about aspects of human experience and about aesthetic value.
Content Structure, Texts, Australian Curriculum: English

New and digital literacies

Multimodality: A Literature Companion

What is multimodality? Multimodality incorporates different semiotic or meaningmaking systems. A picture book, information book or movie can be described as a multimodal text because each combine both print and still or moving images. Even more so, a screen based digital text is multimodal if it combines moving images, graphics, sound and hypertext Random House Website Moving images, links. videos, hypertext links, blog, quiz, apps, podcasts ( p.181 A Literature Companion)

Can picture books and other paper texts be multimodal?

Recent discussion about the texts delivered via new technologies has focused on the fact that they comprise more than one semiotic system. Texts may utilise visual images as well as linguistic text. Texts delivered via traditional communications technologies (such as paper technologies) have long employed more than one semiotic system but we have conveniently overlooked this and focused mainly on the linguistic semiotic system. Texts delivered via paper technology are often multimodal (they may comprise linguistic and visual semiotic systems) while texts delivered live are more often multimodal (comprising gesture, audio and linguistic semiotic systems). Thus all these texts can be viewed as multimodal rather than mono-modal.
(Kress, 2000:184) in Evolving Pedagogies by Bull and Anstey

Four Resources Model: Luke & Freebody

Code Breaker Practices How do I crack this text?
How does this text work? Is there more than one semiotic system or mode operating here? How do they relate? How do the parts of the text relate to one another? (layout and organisation?

Meaning Maker Practices What does this text mean to me?

How will the purpose and context for my reading influence my meaning making? What social, cultural, literacy and technology knowledge do I have that might help me make meaning of this text?

Limit the responder to what is contained Limit the responder to what is contained in the text and their own experiences in the text and their own experiences Text User Practices What do I do to use this text purposefully?

Text Analyst Practices How might I be shaped through engagement with this text?
What are its origins? What is the producer of this text trying to make me believe or do? What beliefs and positions are dominant or silenced in the text?

What is the purpose of this text and what is my purpose in using it? What should I do with this text in this context?

Transformative as they go beyond the text.

Transformative as they go beyond the text.

Applying the Four Resources Model to these two texts. Plus the glossarys definition of Visual Features **
Olive Cotton Only to taste the warmth, the light, the wind - 1939

Georgia OKeeffe 1939

(anonymous)

Quality picture books are literary texts

Finding congruence (the quality of agreeing) between literary elements and verbal/visual details: Is there congruence between the literary elements and the verbal and visual details on an illustrated page? Readers/viewers can notice where the verbal and visual correspond. Extra information can be provided in the visual or written text.

Ziba Came on a Boat importance of

cohesion/congruence between the two modes Azadi, her mother whispered. Freedom.

Literary Elements

Visual and Verbal Choices Visual

Setting

Verbal/written
Visual

Characters/ characterisation

Verbal/written Visual

Plot

Verbal Visual

Mood/ Atmosphere

Verbal Visual

Using a chart like this can help locate congruence between written and visual

Theme

Verbal Visual

Symbol

Verbal
Visual

Style

Verbal Visual

Vocabulary

Verbal

Finding congruence
Literary Elements Visual and Verbal Choices in Collecting Colour Visual Verbal Visual Verbal Visual Verbal Visual Verbal Images of a girl This is Rose On a swing

Setting Characters/
characterisation

Plot

Mood/ Atmosphere

Relaxed playful smile on her face: eyes closed

Finding congruence between written and visual text

Literary Elements

Visual and Verbal Choices in Alexanders Outing

Visual

Setting

Verbal Visual

Characters/ characterisation

Verbal Visual

Plot

Mood/ Atmosphere

Verbal Visual Verbal Visual

Sydney Harbour Bridge, Opera House, tall buildings at North Sydney, all viewed from Farm Cove and park (using my personal repertoire knowledge). in the most beautiful place in the whole of Sydney one warm sunny morning duck family, mother duck, mother is turning around maybe to look at or call to Alexander assume the last duckling is Alexander (literary repertoire); he straggles behind and is looking backwards Alexander lived with his mother and his four brothers and sisters: Alexanders mother was bored the straggling Alexander infers that he will cause some complication; the books title also implies this (literary repertoire) they all set of in search of adventure relaxed and enjoyable beautiful and famous view of Sydney, people walking, ducks swimming, an ibis by the pond, a sunny day; ready for an adventure most beautiful place, bored, warm sunny morning, adventure the Harbour Bridge and the Opera House are symbols of Sydney, Australia; green and gold colours are symbols of Australian identity; no other suggestion of a symbol for the storyline evident in the visual text no suggestion of a symbol evident in the verbal text watercolour media, green and gold colours, detailed drawings typical narrative orientation; everyday language naming the iconic buildings and maybe the types of birds in the image would build emergent readers vocabulary beautiful, bored, adventure could be difficult words for emergent readers; Sydney could be a special word for the setting and for readers from other places

Symbol
Verbal Visual Verbal Visual Verbal

Style

Vocabulary

Theme

Visual
Verbal

Alexander is straggling behind and not looking where he is going, so maybe the significant message to young readers will be pay attention and be obedient
no evidence of a suggested theme/significant message; this is unlikely as it is the orientation of the narrative on the first page

The Grammar of Visual Design

Kress & van Leeuwen argue for the use of the term grammar re visual design. Visual grammar to describe how the elements of the picture combine in different design ways to construct visual meaning. Kress & van Leeuwen make the point that theirs is not a universal grammar, but one that is culturally based in Western Societys mores.

Model of Visual Design

Australian Curriculum: English

Representational Meanings The visual grammar components that represent the topic: they construct who or what participates in the image, in what kind of activity or process they are engaged and in which set of circumstances. Interactive Meanings The visual grammar components that create the interaction: they construct the relationship between the participants in the image. Compositional Meanings They structure how the entire image (constructed through the representational and interactive meanings) is composed as a coherent and cohesive meaningful whole.
(This is the same for written grammar see A New Grammar Companion)

5 main components in Representational Meanings

Participants Any objects, human or animal characters, or any parts of the natural world that are presented in the image Sight line or reading pathway that the reader follows around the image. They may be curved, vertical, horizontal etc. These often follow where the character is looking. Circular and curved seen as natural organic and part of the female world. Squares, rectangles, straight-edged shapes are seen as mechanical or made by humankind. Objects, actions or ideas that represent something else normally an abstract concept. Highly significant in visual design. Vectors

Shapes

Symbols

Colour

The Arrival by Shaun Tan

Symbols and their importance Symbolism ***

The Arrival by Shaun Tan

Colour Red

Western

Other countries

passion /emotion(love/hate) danger (stop lights, signs, warnings) masculine excitement, desire, energy, vitality affordable or inexpensive items, Halloween (with black) Ireland: colour for Protestants warm colours, cheerful, joy, enlightenment, lighthearted

Orange

China red signifies prosperity, happiness and luck South Africa red is the colour of mourning India: purity, fertility, love, used in weddings Australian Aboriginals: represents land and earth Eastern: happiness, spirituality Hinduism: Saffron is considered a sacred colour

Yellow

Chinese culture: sacred, imperial, honour, masculine Thailand: represents Buddhism Egypt: Colour of mourning Africa: Usually reserved for those of high rank Chinese culture: immortality, feminine Korea: mourning Korea: trust Chinese culture: associated with health, prosperity, harmony, fertility Colour of Islam Ireland: colour for Catholics South America: death USA: money Chinese culture: neutral, Heavens colour, times of unknown/winter months; colour for young boys India: evil, negativity, anger Africa: Age and wisdom Chinese culture: white is associated with death and mourning, the unknown and purity. Eastern: wealth India: sorrow, comforting Brazil: death and mourning

Blue Pink Green

cool, calm, wisdom, sadness, trust, conservative sensuous, emotion, feminine, romantic knowledge, hope, promise, nature, abundance jealousy, greed

Black

evil, mysterious, powerful, fear, elegance

White Purple Grey Sepia Brown

purity, innocence, timeless, mystical, cleanliness, virtue passionate, smouldering, caring, creativity, royalty, wealth

neutral, uncommitted, non-involvement, Associated with Christianity, prayers and lent. conservative, moody Eastern cultures: associated with helping attitude and travel the past or age nature, wholesomeness, dependability, blending in Japan: no specific name for brown, refer like fox or tea colour.

Colour- Warm Blue and Cool Blue

Sea Shepherd website

Using the 5 components to analyse an image

Collecting Colour
Participants The girl and the swing Vectors: The visible swing vector is the strongest, highlighting the light-hearted mood of the scene. The Rose vectors are the lines of her body and recreate the back-forth movement she is enjoying. Shapes The curved shapes in the image, indicated by yellow lines, denote a female world (which is continued as the story progresses) and infer the naturalness of a child playing. Colour The intense red-brown colour of the background could symbolise the red-brown earth of the Northern Territory. Symbol The swing could be represented as a symbol of carefree play or relaxation.

Representational and interactive meanings to analyse an image

Participants
The large range of participants in the foreground and background of Alexanders outing indicate the setting clearly.

Interactive
The interaction is a reaction as it is one-way mother duck reacts to Alexanders inattention.

Vectors three sets of vectors

A long horizontal line marks a strong visible vector of movement the importance of the ducks walking in search of adventure is evident and reinforced in the forward motion of the ducklings. A second, invisible vector can be seen in mother ducks backward gaze at Alexander, who continues the vector with his own turning backward. These opposing vectors presuppose the trouble that is to come. A third set of vectors follow the palm trees to lead the eye to the background of the image, with more details of the setting. Icon Sydney Harbour Bridge iconic status

Alexanders Outing

Representational and interactive meanings to analyse an image

Shapes The accumulation of the curved shapes indicated by red lines, create a dominant natural world of ducks, palm trees, the pond, the sand, the ibis and the white cockatoo, despite the fact that it is near the centre of the biggest Australian city, Sydney. The rectangular buildings and pylons in the background denote their human construction, as befits a city, but the combination of organic curves and robust rectangles remind us why our Harbour Bridge and Opera House are National Heritage listed and known internationally for their beauty of form. Symbols Symbols of Sydney are represented in the image to locate the setting visually, as noted above. The green and gold colours are clearly symbolic: they are Australias national colours, and variations of the green and gold are worn as the traditional team colours of our national sporting teams.

Alexanders Outing

Components that realise interactive meanings

Demand and Offer
Social Distance Angle/Point of View Modality Colour

The Arrival

NSW English K-10 Syllabus

Glossary Gaze: the directed look of either a viewer or figure in an image, including demand and offer.

Interactive: Demand and Offer ***

Demand and offer are terms which name the degree of interest or detachment readers/viewers are positioned to have towards the participants. They are contrasts applied only to animate, or living, participants. Demand denotes when a participant gazes at the readers/viewers: the character demands that viewers look back directly.

Analyse: demand and offer/colour participants/colour

Offer denotes when a participant gazes away from the readers/viewers: no contact is established. We become invisible viewers who can choose whether and when we look at the character.

Interactive: Social distance

A long shot denotes an impersonal public relationship as though the participant and the viewer are strangers. A medium shot denotes a social relationship the participant and the viewer can have a conversation, they are acquainted and friendly. A close-up shot denotes a personal relationship with frequent contact we are family, or best friends. Extreme close-up denotes a highly intimate relationship viewers are closely involved with the participant.

How close or distant the participants are from the viewer. Visual Grammar: the type of shot

Analyse: participants/colours/shapes/dem and and offer/social distance

Interactive: Angle or point of view

Angle/point of view names the degree of involvement and degree of power the view has in relation to the participants, which depends on the angle of the viewer. Participants are basically placed at different angles: At a low viewing angle, with viewers looking down on them, participants are perceived as having low status/power At a high angle, with viewers looking up at them, the participants superiority in knowledge or power/status is signified At eye level, an equal point of view is maintained.

What social distance? What type of shot? Why?

What is the effect of the type of shot?

Demand and offer? *** Social distance? Type of shot?

Interactive: Modality
Modality names the degree of naturalistic realism in the illustration High modality indicates a more natural image, which the viewer is more likely to regard as real. Naturalness is indicated by the degree of colour saturation and the use of heavy or faint lines, shapes and shading. Modality works on a continuum from high to low typically, strong colour and clear lines construct a high level of realism/modality. Low modality is indicated by pastels and faint lines, which are not perceived as natural or as real. At the other end of the continuum, very heavily saturated colour or extremely thick lines will appear exaggerated and unreal.

Analyse Alexanders Outing in terms of modality, social distance and angle. What are the effects of each?

Modality What level of modality? What is the effect?

An ordinary day

*** Analyse the interactive in terms of: demand and offer, modality and colour, angle, social distance. Representational meanings?

Interactive: demand and offer, social distance, angle, modality, colour ***Do they all apply? Representational meanings?

Rose Blanche

Theres a sea in my bedroom

The Arrival by Shaun Tan

Shaun Tan said: In The Arrival, the absence of any written description plants the reader more firmly in the shoes of an immigrant character. There is no guidance as to how the images might be interpreted, and we must ourselves search for meaning and seek familiarity in world where such things are scarce or concealed. Words have a remarkable pull on our attention, and how we interpret attendant images. In their absence, an image can invite more lingering attention from a reader.
** Use the five main components of representational meaning to analyse this image. Also interactive meanings demand and offer etc. Stage 3: Interpret picture books which do not contain written text

The Arrival ***

The reason for migrating
Another migrants reason for leaving his homeland

Components that realise compositional meanings

Visual composition focuses on the configuration of the image, that is, how the people, places and things, and relationships they convey are placed within the image and the meanings attached to these positions.
(Kress & van Leeuwen, 2006)

Components

1. Information Value 2. Salience 3. Framing

What does the Syllabus say about composition?

Stage 2 Identify organisational patterns and language features of print and visual text

Composition: Information Value

Information value names the values attributed to the arrangement of the participants (people, places and things) in the image.
Each set of spaces contains a certain distinct value in the information it represents. This information value can be applied to all kinds of images factual images, diagrams, advertisements, television news and digital texts etc. Set 1 Given is the information placed on the left of a page; it is known, or familiar, or previously introduced information; the viewer already knows, or is assumed to know. New is the information placed on the right of a page; it is unknown previously, newly introduced, or innovative information; it is the new message to be attended to. Set 2 Ideal is the information placed towards the top of a page; this space represents the ideal or optimum position Real is the information placed at the bottom of a page; this space represents reality, the practicalities of the real world, what is. Set 3 Centre is the information placed in the middle of a page; it is the focus of attention and the nucleus of the information; it is the strongest, most dominant position, even if it is an empty space, and its value overrides other components. Margins is the information placed towards (or on) the edges of a page; it indicates that the participant is dependent on the Centre focus, or peripheral, or subservient to what is in the Centre.

Given New

Information Value: Which sets in these images? Any other visual features you can recognise? ***

Shaun Tan: The Arrival and The Red Tree

Salience the most attention-grabbing part of the image

Salience (Syllabus Glossary) A strategy of emphasis, highlighting what is important in a text. In images, salience is created through strategies like placement of an item in the foreground, size and contrast in tone or colour. Salience: NSW English K-10 Syllabus Stage 2 Explore the effect of choices when framing an image, and salience on composition of still and moving images

Salience is the most attentiongrabbing part of the image.

Salience is constructed through a number of elements Relative size Sharpness of focus High contrast (eg. black/white borders) Human figures Animated participants Being in the foreground not the background Colour the application of saturated colours (next to softer colours) and use of red and/or yellow add Salience. Information value components (Given/New/Real etc) can be Salient. All of these can draw the viewers eye and create Salience.

Composition: Salience
Composition in Collecting Colour (page 1) The salient component is Rose: she grabs attention in a number of ways. The colours are saturated and contrast to the background. The illustrator has used most of the elements of salience to draw our attention to her.

Salience use the elements on the earlier slide to determine the salience in these images

What is the most attention-grabbing part of the image?

Composition: Framing
English K-10 Syllabus Stage 2 Explore the effect of choices when framing an image, and salience on composition of still and moving images Syllabus Glossary: The way in which elements in a still or moving image are arranged to create a specific interpretation of the whole. Strong framing creates a sense of enclosure around elements while a weak framing creates a sense of openness.

Composition: Framing
A frame creates boundaries: these connect or separate participants in the illustration, implying they belong, or do not belong together. Framing can occur within an image as well as around an image; it can be heavy or light, or absent altogether. Framing does not have to be an actual frame parts in an image may form a Frame. Heavy framing can imply oppression, containment, individuality and/or difference. Light or no framing can imply freedom, lack of confinement and/or group identity.

Composition: Framing
The Margins the repeated palm trees form a frame for the participants in the Centre reinforce the participants importance to the story.

The lack of framing implies freedom and as she swings unhindered, she certainly seems so.

Composition: Framing *** What type of framing? To what effect?

Stage 3: Recognise how the use of language and visual features can depict cultural assumptions in texts

Framing*** What type of framing? To what effect?

Framing What type of framing? To what effect? Other visual features?

Stage 3 recognise how the use of visual features can depict cultural assumptions in texts

The three types of Meanings: representational, interactive and composition ****

Stage 3 recognise how the use of visual features can depict cultural assumptions in texts

Previous page: In her mind, she sat with her father, playing with the doll he had given her. This page: He told her stories and poems of long ago. She felt the strength of his arms and she gazed into his peaceful face.

Comparing texts- from different times

Stage 2: identify and explain language features of texts from earlier times and compare vocabulary, images, layout and content of contemporary texts.

Suggested Text Stage 2 & 3 The Sneetches

Suggested Text Stage 2 & 3 The Island

Comparing texts by same illustrator

Stage 3: identify, describe and discuss similarities and differences between texts by the same illustrator and evaluate characteristics that define an authors individual style

Anthony Browne Into the Forest Stage 3

The Tunnel Stage 4

Comparing texts by the same illustrator/author

Piggybook: Suggested Text Stage 3

Gorilla - ??