Visual Communication

A multimodal approach to visual thinking: the scientific sketchnote

by Almudena Fernández-Fontecha, Kay L. O’Halloran, Sabine Tan, & Peter Wignell (2019)

There is a growing interest in the use of visual thinking techniques for promoting conceptual thinking in problem solving tasks as well as for reducing the complexity of ideas expressed in scientific and technical formats. The products of visual thinking, such as sketchnotes, graphics and diagrams, consist of ‘multimodal complexes’ that combine language, images, mathematical symbolism and various other semiotic resources. This article adopts a social semiotic perspective, more specifically a Systemic Functional Multimodal Discourse Analysis approach, to study the underlying semiotic mechanisms through which visual thinking makes complex scientific content accessible. To illustrate the approach, the authors analyse the roles of language, images, and mathematical graphs and symbolism in four sketchnotes based on scientific literature in physics. The analysis reveals that through the process of resemiotization, where meanings are transformed from one semiotic system to another, the abstractness of specialized discourses such as physics and mathematics is reduced by multimodal strategies which include reformulating the content in terms of entities which participate in observable (i.e. tangible) processes and enhancing the reader/viewer’s engagement with the text. Moreover, the compositional arrangement creates clear stages in the development of the ideas and arguments that are presented. In this regard, visual thinking is a form of cultural communication through which abstract ideas are translated and explained using a multimodal outline or summary of essential parts by adapting resources (e.g. linguistic resources and mathematical graphs), using new resources (e.g. stick figures and other simple schematic drawings) and maintaining others from the original text (e.g. mathematical symbolic notation), resulting in a congruent (or concrete) depiction of abstract concepts and ideas for a non-specialist audience.

multimodal discourse analysis, multimodality, resemiotization, sketchnote, social semiotics, systemic functional theory, visual thinking.

Biographical Notes:
Almudena Fernández-Fontecha is a Lecturer in the Department of Modern Languages at the University of La Rioja (UR), Spain. She is a member of the UR Applied Linguistics Group (GLAUR). Her research interests include different dimensions of Content and Language Integrated Learning (CLIL), such as the study of multimodality for scaffolding language learning, foreign vocabulary learning and the study of learners’ factors in foreign language acquisition, such as motivation and creativity. Address: Department of Modern Languages, University of La Rioja, C/San Jose de Calasanz 33, Logrono, La Rioja 26004, Spain. [email:]
Kay O’Halloran is Professor in the School of Education, Faculty of Humanities at Curtin University. Her areas of research include multimodal analysis, social semiotics, mathematics discourse, and the development of interactive digital media technologies and visualization techniques for multimodal and sociocultural analytics. Address: School of Education, Curtin University, GPO Box U1987, Perth, Western Australia 6845, Australia. [email:]
Sabine Tan is a Senior Research Fellow in the School of Education, Faculty of Humanities at Curtin University. Her research interests include critical multimodal discourse analysis, social semiotics, and visual communication. She is particularly interested in the application of multidisciplinary perspectives within social semiotic theory to the analysis of institutional discourses involving traditional and new media. Address: as Kay O’Halloran. [email:]
Peter Wignell is a Senior Research Fellow in the School of Education, Faculty of Humanities at Curtin University. Peter’s current research interests are in Systemic Functional Linguistics, especially in its application to the analysis of multimodal texts. His research has also focused on the role of language in the construction of specialized knowledge. Address: as Kay O’Halloran. [email:]


Fernández-Fontecha, Almudena / O’Halloran, Kay L / Tan, Sabine / Wignell, Peter
A multimodal approach to visual thinking: the scientific sketchnote.
In: Visual Communication 18,1 (Feb 2019), pp. 5-29.