Comics as a Tool for Understanding Cognition

These are further reflections on thoughts from a previous post to help me prepare my thoughts for a talk that I’m giving at the Small Press Expo in Bethesda, Maryland.


I look at comics a lot. But I don’t look at them the way I think you think I do.

Deep down I’m a linguist trying to understand language, particularly interested in visual language. I’m convinced that comics are attractive to us as a mode of expression because they allow our human eyes to read images that line up with the mental models that we create for how we “see” the world. You can think of cartoons as diagrams of our existing, procedural abstractions of the world and comics sequences as visual abstractions of how we model the passage of time in our minds.

Why do we cartoon the way we do? Is there a biological reason for it? I believe so.

I recently read an article on the first anatomically accurate mathematical model that explains how vision is possible. It was interesting. It’s a model that recognizes how we can create such seemingly rich visual depictions of the world in our mind despite there only being just 10 or so nerve cells that connect the retina to the visual cortex. The part that really got to me was the idea that these 10 Lateral Genticulate Nucleus cells are scarce and can’t do much. They send a pulse to the visual cortex when they detect a change from dark to light, or vice versa, in their tiny section of the visual field. And that’s it. On. Off.

Information from the eye passes through a bottleneck before it gets to the brain’s visual cortex, which heavily processes the sparse signal. – DVDP for Quanta Magazine

From blips of information, we build the world in our minds. This struck me.

In essence, the world that we “see” is a reconstruction that your mind makes up. Therefore, the neural cortex must process these simple signals in a way that builds + creates representational models of the world. I imagine that there might be a procedural grammar to vision. What does the process of this reconstruction look like? What abstractions are used to model it in our “mind’s eye”? How can we understand what this might look like?

Figure 3.7
Reading in the Brain – Stanislas Dehaene Figure 3.7

I believe that comics offer us a mirror to see an expression of the tendencies and limits of our biology. That we can glean the beginnings of what such a grammar may look like by analyzing the cross-cultural visual syntaxes of comics from around the world. It can offer theories that can guide further research.
In a sense, comics are especially equipped to help us see how we see. I wish to clarify my use of the word equipped. Comics don’t exist outside of us. We created them. As such, it is our mental apparatus that has equipped them with this capacity.

What is especially interesting to me with comics as a lens through which to observe human thought and perception is that it’s a universally accessible and naturally occurring petri dish. The reason for this is that comics, in my mind, happens to be an egalitarian stage of expression that allows for us as individuals and also as a society to make high-level abstractions with accessible, immediate tools and technologies. This mode of creation cuts across race and class on Earth. All you need is a mark-making tool and a surface to make marks on.

Comics help us see how we are collectively and individually “seeing”, most notably how we experience and structurally breakdown our experience of space and time. Through comics, we break down the visible world into simple marks, into images that encode reality. As I mentioned earlier, my understanding of cartooning is that it diagrams our existing, procedural abstractions of the world. And comics sequences visually abstract how we model the passage of time in our minds. We break it all down on the page and then a reader reconstructs a model of what we were trying to express.

I believe there is usually a conscious or intuitive understanding of this process when a creator is making a comic. It is when this process is understood as a communicative act that successful comics emerge.

This is important in that it provides an analogous process to how we process sound to understand oral language and music. We don’t hear words, or notes, we decode them from complex signals. This means that we have a way of decoding. And in this decoding, there is an order, a grammar. That is perhaps innate or developed over time through socialization.

Ray Jackendoff‘s research is especially interesting in this front – He has always straddled the boundary between generative linguistics and cognitive linguistics, committed to both the existence of an innate universal grammar (an important thesis of generative linguistics) and to giving an account of language that is consistent with the current understanding of the human mind and cognition (the main purpose of cognitive linguistics).

Ray Jackendoff - Foundations of Language: Structure of a simple sentence. Fig. 1.1
Ray Jackendoff – Foundations of Language: Structure of a simple sentence. Fig. 1.1

So what? How does affect us on a level of readers and makers of comics?

I find that an immediate use for this idea is that it helps us see comics as a cognitive bridge. It is a cultural tool that allows us to cross the gulf and better understand others. Particularly, it helps us understand how others experience and perceive the world. If it is true that comics allow us to see the ways that others encode their perception of reality into the two-dimensional substrate, we can better understand how their minds genuinely see and construct their experiences of the world.

When we look at someone’s comics, or even our own, we can ask: What is there? What is not? Why might this be?

Through comics, we can “step into someone else’s shoes” at a level that is unavailable through other media, including writing. That is something that I find fascinating.

Thinking the Unthinkable

This weekend Jenn and I took a trip to visit her parents in Garrettsville. It was quiet little weekend. We got in late Friday night as we usually do, traveling from Pittsburgh via the Turnpike. The sunset was especially glorious that night. Clouds tipped with fluorescent pink.

On Saturday, Jenn and Joyce went out to Goodwill, as they are want to do. The thrift store goods shown their light on Jenn. She found pair after pair of pants, exactly her size. Great brands. All for around $3 each. Ace. I spent some time walking around outside while they were gone. I poked around in the garden. Joyce’s garden is bursting, though a little overgrown at the moment, it’s full of life. In the nearby coop she has new chickens that she and her neighbor have been incorporating into their flock. They seem to be getting on well. Saturday saw 5 eggs. Sunday just 1.

While they were out, I watched an incredible talk presented by Bret Victor at the MIT Media Lab in 2013. It was called “Media for Thinking the Unthinkable“. It was recommended by my friend Max Krieger, who is a current undergraduate in Mathematics at CMU. It was amazing.

The big take away was the inspiration of thinking about looking and creating for new tools to express and capture the systems that we use notation systems to represent. Computers allow us to create dynamic, interactive system representations that can adapt to our modifications. Real Time. And in the playing around with these models, we gain intuitions about the system that are invisible to us. These depictions expand what we can understand and make the unthinkable suddenly thinkable. (Needless to say this makes my mind immediately jump to thinking of Lynda Barry’s Writing the Unthinkable talks.)

Just as we augment the range of our observable reality via microscopes, telescopes, infrared and UV light detection, we can augment the range of our thinkable reality by rethinking how we represent the world.

I mention all of this because earlier int the day I had read an article on the first anatomically accurate mathematical model that explains how vision is possible.Max had recommended Victor’s talk after I shared my initial impressions of this article via Instagram. It’s a model that recognizes how we can create such seemingly rich visual depictions of the world in our mind despite there only being just 10 or so nerve cells that connect the retina to the visual cortex.

Not only are Lateral Genticulate Nucleus cells scarce — they can’t do much either. LGN cells send a pulse to the visual cortex when they detect a change from dark to light, or vice versa, in their tiny section of the visual field. And that’s all. The lighted world bombards the retina with data, but all the brain has to go on is the meager signaling of a tiny collection of LGN cells. To see the world based on so little information is like trying to reconstruct Moby-Dick from notes on a napkin.

In essence, the world that you “see” is a reconstruction that your mind is just making up. We are creating representational models of the world. What is the process of this reconstruction? What abstractions are used to model it in our mind’s eye?

Unsurprisingly I started to think about comics, especially along a framework of Expanded Comics as Kim Jooha has recently invited contemporary makers and readers to think along. Why comics? Why do we care about them and why do they keep appearing with us as mode across human history?

I’m convinced that comics are attractive to us as a mode of expression because they allow our eyes to read images that line up with the mental models that we create for how to see the world. Think of cartoons as diagrams of our existing, procedural abstractions of the world. And comics sequences as visual abstractions of how we model the passage of time in our minds.

Why do we cartoon the way we do? Is there a biological reason for it? I believe so. I believe that comics offer us a mirror to see an expression of the tendencies and limits of our biology. This is not to say other forms of culture don’t allow for that, but just that comics are especially equipped to help us see how we see.  Comics, in my mind, happens to be an egalitarian stage of expression that allows for us as individuals and also as a society to make high level abstractions with accessible, immediate tools and technologies. This mode cuts across race and class on Earth. All you need is a mark making tool and a surface to make marks on.

Comics help us see how we are collectively and individually seeing, most notably how we experience space and time.

I feel this ties into the Sapir-Whorf hypothesis in some way but I can’t articulate it at the moment… The concept of Linguistic Relativity is extremely controversial, so I’m not making any claims this way or that. Just that there’s something here. It’s something that I would like to explore. Unlike the strong hypothesis in Linguistic Relativism, that says that language determines thought and that linguistic categories limit and determine cognitive categories, I believe there’s a flex and flow with it all. Taking up the idea of Visual Language, there’s an inevitable creep of signifier and signified relationships. These relationships can morph due to accidental depiction in drawings that inevitably shift how we see what we see. I hope that makes sense. We depict via representation, but we also wind up abstracting, simplifying, transforming. These transformations, accidental or intentional in turn have an influence in how we see what we were depicting when we experience it in the future. This is a powerful quality of the back and forth between the world and the model of the world that we build with language, one that can be seen extremely vividly in the use of racial caricature. Needless to say I need to think about this more.

Pulling back, though, comics and cartooning endlessly amaze me as I see them as an accessible, evolutionary rich linguistic lens onto our cognitive apparatus through our very cognition.

So that’s where my mind was Saturday. Joyce and Jenn came home. We had a comforting dinner of macaroni and cheese. We stepped out with Jeff to enjoy the cool late August air with a fire. In the garden by the chicken coop.

It was nice.