On Kernel Image Transformations

I’d like to share the paper I wrote for my senior thesis in linguistics from Carnegie Mellon University, Kernel Image Transformations: An exploration of transformational grammar in visual language. My advisor for this was the ever helpful and patient, Thomas Werner. A huge thank you to him for helping me cross the finish-line on this.

So, what’s it about? Well, essentially it’s a playfully pedantic look at what is going on at the granular level of communicative drawing. My goal here is to show the degree of intuitive meaning based knob twiddling that is involved when a drawer’s brain is set to work on communicating meaning.
Why did I spend a ton of time working on this? Bottom-line, I hadn’t seen this done. So, rather than keep looking for someone to talk about this in a way that made sense to me, I sought to make my own way.

Deep down, this way of structurally looking at the ways that a drawer reconfigures the different elements of the images they create is critical to how I teach visual composition to my students and that I use in my own work. It is also what I use to evaluate the qualities of the communications that I experience when looking at comics. To me, these transformations are the core act in visual communication.

This mechanism of transforming base images cuts across all aesthetics and narrative forms. I believe that so long as someone is drawing and wants those drawings to be understood a certain way, with intentionality, the processes I outline in this thesis are involved. As a maker, you learn these transformation techniques intuitively and don’t think much of what they are. It’s the blueprint of visual meaning making. I wanted to take a closer look at what seems to be written in invisible ink in our minds…

To get to the nuts and bolts of it, this paper presents a linguistic exploration of visual language from a drawer’s perspective. I argue that there is much to be gained by exploring this particular perspective in the nascent field of Visual Language Theory. It presents a working concept of the kernel image and its subsequent transformations, which I propose to operate similar to the concept of the kernel sentence as proposed by Zellig Harris in early notions of transformational grammar.

Big picture-wise, this paper  presents an exploratory catalogue of transformations of a single kernel image which I feel illuminate the array of processes involved in the grammatical and semantic composition used by the cartoonist.
It covers locational, temporal, and aspectual transformations. Altogether, this review shows how the kernel image transformation framework is especially revealing when trying to understand the observably systematized rules that are at play in a cartoonist’s encoding of meaning into sequences of visual language.

This is an example what the guts of this paper are like.
If that sounds interesting, be my guest and check it out! 
Download: Kernel Image Transformations – Juan Fernandez

 

In the near future, I’m hoping to turn this into a zine that breaks things down into useful chunks for cartoonists/ visual communicators. Above all I just want to be able to have conversations with people about this compositional process, as it pertains to meaning making in images! If you wind up reading this and have some thoughts regarding this process, I’d love to talk. This is me building up on A LOT of people’s work, so I’m excited to be part of a larger conversation on this subject.

I’d love to know what you think!

<3 June 13th, 2020 <3

Collaborative drawing by me and Jenn.
Collaborative drawing made by me and Jenn Lisa to celebrate our wedding.

Dear friends, Jenn and I have been sitting on an announcement that a lot of you already know: we’re getting married!

We are so thankful to have each other. It’s a real blessing to have someone who is there with you through it all. Thick and thin. It’s been an amazing 6 years that have helped us realize that we want to spend our lives together.

We had been planning our wedding for this coming June, but the state of the world is quite different from what we expected when we started planning last year. Who could have imagined a pandemic to have struck the world the way it has? We aren’t sure if the wedding we end up having will be anything like the one we have been planning. We’ll see how it all shakes out…

It’s a tumultuous time for everyone with this pandemic and we’re very lost as to how best to move forward. We’re hanging in there. Jenn’s working at Reed Smith remotely and I’m completing my Web Developer coding bootcamp at Academy Pittsburgh remotely. Our finances aren’t totally wrecked by the economic windfall of Covid-19. I’d planned to be without an income right now to study full time and Jenn’s work adapted to remote without a hitch. Phew.

In these wild times, I wish to share our love and joy with you, because it’s clear we all need a little more of both in our lives. Sending you all love from our little corner of the world.

Thanks for being there for me and Jenn on this journey.
No matter what happens, we’re getting married and I will share it with you in some way!

J & J

An Image of Juan and Jennifer

A list of breath lost

I got on the wrong bus, mistaking a 56 for a 58. eyes grow weary, roads become rivers

Tom Spurgeon, THE COMICS REPORTER, is gone.

I keep thinking about Tom. I barely knew him. And yet, I’ve been thinking about him everyday. I miss him. I couldn’t exist making, reading and theorizing about comics and visual language in America without someone like Tom, building the internet.

Spend some time in the archives of his website to remember him, if you can http://www.comicsreporter.com/.

Editing and transcribing revisions to my draft on Kernel Images and Visual Transformation Grammar.

I am currently in the home stretch of finishing my degree in Linguistics at Carnegie Mellon University. This shit is grueling on top of my day job. I’m tired, sad, and often angry. But I’m moving forward and I won’t stop until I’m done. Jenn is by my side. I have a sweet cat with us at home named Fran. She is kind and sweet and spending time petting, playing and resting with her has helped me so much this fall.

I feel like I’m hanging in there by a thread of energy and sanity. Everyday is a climb. Problem sets, tests, presentations and my Thesis. I’m just so tired. This post is to serve as a smoke signal for you, friend. It’s also for myself, for my future self to remember how hard this has been. I’m ok. There is light up ahead. I just need to keep moving.

This is what I listen to to stay awake and focused after work and into the night:

Cheesy? Trashy? Who cares. It works.

until next time ❤
juan

On “writing” in comics.

The other night, over dinner here in Pittsburgh, Jenn and I were talking about making autobio comics and she asked “Are comics mostly writing? Only?”. It made me stop in my tracks.

That’s a good question.

To me, comics are more writing than they are drawing. That writing process, though, is one that far different from this common notion:

And the drawing? In comics, the drawing is really about the recording of images. The drawing is the creating of a signal and the “writing” is drawn. To clarify, the writing in my view is how those recorded images are arranged and presented. This is a semiotically driven view.

I just heard someone say that the dirty secret of comics is that it is mostly writing and if you haven’t done good writing all you made was a pretty picture.

And everything in me wants to dispute that.

It isn’t that cut and dry to me. – Jenn Lisa

I agree with Jenn. It isn’t that cut and dry to me. That is because the only people that I ever hear tout that idea usually have a very narrow definition of writing. This is where that idea is wrong.

If you use a broader notion of understanding visual language, you might see things differently. When we make comics we are “writing” via arrangement, mark-making and coloring, not just through writing scripts or thumbnails. When people talk about writing, they are talking about scripting, they are talking about the comics in which the drawings are manifesting an already decided meaning, according to an intuitive set of aesthetic and styling rules that vary from maker to maker according to their abilities and interests.

The turth is that we are encoding meaning through signals that can travel across time and space. Sometimes that encoding happens in a mode that uses no “words”. It may include “words” and  if it does, there can be more going visually on that is not verbalizable by a reader, though it may be understood.

I do believe that the dirty secret is that comics is mostly writing, but just as there is a difference between speech and speeches there is a difference between writing and writings.


So how does that affect one as a comics maker?

If you would, please indulge me in a tangent. It’s an idea you might already be familiar with, but I just want to sing the song more fully into the ether.

We don’t hone in on meaning by having a creative process that follows a straight line, the reality is that most of us spiral in on that meaning, by diligent work. But what if you don’t know what you’re trying to say? That’s ok. You can still make comics without knowing what you’re trying to say, so long as you know that you’re trying to say something. Your comics will still have an impact, even if you’re not communicating exactly what you intend.

How do you get to that meaning? There’s this thing that happens, in comics making, and art in general that I’d like to talk about. What is the nature of the back and forth between the work a reader does and the work a writer does? Writer here being the operative name for the person making the comic.

Sam Ombiri and I have talked about it at length during the Pittsburgh Comics Salons monthly meet ups.

If I take 14 random drawings from my pile of post-it notes and arrange them across 4 pages, and I give it to you in a little booklet, your brain, the reader’s brain, stitches together a patter of cohesion. That pattern will vary connect disparate elements. Make order out of chaos.

They try to figure out a pattern. Why are these images together? The reader does all this work without trying. They can’t help it. Our brains can’t help it. We are trying to understand according to the grammars we know and trying to find semantic relevance. Sometimes we do. And it feels good. Sometimes we don’t and we don’t usually like it. Is it our decoding? Is their encoding? Were they encoding anything at all? (We can talk at another time about encoding-less comics experiences, they exist and are interesting in their own right.)

As a comics maker, that idea is a tool that we are using.  A reader is naturally going to be interpreting what we assemble together. As a result, we can be playful, intuitive in what we draw and how we arrange what we draw. It will be stitched together by the reader. You don’t need to shove the connective tissue of narrative and symbolism down a reader’s throat. You don’t need to know everything you are trying to do.

That’s all pretty explored territory conceptually among many comics makers. The part where my conversations with Sam get interesting is when we talk about signal processing a little more in-depth.

The signal interpreted by the listener’s brain CAN be the same one that you are making. BUT, it can also not be that, while still being an interesting signal. The meaning you intend can be communicated, and you can work at figuring out how to effectively communicate that meaning so that they get what you intended them to get out of the written sequence. There’s a beautiful dance between signal and noise that we are working through when we create art for others to interpret.

1. Signal initiates in speaker’s brain. 5. Signal interpreted by listener’s brain. 2. Signal articulated by the speech organs. motor. neurons. ear. sensory. neurons. 3. Sound waves travel through the air. sensory. neurons. 4. Listener detects the sound waves. feedback loop. ear. Linguistics 450/550: Introduction to Phonetics Slides by Richard Wright & Dan McCloy, University of Washington.

But no matter what, their brain is going to interpret a signal. The realization was a breakthrough. You don’t have to worry about there not being anything on the other end. There will ALWAYS be something. You just have to focus on how accurately and uniformly you want that something to be experienced across readers.

You can choose how intentional you are about the signal you are transmitting.

With that knowledge, you can allow the process of drawing and writing and editing to flow more intuitively. Allowing yourself. You can choose to simply be a radio antenna with your drawings on the page, and how you arrange those drawings on the page in ways that feel intuitively, “right” or “good” to you – your comic will beam a signal to the brain of the other through the reading experience, whether you are aware of that signal or not.

This is a model of aural communication, just swap out aural for visual 😉

Don’t be afraid to allow the process to be a Rorshach test of your subconscious. You can see what comes out in your drawing, your arranging. Allow for the accidents! Keep moving. Keep making marks, keep tracing, keep moving your index cards. You don’t have to know what you’re looking for, you just need to know that you’re looking for something. The paper trail you leave in your search for your work’s “truth” in comics, that’s your writing.

All that said, in a more grounded sense,  the process of editing, that traditional refining that we do as cartoonists, that’s where writing is also happening. Every step of the way, where you change, add or subtract meaning, that’s writing. Color blocking is writing. How you scoot images, how you erase, what you add after initial drawings and after you let work sit for a little while, that’s all writing.

The intentional arrangement of anything, to be parsed and understood a certain way, that’s writing.

So yes, comics are mostly/only writing. But not in the way you may think they are.


* It’s funny to me that the “writing” in panels and across pages in comics found as the lettering is fundamentally a kind of drawing. That’s a point Kevin Huizenga drives home to his students. More on that, another day.

Comics as a Tool for Understanding Cognition

These are further reflections on thoughts from a previous post to help me prepare my thoughts for a talk that I’m giving at the Small Press Expo in Bethesda, Maryland.


I look at comics a lot. But I don’t look at them the way I think you think I do.

Deep down I’m a linguist trying to understand language, particularly interested in visual language. I’m convinced that comics are attractive to us as a mode of expression because they allow our human eyes to read images that line up with the mental models that we create for how we “see” the world. You can think of cartoons as diagrams of our existing, procedural abstractions of the world and comics sequences as visual abstractions of how we model the passage of time in our minds.

Why do we cartoon the way we do? Is there a biological reason for it? I believe so.

I recently read an article on the first anatomically accurate mathematical model that explains how vision is possible. It was interesting. It’s a model that recognizes how we can create such seemingly rich visual depictions of the world in our mind despite there only being just 10 or so nerve cells that connect the retina to the visual cortex. The part that really got to me was the idea that these 10 Lateral Genticulate Nucleus cells are scarce and can’t do much. They send a pulse to the visual cortex when they detect a change from dark to light, or vice versa, in their tiny section of the visual field. And that’s it. On. Off.

Information from the eye passes through a bottleneck before it gets to the brain’s visual cortex, which heavily processes the sparse signal. – DVDP for Quanta Magazine

From blips of information, we build the world in our minds. This struck me.

In essence, the world that we “see” is a reconstruction that your mind makes up. Therefore, the neural cortex must process these simple signals in a way that builds + creates representational models of the world. I imagine that there might be a procedural grammar to vision. What does the process of this reconstruction look like? What abstractions are used to model it in our “mind’s eye”? How can we understand what this might look like?

Figure 3.7
Reading in the Brain – Stanislas Dehaene Figure 3.7

I believe that comics offer us a mirror to see an expression of the tendencies and limits of our biology. That we can glean the beginnings of what such a grammar may look like by analyzing the cross-cultural visual syntaxes of comics from around the world. It can offer theories that can guide further research.
In a sense, comics are especially equipped to help us see how we see. I wish to clarify my use of the word equipped. Comics don’t exist outside of us. We created them. As such, it is our mental apparatus that has equipped them with this capacity.

What is especially interesting to me with comics as a lens through which to observe human thought and perception is that it’s a universally accessible and naturally occurring petri dish. The reason for this is that comics, in my mind, happens to be an egalitarian stage of expression that allows for us as individuals and also as a society to make high-level abstractions with accessible, immediate tools and technologies. This mode of creation cuts across race and class on Earth. All you need is a mark-making tool and a surface to make marks on.

Comics help us see how we are collectively and individually “seeing”, most notably how we experience and structurally breakdown our experience of space and time. Through comics, we break down the visible world into simple marks, into images that encode reality. As I mentioned earlier, my understanding of cartooning is that it diagrams our existing, procedural abstractions of the world. And comics sequences visually abstract how we model the passage of time in our minds. We break it all down on the page and then a reader reconstructs a model of what we were trying to express.

I believe there is usually a conscious or intuitive understanding of this process when a creator is making a comic. It is when this process is understood as a communicative act that successful comics emerge.

This is important in that it provides an analogous process to how we process sound to understand oral language and music. We don’t hear words, or notes, we decode them from complex signals. This means that we have a way of decoding. And in this decoding, there is an order, a grammar. That is perhaps innate or developed over time through socialization.

Ray Jackendoff‘s research is especially interesting in this front – He has always straddled the boundary between generative linguistics and cognitive linguistics, committed to both the existence of an innate universal grammar (an important thesis of generative linguistics) and to giving an account of language that is consistent with the current understanding of the human mind and cognition (the main purpose of cognitive linguistics).

Ray Jackendoff - Foundations of Language: Structure of a simple sentence. Fig. 1.1
Ray Jackendoff – Foundations of Language: Structure of a simple sentence. Fig. 1.1

So what? How does affect us on a level of readers and makers of comics?

I find that an immediate use for this idea is that it helps us see comics as a cognitive bridge. It is a cultural tool that allows us to cross the gulf and better understand others. Particularly, it helps us understand how others experience and perceive the world. If it is true that comics allow us to see the ways that others encode their perception of reality into the two-dimensional substrate, we can better understand how their minds genuinely see and construct their experiences of the world.

When we look at someone’s comics, or even our own, we can ask: What is there? What is not? Why might this be?

Through comics, we can “step into someone else’s shoes” at a level that is unavailable through other media, including writing. That is something that I find fascinating.

Digital Ways of Seeing and a call for Serendipity

Lately, I’ve been really interested in how we create and share information (and truth) across the web.

Did you know that more than 180,000 of the items in the New York Public Library’s Digital Collections are in the public domain? It’s true! My friend, Aaron Cockle, showed me that a year ago, the NYPL had announced this new part of their digitally accessible collection. Thanks Aaron.

Now everyone has the freedom to enjoy, modify, reuse and share the materials in almost limitless ways! NYPL now makes it possible to download these items in the highest resolution available, directly from the Digital Collections website. They’re BIG files.

Explore the NYPL Public Domain Digital Collection with their visualization tool.

It’s true that “the interactive visual library” has been upon us for a while thanks to search engines that could deliver images. Nevertheless, searching in that kind of library is for the most part garbage. You have to sift through mountains visual of information just to potentially find what you need. Bummer. Web art that is allowed to grow wildly from that collective visual library, in an art-for-its -own-sake kind of way, has for the most part been visually chaotic and noise of visual resolution has usually been used as an aesthetic tool. Think Paper Rad. (Of course, there are lots of interesting exceptions like Dina Kleberman’s I’m Google project, not chaotic at all!) What kind of art is possible with tools like Public Domain visual collections? You tell me.

Screenshot of Dina Kleberman’s “I’m Google”

More broadly, this is kind of thing reveals one of the many reasons why public libraries are so important. They point to a simple truth that everyone is denying. Searching via tools made by creators that work  hand-in-hand with corporate interests sucks. It’s never deep enough. It’s never a rich enough experience.the paths to finding high quality, curated collection of visuals, for free are long, windy, and sometimes non-existent. A private browser like google, isn’t the most stable place to reliably find images across time. Sorry, private browsers. 

What we have is the exciting arrival of another institutionally supported manifestation of archive-able visual memory. I’m very thankful. We have Wikipedia, Archive.org, and others, but we still need more. Yes, this news from last January, but I want to shout about these kinds of projects from the hilltops. Especially in these times of “#alternativefacts”.

Why? Because.
With the explosion of instantaneous visual communication across the internet, visual forms, digitally transferred, can essentially be considered as shared experiences. The shared experience of “having seen”. This aspect of highly networked communication allows for users to collectively “see” the world through a shared viewpoint. Permission to transform artifacts of visual ephemera, of past visual memories, invites us to step across time and create and modify languages using pre-existing systems of visual communication. It is the power to create knowledge. Captured thought held into stable states that can be comprehended by another person across the network. Truth as recordable and transferrable. Props to the late John Berger for helping me understand this. My hope is to expand these thoughts from the seventies into the present day.

Simple Net Art Diagram by MTAA

Expanding this all out to copyright, websites ARE the expressions of that thought. And, as such function as the basis for the merging of idea and expression of idea. Everything expressed in “recordable” communication IS an idea within the legal framework. Organizations with corporate money have the power to take away certain “sides” of information. Duh. Information nowadays essentially being the mass of recorded experience. That kind of power can shape the mass of collective knowledge, that is the basis from which we collectively determine “truth”. Turning the creation of ideas into a commodity form.

As a society, it is only through collectively supported libraries, physical and digital that we can expand what it means be creating, sharing and finding Truth. Commodity form as community form.

Please visit your local library today. It may have been a while since you last visited… It may have been just yesterday! Either way, do yourself a favor and open yourself up to serendipity. Get out there, find something interesting and check it out.

Thinking the Unthinkable

This weekend Jenn and I took a trip to visit her parents in Garrettsville. It was quiet little weekend. We got in late Friday night as we usually do, traveling from Pittsburgh via the Turnpike. The sunset was especially glorious that night. Clouds tipped with fluorescent pink.

On Saturday, Jenn and Joyce went out to Goodwill, as they are want to do. The thrift store goods shown their light on Jenn. She found pair after pair of pants, exactly her size. Great brands. All for around $3 each. Ace. I spent some time walking around outside while they were gone. I poked around in the garden. Joyce’s garden is bursting, though a little overgrown at the moment, it’s full of life. In the nearby coop she has new chickens that she and her neighbor have been incorporating into their flock. They seem to be getting on well. Saturday saw 5 eggs. Sunday just 1.

While they were out, I watched an incredible talk presented by Bret Victor at the MIT Media Lab in 2013. It was called “Media for Thinking the Unthinkable“. It was recommended by my friend Max Krieger, who is a current undergraduate in Mathematics at CMU. It was amazing.

The big take away was the inspiration of thinking about looking and creating for new tools to express and capture the systems that we use notation systems to represent. Computers allow us to create dynamic, interactive system representations that can adapt to our modifications. Real Time. And in the playing around with these models, we gain intuitions about the system that are invisible to us. These depictions expand what we can understand and make the unthinkable suddenly thinkable. (Needless to say this makes my mind immediately jump to thinking of Lynda Barry’s Writing the Unthinkable talks.)

Just as we augment the range of our observable reality via microscopes, telescopes, infrared and UV light detection, we can augment the range of our thinkable reality by rethinking how we represent the world.

I mention all of this because earlier int the day I had read an article on the first anatomically accurate mathematical model that explains how vision is possible.Max had recommended Victor’s talk after I shared my initial impressions of this article via Instagram. It’s a model that recognizes how we can create such seemingly rich visual depictions of the world in our mind despite there only being just 10 or so nerve cells that connect the retina to the visual cortex.

Not only are Lateral Genticulate Nucleus cells scarce — they can’t do much either. LGN cells send a pulse to the visual cortex when they detect a change from dark to light, or vice versa, in their tiny section of the visual field. And that’s all. The lighted world bombards the retina with data, but all the brain has to go on is the meager signaling of a tiny collection of LGN cells. To see the world based on so little information is like trying to reconstruct Moby-Dick from notes on a napkin.

In essence, the world that you “see” is a reconstruction that your mind is just making up. We are creating representational models of the world. What is the process of this reconstruction? What abstractions are used to model it in our mind’s eye?

Unsurprisingly I started to think about comics, especially along a framework of Expanded Comics as Kim Jooha has recently invited contemporary makers and readers to think along. Why comics? Why do we care about them and why do they keep appearing with us as mode across human history?

I’m convinced that comics are attractive to us as a mode of expression because they allow our eyes to read images that line up with the mental models that we create for how to see the world. Think of cartoons as diagrams of our existing, procedural abstractions of the world. And comics sequences as visual abstractions of how we model the passage of time in our minds.

Why do we cartoon the way we do? Is there a biological reason for it? I believe so. I believe that comics offer us a mirror to see an expression of the tendencies and limits of our biology. This is not to say other forms of culture don’t allow for that, but just that comics are especially equipped to help us see how we see.  Comics, in my mind, happens to be an egalitarian stage of expression that allows for us as individuals and also as a society to make high level abstractions with accessible, immediate tools and technologies. This mode cuts across race and class on Earth. All you need is a mark making tool and a surface to make marks on.

Comics help us see how we are collectively and individually seeing, most notably how we experience space and time.

I feel this ties into the Sapir-Whorf hypothesis in some way but I can’t articulate it at the moment… The concept of Linguistic Relativity is extremely controversial, so I’m not making any claims this way or that. Just that there’s something here. It’s something that I would like to explore. Unlike the strong hypothesis in Linguistic Relativism, that says that language determines thought and that linguistic categories limit and determine cognitive categories, I believe there’s a flex and flow with it all. Taking up the idea of Visual Language, there’s an inevitable creep of signifier and signified relationships. These relationships can morph due to accidental depiction in drawings that inevitably shift how we see what we see. I hope that makes sense. We depict via representation, but we also wind up abstracting, simplifying, transforming. These transformations, accidental or intentional in turn have an influence in how we see what we were depicting when we experience it in the future. This is a powerful quality of the back and forth between the world and the model of the world that we build with language, one that can be seen extremely vividly in the use of racial caricature. Needless to say I need to think about this more.

Pulling back, though, comics and cartooning endlessly amaze me as I see them as an accessible, evolutionary rich linguistic lens onto our cognitive apparatus through our very cognition.

So that’s where my mind was Saturday. Joyce and Jenn came home. We had a comforting dinner of macaroni and cheese. We stepped out with Jeff to enjoy the cool late August air with a fire. In the garden by the chicken coop.

It was nice.