Remaking IO, remaking rhetoric: Semiotic remediation as situated rhetorical practice.

by Paul A. Prior

Introduction

Here I examine a case of a group engaged in remediating an art object called IO, an interactive web site mixing words and images. I followed the collaborative re-design and re-mediation of IO, which was primarily accomplished by Joseph Squier and Nan Goggin, Professors of Art and Design, and by two graduate student research assistants, Tony and Eunah. Joseph is a photographer who has largely stopped taking photographs and become a recycler of cultural images. He describes his work as collecting and repurposing. One of the best illustrations of his practice might be Urban Diary (theplace.walkerart.org). (Some of you may recall seeing an image from Urban Diary in Bolter and Grusin's Remediation, 1999.) Joseph explained that all objects and texts represented in that piece were ones he had found in a baggie in an alley. He still stores the original images in the baggie in his office at the university. Nan Goggin is a graphic designer and director of the Narrative Media Program. Joseph and Nan have worked on a number of projects together, from their current work on Ninth Letter (www.ninthletter.com), a literary-design journal with paper and electronic versions, to a CD publication (think immersive video game meets artistic film meets art work). Tony and Eunah were both international graduate students in Art and Design and both specialized in computer programming.

Data collection, with assistance from Jody Shipka and Kathie Gossett, involved interviews, document collection, videotaped observations, and online use of the sites (including screen captures). In a few cases, participants videotaped their own groups or individual work sessions.

This case calls attention to several key dimensions of the cultural-historical mapping of rhetorical activity we have proposed in the core text (see Take 2 and Mapping literate activity) . IO and its production involve multiple, often dynamic means of representation. Attempting to analyze these means in terms of the canons of style and arrangement would offer little insight into the visual, gestural, and filmic quality of the semiotic objects involved. If we were to take up the internet computer screen as the medium of IO, as the way that IO is delivered, we would miss much of the story here. The computer screen, and the programs underlying it, pervade all aspects of the group's rhetorical activity. For example, the release of Flash 5 prompted Joseph to revise IO. Its new features, in effect, initiated a new round of invention. Likewise, the architecture and capabilities of the database program shaped the goals of the revision and redesign as well as the realization of those goals. This case also focuses on the situated practices of new media production and on the way those practices implicated socialization. Finally, seeing IO simply as a computer-mediated art object misses the extent to which IO was drawn, talked, and gestured into existence.

The original IO

In the (silent) clip below, you can see a video screen capture of the origninal version of IO, the version that Joseph made and that remains archived at the Walker Art Museum site. As this screen capture begins to play (for all the Flash videos on this page, use the rollover panel on the video to start, pause and stop play), the three columns of words that introduce IO are replaced by a layout with words on the left and images on the right. The found images are photoshopped, repurposed and complexly layered. Clicking on an image calls up a new image that appears in a part of the image space, replacing an older image but also sometimes overlaying existing images. The text on the left consists of words that Joseph had originally written in a notebook. On this screen, the text is called up in response to my typing IO into the rectangular input box. The text on the left reads: here is where I live, this is the place of my origin, this is the landscape of my consciousness. Presented in a sans-serif, reversed (white on black) typeface, without capital letters or punctuation, set up in broken lines and arranged asymmetrically in the column-like text space, the words invite us to read them as poetry. Each clause begins, somewhat ironically for a piece of virtual art, with a deictic reference to place (here, this, this).

As you watch the clip above, you can see that the text is animated in a very particular fashion. An open-source artificial intelligence (AI) program named Bob (a distant cousin of the famous Eliza, who was featured in many tales of the Turing Test) feeds the text to the screen letter by letter at an uneven pace and randomizes certain responses to input to prevent the site from appearing predictable or mechanical. Joseph explained that he liked this way of animating the text because it mimicked features of human behavior. In an interview, Joseph described the production of the original IO:

I actually started IO in Italy. I went to Italy for a month and worked on this piece every day...and I went with nothing but still images and this camera and my journal, my journals, that's all I had [actually he noted later that he also, of course, had his laptop computer] and when I came back after a month this is what I had. I did it all. Um, and I, I had a studio and I did the same thing except that my bedroom was [inaudible], I'd, like, get up in the morning and go sit in the studio all day and I would actually close the blinds so I couldn't look out the window and turn the lights out and I sat in there in the dark, practically, for almost a month and, um, that's the best situation for me.

Revising IO

As I noted above, what led Joseph to decide to revise IO, the first art object he had ever chosen to redo after exhibition, was a change in technology. The first version of IO had many interesting features, but Joseph had no way to relate the images to the words directly so that a particular input word would trigger certain images as well as certain words. In addition, Joseph began to think about adding sounds to the site and adding an interactive component where users could offer potential new text and new images. What triggered his decision to revise was the release of Macromedia Flash 5, which brought the integration of sound, image, and text into the realm of the possible as long as Joseph could work with a team, distributing expertise (skill sets). The movie clip below presents images of the revised IO interface. Eunah took the lead in creating the new look of the screen. However, in this clip we hear Tony describing the interface to Nan and Joseph. Prompted by typing words in the input window, the text still feeds to the screen. Now, however, the words no longer appear in a separate space but are blended into the space of the images. The box where words were input in the old IO (upper left light colored rectangle, as seen in the first video) has been replaced by another rectangle, but now in a circle that can be dragged around the screen by the user. The images layered in the background are still complex, but at this point more blended, more seamless, less distinct. The boundary between the black border and the display space is not a straight line, but fuzzy and uneven, a difficult effect to produce.

 

Articulating a Flash template for images

A complex history of mediated activity stood behind the remediation of IO. The next film clip is from an early planning meeting. It displays part of the progressive articulation of a Flash template for how the images from the IO database should be delivered to the screen. Nan had earlier drawn the box, subdivided into smaller boxes on the whiteboard, sketching out the design for Tony, who is in charge of (and most expert on) the PHP database and MySQL program that manages input queries. The template is drawn, talked and gestured into existence. Nan uses different color markers on the whiteboard, hand gestures in the air and over the drawing, and talk to depict the goal for Tony. Tony checks his understanding by annotating the drawing (and earlier had walked over to a laptop on the table and shown Nan the database entry screen). Nan's hand gestures dynamically indicate the spaces to which images from the database will be directed. Much later, I learned that Nan was also remediating ancient Pythagorean theories of harmonic numbers, as her design approximated the golden section proportions that appear in many natural phenomena and that artists and architects have oriented to for over two millennia (see Elam, 2001). The video also shows Christian Cherry, a Dance professor who was involved early in the redesign, eating a sandwich wrap as he watches, listens, and at times (though not in this clip) participates in the discussion. The space of this work (with the whiteboards, table, multiple computers and displays) and its informal atmosphere (with various people working, eating, drinking coffee, and chatting) is also worth noting as a signature of this kind of new media production activity.

 

The movie clip below shows another moment in the emergence of the template. It begins with Tony pointing on the screen to the way the database encodes the image types and, thus, their possible positions in the screen template. This conversation had begun seven months earlier: a part of it was presented above in the clip at the whiteboard. The clip below fades out and resumes at a point a few moments later in the discussion as Nan locates and lays out a pencil drawing (annotated with pixel measurments) of the screen template. The paper offers a mobile and modifiable representational artifact. As with the whiteboard drawings, Tony and Nan talk and gesture over the paper that represents the screen and that then coordinates with the database entries.

What would IO say?

Nan: So do you think—ok, this, should say, um, do you want it to say like Are you sure you want to leave?

Eunah: [nods] I just put the word [laughs]

Nan: Or maybe you should say, uh, you wouldn't, I mean the machine's talking to us, right?

Eunah: Umhm

Nan: So maybe we wouldn't say um Do you want to leave IO? Because you're—you'd say Are you sure you want to leave me? Maybe that's what it should say?

Eunah: Yeah

Nan: Are you sure you have to go?

In the short transcript above, we see a conversation (about 5 months after the whiteboard drawing shown in the third clip). Nan and Eunah are discussing the redesigned interface. Interestingly, this conversation focused on what IO should say to users, in the written text dialog box of the circle interface, before they quit the program. In the first line, Nan points to an IO prompt on the screen, Do you want to leave IO?, and suggests possible alternatives. Indexically, Nan quickly shifts from what this should say to what you and we would say, making the point explicit in her second turn (the machine is talking to us). She offers three direct representations of what the machine might say, intended to function as candidate revisions: Are you sure you want to go?, Are you sure you want to leave me?, and Are you sure you have to go? This kind of externalization of the motives for a specific revision of wording and the way it is being orally composed is interesting. Nan here seems to be directing her comments at Eunah as well as at revising the text, trying to ensure that Eunah takes up the perspective that IO should be designed to simulate a human interacting with the users (cf. Engestrom and Escalante's 1995 tale of Postal Buddy). Nan explicitly highlights the personalization of IO and offers a somewhat petulant sounding dialog box (Are you sure you want to leave me?) before moving back to a more neutral form (Are you sure you have to go?). In other words, I read Nan's responses here as directed at the socialization (see Society and socialization) of Eunah as well as the entextualization of IO's personality.

Conclusion

While a growing body of work (e.g., Bolter & Grusin, 1999; Manovich, 2001; Lemke, 1998, 2003; Kress & Van Leeuwen, 2001; Wysocki, 1998, 2004) has begun exploring ways to understand multimodal artifacts and some attention has gone to reworking rhetoric to account for electracy (e.g., Welch, 1999), the situated and mediated practices of producing new media have received limited attention (see, however, Johnson-Eilola, 2005). Even in this very partial tracing of the redesign process for IO, we begin to see multiple levels and types of mediation:

These mediational means display a chronotopic lamination (see Prior and Shipka, 2003), or what Hutchins (1995) refers to as heterochronicity, with multiple times and places uniting in the present. This lamination of history can be seen, for example, in the way the site is organized around both PHP and Pythagoras, the way the web is remediating photography, the way Flash templates and PHP databases are negotiated through gesture (presumably one of our most ancient semiotic systems), and the way paper-and-pencil drawings guide the programming of screen dynamics. Finally, as we follow the process of the redesign of IO, we also begin to see the way that this site is involved in remaking the people as well as the artifacts. Tony, for example, is doing this project for his master's thesis as well as working as a research assistant. Eunah is learning how a basic concept (that IO should be person-like) guides even the language of routine prompts, and she is learning as well much about the way such groups work. Cultural-historical activity theory (CHAT) offers a rich theoretical framework for exploring such situated, multiply mediated, semiotic and social practices.

This research also captured a key sociohistorc transition in the artistic practice of the professors, Joseph and Nan. In an interview at the end of the semester, they both noted that the revised IO marked the first time they had completed an object that had elements they did not understand and could not reproduce or repair. Tony and Eunah had areas of expertise that were basic to the revised IO and that Joseph and Nan were unlikely to ever develop. Nor was this transition a temporary aberration. Their next planned project would take them into the CAVE, a virtual reality space, where the programming team would be much larger and the programming much more deeply embedded and, thus, farther out of reach. Like the large-scale shift from individual laboratories to team science and engineering that Andrew Pickering (1995) notes happened in physics around World War II, the revision of IO captures a transition, at least for these artists, into team art.

Finally, as interviews with Joseph made explicit, through projects like IO, Joseph and Nan were engaged in a larger rhetorical campaign (see Society and socialization and Using CHAT), an effort to infuse digital spaces with complex human experiences and to combat the potential of computers to dehumanize our world. I conclude then by suggesting that the character of their rhetorical practices requires rich tools for mapping the laminated chronotopes and local functional systems through which their semiotic activity was achieved (see Take 2 and Mapping literate activity).

All works cited here available at the webtext's common references.

Return to mapping page link to mapping page