What do students find reliable?
Some findings of our study are straightforward: Students find visuals (pictures, graphs, and videos) reliable and are hyperaware of advertisements and web design features of sources (Silva, Green, & Walker, 2018). Fewer, however, are able to dig deeper and articulate why certain design choices have the effect they have, and what that might suggest about the information source. Given the fact that misinformation online about racial histories and realities of oppression in our country abound, white supremacist groups may be just one good web designer away from convincing a new group of young folks that their viewpoints are valid.
The trickier finding of our study is that students find authority important while evaluating a source, but there is little agreement about what authority actually means. What makes something authoritative or not? When you have the current Trump administration at odds with the CDC, which avenue of authority are students supposed to trust? How do we define authority, and who gets to decide? What parts of an article speak to authority? When different parts of an article have competing authority claims, which ones are more or less important? Given students' difficulty rating sources, it might seem that students did not understand the importance of authority. However, we contend that our students clearly understood that authority was a significant component of source evaluation. They knew they should look for authority markers, but they struggled to know how sources were created clearly enough to put their finger on what might make them authoritative to a particular audience, for a particular research task. This might lead students to think that the answer in these situations is to believe whatever they want. In our current moment of civil unrest and an awakening of public consciousness surrounding Black Lives Matter and police brutality, believing whatever you want when authority markers are unconventional has life and death consequences for members of our communities.
This finding is both unsurprising but also very difficult to confront. In an era of teaching students pithy rules of thumb about source evaluation, we have told students that authority matters, but we have not sufficiently unpacked how complicated authority markers are. For example, one test that librarians have utilized is the CRAAP test (currency, relevance, authority, accuracy, purpose). These criteria are fairly straightforward to teach but do little to confront the complicated authority markers on the web. While each criterion is a subset of authority, teaching blanket truths about something as contextual as authority unduly simplifies the issue. Indeed, compositionist Jonathan Ostenson (2014) has asked us to move away from these kinds of rules of thumb, but teaching the CRAAP test is still ubiquitous in high school and university information literacy pedagogy.
Furthermore, the students' misunderstanding of information types, like the inability to evaluate mainstream media any differently than fringe media, showcases a naiveté about how online publications are written, edited, and for what reason these publications exist. And while some immaturity is to be expected of students at this level, the amount of time our students spend online begs for better instruction on how to evaluate online source material.
Changing Praxis
As librarians who partner with the first-year writing course on our campus, we have had to think about how source evaluation instruction might change based on the findings of this study. We recognize that many librarians feel deeply conflicted about what to teach in library sessions because we generally only have students for a short period of time. For some of us, we might only have one shot to teach everything we need to teach. Historically we’ve used this time for point and click instruction, instruction that is more task-oriented and less concept-oriented. In other words, our instruction has been very skills-based and not metacognitively oriented (Mackey & Jacobson, 2011, p. 62). To truly change the way librarians teach about source evaluation would be to incorporate them into the curriculum more integrally from the beginning—something that not all library staff are prepared to do given budget and personnel constraints. We are a far cry from the first scholars to suggest such an approach. Indeed, Margaret Artman, Erica Frisicaro-Pawlowski, and Robert Monge (2010) argued that such a collaborative approach will improve the transferability of information literacy competencies for first-year students (p. 105).
In our praxis at BYU, we have set aside about 30 minutes to teach about source evaluation, and have radically shifted from teaching general rules of thumb (accuracy, authority, relevance, currency) to more contextualized instruction about what to look for in different publication types. Instead of suggesting a few markers that can be used in all source evaluation contexts, we instead teach that the context should dictate how the source is evaluated. This helps students move beyond shallow markers of credibility like simply looking at the website design to asking what design means about credibility. It helps students enact more sophisticated fact-checking behaviors like corroborating information or finding out who created the information at hand.
Furthermore, we focus on teaching the difference between evaluating academic sources and popular sources and focus on lateral reading strategies when talking about online, open source material. This comes directly from Sam Wineburg and Sarah McGrew's (2017) work through the Stanford History Education Group, where researchers found that advanced fact checkers spend little time on a website itself (reading vertically up and down the page); rather, fact checkers got off of the website to check its credibility through Google searches and other methods, on average opening up seven tabs in the first few seconds of evaluating a source.
These strategies do not come naturally to students, especially if they have the inclination to assess a website's visual presence for markers of authority rather than leaving the site to figure out who created it and what bias the author/publishing body might have. In discussions with our research team in the library we have come to the conclusion that if information literacy concepts are to be taught effectively in the classroom, librarians (especially in the way they are traditionally incorporated into composition courses) may not be the ones best equipped to teach it. That is not because we are not qualified, but rather because the composition teachers are the ones with the strongest ethos and extended face time with student researchers.
In short, 30 minutes, no matter how well spent, cannot teach all of the ways in which source evaluative behaviors might take place in a research process. In some ways, training the English teachers to think a little bit more like librarians might be the best way to teach source evaluation to writing students.
Pedagogical Takeaways
To restate: composition teachers may be best able to reach student researchers with information literacy skills to help with source evaluation. Though the principles we offer here are targeted for composition instructors, there are certainly many instructional contexts in which you might apply these principles. Our suggestions are based on the major findings of our study, especially in the issues we observed in novice student comments.
1. How does information come to be?
Students struggled with understanding the differences between fringe and mainstream news media outlets. They exhibited distrust of the media in general with little understanding of how different kinds of information are created and how that affects the value of the information. The Framework for Information Literacy for Higher Education, the document that outlines major competencies for information literacy standards, puts it this way: "The unique capabilities and constraints of each creation process as well as the specific information need determine how the product is used. Experts recognize that information creations are valued differently in different contexts, such as academia or the workplace. Elements that affect or reflect on the creation, such as a pre- or post-publication editing or reviewing process, may be indicators of quality" (ACRL, 2015, "Information Creation as a Process," para. 2). Showing students examples of different sources and unpacking the ways they were created, may help students grasp this concept in their research processes (Ostenson & Silva, 2019). Current examples of wildly differing source material surrounding the #BLM conversations and coronavirus would surely prove salient jumping off points for students right now.
2. Authority is contextual.
Given the fact that many composition courses also include elements of rhetorical theory, extending such discussions to how authority functions in information contexts could be helpful. In our study, students struggled to understand and articulate what makes sources authoritative. We believe this has to do with students' confusion over why different sources might be consulted depending on different research tasks. While there are traditional markers of authority that carry from one source type to another (venue quality, author’s credentials), much of what would help a student choose one source over another is what their audience would consider authoritative and persuasive. As the Association of College and Research Libraries (2015) put it: "Information resources reflect their creators’ expertise and credibility, and are evaluated based on the information need and the context in which the information will be used. Authority is constructed in that various communities may recognize different types of authority. It is contextual in that the information need may help to determine the level of authority required" ("Authority is Constructed and Contextual," para. 1). Extending discussions of authority from analyzing texts within their rhetorical contexts to analyzing sources within their research contexts may help students understand this principle. Indeed, as Andrea Baer (2018) argued, moving away from the good/bad dichotomy within information evaluation discussions will help students situate information within its contextual ecosystem and become more sophisticated information users (p. 72).
3. Check your bias.
While unsurprising, confirmation bias was something that students relied on strongly to decide whether information was credible or not. Writing teachers may be able to help students create a more open posture when searching, or at least help students understand how their preconceived notions on a subject might shut them off from learning. Furthermore, discussions on intellectual silos and echo chambers are helpful to aid students in seeing how the information environments in which they live affect their beliefs. Mike Caulfield (2017), media professor and Director of Blended Network Learning at Washington State University in Vancouver, wrote in Web Literacy for Student Fact-Checkers that in addition to developing good fact-checking habits, we must always check ourselves. He cautioned, "When you feel strong emotion–happiness, anger, pride, vindication–and that emotion pushes you to share a 'fact' with others, STOP. Above all, these are the claims that you must fact check. Why? Because you’re already likely to check things you know are important to get right, and you’re predisposed to analyze things that put you [in] an intellectual frame of mind. But things that make you angry or overjoyed, well… our record as humans are not good with these things" ("Building a Fact-Checking Habit by Checking Your Emotions," para. 3–4). While librarians may be able to start discussions about bias, it's unlikely that in 30 or 50 minutes, or even in one to two class periods, we would be able to address the issue in enough depth to help students confront their own beliefs and question their own biases.
Given the extent of our study, the nearly 100 hours of recorded research behavior and 500 pages of transcriptions we coded, our observations both corroborate previous discussions on source evaluation in composition classes and complicate those findings. We find much to be hopeful for. Students seem to have impulses in assessing information sources productively, but lack the context and instruction to help them successfully place themselves within information ecosystems. Indeed, many of us that fall on the more expert side of the spectrum also struggle with our own biases and issues of trust. For instance, throughout this article we have consistently discussed the Washington Post as a reputable source because of our previous experience with that source, and, perhaps, our own biases regarding the kind of journalism we have grown to trust. However, the Washington Post also publishes humor and opinion editorials. If we teach students that the Washington Post is always trustworthy, students will struggle when they come across alternative genres in that venue. Instead, we should teach students to consider their biases, including previous experience, and help them realize when the venue's reputation coincides with other markers of authority as well (like relevant evidence, links to corroborating sources, prior work on the same subject by authors, neutral tone, etc.). In this example, the publication venue is a shortcut, but not a substitute, for various authority markers. Checking bias can help students, and ourselves, make sure we are looking for all of these markers and weighing them within the information's context.
Overall, we believe that first-year writing courses can help with such media and information literacy challenges by focusing on teaching information creation patterns, contextual authority, and bias-checking. These are places where further research with writing students and first-year students at large could uncover practical pedagogical interventions to make a difference in students' critical literacy skills.
Indeed, if our current moment of COVID-19 and a resurgence of discussions surrounding race in America has taught us anything, it is that with the ever-changing information engine that is the web, we are all of us relearning how to evaluate source material. Part of showing students how to be better evaluators of information is to reflect on our own information consumption and usage, and ask ourselves how we came to be the experts we are? COVID-19 and #BLM protests have shown us all our blind spots as fear and confusion have overtaken our information patterns. Perhaps starting with personal reflection will better ground us to be more thoughtful information literacy instructors for our first-year students.