The off ramp: AI literacy obstructs learning and academic freedom

A series outlining my concerns with the “two-lane” approach to assessment security. In this article I explain why scaffolding AI skills via "open" assessments is terrible pedagogy.

A refresher: what is the two-lane approach?

Australian universities are rolling out tough new assessment security approaches in response to the so-called “Age of AI”, which is seeing a tidal wave of cheating that is rendering university degrees worthless as currency for the job market.

For a quick refresher on the two-lane approach, you can review the University of Sydney’s explainer, or my summary here. In even-more-brief:

• Lane 1 assessments are “secure”, meaning the conditions for student work are tightly controlled and supervised. Generative AI can be banned.

• Lane 2 assessments are “open”. This means the conditions for completing them are flexible and unrestricted, and generative AI cannot be banned.

I have two problems with the two-lane approach: Lane 1, and Lane 2. I’ve written before about my concerns with Lane 1 and its high potential to obliterate decades of advancement in educational equity and assessment validity.

Now, I want to focus on the insidious political and pedagogical agenda embedded in Lane 2 — “open” assessments. This is a longer one, so I’ve split it into subheadings, which I hope you can use as bookmarks if you need to split your reading time.

“Open” assessments promote AI literacy

The nature of Lane 2 is such that generative AI “cannot be banned” (indeed this was cutely phrased as “banning the banning of AI” in a recent USyd editorial).

“Open” assessments are, you’ll recall, assessment for learning. The idea is that this is where unit-level assessments happen; where students develop various elements of their desired skillsets in feedback-supported ways. The formats of assessment are flexible because the conditions are not secured, in the sense that they are not required to be in person, supervised, time-constrained and resource-controlled.

(I presume that standard cheating checks will still apply, such as Turnitin similarity reports and metadata checks to ensure the student is named as the author of the submitted document. Turnitin is also an absolute steaming garbage heap of a tool, but I’ll leave that argument to an expert.)

Important to the conceptualisation of “open” assessments is that, because students cannot be prevented from using generative AI to complete them, generative AI use should be “supported and scaffolded” to ensure that students don’t just prompt them all willy-nilly. The purpose: build AI literacy to equip students with vital skills needed for the world ahead.

But, as I’ve stated before, the evidence to date indicates that generative AI use is degenerative to literacy. Significant studies have shown that generative AI use not only does not improve learning gains, but actually appears to prevent them. Punya Mishra analyses Microsoft research to show that people with high expertise in a discipline actually have to manage higher cognitive demands if they use generative AI than if they don’t use it. This is because it takes considerable additional effort to evaluate and refine the output of an app that uses probability, not reason, to produce claims; and experts are able to recognise that a great deal is wrong with those claims. Let me, ahem, underscore that one.

Using generative AI makes life harder, not easier, for experts.

But for students, who by definition are not experts, Mishra suggests that using generative AI can lead to either:

• false confidence, if the student has “AI literacy” and can produce prolific output, but lacks the expertise to notice and correct content errors

• problematic vulnerability, if the student has no expertise in either domain and is being expected to use generative AI to learn.

To be fair, the research, though insistent and numerous, is nascent and inconclusive. We don’t actually understand generative AI’s impacts on learning. But the evidence so far is pointing to this:

“AI literacy” doesn’t improve desired learning. It obstructs it.

“Open” assessments close avenues for student autonomy and agency

For the above reason and many others, both university students and educators may prefer not to use generative AI in every case where it’s possible to avoid it.¹ A curriculum approach that integrates generative AI use into assessment for learning is one that forcibly obstructs academic agency, coercing students and educators to engage with the technologies whether they wish to or not.

As a teacher, a student and a learning designer in universities, I object.

I would like to reserve the right to tell my students I expect them not to use generative AI in their assessments — whether I can enforce this or not. It wouldn’t be about “banning” the technology, but about establishing a contract between me and my students. If you use generative AI, and I don’t know it, then I can’t teach you properly. If you don’t feel you need to be taught, then you are wasting your own time in my classes.

Of course, I don’t currently have the right to say this — I’m a sessional teacher and I don’t get to make these decisions. And even if I did, I might not say it. I’m generally very happy to give feedback to students who tell me how they’re using generative AI in their workflows. And the more I’m able to speak honestly to my students, the more they are willing to be honest with me.

But you know what? I’m an academic. I’m supposed to have academic freedom. We’re supposed to be asking important questions, not merely following instructions. I should be able to choose. And so should our students.

“Open” assessments position ubiquitous AI as inevitable

Supporting and scaffolding generative AI use through “open” assessments plays into the narrative that AI literacy is a must-have for all graduates. I don’t just mean it assumes it. I mean it ensures it.

Higher education curricula don’t just echo the rhythms of future professional practice. They inform them, by equipping graduates with various knowledges, habits and expectations about the norms of their disciplines.

Of course, plenty of the habits we learn in university make it no farther than the garbage bin at the campus gate — how many professional workplaces use Chicago style? But plenty more shapes our future professional practices. My copyediting teacher told us never to use a red pen to correct an author’s work (too threatening). To this day I keep a purple pen on hand for markup.

When students are persistently instructed and encouraged to integrate generative AI into their assignment workflows, the products become naturalised, rhythmic. Whether future workplaces seek generative AI users or not, many graduates will struggle to operate without them.

This isn’t “inevitable”. We’re choosing to make it happen if we tell our students we expect them to use generative AI in the work they produce for us.

“Open” assessments aren’t constructively aligned with “secure” ones

All of this is delightfully problematic from a pedagogical perspective when we consider that the “secure” summative assessments students will face are AI-restricted or, indeed, AI-prohibited. It continues to be unclear to me how this misalignment of assessment “for” and “of” learning can be considered logical or valid.

The University of Sydney is one that encourages program-level assessment — that is, a total assessment strategy that measures students’ achievement of program-level learning outcomes, not only unit-level. This is a great approach, for many reasons.

Basically: all parts of the course — every lesson in every week of every unit — align to some part of the overall desired learning for the program as a whole. Assessments are the way that students actively, constructively demonstrate their achievement of these desired learning outcomes.

But USyd’s program-level assessments must be “secure”. Which means that they will not allow the flexibility of generative AI use that students have …enjoyed… in their unit-level “open” assessments. In all likelihood, generative AI will be prohibited — though in some cases it may be allowed but tightly restricted.

Not only does this not align, but it is precisely opposite to the emerging finding from Kosmyna et al. that students who start writing essays using LLMs perform far worse later without LLMs. It appears that it’s much better for students to learn such skills without generative AI, because they will perform them better later regardless of whether they use it in future.

This aligns neatly with Punya Mishra’s earlier warning that discipline expertise is needed before generative AI skills (if generative AI skills are to be used at all).

In a nutshell

In sum, “open” assessments are corrosive to good pedagogy, academic freedom, student and teacher agency, and the potential for humanity to disentangle itself from the harms of the present commercially and politically corrupt generative AI value chain.

Setting aside the larger-scale environmental, legal, social and political impacts of generative AI development and use (although we shouldn’t), we don’t know what the long term educational effects will be. So far we have evidence of GenAI’s educational corrosion in studies lasting months. What will this look like over years or decades?

Perhaps we can’t enforce a ban on generative AI in unsupervised assessment. But is an inability to detect it a valid reason not only to allow but encourage its use?

If we applied the same reasoning to other matters of moral and legal import, society would collapse. Not all murders are provable. Not all lies are detectable. We still don’t ban their banning.

We can, and must, teach our students better than this.

1. I will not be engaging with responses that claim that avoiding generative AI isn’t possible. I’m not claiming it’s possible. I don’t have a choice, right now, to avoid the Gemini widgets that keep inserting themselves into my Android software. But I can and do choose not to create a ChatGPT account or actively use OpenAI products. Don’t tell me limited choice is the same as no choice at all.