Headagogy 10 - Peer Assessment Ep1
Thu, Oct 20, 2022 8:56AM • 53:04
SUMMARY KEYWORDS
students, grades, peer assessment, educators, engraving, assessment, excalibur, arthur, discipline, knights, power, practices, rubric, standards, education, understand, writers, class, assess, sword
SPEAKERS
Steve Pearlman, Excalibur
Steve Pearlman 00:00
I have been putting this podcast off for a long time. And I am, in so many ways reticent to try to tackle it, because what I'm going to talk about is the importance of peer assessment, which means involving students in grading one another, and its relationship to engraving, which I think is particularly important. And my problem and talking about this today is simply this. The issue of peer assessment, not to mention the greater issue of assessment itself has so many interrelated overlapping, interconnected forces around power, learning, engagement, critical thinking, and frankly, just how much is learned that this is a difficult discussion to work my way through in any way that I hope you all will find even remotely cogent. And that's not because I don't have enough to say about it, it's because I have too much to say about it, I need to talk to you about not only how peer assessment affects learning outcomes, writing, skill development, critical thinking, development, but also about students attitudes about it about how to implement it successfully, and why it can work, I need to talk about how it can elevate instead of diminished rigor, while doing so in a way that students actually appreciate. And by which they feel empowered, and that they are given agency and respect, I need to talk about how it does not undermine but rather improve our standing as educators, not only in our own knowledge of our fields, but in how much our students respect us. And what it is that we do, I certainly need to talk about the relationship between peer assessment and democracy, and how critical it is that we're not fostering a population of people, as I would argue that we have been, who are taught to listen to the answers given to them by authoritarian figures, instead of finding the agency and the critical thinking skills to play a role in constructing an understanding of truth for themselves. And I'm entirely omitting another dozen or so things that I need to talk about with respect to peer assessment. So I'm hoping I'm able to trace a path through this that is at least something you feel as though was cogent enough for you to gain a modicum of understanding and appreciation for what I'm describing. And if I fail at doing that, then I apologize. But I would have to think that there's going to be a lot of interesting and thoughtful research along the way, if not some important provocation, about how we treat assessment and the critical importance of involving students in it. And I'm going to argue that I feel that's actually the ethical way to grade students. Now take that with a grain of salt, please take that with a grain of salt, because I don't really mean it. And I don't really mean it, because I don't think I'm wise enough to determine the only anything about education. But I'm gonna put it out there as a polemic for the sake of the podcast, with the agreement that you're going to understand that there's a grain of salt that goes with it, because I do think there's an argument to be made for that case. And I'm going to be the one to make that argument. And ultimately, if you see something that's better than what I'm describing, then that's really what we want. In the end, as a way to articulate the importance of peer assessment. I'm going to contrast it against a couple of other assessment methodologies. I'm going to try my darndest to make sure that I don't essential eyes, those other assessment methods into straw person arguments, because that's really not my intention. And I'm sure I'm going to do that to a certain degree, because I'm not going to talk about nor tried to justify any of them as much as I'm going to try to justify peer assessment. And so while I might give some of those other methods, a little bit of a short shrift, I hope I'm not in any way doing them injustice. And if you find me doing that, then that could be one of the grains of salt with which my argument must be taken. But the other types of assessment I'm going to use as foils for peer assessment are upgrading peer feedback, which is different than peer assessment, we'll get into those distinctions and why they're of critical importance, and then more traditional summative assessment methods that are more authoritarian by the educator. And those are the ones I think with which we are most familiar. Having now foregrounded the discussion that's to come about peer assessment. I want to set up the overarching metaphor for what peer assessment does and just how powerful it is. And in order to do that, I need to tell you about one of my favorite movie scenes of All Time, one that has carried with me since I was a child and I got to watch this movie over and over again, not only because I really liked it, but because it was one of the few movies on Showtime in early cable and that movie is Excalibur. Excalibur was a film made of the Arthurian legend of King Arthur and the sword and Merlin and so forth. It was produced in 1981, and it was based on landmark to Arthur by Sir Thomas Mallory. And as a bonus for any of you Trekkies out there, you'll get a glimpse of Patrick Stewart, who played King leon de Grant's, also of note are earlier appearances by people like Liam Neeson, and Gabriel Byrne, not to mention the presence of Helen Mirren, who was fantastically gorgeous in the film, and obviously offered just a magnificent performance. And so I'm about to play you the audio from my favorite scene in that movie, which again, is one of my favorite movie scenes of all time. And so let me just afford you a little bit of a context for what you're about to hear. If you recall the tale of the Sword and the Stone, what happened was that the previous rightful King of England, whose name was Arthur, Penn Dragon, because he did not conduct himself very well, I won't give that away entirely, but because he did not conduct himself very well was being hunted down by a group of other knights and with him was Excalibur mythical sort of legendary power that could only be wielded by the rightful King of England, which of course, is very convenient for England to have that sword. But nevertheless, as Luther pin Dragon was meeting his fate, rather than give the sword to anyone else, he took Excalibur and he drove it into a boulder hence to be known as the stone of the Sword and the Stone. So for years after that, the knights would gather on Easter, and they would joust and they would battle and they would fight for the right to have an opportunity to draw the sword from the stone, and many knights would win their battles, and they would ride over to where Excalibur was driven into the stone, and they would make every effort they could to extract Excalibur, but it wouldn't budge. Through a series of circumstances I don't need to discuss a young Squire and a squire being of course, a boy who is not in any way yet qualified to be a knight. But a young squire named Arthur, who unbeknownst to him was the son of wizard pin Dragon, and therefore the rightful King of England drew the sword from the stone, and that created quite a bit of controversy between the Knights because none of them knew who Arthur was. He was some boy or at least teen and they felt as if his capacity to draw the sword from the stone must have been a deception created by Merlin the wizard, and they had no intention of following a boy as the king. So in the scene, I'm about to show you the Knights are battling between each other. Some of them are loyal to Arthur because if the rule was that whoever drew the sword from the stone would be king, and Arthur drew the sword from the stone. Then Arthur was king. And some of them were against Arthur because he was just a boy and they didn't want to follow some snot nose boys king. The leader of the opposition against Arthur was Sir uriens, played by Keith Buckley did a wonderful job. And so uriens is a well respected and seasoned night and warrior and leader after the battle ensues for some time, and where we're going to pick up on the clip Arthur has somehow unhorsed uriens and uriens is kneeling waist deep in a river and Arthur is standing over him with Excalibur pointed at his throat. And what you're going to hear is that Arthur nobly attempts to offer sir uriens the opportunity to yield to him and therefore become one of his knights. But uriens a man of honor is going to object and he's going to object because according to the rules of knighthood, a squire can offer mercy to anybody. A squire lacks the authority to do anything for a higher stationed night in that regard. And then you're going to hear how this plays out. You'll also hear a few snippets of Merlin played by Nicola Williamson in a magnificent performance, mentioning a few things about what he had and had not foresaw with respect to how this scene goes down. But mostly what you're going to hear is the conversation between Arthur and Europeans were uriens, a noble knight is going to say there's no way I'm submitting to you. You're a lowly squire, that would dishonor me and Arthur concedes his point, and what you'll hear but what you won't be able to see happen Is that inconceivable his point Arthur then hands Excalibur to the otherwise disarmed uriens and Arthur then kneels down in the moat, and EURion stands up and has Excalibur the ancient sword of power, and he could easily kill Arthur and become king. And you will hear a couple of nights in the background telling him to keep the sword. But since you all know that Arthur goes on to be king, he obviously doesn't. And this is how it plays out. Again from the movie Excalibur a young King Arthur and the knight Sir uriens.
Excalibur 10:38
Swear faith to me. Mercy I need by the Lord suckers you You're right. I'm not gonna lie to you. Here Ian's like me. Tonight, I count off your mercy What's this? What's this I've got son Michael and St. George. I give you the right to bear arms and a part of me justice. A duty I will solemnly obey as Knight and King. I never saw this. Rise, King. I am your humble Knight. And I swear allegiance to the courage in your veins. so strong it is. It's psoas muscle.
Steve Pearlman 12:13
I doubt you're noble. So Arthur knelt down and Europeans had all of the power in his hand. Arthur gave up the power to Europeans, he gave him the sword. And Europeans though tempted for a moment to just kill Arthur and therefore in holding Excalibur become king finally realized Arthur's true power for only the son of uth, or pen dragon, the real king would have the balls to hand Excalibur over to another night, an enemy night so that he could be knighted by him. Nobody else would be powerful enough to do that. And I suspect at this point that you can see where this is setting up my discussion of peer assessment. And I'd like you just to hold it in mind as I work through this discussion of peer assessment. Because what peer assessment is advocating is roughly what plays out in the scene. It's advocating that we hand Excalibur over to our students. But in doing so, they are going to realize and come to new appreciation for just how powerful we really are, as intellectuals, as educators, as experts in our disciplines, and they effectively will then willingly and eagerly serve nobly as knights working with us towards the greater cause of a better state of the Britons, I guess, to torture the metaphor as much as I can. So please keep that scene in mind. Keep in mind the value and positive outcome of handing over Excalibur and how it did not undermine Arthur, but rather revealed him and empowered him to go on and build the Knights of the Round Table and forge the utopian mystic, Camelot. And now let me get to the real business of the podcast by first talking about engraving more generally. I'm going to start by saying a little something about engraving in itself. And the first thing I have to say is that anyone who is moving towards engraving who is questioning traditional assessment methodologies, I think we are all on the same team. Here we are all recognizing that traditional assessment in the sense of a single summative grade brought down from on high onto the student by the educator is simply not the best way to educate anybody, and that all of our goals must be to move towards whatever serves our students best in terms of their education and empowerment and summative assessment. Traditional summative assessment is not that way. I will go into that in more detail, but the point Bing for now, as I'm about to put a critique on engraving, as it is sometimes enacted in education, I want to note again, that these might be my predilections about those other methods of engraving. And that ultimately, I think we're all really allies in trying to find better ways to educate students than through the most traditional assessment methods. So I'm a fan of engraving as a general principle, and generally how people are applying it. I'm going to argue that there are a couple of challenges with that. And I think that peer assessment helps us to resolve some of those challenges on grading certainly involves a spectrum of activities, it absolutely does not necessarily mean that we are immediately trying to do away with all grading entirely. Grading assessment in education is not going anywhere, anytime soon. And even if we could get it to go away, I don't believe that's necessarily the best thing for us to do. The truth is that in life formally, and informally, we are going to be assessed in different ways, we're going to be evaluated in different ways, in our careers, by our peers, by our friends, by our partners, people will make judgments of us and we need to learn how to contend with that. And we need to learn how to contend with that constructively such that it empowers us all rather than disempowers, ourselves or anyone in our constructs. And I don't think education needs to be accepted from that reality. And I don't think education could be accepted from that reality. But let's take an example of engraving. And let's take one of the more extreme ones. And let's think about it for a minute. If we were to say to students in a class that at the end of this class, after, let's say, a formidable amount of reflection on their behalf, perhaps written reflection perhaps dialogue, that they are nevertheless going to assign themselves their own grades individually, and that those grades will, for all intents and purposes be the grades that they receive for the course, maybe with some loose boundaries, perhaps not perhaps we might very well want to eliminate the most extreme cases of the student who never attends and does any work and then tries to assign themselves an A for the course. But for the most part, we're allowing students to assign their own grades. And they might assign those grades based on how they feel they performed in the class. And let's put aside right away those students who are not engaging that experience with integrity, because we already know why that's problematic. But let's work from the premise that we have students who are acting with integrity, students who feel as though they devoted a great deal of effort, perhaps more than in their other classes, students who feel as though they made certain epiphanies and strides within their writing or in the subject matter of the course. And they genuinely feel as though that they have grown as a human being and grown within the discipline. And they're going to assign themselves a grade, again, that might have come through some dialogue and reflection on the student's behalf with us or just in their own introspective space. So I don't want to suggest that this grade is emerging out of nowhere. And perhaps it is relative to the student's understanding of some criteria that were put forward or perhaps not. But nevertheless, what is certain in this construct is that we have communicated to the students that we are not going to be the dominant active agents, and how those grades emerge that the students are going to be, if not the exclusive active agents, and how those grades emerge, but at least the dominant active agents for their own individual grades. In that case, I think we need to think very carefully about what we just capitulated, what we just capitulated were several dangerous points. First of all, the message is that whatever they think about their own growth and education, and achievement is correct. And I'm going to use writing as the general locus for this discussion, because my backgrounds in composition, but this applies to any modality of assessment. But if we're dealing with writers and student feels as though student made great strides in their writing, then what we are effectively telling them is that if you feel as though you made great strides, you also therefore did make great strides. Now, that might not be the literal commentary, but I think that there's an implicit message to the student there. And yet what we know from so much research that has been done on student self perception of their own learning is that it is frankly not always accurate. How many times have we encountered a student who comes to us and says that they think they did great on their paper. They think that they really achieved something powerful. They think they revised it in accordance with our comments or the comments of their peers. They think that they have presented something that's pretty ticularly thoughtful or innovative or forward thinking or unique for the field, or the discipline, or the class or the assignment, when in fact, they simply haven't. Perhaps what they're seeing as this sudden, exceptional growth might be something that feels dramatic and exceptional to them for themselves. But if we were to try to assess it from any other particular scale of how college writers might perform, we might find that it's just a little bit of a blip of growth, if any growth at all. Now, that, of course, is not true for all the students, some students might feel as though they haven't made much growth and have made great growth. Some students might feel as though they've made great growth and haven't made much at all. And some students might be accurate in assessing the amount of growth they've made in their capacity as a writer. But isn't that the point, if we're leaving this up to the students, we've sacrificed the notion that there can be any observable standard for whether or not they've made the progress they think they've made, and that when any student thinks about their own growth is accurate. Now, that's bad for the student who thinks that they've grown a lot when they haven't. But I think that's an even greater injustice to the student, some of those with the most integrity and some of those who are most critical on themselves as sometimes we would want students and intellectuals and writers and critical thinkers to be that they're saying, I didn't really make a lot of growth as a writer, when in fact, they did. And so these other students who believe they made a lot of growth, but haven't are getting good grades for themselves, where the students who have actually made great gains are not rewarding themselves, because they are more self critical and hold themselves, perhaps, to a higher standard. And so somehow, because grades do have currency, and everyone in engraving acknowledges that, in an effort to be more fair to the students to take off the authoritarian pressure of grades, to help them focus on their self growth and self worth, and self actualization and writing skills and critical thinking skills instead of on the grade itself. What we've nevertheless ended up doing is an injustice to those students. Now, the counterpoint to that might very well be that the more important message to the student is perhaps that grades shouldn't really matter. Anyway, they're a momentary snapshot of perception, or what have you, and we shouldn't put so much weight on them. But nevertheless, the great is still happening. And if it's happening, shouldn't it need to happen so that it's fair to all the students. And there is an argument that these are all students who are different. And so it's challenging to apply any kind of consistent standard to a roomful of different people who are coming from different places with different levels of experience, and so on and so forth. And that we should respect and value their uniqueness and personal understanding of their own growth and development over what might be a more authoritarian standard for that. But there's a difference, a very important difference, I think, between valuing their uniqueness and their capacity for self reflection and their agency. And then also at the same time, saying, therefore, that everyone's grades are good, saying that whatever anyone graded themselves is equally good. I think one of the errors that's being made within the engraving movement is the false dichotomy between either the authoritarian model, or that more individualized and decentralized on grading model. And I'm hoping to be able to make a good case for why peer assessment is the Goldilocks solution between those things, because I think engraving also makes another capitulation to the students that we don't want to make. And that capitulation is, well, there's just really no way to assess what anyone is doing, especially if you're all coming from different backgrounds in different situations with different skills and during the course, who could tell who is developed as a writer and who hasn't? Well, now let's think of the pedagogical message and theory that's being carried in that implicit message. If I can tell how my students are being developed as writers or historians or physicists, or what have you, how can I possibly know how to teach them? How can I possibly know which methods work better than others? Because we know that students self report is not necessarily a reliable source for that. The fact that students like it or feel as though they're learning is not always an indicator of whether or not they actually are. That's not to say if students found something particularly meaningful that we could say it wasn't meaningful to them. But whether or not they're growing within the discipline is a different question, though. Obviously, we want to try to merge those down the road. But if I cannot tell as an educator to what extent within my discipline that what I'm doing as a pedagogical practice is having an impact on the students capacity within that discipline. How can I know how to improve my practice? How can I know which pedagogies are serving the discipline? Well, and serving my students well, and which pedagogies are not, I can't. And I would contend that that's not a little dangerous. I would contend that that's fatally dangerous. How can we say you go off and learn in this discipline, you engage in the practices of this class, you do the things that I'm telling you, but I have no idea whether or not they're really working, or for whom they're working better than others, or which practices are more successful than others, or how well you're advancing in your goals and in your needs to become a member of this discipline, a colleague, an intellectual in this field. And again, I want to be equally leery of the contrasting point of the far end of the other spectrum, which is that because I've taught something in class, because you have regurgitated on a test, I can therefore determine that you have learned something meaningful and important within the discipline. And the grade on that single summative multiple choice exam, based on the lectures and the readings that I gave is an accurate reflection of your movement within the discipline. That's also not the case. And again, I want to get us out of this either or construct where it's either that authoritarian model on one side, or the individualistic student driven model on the other, neither is a good answer. And I understand the desire to push back against the authoritarian model by adopting the engraving methodology and mindset. And I affirm that again, I support that, even in ways where I think it's problematic, I think that we have to push back against highly authoritarian grading. And we have to do so for democratic purposes, if nothing else, because I think it will give us a better society. But from a pedagogical perspective, I want to return to my point, if our message to students becomes we can't really assess the extent to which you're improving in our discipline, then we're also confessing that we don't really know how to teach it. And worse, we actually then don't know how to teach it because we're not making assessments of our students progress along the way, not authentic assessments, valid assessments, reliable assessments, I understand why those terms are problematic with respect to assessment, but I'm going to try to work back into them later as we get into pure assessment. And once again, I want to be really clear that I've chosen a fairly extreme example of Engraving and a fairly extreme example of traditional assessment, again, using those as foils for my argument. If we backtrack that a little bit, then my argument retreats in some degrees proportionally, which is to say that if we want to start engraving by having a conversation with our students about what grades mean to them, and how they see their role within grading, what messages they hear from grading, how they perceive the validity of grading, we're having that conversation and trying to inform our practice based on some of the things that our students say, Well, I don't really have any objections to that honest, earnest dialogue between humans about matters that matter are important and valuable. In fact, part of my move into peer assessment has occurred because of those kinds of discussions with my students. Another way that it's occurred was that the first semester I taught writing at American University back in 1992, when I was a graduate student there in the literature program, and I volunteered to take on a freshman comp course, I looked at my students papers at the end of the semester, and compared them to their papers at the beginning of the semester. And it was painfully evident to me that my students were really no better writers at the end of my course than at the beginning of my course, they had written papers, but I had not made them notably better. I don't think even marginally better writers than they were when they entered, I certainly did not lead them to do so. And my feedback to them was probably pretty useless. The flaw in my practice was not only limited to how I assessed there were all kinds of things that for me, as a first time educator in front of the classroom, I simply didn't do very well, despite having I will tell you have a passion for teaching. I could not wait for my class to happen every single week. I look forward to it. I could not wait. I enjoyed the process of teaching them. I think we were successfully good as a group for the most part, but I sucked as an educator. And in fact, the reviews suggested that I did some positive things for the students I had inspired them in certain ways and gotten them to look at the world in certain ways that were different and feel empowered about themselves. And those things were great. And I took pride in those and take pride in those. But their papers aren't any better at the end than at the beginning. And yes, there is no question that I have ever since been somewhat driven by the question of how can I know for sure, then when the end of my semester comes around as a right instructor that my students have really grown as writers, and not that I might just think that they've grown as writers, because I might feel as though some of the things they've written are better. And we know from a lot of research that's been done on writing instruction, as well as anyone who grades writing across all of the disciplines, not to mention many other forms of assessment, that educators will tend to see a self efficacy and what they do with respect to their students work, we tend to see growth when it isn't there. But nevertheless, my point is that 30 years ago, after teaching my first class, though, I didn't really recognize it at the time in any self aware kind of way. My recognition that I wasn't accomplishing very much with my students actually began my journey into engraving and Peer Assessment, which I really started to do, maybe about 25 years ago with students was my way to start on grading my students and start to help them focus more on the important work of becoming better writers, de emphasizing grades within the class, and forging a way to look at Academia such that we could truly understand whether or not growth is occurring, and do so in a way that was fair to students and empowering to students at the same time. But let me speak now to the impetus for engraving in general, which is entirely noble and correct. I'll talk to that on a couple of different levels as we go forward. I'll talk about simply its relationship to actual learning outcomes. But I think before we get to that, the more important discussion and the theme that I'll continue to run through this discussion more consistently, is that of power. And there's just no question that within academia, power is held in the capacity to grade and students know that they are disempowered in that regard because they don't hold the keys to the kingdom and the keys to the kingdom are in assessment. What is right what is good, what is growth, what is learning is communicated through the grade. As Reynolds and tree Han put it, assessment is not simply another aspect of the educational method. Its function in providing the basis for granting or withholding qualifications makes it a primary location for power relations. In fact, Reynolds and tree Han go on to argue that more than any other aspect of education assessment embodies power relations between the institution and its students with tutors or educators as custodians of the institution's rules and practices. The effects of judgments made on individuals careers, as well as the evaluation of their worth by themselves or by others ensures that assessment is experienced by students as being of considerable significance. As heroine put it all way back in 1979. Assessment is the most political of all educational processes. It is where issues of power are most at stake. As Abdulmalik put it in 2016, students come to see themselves as powerless in their own education and see professors as having a majority of power to educate and to produce learning. And isn't that Locus of Power Within grading and how it has been so tightly held by educators not what on grading nobili pushes against. And here's why it's so important that we push against that, because there's just all kinds of research studies upon studies that students have absolutely no idea from where their grades come, especially when it comes to things like writing. They understand it more with respect to a multiple choice test, though, they don't always understand why certain questions appear on a multiple choice test instead of others, they don't understand the significance. And they also don't see particular meaning or value in multiple choice testing. So they might understand where the grade comes from better with a multiple choice tests, but they don't necessarily see it as particularly meaningful learning experience. Whereas with something more like writing or presentations or something more where critical thinking is involved, they simply don't know how grades emerge. And let's problematize this even more, many educators confess that they cannot really articulate all of the distinctions between the grades they award to students, especially with respect to writing or presentations. And we know in fact that much of it is subconscious for educators and idiosyncratic we know as well without question that if we take a number of educators and we put them in the same room, and we give them the same paper, even if they are in the same department, that the grades can vary greatly. In fact, those grades can vary between F and A. We know as well that educators tend to grade their own students writing or presentations, or what have you higher than they will grade those of their colleagues. And that's not a terrible thing of which to be guilty. We get to know our students, we get to also understand what it is they're trying to say or what discussion they might be trying to have, or how what they're saying reflects something that we've talked about in class. And so if nothing else, the fact that we know our students and want them to succeed might affect and elevate grades, maybe isn't terrible, but it's really not great for the students. I mean, it's a rather noble, humanistic thing to happen, but it's not helping the students grow if grades are distorted, elevated, or mysterious. And now let me problematize that construct even more and further justify on grading even the engraving for which I take issue. In the typical construct. The problem is not that students do not understand where grades come from, and their meaning and what they're really communicating. The problem is that they can't, it is under most constructs, a literal impossibility for students to understand what grades are really signifying. And I want to paint this picture for you because it's so important to the discussion that's going to follow that we understand the real nature of this paradigm, this could not be more critical. What we know for certain is that faculty assign grades and conduct assessment not based on criteria they necessarily articulate to students, but on a tremendous amount of tacit knowledge and conversation to which the students have no access at all within the academy, as Fienberg peoples and Hart Davidson wrote in 1997 grading artifacts such as assignments, and rubrics are only half of the technology of grading, the other half being the network of cultural, institutional and personal values, rules and decisions that make up the technical code of grading. Seen in this light, the teacher's job of grading papers amounts to participating in a number of arguments within and among the institutional frameworks in which she works. And with any number of specific subjects, students writing program administrators, other teachers, parents, et cetera. In other words, the students submit their papers. And perhaps though most faculty members do not perhaps you even presented them with a rubric for how that will be graded. And we'll talk more about that in a minute. Because it doesn't necessarily make things better, at least not as rubrics are typically constructed or exercised. But how do we generate the grade for that paper, we're trying to be fair, we're looking at whether or not they're insightful or whether or not they're analyzing or whether or not they're doing certain things within the field that the field would recognize is important. What's influencing us in that decision to which the student can have no access is all the conversations we've had about grading prior to that all the conversations we've had about what it is to be exceptional within the discipline, all the ways that we've been graded as individuals prior to that, they were not in the faculty meetings for the last 10 years, that are talking about where the field needs to go, and what's important. They're not privy to how we read texts as experts within the field, and therefore they're not privy to how we're reading their texts. In fact, research on assessment firms what I'm saying here, and it's very clearly doesn't. And we can look at Chun in 2017, or Duric, in 2016, and a number of other studies, that students lack what's called an assessment literacy, even if given some terminology, they don't know how to apply that terminology as we would apply that terminology. And if they don't know how to apply that terminology, as we would apply that terminology or the criteria as we would apply the criteria or the standards as we would apply the standards. How a fuck can they be expected to meet those standards? That's an insanity. And that goes back to what other theorists have already described. And I'll refer for example to Kenneth Brophy in his article social construction language and the authority of knowledge way back in 1986, where he writes, a social constructionist position in any discipline assumes that entities we normally call reality, knowledge, thought facts, texts, selves and so on are constructs generated by communities of like minded peers, social constructionism understands reality, knowledge, thought facts, text selves and so on as community generated and community maintained linguistic entities or more broadly speaking, symbolic entities that define or constitute the communities that gender read them. That was a lot of words. So let me break that down a little bit and try to explain what I mean and what Brophy means historians will understand what effect means in history, what a text means in history, how to interpret a text in history, how to write about a text in history, how to perceive reality through history as historians. And as historians, they will do so probably a little bit differently than philosophers who will certainly do so a little bit differently than physicists who will do so a little bit differently than people in business. That's not wrong. That's inescapable. We know that among educators, there are things we can talk about with educators, among educators of any kind, that are conversations, we cannot have the same way or at all, with people who aren't educators, that trying to inculcate non educators into some of our discussions as educators is exceptionally challenging. And haven't we seen that play out over COVID? Haven't we seen so many opinions emerge about what education is and needs to do, and how it should treat our kids over COVID by non educators, and haven't we looked at those discussions as educators and said, Oh, man, they don't know what they're talking about. Because they're not educators. Even sometimes, if they've had good ideas, those ideas have not been couched in ways and contextualize in ways that we as educators can really appreciate because they're not speaking to some of the other factors within those discussions that we as educators know about just because we're educators, that doesn't mean that we all agree as educators at all, it means that we have a culture means that we have a language, it means that we have some common practices and understandings of the world and perspectives on the world that non educators don't, that's not their fault. And it doesn't mean they can't have valuable things to say about education, it just means it's different. So we are a community, as historians, our community, as your history department, or your physics department is a community as your institution is a community. But students aren't part of that culture. And so when we say to students, we want you to meet the standard, we want you to do this thing as historian, we're screwing them, because they're not part of the culture of faculty members who are historians. So we're saying, We want you to meet the standard, you can't understand what it is because it's cultural behind this veil behind the curtain, where we have conversations, and we talk about things in certain ways. And we understand the world through certain lenses that unless you are historian you really can't fully understand or appreciate. But nevertheless, we want you to write a paper as historian, go do that for us, please, but they can't. And this goes back to what David bartholomae wrote in inventing the university, a very famous article, especially read in social constructionism. And in composition theory. And he writes, to speak with authority, and he could read authority as competency. To speak with authority, students have to speak not only in another's voice, but through another's code. And they not only have to do this, they have to speak in the voice and through the codes of those of us with power and wisdom. And they not only have to do this, they have to do it before they know what they're doing before they have a project to participate in. And before at least in terms of our disciplines, they have anything to say. So again, we're saying we want you to take this history class, and we want you to write as a story in a way I want you to write successfully as an intellectual in my writing class or, again, translate this into any discipline, but really to know what it is to do that is to be part of the conversations that are behind the wall that we in the discipline have, and you're not part of the discipline, you haven't had those conversations. So it's really impossible for you to do that. But I'm going to ask you to do it anyway. And then I'm going to judge whether or not you did that based on the standards to which you have no access based on the culture to which you have no real access. That's not your fault, you can't possibly access it. But nevertheless, that's how I'm going to assess you. bartholomae goes on to add into clarify that as follows. The student has to appropriate or be appropriated by a specialized discourse. And he has to do this as though he were easily and comfortably one with his audience as though he were a member of the Academy or when historian or an anthropologist or an economist, he has to invent the university by assembling and mimicking its language while finding some compromise between idiosyncrasy, a personal history on one hand, and the requirements of convention, the history of a discipline. On the other hand, now some of us might be saying, well, look, I tell my students what it is that I want them to do. I give them clear criteria for how they are to approach this assignment. And I'm not really grading them as a member of a discipline to which they have no access. I'm grading them by those standards. Well, first and foremost, we're always grading them as a member of a discipline through which they have no access, because we are members of those disciplines. And we cannot cease being members of those disciplines. For the moments we choose to grade our students. We cannot cease being intellectuals, we cannot cease having a history of how we've been assessed for the moments we choose to assess our students, we cannot put those things aside, we can try. And we we try to be as objective as we can and fair as we can. Of course we do. It's just impossible to do that, in totality, much less than any meaningful measure. But here's what's worse. And here's how I'm going to close this aspect of the argument. There's just ample research that even if we offer students a rubric, which by the way, they typically appreciate, and which typically, by the way, has some perceived value in terms of trying to help students understand what it is we're looking for. We know that for the most part that students don't understand the criteria we put forward. They might say they understand it, but they don't understand it. In fact, one study by O'Donovan at all in 2004, made a significant attempt to make criteria plain to students. But here's what they wrote. Initially, we thought making assessment criteria and standards transparent and understandable to staff and students alike could be achieved fairly simply through the development and application of an explicit school wide assessment criteria and grade descriptors, a criterion assessment grid or rubric was developed that plotted commonly used assessment criteria in matrix format against grades, resulting in grade descriptors that detailed acceptable performance for each criterion at each grade. Despite our best efforts on their own the explicit assessment criteria and grade descriptors failed to transfer meaningful knowledge on assessment standards and criteria to students difficulties encountered first, in the clear and precise articulation of marking criteria and standards. And secondly, in the accurate receipt of this understanding by relevant participants, undermined the effectiveness of the project, merely putting words out doesn't Empower an understanding of what's being valued, because we will all look at those words differently unless we come together and actually practice their application, which can change all of that. But the mere attempt to describe these things without practice without engagement is ineffective. And this becomes critical to the idea of peer assessment, we can put a whole bunch of language down, but the students don't know what we mean by those terms. So then you might say, Well, wait a second. Now, I go over the rubric with my students, or whether there is a rubric or not, I go over and explain what I'm looking for. We spend the class doing it, I hand out examples of successful essays to the students. And yet the research shows us that most of those efforts, though noble, and perhaps not entirely useless, are nevertheless not particularly fruitful. And I think many of you have had this experience where you said, Well, I've gone over what I wanted with my students, I've told them what to do. In the paper, I've explained what I wanted, we spent a whole class talking about it, I gave them examples of what I wanted them to do. And then they still didn't do it, or they didn't do it particularly well, or for some of them, it was like they weren't even there in the discussion at all. We've all been there, we've all encountered those students. Sometimes we've encountered classes of students who do that on maths. And that's because when you say that you want your students to analyze a text, no matter what you say, in that particular moment, analyze means things to you, that are part of your culture, as an academic behind the wall, that they can access and analyze will mean certain things to them based on their culture, as students based on who had asked them to analyze before where they had been successful in analyzing before, where they had been successful in analyzing before is how they are going to approach analyzing in your paper, even if you try to explain to them that analyzing is something different. And that is because until they start to do analysis, in the sense that you want them to do analysis, they literally do not have neural pathways to accomplish that intellectual act. We have to build those neural pathways, which means we have to bring them into the practice. And I will talk about how peer assessment competitions that so meaningfully and how that statement I am making to you is supported by a tremendous amount of research. So to synopsize that point, we have this problem whereby despite however nobly, we want To work with students and educate them, no matter how good our intentions and in certain respects how thorough our efforts, we have this veil between ourselves and the students, we are members of a discourse community, we are members of a culture that they are not, they don't have a real understanding of our language and practices and culture. And they can't, it's going to take a while for them to acquire that. But we're asking them to do things as if they were members of that culture. And even if we tried to give them standards, and language, and rubrics, and so forth. And even though those things can be of assistance, generally, the research is saying that they still don't really know how to interpret those things as we interpret those things. And I think you can see an immediate segue into why I would advocate for peer assessment because of assessment is the locus of power in the institution. But assessment is governed by tacit knowledge and tacit practices that we have, then the only ethical thing I think we can do is not to take grades off the table or let students grade themselves, but rather to invite them to join us in the process. And I don't mean that they should do that as equal participants with equal power, not at first. But I think you can start to see the argument emerging, that if we want to do the most ethical, if not most rewarding thing for the students in terms of their education and their agency and their empowerment and engagement in the material, the best thing to do is to start to show them how we do that most powerful thing that we do, which is to assess Sure, we can show them how we do lots of the less powerful things that we do, how we talk, how we read how we do some different things. But none of those are the most powerful thing that we do. The most powerful thing that we do is assess grades still have the most power. So if we're not going to bring them into the process and let them see how we grade things, and acculturate them to looking at work in the field through how we, as people in the field, look at work in the field, then they're never going to really get there, or at least that process is greatly slowed by their lack of participation in it. We can jumpstart it, we can accelerate it, we can empower it, we can make it more engaging and interesting instead of scary and disempowering by simply showing them what we do when we grade and invite them to start doing it with us. So in the next episode of pedagogy, I'm going to continue this discussion. I'm going to talk about why students obsess over grades the way they do. I'm going to talk about the difference between grades as symbol versus grades as sign. I'm going to talk about how grades are few skate learning and the subject matter we want students to understand. Of course, I'm also going to talk about Bruce Lee. But then I'm also going to talk about termina stick screens. I'm going to talk about the contrast between peer assessment and peer review, which is critical, and I'm going to talk about how we start to invoke the process and practices of good peer assessment.
