Assessing Writing

I am  collaborating with a Japanese colleague to revise a rating scale for the undergraduate English for Academic Purposes program at her university. She had noticed that even though the program had developed an analytic rubric, some high scoring essays didn’t actually seem to be very good, and conversely, some low scoring essays actually seemed much better than their scores reflected.

Talking about this dilemma, we realized that the missing piece in the analytic rubric was something like “sophistication” or “insight.” Reading students’ essays, we could sense the writing was sophisticated or the writer had insight into the topic, but these qualities were not reflected in their numerical scores on the analytic rubric. In order to add an element to the rubric for sophistication, we had to figure out how to explain to the dozens of lecturers using that rubric what we meant by sophistication.

On the advice of another colleague, we read a recent article by assessment experts Knoch,  Deygers, and Khamboonruang (2021) and followed their guidelines to operationalize the construct of sophistication. Two key processes Knoch and colleagues identified for developing rubrics were reviews of performance samples and documentation of rater cognition. My Japanese colleague and I took printed copies of her students’ writing (these are performance samples) and independently sorted them into piles based on our perceptions of their relative sophistication and insight. We then discussed each text, explaining to each other why we had placed it in a particular pile and what it showed about the writer’s ability to demonstrate sophistication and insight (this is rater cognition).

Having sorted and discussed a subset of the texts, we are now creating a four-level rubric reflecting our description of what insight is. The next step will be to try out our rubric with other writing teachers to see if they understand sophistication and insight in the same way we do and if they can use the rubric to evaluate student writing in the same way we did. We may need to revise our rubric if they do not.

You may be thinking that this sounds like a rather onerous process. Do you really need to go to this much effort just to evaluate your students’ writing in your classroom? The answer is no, but in this blog post I provide a few pointers for making sure that your classroom writing assessment practices give you and your students useful information about their writing abilities.

Developing a Classroom Writing Assessment Process

1. What Are You Really Assessing?

The first step in any assessment process is to clearly describe what you are assessing so you don’t unintentionally assess other things at the same time. Consider a test that requires students to first read a passage and then write a summary of the text. While this is a writing assessment—the student must write a summary in complete sentences, following appropriate grammar rules and norms for the genre—it is just as much a test of reading ability. Students who did not understand the reading passage will not be able to write a summary of it no matter how well they can write summaries of texts they understand. If, however, your teaching has also covered reading comprehension, and students have learned how to read the type of text used on the assessment, then you can use the task as long as you keep in mind that you are testing both reading and writing.

2. What Aspects of Writing Are You Assessing?

I often ask teachers to answer the question “What is writing?” Their answers usually include the following:

• Spelling
• Correct grammar
• Varied sentence structure
• Creativity
• Original ideas
• Organization
• Support for a thesis
• Genre

This list covers a broad range of points, from the formation of letters to drafting multipage essays. None of these answers is wrong, and a writing assessment task will challenge students to do many of these actions. The imperative for us as writing teachers, however, is to decide what aspects of writing we want to assess and make sure that we keep the focus on those aspects in our evaluation.

3. How Will You Assess Your Students’ Writing?

In theory, if we believe that writing can easily be broken into discrete pieces (grammatical structures, word choice, or style rules, for example), we can assess it with a multiple choice test where each piece is an item. You may have taken standardized tests that include sections like this. For each item, there is one correct answer. The idea is that a student who can correctly answer all of these questions knows the rules for writing and can therefore write well.

Most of us now feel that a more appropriate way to assess writing is by asking our students to write. Depending on their language proficiency, we may ask them to write separate sentences or a paragraph or essay. They may have to write in response to a prompt or use certain language structures in their texts. As long as they are putting pen to paper (or fingers to keyboard), they are writing. In developing a performance assessment for writing, we need to choose the appropriate text type for our students in the given moment.

4. What Will Students Write?

In their classic article about writing assessment, Reid and Kroll (1995) provide guidelines for designing writing tasks that allow students to effectively demonstrate their learning. Teachers should consider the following:

• Context: how the task fits with the course and with real-world writing that students need to do. They need to know why they are writing.
• Content: students should know what they are writing about and have some choice in topics. Unless the task is also a test of their content knowledge, the focus should be on whether they can write, not what they know.
• Language: The assignment should be easy for students to understand. It should not be a test of whether they understand the prompt, but rather how well they can write in response to it.
• Tasks: what should be done and what process should be followed. If the task is written in stages following a process, clearly state what those are and when each part is due.
• Rhetorical Specifications: what genre and what audience are expected.
• Evaluation: how you will assess the students’ accomplishment of the task. Let students know if there is a rubric and what aspects of writing will be graded.

5. How Will You Evaluate Your Students’ Writing?

With respect to the final item in the list above, try to develop the rubric at the same time as you are developing the assignment task. Do not wait until after students have turned in their final essays! Share the rubric with your students so they know what matters in their writing; if part of the writing process includes peer response, students can give each other feedback with reference to the rubric as well. When developing your rubric, focus on the learning objectives for the course and module.

6. What Will You and Your Students Do With the Results?

Writing assessment should provide you (the teacher) and your students with useful information that can lead to further learning. As the teacher, you may see that your students are doing better than you had expected, so you can skip ahead to other goals. On the other hand, you may realize that they have not progressed as far as you had hoped, so you need to reinforce and revisit certain topics. Students benefit from getting individualized feedback on their writing that shows them what they can do and what they are still developing. Teach students to understand the feedback and rubric rather than assuming that they will know what it means and how to apply it in their future writing.

Conclusion

Writing assessment is important for us and for our students, but it is often an overlooked practice in second language teaching. While it does take a bit more time to think through the above steps and to develop a strong assessment task and rubric, doing so will allow for much more meaningful and useful evaluation of your students’ progress in writing.