BEYOND IESIS: ALTERNATIVES IN ASSESSMENT.
To speak of alter'native assessments is counterproductive because the term implies something new and different that may be "exempt from the requirements ofresponsible test construction" (p. 657). So they proposed to refer to "alternatives" in assessment instead. Their term is a perfect fit within a model that considers tests as a subset of assessJllent. Throughout this book, you have been reminded· that all tests are assessments but, more important, that not all assessments are tests.
The defming characteristics of the various alternatives in assessment that have been commonly used across the profession were aptly summed up by Brown and Hudson (1998, pp. 654-655). Alternatives in assessments
1. require students to perform, create, produce, or do something;
2. use real-world contexts or Simulations;
3. are nonintrusive in that they extend the day-to-day classroom activities;
4. allow students to be assessed on what they normap.y; do in class every day;
5. use tasks that represent meaningful instructional activities;
6. focus on processes as well as products;
7. tap into higher-level thinking and problem-solving skills;
8. provide information about both the strengths and weaknesses of students;
9. are multiculturally sensitive whenpropedy administered;
10. ensure that people, not machines, do the scoring, using human judgment;
11. encourage open disclosure of standards and rating criteria; and
12. call upon teachers to perform new instructional and assessment roles.
THE DILEMMA OF MAXIMIZING BOTH PRACTICALITY AND WASHBACK.
The principal purposeof this chapter is to examine some of the alternatives in assessment that are markedly different from formal tests. Tests, especially large-scale standardized tests, tend to be one-shot performances that are timed, multiple-choice, decontextualized, norm-referenced, and that foster extrinsic motivation. On the other hand, tasks like portfolios, journals, and self-assessment are
• open-ended in their time orientation and format,
• contextualized to a curriculum,
• referenced to the criteria (objectives) of that curriculum, and
• likely to build intrinsic motivation.
One way of looking at this contrast poses a challenge to you as a teacher and test designer. Formal standardized tests are almost by definition highly practical, reliable instruments. They are designed to minimize time and money on the part of test designer and test-taker, and to be painstakingly accurate in their scoring. AJternatives such as portfOlios, or conferencing with students on drafts of written work, or observations of learners over time all require considerable time and effort on the part of the teacher and the student. Even more time must be spent if the teacher hopes to offer a reliable evaluation within students across time, as well as across students (taking care not to favor one student or group of students). But the alternative techniques also offer markedly greater washback, are superior formative measures, and, because of their authenticity, usually carry greater face validity.
PERFORMANCE-BASED ASSESSMENT.
Performance-based assessment implies productive, observable skills, such as speaking and writing, of content-valid tasks. Such performance usually, but not always, brings with it an air of authenticity-real-world tasks that students have had time to develop. It often implies an integration of language skills, perhaps all four skills in the case of project work. Because the tasks that students perform are consistent with course goals and curriculum, students and teachers are likely to be more motivated to perform them, as opposed to a set of multiple-choice questions about facts and figures regarding the solar system.
Performance-based assessment needs to be approached with caution. It is tempting for teachers to assume that if a student is doing something, then the process hasfulfilled its own goal and the evaluator-needs only to make a mark inthe grade book that says "accomplished» next to a particular competency. In reality, performances as assessment procedures need to be treated with the same rigor as traditional tests. This implies that teachers should:
• state the overall goal of the performance,
• specify the objectives (criteria) of the performance in detail,
• prepare students for performance in stepwise progressions,
• use a reliable evaluation form, checklist; or rating sheet,
• treat performances as opportunities for giving feedback and provide that feedback systematically, and
• if possible, utilize self- and peer-assessments judiciously.
PORTFOLIOS.
One of the most popular alternatives in assessment, especially within a framework of communicative language teaching, is portfolio development. According to Genesee and Upshur (1996), a portfolio is "a purposeful collection ofstudents' work that demonstrates their efforts, progress, and achievements in given areas" (p. 99). Portfolios iBclude materials such as:
• essays and compositions in draft and fmal forms;
• reports, project outlines;
• poetry and creative prose;
• artwork, photos, newspaper or magazine clippings;
• audio and/or video recordings of presentations, demonstrations, etc.;
• journals, diaries, and other personal reflections; .
• tests, test scores, and written hom~wor~ exercises;'
• notes on lectures; and
• self· and peer-assessments~comments, evaluations, and checklists.
Until recently, portfolios were thought to be applicable only to younger children who assemble a portfolio of artwork and written work for presentation to a teacher and/or a parent. Now learners of all ages and in all fields ofstudy are benefiting from the tangible, hands-on nature of portfolio development.
Successful portfolio development will depend on following a number of steps and guidelines.
1. State objectives clearly. Pick one or more of the CRADLE attributes named above and specify them as objectives of developing a portfolio. Show how tho:5e purposes are connected to, integrated with, and/or a reinforcement of your already stated curricular goals.
2. Give guidelines on what materials to include. Once the objectives have been determined, name the types of work that should be included.There is some disagreement among "experts" about how much negotiation should take place between student and,teacher over those materials.
3. Communicate assessment criteria to students. This is both the most important aspect of portfolio development and the most complex.
4. Designate time within the curriculum for portfolio development. If students feel rushed to gather materials and reflect on them, the effectiveness of the portfolio process is diminished.
5. Establish periodic schedules for review and conferencing. By doing so, you will prevent students from throwing everything together at the end of a term.
6. Designate an accessible place to keep portfolios. It is inconvenient for students to carry collections of papers and artwork. Ifyou have a self-contained classroom or a place in a reading room or library to keep the materials, that may provide a good option.
7. Provide positive washback-giving final assessments. When a portfoliO has been completed and the end of a term has arrived, a final summation is in order.
JOURNAlS.
Fifty years ago, journals had no place in the second language classroom. When language production was believed to be best taught under controlled conditions, the concept of "free" writing. was confmed alnlost exclusively to producing essays on assigned topics~ Today, journals occupy a prominent role in a pedagogical model that stresse.s the importance of self-reflection in the process of students taking control of their own destiny.
A journal is a log (or "account") of one's thoughts, feelings, reactions, assessments, ideas, or progress toward goals, usually written with little attention to structure, form, or correctness. Learners can articulate their thoughts without the threat of those thoughts being judged later (usually by the teacher). Sometimes journals are rambling sets of verbiage that represent a stream of consciousness with no particular point, purpose, or audience. Fortunately, models of journal use in educational practice have sought to tighten up this style of journal in· order to give them some focus (Staton et al., 1987). The result is the emergence of a number of overlapping categories or purposes in journal writing, such as the following:
• language-learning logs
• grammar journals
• responses to readings
• strategies-based learning logs
• self-assessment reflections
• diaries of attitudes, feelings, and other affective factors
• acculturation logs
Most classroom-oriented journals are what have now come to be known as dialogue journals.They imply an interaction between a reader (the teacher) and the student through dialogues or responses. For the best results, those responses should be dispersed across a course at regular intervals, perhaps weekly or biweekly.
It is important to turn the advantages and potential drawbacks of journals into positive general steps and guidelines for using journals as assessment instruments. The following steps are not coincidentally parallel to those cited above for portfolio development:
1. Sensitively introduce students to the concept ofjournal writing. For many students, especially those from educational systems that play down the, notion of teacher-student dialogue and collaboration, journal writing' will be difficult at frrst.
2. State the objective(s) ofthe journal. Integrate journal writing into the objectives of the curriculum in some way, especially ifjournal entries become topics of class discussion.
3. Give guidelines on what kinds oftopics to include. Once the purpose or type of journal is clear, students will benefit from models or suggestions on what kinds oftopics to incorporate into their journals.
4. Carefully specify the criteria for assessing or grading journals. Students need to understand the freewriting involved in journals, but at the same time, they need to know assessment criteria.
5. Provide optimalfeedback in your responses.
6. Designate appropriate time frames and scbedules for review. Journals, like portfolies, need to be esteemed by students as integral parts of a course.
7. Provide formative, wasbback-giving final comments. Journals, perhaps even more than portfolios, are the most formative of all the alternatives in assess· menteThey are day-by-day (or at least weekly) chronicles of progress whose pur· pose is to provide a thread ofcontinuous assessment and reassessment, to recognize mid-stream direction changes, and/or to refocus on goals.
CONFERENCES AND INTERVIEWS.
For a number ofyears, conferences have been a routine part of language classrooms, especially of courses in writing. In Chapter 9, reference was made to conferencing as a standard part of the process approach to teaching writing, in which the teacher, in a conversation about a draft, facilitates the improvement of the written work. Such interaction has the advantage ofone-on-one interaction between teacher and student, and the teacher's being able to direct feedback toward a student's specific needs.
Conferences are not limited to drafts of written work. Including portfolios and journals discussed above, the list of possible functions and subject matter for conferencing is substantial:
• commenting on drafts of essays and reports
• reviewing portfolios
• responding to journals
• advising on a student's plan for an oral presentation
• assessing a proposal for a project
• giving feedback on the results of performance on a test
• clarifying understanding of a reading
• exporing strategies-based options for enhancement or compensation
• focusing on aspects of oral production
• checking a student's self-assessment of a performance
• setting personal goals for the near future
• assessing general progress in a course
Conferences must assume that the teacher plays the role of a facilitator and guide, not of an administrator, of a formal assessment. In this intlinsically motivating atmosphere, students need to understand that the teacher is an ally who is encouraging self-reflection and improvement. So that the student will be as candid as possible in self-asseSSing, the teacher should not consider a conference as something to be scored or graded. Conferences are by nature formative, not summative, and their primary purpose is to ·offer positive washback.
Because interviews have multiple objectives, as noted above, it is difficult to generalize principles for conducting them, but the following ,guidelines may help to frame the questions efficiently:
1. Offer an initial atmosphere of warmth and anxiety-lowering (warm-up).
2. Begin with relatively simple questions.
3. Continue with level<heck and probe questions, but adapt to the interviewee as needed.
4. Frame questions simply and directly.
5. Focus on only one factor for each question. Do not combine several objectives in the same question.
6. Be prepared to repeat or reframe questions that are not understood. 7. Wind down with friendly and reassuring closing comments.
How do conferences and interviews score in terms of principles ofassessment? Their practicality, as is true. for many ofthe alternatives to assessment,is low because they are time onsuming. Reliability will vary between conferences and interviews. In the case of conferences, it may not be important to have rater reliability because the whole purpose is to offer individualized attention, which will vary greatly from student to student. For interviews, a relatively high level of reliability should be maintained with careful attention to objectives and procedures.
OBSERVATIONS.
All teachers, whether they are aware of it or not, observe their students in the classroom almost constantly_ Virtually every question. every response, and almost every nonverbal behavior is, at some level of perception, noticed. All those intuitive perceptions are stored as little bits and pieces of information about students that can form a composite impression of a student's ability. Without eyer administering a test or a quiz, teachers know a lot about their students. In fact, experienced teachers are so good at this almost subliminal process of assessment that their estimates of a student's competence are often highly correlated with actual independently administered test scores. (See Acton, 1979, for an example.)
The list could be even more specific to suit the characteristics of students, the focus ofa lesson or module, the objectives of a curriculum, and other factors.The list might expand, as well, to include other possible observed performance. In order to carry out classroom observation, it is ofcourse important to take the following steps:
1. Determine the specific objectives of the observation.
2. Decide how many students will be observed at one time.
3. Set up the logistics for making t:mn0ticed observations.
4. Design a system for recording observed performances.
5. Do not overestimate the number of different elements you can observe at one time-keep them very limited.
6. Plan how many observations you will make.
7. Determine specifically how you will use the results.
Designing a system for observing is no simple task. Recording your observations can take the form of anecdotal records, checklists, or rating scales. Anecdotal records should be as specific as possible in focusing on the objective of the observation, but they are so varied in form that to suggest formats here would be counter productive. Their very purpose is more note-taking than record keeping.The key is to devise a system that maintains the principle of reliability as closely as possible. Checklists are a viable alternative for recording observation results. Some checklists ofstudent classroom performance, such as the COLT observation scheme devised by Spada and Frohlich (1995), are elaborate grids referring to such variables as : • whole-class, group, and individual participation,
• content of the topic,
• linguistic competence (form, function, discourse, Sociolinguistic),
• materials being used, and
• skill (listening, speaking, reading, writing),
SELF- AND PEER ASSESSMENTS.
A conventional view of language assessment might consider the notion of selfand peer-assessment as an absurd reversal of politically correct power relationships. After all, how could learners who are still in the process of acquisition, especially the early processes, be capable of rendering an accurate assessment of their" own performance? Nevertheless, a closer look at the acquisition of any skill reveals the importance, if not the necessity, of self-assessment and the benefit of peer·assessment.
Self-assessment derives its theoretical justification from a number of wellestablished principles of second language acquisition. The prinCiple of autonomy starids_ qut as one of the primary foundation stones of successful learning. The ability to set one's own goals both within and beyond the structure ofa classroom curriculum, to pursue them without the presence of an external prod, and to independently monitor that pursuit are all keys to success. Developing intrinsic motivation that comes from a self-propelled desire to excel is at the top of the list of successful acquisition of any set of skills.
Peer-assessment appeals to similar principles, the most obvious ofwhich is cooperative learning. Many people go through a whole regimen of education from kindergarten up through a graduate degree and never come to appreciate the value of collaboration in learning-the benefit of a community oflearners capable of teaching each 'other something. Peer-assessment is simply one arm of a plethoIa of tasks and procedures within the domain of learner-centered and collaborative education.
Researchers (such as Brown & Hudson, 1998) agree that the above theoretical underpinnings of self- and peer-assessment offer certain benefits: direct involvement of students in their own destiny, the encouragement of autonomy, and increased motivation because of their self-involvement. Of course, some noteworthy drawbacks must also be taken into account. Subjectivity is a primary obstacle, to overcome. Students may be either too harsh on themselves or too self-flattering, or they may not have the necessary tools to make an accurate assessment. Also, especially in the case of direct assessments of performance (see below), they may not be able to discern their own errors.
Types of Self- and Peer-Assessment.
It is important to distinguish among several different types ofself- and peer-assessment and to apply them accordingly. I have borrowed from widely accepted classifications of strategic options to create five categories of self- and peer-assessment: (1) direct assessment of performance, (2) indirect assessment of performance, (3) metacognitive assessment, (4) assessment of socioaffective factors, and (5) student self-generated tests.
1. Assessment offa specific} performance. In this category, a student typically monitors him- or herself-in either oral or written production-and renders some kind of evaluation of performance. The evaluation takes place immediately or very soon after the performance. Thus, having made an oral presentation, the student (or a pe
er) fills out a checklist that rates performance on a defined scale.
2. Indirect assessment of[general) competence. Indirect self- or peer-assessment targets larger slices of time with a view to rendering an evaluatioIl'of general ability, as opposed to one specific, relatively time-cortstrained performance. The distinction between direct and indirect assessments is the classic competence-performance distinction. Self- and peer-assessments of performance are limited in time and focus to a relatively short performance.
3. Metacognitive assessment [for setting goals}. Some kinds of evaluation are more strategic in nature, with the purpose not just ofviewing past performance-or competence but of setting goals and maintaining an eye on the process oftheir pursuit. Personal goal-setting has the advantage offostering intrinsic motivation and of providing learners with that extra-special impetus from having set and accomplished one's own goals.
4. Sociol adjective assessment. Yet another type of self- and peer-assessment comes in the form of methods of examining affective factors in learning. Such assessment is quite different from looking at and planning linguistic aspects of acquisition. It requires looking at oneself through a psychological lens and may not differ greatly from self.
5. Student-generated tests. A final type of assessment that is not usually classified strictly as self- or peer-assessment is the technique of engaging students in the process of constructing tests themselves. The traditional view of what a test is would never allow students to engage in test construction, but student-generated tests can be productive, intrinsically motivating, autonomy-building processes.
Guidelines for Self- and Peer-Assessment .
Self- and peer-assessment are among the best possible formative types of assessment and possibly the most rewarding, but they must be ~arefully designed and administered for them to reach their potential. Four guidelines will help teachers bring this intrinsically motivating task into the classroom successfully :
1. Tell students the purpose ofthe assessment. Self-assessment is a process that many students-especially those in traditional educational systems-will initially fmd quite uncomfortable. They need to be sold on the' concept. It is therefore essential that you carefully analyze the needs that will be met in offering both selfand peer-assessment opportunities, and then convey this information to students.
2. Define the task(s) clearly. Make sure the students know exactly what they are supposed to do. Ifyou are offering a rating sheet or questionnaire, the task is not complex,but an open-ended journal entry could leave students perplexed about what to write. Guidelines and models will be of great help in clarifying the procedures.
3. Encourage impartial evaluation ofperformance or ability. One of the greatest drawbacks to self-assessment is the threat of subjectivity. By showing students the advantage of honest, objective opinions, you can maximize the beneficial washback of self-assessments. Peer-assessments, too, are vulnerable to unreliability as students apply varying standards to their peers. Clear assessment criteria can go a long way toward encouraging objectivity.
4. Ensure benefictal washback through follow-up tasks. It is not enough to simply toss aself-checklist at students and then walk away. Systematic follow-up can be accomplished through further self-analysis, journal reflection, written feedback from the teacher, conferencing with the teacher, purposeful goal-setting by the student, or any combination of the above.
A Taxonomy of Self and Peer Assessment Tasks.
To sum up the possibilities for self- and peer-assessment, it is helpful to consider a variety of tasks within each of the four skills.
Seff- and peer-assessment tasks:
1. Listening Tasks: listening to TV or radio broadcasts a.nd checking comprehension with a partner listening to bilingual versions of a broadcast and checking comprehension asking when you don't understand something in pair or group work listening to an academic lecture and checking yourself on a "quiz" of the content setting goals for creating/increasing opportunities for listening.
2. Speaking Tasks: filling out student self-checklists and questionnaires using peer checklists and questionnaires rating someone's oral presentation (holistically) detecting pronunciation or grammar errors On a self-recordi ng asking others for confirmation checks in conversational settings setting goals for creating/increasing opportuhities for speaking .
3. Reading Tasks: reading passages with self-check comprehension questions following readi ng and checki ng comprehension with a partner taking vocabylary quizzes taking grammar and vocabulary quizzes on the Internet conductirfg self-assessment of reading habits setting goals for creating/increasing opportunities for reading.
4. Writing Tasks: revising written work on your own revisi ng written work with a peer (peer editi ng) proofreadi ng using journal writing for reflection, assessment, and goal-setting setting goals for creating/increasing opportunities for writing
An evaluation of self- and peer-assessment according to our classic principles of assessment yields a pattern that is quite consistent with other alternatives to assessment that have been analyzed in this chapter. Practicality can achieve a moderate level with such procedures as checklists· and questionnaires, while reliability risks remaining at a low level, given the variation within and across learners. Once students accept the notion that they can legitimately assess themselves, then face validity can be raised from what might otherwise be a low level. Adherence to course objectives will maintain a high degree of content validity. Authenticity and washback both have very high potential because students are centering on their own linguistic needs and are receiving useful feedback.
References:
Brown, H. Douglas. 2004. Language Assessment: Particple and Classroom Practices. New York: Person Education
