Can standardized exams adequately assess writing skills?
Supporters and opponents of standardized testing in schools agree on at least one thing — that the requirement has changed public education. Disinterested students skip through questions with little wonder about the relevance or consequences of their answers; worried school administrators pore over the results with a lingering fear that the state Department of Education will penalize them with punitive sanctions if the scores don’t measure up.
In such an environment, companies like Dover-based Measured Progress, one of the nation’s top standardized testing companies, are thriving. While computers correct the vast majority of the multiple choice sections of the tests, the job of determining proficiency in writing, an area in which New Hampshire students have been performing poorly, falls primarily on temporary, part-time low-wage employees.
Measured Progress uses Kelly Services, a national employment agency, to hire scorers to read and grade the volume of constructed written responses from schools outside of New Hampshire.
Measured Progress scores New Hampshire’s assessment tests, but the scorers work out of the firm’s other offices in Colorado and Kentucky.
Published ads invite applications to apply for jobs that pay $10.75 per hour, looking for candidates with a minimum of an associate degree or two years of college.
While Measured Progress wouldn’t participate in this story — referring questions about its practices to the New Hampshire Department of Education — former employees of the company praised it for its razor-sharp efficiency, but many of them also criticize its adherence to lockstep conformity and an assembly-line mentality that punishes creativity, innovation and style that are the hallmarks of good writing.
“It was all about how many tests you can crank out,” said Michael Deragon, currently a graduate student in Los Angeles and former professor at Chester College in Chester, N.H., who worked for Measured Progress from 1998 to 1999.
To succeed, Deragon said, scorers learned quickly to kowtow to the strict conventional response and “balance out” scores by not grading too high or too low.
The scorers typically read 100 scanned constructed responses in an hour on a computer monitor, then fill out a content-related rubric and provide a grade ranging from a low of zero to high of four.
Timothy Horrigan, who was a scorer from 2002 to 2003 and is now a state representative from Durham, described the scoring process: “A four is a perfect answer. A three is a pretty good one with one or two errors. Most students get twos, which is a mediocre answer with multiple errors. If there is anything even remotely relevant to the question, the kid gets a one. A zero is given when there are marks in the space which totally fail to answer the question, and a blank is given when the space is blank.”
Ironically, he said, most of the effort was devoted to ensure that tests with zeroes that had any relevant writing on them could be advanced to a score of one.
Scorers interviewed insisted that, from beginning to end, the process, seemed designed to produce grades that were just below the satisfactory level. Over the last two school years, approximately half of students tested fell within the second level, which is “Partially Proficient.”
Specifically, such a score means students’ writing lacked focus and organization, details were not elaborated, and sentence structure and language were unsophisticated, repetitive and inconsistent.
Over the past two years, 33 percent of students received threes, or “Proficient” scores, 14 percent received ones, “Substantially below Proficient.” Three percent were awarded fours, “Proficient with Distinction.”
According to Deb Wiswell, administrator for accountability in the state Department of Education, the process that governs New Hampshire tests is different from the one employed by Measured Progress.
“The majority of the points earned in New Hampshire writing assessment are from extended writing prompts,” which are scored using a six-point rubric and by two readers, she said. She also noted that state officials each year observe the scoring process and choose anchor papers to use as models for scorers.
Former Measured Progress scorers indicated that there was little training and feedback and that they felt very disconnected from the inner working of the company.
Part of the problem, Horrigan said, was the layers of supervisors above him that also were temporary Kelly employees. All of them were called in to complete a job, and when the job was finished they were laid off.
“Our immediate supervisors tended to get very frustrated,” he said, “They were expected to manage without any means of rewarding good scorers, of retraining mediocre ones, or even of punishing bad ones.”
Scoring in the middle
“They were pretty rigid on how they wanted us to grade,” said Justin Carloni, a guidance counselor at Prospect Mountain High School in Alton, who worked as a scorer while a student at the University of New Hampshire in the late 1990s.
Students were given a content-related question and were required to construct a response that included several key points that needed to be addressed.
“The responses had to fit the mold,” Carloni said.
He recalled some disagreements with Measured Progress supervisors over some responses by students who Carloni considered “clearly intelligent” but who didn’t play the game, by challenging, mocking or failing to directly answer the question.
Students, he said, “were not allowed to be creative.”
Avoiding such conflicts was the measure of a good job, for these disputes needed to be resolved, a process that bogged down the scoring flow. This most often occurred with a random selected test, which was scored by two separate scorers. If there was a difference of more than one point, then the score went to arbitration.
“The pressure was to avoid arbitration because supervisors had to sort it out,” said Horrigan. “That was the main rubric for (a scorer’s) success.”
Wally Keniston, a paraprofessional at Oyster River High School in Durham who worked at Measured Progress in 2002, had a different experience. He found supervisors to be reasonable in resolving conflicts over grading disputes.
The Dover office handles many different tests, and not all tests and the process for correcting them are the same. Each state or region has a different set of standards and accompanying rubrics, and varying consequences for poor performance.
Testing companies vie to provide testing services to states and school districts through a competitive bidding process. Industry observers say some test companies are willing to lowball a contract in order to win the right to sell additional products, such as study guides, to local school districts.
“The scoring gets less attention and fewer resources than the other step,” said Horrigan.
The world of standardized education seems poorly equipped to teach or assess the quality of something as personal and varied as writing skills. Still, education leaders worry about the decline in those skills, especially in a technological environment, which has dramatically decreased the volume and importance of writing in daily life.
The goal is to embed writing across the vast and crowded curriculum, but this is easier said than done.
This may be among the reasons the New England Common Assessments Program writing assessment administered in New Hampshire schools is not used to determine a school’s “adequate yearly progress.” As Deb Wiswell told the Concord Monitor in February 2007: “It’s such a subjective method.”
But test-makers say they are not trying to distinguish among styles, rather they seek to establish a minimum level of competency, and that it is but one of many assessments.
Lynda Bettcher, who has taught for 37 years at Portsmouth High School, along with some of her colleagues in the English department, noticed last spring that their students’ NECAP writing grades were, as she said, “way out of whack.”
It seemed that some of their most talented writers performed poorly. They decided to take a closer look with their own targeted assessment and found “what we knew to be the case,” she said. “The problem I have with the current New Hampshire state assessment of writing is that the inadequate per pupil dollars spent on scoring it often result in subpar scorers and flawed results.”
Some still see more good than harm. Despite any imperfections in the process, Rep. Emma Rouss of Durham, chair of the House Education Committee, is glad to see students tested for writing skills because it ensures that schools will teach those skills.
Dr. Monty Neill, deputy director of education, teaching and administration at FairTest, an organization opposed to standardized testing, agrees that the key to improving writing is repetition, but said testing doesn’t help.
“The problem is testing, not the low pay (or quality) of scorers — that is a symbol of the problem.”
Teachers in the classroom, like Portsmouth High School’s Bettcher, argue that assessment must be locally driven and checked by a state process that is competent and impartial. “Testing writing is critical and every school should conduct its own assessment,” she said. “This important task should not be left to a state unable to fund it properly.”