The results below summarizes information on the Writing Assessment as reported in the Student Learning Section of the WASC Educational Effectiveness Report (pages 9-11).
The WASC EER Report (PDF) contains the full reports, including all charts and appendix information.
To measure value added, the ULO Project on Writing assessed skill attainment at three key educational levels:
- First-year, 100-level GE composition courses
- 200- and 300-level GE writing-intensive courses
- Discipline-specific senior courses that emphasize writing.
The chair of the ULO Writing Committee was the English Department’s Director of Writing, whose specialty is composition assessment and pedagogy. To obtain a consistent framework, the committee developed the four-point University Expository Writing Rubric based on five traits of effective writing: purpose, synthesis, support, style, and mechanics. The committee examined persuasive essays of four to six pages in length because curricula across all levels and majors emphasize this type of writing.
- The committee collected work from 56 class sections that either had a GE designation of “writing intensive” or were taught by faculty members who made writing a priority.
- In total, the committee collected 1,147 essays. From this pool, the committee randomly selected 272 essays for scoring: 88 from freshmen, 41 from sophomores, 54 from juniors, and 89 from seniors. 153 of the essays were from men (56%), and 119 were from women (44%), which approximates the university’s gender mix.
- There were three norming and scoring sessions. Once inter-rater reliability was established, two readers scored each essay, from which all identifying information about student or class level had been removed. Because of time constraints, the two scores were averaged rather than using a third reader to resolve discrepancies. The average scores were used in the following analyses.
Class Level Comparisons
A statistical analysis compared the variables of Class Level (freshman, sophomore, junior, senior), College, Gender, and Trait. Only Class Level and Trait were significant (see Appendix 1.1 for full statistical analysis). A follow-up analysis showed that freshmen scored significantly lower than sophomores, juniors, and seniors; no additional progress in the mean total was evident after students’ sophomore year. In other words, seniors differed from freshmen in skill attainment but did not differ from sophomores and juniors. No other significant differences were found for Class Level. The data also show that about 20- 25% of sophomores, juniors, and especially seniors did not earn a score of 2 (average attainment) in their writing overall.
Follow-up comparisons showed that students were significantly stronger on both Purpose and Mechanics, which did not differ from each other, than on Synthesis, Support, and Style, which also did not differ from each other. The trait results suggest that these three higher-level writing skills need further development regardless of class level. For each trait, the figure shows the percentages of students earning a score of 2 or better on the rubric, as well as the mean score for each trait, all as a function of Class Level.
- For Purpose, freshmen scored significantly lower than both sophomores and seniors. No other Class Level comparisons were significant.
- For Synthesis, freshmen scored lower than both juniors and seniors. For Style, only the difference between seniors and freshmen was significant, with freshmen scoring lower.
- For both Support and Mechanics, follow-up comparisons showed that freshmen scored significantly lower than sophomores, juniors, and seniors, with no significant differences among these latter groups.
It should be noted that most students reached average attainment on at least one trait.
- Mechanics was especially strong, with 73% of freshmen reaching average attainment or above; this increased to 83% of seniors, 89% of juniors, and 93% of sophomores. In sum, analyses of the mean scores for each trait yielded the following observations:
- Seniors had higher scores across all rubric traits than freshmen.
- Juniors scored higher than freshmen on Synthesis, Mechanics, and Support.
- Sophomores scored higher than freshmen on Purpose, Mechanics, and Support.
- Sophomores, juniors, and seniors exhibited statistically equivalent levels of attainment across all traits.
Other Writing Assignments- English 134
In AY 2008-2009, the Associate Dean in the College of Liberal Arts and the ULO Writing Consultant conducted an assessment that compared students’ initial and final essays in the first-year composition course,
English 134 Writing and Rhetoric.
- The original sample was 156 students from 7 classes. First and last essays from 56 students—8 from each section—were randomly selected for assessment. Essays were scored using an earlier, holistic draft of the expository writing rubric. Final essay scores were significantly higher than those on the initial essays.
- As a follow-up, scores for both initial and final essays were compared to a constant of 3, indicating average attainment on the holistic rubric. Initial essay scores were significantly lower than 3; in contrast, final essay scores did not differ significantly from the constant.
- A separate test showed that initial and final essay scores were both correlated with final grades. Initial essay scores were weakly correlated with final grades, whereas final essay scores were significantly correlated with final grades.
- The overall pattern of results with regard to the initial and final essay scores yielded promising evidence that students significantly improved in their writing during the quarter, that this improvement moved students to an average and acceptable level of attainment, and that the final essay scores were indicative of final grades.
- Importantly, the data showed that students progressed from minimal to average attainment of writing skills during the quarter. This finding is consistent with the ULO-based assessment results reported above that show gains following the freshman writing experience and suggest that students retain these initial gains.
Graduation Writing Requirement
All CSU students must satisfy the Graduation Writing Requirement (GWR). Cal Poly students can meet this requirement in two ways:
- Earn a C or better and successfully complete a timed essay in a GWR-designated, 300-level, writing-intensive GE course. Students who are unsuccessful receive feedback and at least one more opportunity to complete the essay. The pass rate was 84% for AY 2010-11.
- Pass the Writing Proficiency Exam (WPE), a 350-500 word, timed, expository essay test scored by writing experts and other faculty members. The WPE pass rate was 70% for AY 2010-11. The essay and exam results likely constitute non-comparable samples for several reasons: students select the method of administration; the tests are administered in different environments; the content differs from test to test; the scoring differs across test types; and students taking the GWR course receive feedback and have a second opportunity to write the essay. In addition, each test may attract a different population, a factor that may interact with variables such as college, ethnicity, interest in writing, etc.
To date, this question has not been looked at in a systematic way because the data have not been readily available. Finally, the essays administered in a GWR course may not be suitable for drawing university-level conclusions because they are only assessed by the instructors of record. However, multiple readers score the WPE using the WPE scoring criteria, which differ from those of the expository writing rubric.
WPE readers assign a single score ranging from 1, ineffectual paper, to 6, exemplary paper, based on four traits: comprehension, organization, development, and expression. Stronger connections could be made between the WPE and expository writing rubrics. The expository writing rubric could be revised to function holistically, allowing readers to assign one score to an essay. Conversely, the WPE rubric could be revised to function analytically and thus provide more formative results. The latter approach seems appropriate as the WPE rubric was developed some time ago outside the framework of university-wide assessment.
In various surveys, Career Services has asked employers to indicate both the importance they place on certain skills, including written communication, and the degree to which Cal Poly graduates demonstrate attainment of these skills.
The data in Figure 1.4 (PDF) show a discrepancy between the importance employers place on written communication and their perception of the skill level graduates demonstrate. For example, employers of graduates from the College of Engineering gave written communication a mean importance score of 4.41 on a scale of 1 to 5 with 1 being lowest and 5 being highest. Yet in assessing the industry readiness of engineering students, employers gave students a mean score of only 3.86. This discrepancy is especially important because employers consistently rank communication among the skills they value most in employees. Considering the ULO data showing that senior-level Cal Poly students generally do not outperform sophomores and juniors in writing, it would seem that additional instruction or an increased emphasis on this skill may be warranted.
Recommended Action Items
1. Ensure that Cal Poly juniors and seniors continue to improve their writing skills
- Coordinate efforts with the University Writing and Rhetoric Center to develop and raise awareness of outreach programs that target upper-division students
- Identify upper-division students who struggle with writing before their senior year, especially ESL students, and offer additional upper-division writing courses for these students.
- Coordinate efforts with the CTLT and the WINGED (Writing in Generally Every Discipline) program to offer workshops and develop learning communities for faculty members who teach upper-division,
writing-intensive courses in GE and the major.
- Emphasize the value of writing in every discipline by identifying non-GE, upper-division, writing-intensive courses in the majors and across colleges; if such courses are difficult to identify, work with departments to develop discipline-specific, advanced writing courses, possibly tied to the senior project.
- Actively support Cal Poly’s acquisition of an e-portfolio and assessment management system so that students can document and assess their own progress as writers.
2. Align learning experiences so that GE, the GWR, and the senior project form a coordinated assessment of writing skills at the beginning, developing, and mastery levels (5).
- Develop a single expository writing rubric for use by GE or GWR-designated courses, the WPE, and the senior project.
- Require Cal Poly undergraduates to satisfy the GWR as juniors, i.e., as soon as possible after completing ninety units, so that they can receive additional writing instruction if necessary before attempting the senior project.
- Make the WPE a formative assessment. The exam should be re purposed so that it becomes a formative tool for improvement rather than a summative gatekeeper to graduation.
Background/Timeline/Objectives and History of Assessment Activities Writing Project from 2008-11
Message from Writing Learning Objective Chair - Brenda Helmbrecht
As the ULO Writing Committee (WLO) Chair and as the Director of Writing in the English Department, I seek to ensure that students at Cal Poly receive consistent and progressive writing education. To determine the progress you make as writers from the time you enter Cal Poly to the time you graduate, I am working with faculty across the university to collect and assess writing from a wide variety of classes across all grade levels. I firmly believe that Cal Poly students will develop as writers if the skills they develop in their GE Area A: Communication (A1 and A3) courses are built upon in both their upper-level GE courses and their major courses. Writing is a skill that needs to be reinforced and practiced. Constantly working to develop and improve your writing will help ensure that you will pass the WPE in your junior year and that you enter your career with the writing skills that employers tell us they expect Cal Poly graduates to possess.
: To determine the progress Cal Poly students make as writers by assessing writing from three key educational levels: first-year writing classes (beginning), 200- and 300-level GE writing-intensive courses (developing), and discipline-specific senior courses (mastery). This data will be used to identify the areas in which students and faculty need additional support in writing instruction.
Year One (2008 - 2009)
- Set up the Writing Assessment Committee
- Developed a University Writing Rubric. University Writing Rubric (PDF)
- Collect and assess student work from GE Area A1 (Expository Writing).
Year Two (2009- 2010)
- Collect and assess student work from writing-intensive GE Areas C: Arts and Humanities: (lower-division C1 and C2) and upper-division C4, and GE Area D: Society and the Individual (upper-division D5) , as well as from senior-level discipline-specific courses.
The goal of this tiered assessment plan is to obtain a wider perspective of students' writing abilities than can be gained by looking at writing development at a single level (such as with the Writing Proficiency Exam, WPE, which is intended to focus on junior-level writing). In effect, this assessment method will track the nature of writing development across three stages in your educational careers.
- A survey was created that asked students to reflect on their writing experiences at Cal Poly.
More specifically to understand: the kinds of instruction they received; the writing process they used; the kind of support they received on campus; and their impressions of themselves as writers.
Year Three (20010 - 2011)
- Use the data to develop a plan for the following academic year by identifying ways in which Cal Poly can further support both student writers and faculty who teach writing.