Oral Communication Assessment
The results below summarize information on the Writing Assessment as reported in the Student Learning Section of the WASC EER Report (PDF) (pages 11-13).
The report also contains all supplemental data, charts and appendixes.
Background
The ULO Project on Oral Communication began in September 2009. The ULO Oral Communication Committee adopted an operational definition from AAC&U’s Oral Communication VALUE Rubric: “a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners’ attitudes, values, beliefs, or behaviors.” Based on this definition, the committee designed a five-point rubric with seven traits: verbal delivery, nonverbal delivery, presence of a central message, organization, language use, use of supporting material, and use of visual aids.
Method
In the first year, the committee sought to establish a benchmark of students’ performance toward the beginning of their academic careers. The assessment entailed videotaping oral presentations delivered by a sample of 102 freshmen enrolled in COMS 101 and 102 during Spring 2010. The sample was 51% female and 49% male and represented all six colleges: Engineering (24%), Agriculture (23%), Science and Math (20%), Liberal Arts (15%), Business (13%) and Architecture (7%). Frequencies for both gender and college distributions did not differ significantly from what would be expected. Three faculty members from Communication Studies observed and evaluated the speeches. Training sessions ensured norming of scores and provided evaluators the opportunity to discuss, modify, and clarify the rubric as needed. Following these sessions, each evaluator scored a selection of speeches on each rubric trait on a scale of 1 to 5 with 1 being insufficient and 5 being excellent.
Results
Figure 1.5 (PDF) page 58 shows the overall scores, with the rubric traits presented in order from highest to lowest means. In addition, the figure shows the percentages of students scoring at each level of the rubric. Because so few had scores of 1, percentages for scores of 1 and 2 (insufficient and below average) were added together (see Appendix 1.1 for full statistical analysis). Because Use of Visual Aids was not a component of all speeches, two different statistical analyses were run on the differences in mean trait scores. One considered all 7 traits for the 75 students who had scores on all 7, while the second considered all 102 students but excluded Use of Visual Aids. A follow-up comparison showed the same basic pattern in both analyses: students’ trait scores were significantly higher for Language Use and Use of Supporting Materials than for Verbal and Non-Verbal Delivery and for Presence of a Central Message than for Verbal Delivery. In the seven-trait analysis, scores were significantly higher for Presence of a Central Message than for Non-Verbal Delivery. There were no other significant differences.
These data suggest that the vast majority of Cal Poly freshmen meet an average (3) or better level of competence in oral communication, even with only introductory instruction. This is good news, but the data also suggest that students’ verbal and nonverbal delivery could be developed further; only a quarter of the sample achieved a score of good (4) or excellent (5). Improvement in these areas would likely occur over time as students received further instruction and additional speaking opportunities. However, given that Cal Poly requires most students to take only one course focusing on oral communication, instructors of that course should consider spending additional time on improvement of verbal and nonverbal delivery.
During the second year of the project, the committee presented these results to the University Assessment Council and the Communication Studies faculty. In addition, the committee delivered a ULO-based oral communication workshop through the CTL in which twelve participants applied the rubric after watching both a below average speech and a good speech. The first speech received an average score of 2.2 and the second received an average score of 4.4. This consistency indicates that the participants used the rubric to make reliable distinctions of quality between the two speeches. The committee originally planned a third year of activity to assess senior-level presentations perhaps in connection with senior projects, but budget cuts curtailed this aspect of the project.
Recommended Action Items
- Identify areas of the curriculum outside the GE oral communication requirement in which the Communications Studies faculty can partner with other faculties to develop students’ oral communication skills
- Complete the ULO Project on Oral Communication by collecting data on upper-division student performance and making a value-added comparison to lower-division results .
Time line/History of Assessment Activities 2009-2011
Year One and Year Two (2009-11)
- assembled an interdisciplinary Oral Communication Committee
- The Cal Poly Oral Communication ULO Committee utilized the operational definition offered by the AACU for the purpose of this assessment project.
Oral Communication is a prepared, pursposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners' attitudes, beliefs or values. - designed an Oral Communication Rubric (PDF)
- videotaped over 100 student speeches in GE introductory public speaking classes:
- used the rubric to assess students' performance
- tabulated results, and
- Completed: Summary - Oral Communication Report -August 2011 (PDF) - written by Dr. Lorraine Jackson, ULO Consultant for Communication
Brief Statement of Significant Findings:
One goal of the first year of this project was to apply a rubric to obtain a benchmark of student oral communication performance. On a 1-5 scale (1= low and 5 = high), freshmen students in this sample (n=102) achieved average scores ranging from 3.02-3.42 in all categories of the rubric.
- The categories receiving scores on the higher end of average included students' use of supporting materials, their language use, and their ability to develop a central message.
- Categories scoring on the lower end of average included their verbal delivery, their nonverbal delivery, and their organization.
Recommendations for Improvement Based on Assessment Results:
These data suggest that although freshman students are meeting a basic level of average competence, there is room for improvement, particularly in students' verbal and nonverbal delivery where a quarter of the sample was able to achieve a good or excellent score.
- Faculty across the university might be well advised to give meaningful feedback to students and to suggest how they can make positive changes in their organization, as well as their verbal and nonverbal delivery. For example, one area that emerged as needing improvement in approximately one third of the sample concerns giving eye contact (avoiding reading) during public speaking.
Actual Improvements In Progress:
- Assessment results were communicated to faculty and are particularly useful to faculty teaching public speaking.
- The CTL workshop was held in Spring 2011 and was open to interested faculty from various disciplines. It providedan an opportunity to apply the rubric and discuss how to give quality feedback to students
- In addition to the CTL workshop mentioned above, we hope hoping to increase our outreach by working with one or two programs that would like assistance with the assessment of oral communication.
References/background: Including Scoring Rubric:
- Association of American Colleges and Universities (AACU) Oral Communication VALUE Rubric - Retrieved September 7, 2009
- Dunbar, N.E., Brooks, C. F. & Kubicka-Miller, T. (2006). Oral communication skills: Using a performance based evaluation rubric to assess communication skills. Innovative Higher Education, 31 (2), 115- 128.
- Lucas, S. (2010). The art of public speaking. McGraw Hill Publishing.