Background

In its third year of instruction, US12: Computer Games as Art, Culture and Technology is an interdisciplinary course concerned with the study of computer games as artistic, cultural, and technologically phenomena.  In addition to engaging students in the study of computer games from the perspective of three different disciplines, writing played a central role in the course, serving as “the thread that enables students to connect the various perspectives”. (http:/www.ics.edu/~frost/US12A/syllabus.html).  Writing was assigned each quarter with the intention of improving students’ information literacy skills, written expression, and ability to compose a research project, and culminated with the production of a capstone paper in the Spring Quarter, designed to fulfill the second course of the lower-division writing requirement.  Students who participated in US12, and successfully completed the three quarter sequence, would fulfill 4 breadth category requirements (1 from Category I-the equivalent of Writing 39C, and 1 from Categories III, IV, and V). 

Assessment Plan

As the 2008-2009 academic year represented both the third year of instruction for US12 and the First-Year Integrated Program, and in light of the findings from the first and second year assessment and evaluation efforts, the 2008-2009 assessment plan focused on the writing components of the course and is presented in Table 1. 

The assessment plan was designed to gather information within the following three general areas of interest:

1. The students:

            -were US12 students different from other new freshmen at UCI?
-what quarterly grades did students receive in US12 by gender?
-what were the students’ majors when they started at UCI and what were their majors at the
end of their first year?

2. The course:

            -what learning gains did students report?
-did students have a sense of belonging to a learning community as a result of their FIP course?
-did students report understanding how different disciplines ask and answer questions as a result of their FIP experience?
-how did students experience the instruction and overall course quality of US12?

3. The students’ writing skills:

            -when did students complete the first course in the lower division writing requirement?
-what was the quality of their capstone papers?
-was the quality of writing demonstrated via the capstone papers similar to the quality of writing produced in Writing 39C and Humanities Core, the two primary methods, outside of FIP, by which students fulfill the second course of the lower-division writing requirement?

Table 1: Assessment Plan for 2008-2009

The Students

As noted in Table 2, 81 students initially enrolled in US12 in Fall 2008.  In Winter 2009, 3 students (4%) chose not enroll in US12B.  By the end of the 2008-2009 year, 96% of the original 2008-2009 US12 student cohort completed the three-quarter sequence.

Table 2: US12 Enrollments by Quarter

a One student, who completed US12A/B in 2007-2008, enrolled in US12C in Spring 2009 and was excluded from the enrollment totals.

How US12 Students Differ from Other First-Year Students

In reviewing the findings from comparisons made between US12 and non-continuing US12 students, and US12 students and all other new freshmen who passed the Subject A exam, significant differences were found both between students participating in US12 and all non-FIP freshmen, and students who completed all three quarters of US12 and those who left after Fall Quarter.

The results comparing all entering freshman to those students enrolled in US12 during Fall 2008 are displayed in Table 3.

• US12A students had significantly higher mean SAT Verbal and Math scores.
• Though not statistically significant, US12A students had lower High School GPA’s and higher Fall Quarter 2008 GPA’s than all other non-FIP freshmen.

Table 3: Comparison of Students Enrolled in US12A students and to all non-FIP
Freshmen on Entering Characteristics and GPA

Table 4 displays the findings when students who remained enrolled in US12 are compared to those who did not enroll in US12 beyond Fall 2008.

• Though not statistically significant, students who remained enrolled in US12 for the entire year had higher SAT Verbal and Math scores, and higher Fall Quarter, Spring Quarter, and cumulative GPAs than did their colleagues who did not enroll in US12 beyond the Fall Quarter.
• In contrast, students who did not enroll in US12 beyond the Fall Quarter had significantly higher Winter Quarter GPAs and, though not statistically significant, higher SAT Writing scores than those students who completed the three-quarter sequence of US12.

Table 4: Comparison of Continuing and Non-continuing US12 students

Overall, these findings suggest that students who initially enrolled in US12 are significantly different from all other non-FIP freshmen who passed the Subject A exam in their mean SAT Verbal and Math scores.  The findings comparing students who remained enrolled in US12 for the entire year to those who did not enroll in US12 beyond the Fall Quarter suggest that successful US12 students had higher SAT Math and Verbal scores, higher quarterly GPAs in Fall and Spring, and higher cumulative GPA’s as of Spring Quarter 2009.

US12 Course Grades

Table 5 displays the average grades in US12 both by quarter and gender. 

• The mean grades for all students in US12 increased quarterly.
• The mean grades for women were higher than the mean grades for men each quarter, though differences were not statistically significant.

Table 5: Grade Distribution for US12 by Quarter and by Gender

US12 Student Majors at Entry to UCI and at End of their First Year

As the FIP Program is committed to interdisciplinarity, there was interest in collecting student school affiliations at the start of Fall 2008 and again at the end of Spring Quarter 2009.  Table 6 displays the academic school affiliations of US12 students who completed the three quarter sequence at the beginning and end of the 2008-2009 academic year.

• Undecided/Undeclared students represented 32% of the students enrolled in US12 in the Fall Quarter as had been the case in 2007-2008.
• Students affiliated with the School of Information and Computer Science represented 35% of the students enrolled in US12 in the Fall Quarter.  By the end of the Spring Quarter, 39% of the US12 students were affiliated with the School of Information and Computer Science, compared to 48% in 2007-2008.
• The percentage of students with a declared major in Social Sciences rose from 8% in Fall 2008 to 22% in Spring 2009.

Table 6: Academic School Affiliation for US12 students in Fall 2008 and Spring 2009

The US12: Computer Games as Art, Culture, and Technology Course

US12 organized its course content around a central quarterly theme.  The Fall Quarter addressed the history of and general concepts in computer games.  As a function of this focus, students in US12 were required to learn Scratch, a programming language students would then use to develop a game.  In addition, students wrote one paper exploring the connections between the history of computer games and contemporary culture which included a annotated bibliography component.  The Winter Quarter was devoted to an exploration of computer games both as works of art and political or social expressions.  As such, students in US12B moved beyond the exploration of central gaming concepts to the exploration of non-commercial computer games.  In addition, students were introduced to the basics of Java programming which allowed students to further develop their computer game and wrote an analysis of a commercial computer game.  The Spring Quarter focused on gaming criticism, the elements that make computer games unique, and a more in-depth look at the elements needed to construct a computer game.  During the Spring Quarter, US12 students wrote their capstone papers and produced a design document about their computer game.  In addition to weekly lectures, students in US12 participated in both lab and discussion sections.  The faculty in US12 rotated lecturing responsibilities from week to week throughout the year and invited guest speakers to lecture each quarter. 

In order to gather information about students’ perceptions of the gains they made in (1) the content specific learning objectives, (2) information literacy, writing and research skills, (3) their sense of belonging to a learning community, (4) their understanding of how different disciplines ask and answer questions, (5) their experience of the teaching, and (6) their overall assessment of the course, evaluations were distributed electronically each quarter to all enrolled students.   The response rate for the US12 course evaluation was 79% (64 out of 81) in the Fall Quarter, 70% (55 out of 78) in the Winter Quarter, and 70% (55 out of 79) in the Spring Quarter.

US12 Learning Gains

Table 7 displays the four content specific learning outcomes associated with US12: Computer Games as Art, Culture, and Technology, and the percentage of students who either agreed or strongly agreed that they made gains in these areas.

Table 7: US12 Content Specific Learning Outcomes

a The results displayed represent the percentage of survey respondents who agreed or strongly agreed with the statement.

• By the Spring Quarter, over 90% of survey respondents reported making gains in all the content specific learning outcomes.
• Students reported making steady gains in their understanding of how computer game technology and techniques are used for purposes other than entertainment.  Students have consistently reported the greatest gains in this learning outcome for the past three years. 
• All students reported making gains in their understanding of how computer game technology and techniques are used for purposes other than entertainment compared to slightly more than half the students in both 2006-2007 and 2007-2008.
• As has been the case since 2006-2007, fewer students reported making gains in their ability to implement code, art, and sound/music within a game than was the case for any of the other content specific learning outcomes, which likely reflects the complicated nature of this learning component.

Because of the importance given to providing students with the opportunity to practice and demonstrate their written communication skills, 10 learning outcomes, developed in the summer of 2008, and shared across all three FIP courses, were included in the quarterly course evaluations as a means to capture students’ learning gains in the development of information literacy, writing, and research skills.

As shown in Table 8, students’ self-reported gains in information literacy, writing, and research skills in US12 are mixed.

• Students reported making gains from Fall to Spring in six of the ten learning outcomes concerned with their information literacy, writing, and research skills.
• Students reported making no gains over time in their ability to use APA citation style, which likely reflects the limited importance associated with citation style within the discipline.
• In the Fall compared to the Spring, students expressed with greater enthusiasm that they were making gains in their ability to both locate and assess sources and integrate those into their writing, and think critically as demonstrated through the development of well-supported thesis driven essays.
• With the exception of their ability to successfully integrate sources into their writing, students reported the largest gains in their information literacy, writing, and research skills in the Winter Quarter.
• The decline in students’ level of agreement in the Spring Quarter about the learning gains achieved around their writing likely reflects how the production of their capstone paper in the third quarter led students to reappraise the degree of their achievement of information literacy and critical thinking skills.

Table 8: US12 Information Literacy, Writing and Research Learning Outcomes

a The results displayed represent the percentage of survey respondents who agreed or strongly agreed with the statement.
b This statement only appeared on the Winter Quarter course evaluation.
Students’ Perceptions of the Teaching in US12

Table 9 displays the percentage of students who agreed or strongly agreed with the evaluation items designed to capture student perceptions of the US12 instructors and instruction.

• Overall, the results show that students had very positive views of the faculty and the way they worked together to teach US12 as has been the case since US12’s first year of instruction.
• By the end of the academic year, nearly all (95%) students reported that the instructors created an atmosphere that was conducive to learning and taught in a manner that reinforced the interdisciplinary nature of the course.

Table 9: Perceptions of US12 Instructors

a The results displayed represent the percentage of survey respondents who agreed or strongly agreed with the statement.

Table 10 displays students’ perceptions of the teaching assistants in both in US12. Before reviewing the results related to the effectiveness of both the lab and discussion sections in US12, it is important to note that the TAs rotated responsibility quarterly for leading the lab and discussion sections.  As a result, the changes in students’ level of agreement may be a function not only of the content of these sections but also of the TA leading that component of the course.

• Over the course of the year, students reported a decline in the helpfulness of their discussion section to improving their writing and research skills, as well as the effectiveness of the TA in clarifying lecture concepts and materials. 
• For both the discussion and lab sections, students reported lower levels of agreement that their lab/discussion section was well organized as the year progressed.
• In contrast, students reported the highest level of gains in their computer programming skills in the Spring Quarter and remained steady in their level of agreement that their lab section helped them improve their game design skills. 
• For all of these results, it is unclear to what degree these results reflect differing degrees of effectiveness of the TA leading these sections or how students, as the year progressed, gained a more realistic picture both of their skills and the effort required to improve those whether they be writing-related or computer game/design-related.

Table 10: Perceptions of US12 Teaching Assistants

a The results displayed represent the percentage of survey respondents who agreed or strongly agreed with the statement.

            Students’ Overall Assessment of the Course

Table 12 displays students’ perceptions of the overall gains associated with US12: Computer Games as Art, Culture, and Technology.

• Students increasingly reported having a sense of belonging to a learning community as a result of their participation in FIP over the course of the year. 
• Students also reported with steadily increasing levels of agreement that US12 strengthened their understanding of how different disciplines ask and answer questions.
• Perhaps most importantly, when asked about the overall impact of the course on their writing and research skills, there is an increase over time in the percentage of students reporting that the course strengthened both their research and writing skills. 
• Regardless of the quarter, students consistently reported making greater gains in their research skills than their writing skills with students reporting at a slightly higher rate that the course benefited their research skills versus their writing skills.  It is interesting to note that while the gains students report around their research skills remain consistent from Winter to Spring, students report slight gains in their writing skills from Winter to Spring, likely a reflection of their completion of their capstone paper.

Table 12: Overall Learning Gains

a The results displayed represent the percentage of survey respondents who agreed or strongly agreed with the statement.

Students’ assessment of the quality of US12 is presented in Table 13.

• Students’ ratings of both the overall quality of the course and its instruction are incredibly positive, with 93%-98% of respondents in any given quarter describing the overall quality as excellent, very good, or good.
• Interestingly, students’ rating of the overall quality of instruction and the course was at its highest in the Winter, whereas in the previous two years, it was in the Winter Quarter when students were least enthusiastic in their appraisal both of the course and its instruction.  This change in pattern suggests that positive and effective changes were made to curriculum flow of US12.
• Though the percentage of students who reported they would recommend this course to other students declined from the Fall Quarter, it is important to note that a greater percentage of students agreed that they would recommend this course to their peers than was the case in 2007-2008, when this question first appeared in the quarterly course evaluations.

Table 13: Students’ Perceptions of the Overall Quality of US12

a The results displayed represent the percentage of survey respondents who rated the quality as excellent, very good, or good.
b The results displayed represent the percentage of survey respondents who agreed or strongly agreed with the statement.

The results from the course evaluations suggest that students who participated in US12 reported meaningful gains in the content specific learning outcomes for the course.  US12 students improved their overall writing and research skills as well as more specific writing and research skills.  The gains they reported on specific writing and research skills fluctuated slightly from quarter to quarter, likely a reflection of the writing assignments they produced during the year.  Over the course of the year, students viewed the teaching and overall course quality very positively.  Further, the course evaluations show that students increasingly understood the relationship between computer games, art, culture, and technology, how different academic disciplines work, recognized the interdisciplinary nature of the course, and felt they were part of a supportive community of learners.

The Students’ Writing Skills

The central writing requirements for all US12 students were (1) enrollment and successful completion of the first course in the lower-division writing requirement (Writing 39B or its equivalent) with a grade of C or better before Spring Quarter, (2) quarterly writing assignments to facilitate students’ writing, research and information literacy skills, and (3) the production of a capstone paper during the Spring Quarter.

Enrollment in Writing 39B

All students were notified of the expectation that they enroll in Writing 39B during the summer, both through SPOP and promotional materials, and again during the Fall Quarter for those unable to enroll in the first lower-division writing course during their first quarter.  The course enrollments were reviewed at the beginning of Fall in order to ascertain which US12 students would need to take Writing 39B in the Winter and to assist them in getting enrolled in the course.  Ultimately, all students in US12 enrolled in and completed the first quarter of the lower-division writing requirement by the end of the Winter Quarter.

Table 14 displays when US12 students enrolled in Writing 39B and the average grade they received in the course.

• All students who enrolled and completed Writing 39B received a grade of C+ or better in the course.
• The 3 students who were enrolled in Humanities Core, instead of Writing 39B, received grades of B or better.
• The mean writing grades for US12 students in Writing 39B were higher this year than had been the case for the two previous US12 cohorts.

Table 14: Student Enrollments in Writing 39B

a  3 US12 students were enrolled in Humanities Core and therefore
did not enroll in Writing 39B.

Quality of Student Capstone Papers

All US12 students were required to produce a capstone paper in the Spring Quarter.  All capstone papers were collected and a random sample of 7 papers were evaluated to assess the quality of writing produced by the students in US12 and the degree to which those papers were similar in quality to those produced in Writing 39C and Humanities Core, the two primary methods, other than FIP, by which students fulfill the second course of the lower-division writing requirement.  In order to answer these questions about the nature of student writing achieved in fulfilling the lower-division writing requirement, the Campus Writing Coordinator initiated an assessment plan which included such goals as determining the shared writing expectations across all lower-division writing courses and assessing writing products based on learning outcomes for lower-division writing adopted by the Council on Educational Policy in 2008-2009.  As a result, US12 capstone papers were assessed alongside a random sample of final papers produced by students in Writing 39C as well as Humanities Core, the two primary curricular avenues available to students for fulfilling the second course of the lower-division writing requirement.  In preparation for this review of student papers, Jonathan Alexander, the Campus Writing Coordinator, convened the Lower-Division Writing Committee, comprised of all lower-division writing course program directors, to discuss the writing expectations across courses and develop a rubric that captured those expectations and the lower-division writing learning outcomes.  In collaboration with Lynda Haas, Course Director for Writing 39B, Jonathan Alexander worked with the Lower-Division Writing Committee to develop a rubric that would capture the shared learning outcomes across the lower-division writing courses.  The rubric developed to assess the lower-division writing products is an amalgamation of three different rubrics, one that had been used previously to assess FIP capstone papers, one used in a longitudinal study of writing at Stanford, and a third developed in early 2009 through AAC&U’s VALUE Project. The Lower-Division Writing Rubric is comprised of 12 agreed-upon writing traits which can be further collapsed into four general categories of writing: (1) rhetorical knowledge, (2) critical and rhetorical approach to analysis and evidence, (3) structure and organization, and (4) language use.

Because the nature of this writing assessment assessed FIP papers as part of a larger effort to assess lower-division writing at UCI, twelve readers , all of whom have extensive experience in writing and who teach in the lower-division writing courses, were hired to assess the quality of student writing produced in Writing 39B, Writing 39C, FIP, and Humanities Core using the rubric presented in Table 19.

Table 19: Lower-Division Writing Assessment Rubric

In preparation for the assessment of Lower-Division Writing on June 24-26, 2009, all twelve readers were sent the Lower-Division Writing Assessment Rubric along with 2 sample papers identified by Jonathan Alexander and Lynda Haas as reflecting a range of writing quality based on the rubric’s twelve writing traits.  The readers were asked to review these papers as well as the rubric and assign scores to each of these.  The first morning of the assessment was devoted to a review of the goals of the lower-division assessment plan, the lower-division writing rubric, and the 2 sample papers.  Each subsequent day of the assessment, the group collectively reviewed and discussed 6 more sample papers to strengthen agreement of what constitutes achievement of the 12 writing traits contained in the Lower-Division Writing Rubric.  Because this assessment was predicated on the notion that student writing produced in the lower-division writing courses should be able to demonstrate the application of these 12 writing traits, neither the assignment prompts nor the course within which a given paper was produced were provided to the readers.

After the initial training session on the morning of June 24, 2009, readers were divided into 4 reading teams, with all members of a given team reading the same set of papers.  All papers were read by three readers, with each individual reader assigning a score for each writing trait contained within the rubric.  As papers were scored, Natalie Schonfeld monitored and tabulated the results.  In order to strengthen inter-rater reliability, for any paper where the difference between the overall scores assigned by the three readers was greater than 4, one of the readers was asked to re-read the paper in question to confirm the score they initially assigned to the paper in question.  The results of these efforts yielded a reliability coefficient of .95 for the US12 papers and an overall reliability coefficient for all lower-division writing papers of .945.

• The US12 capstone papers were found to be strongest in the following five writing traits: (1) paragraphs, (2) documentation, (3) focus, (4) expertise on topic, and (5) language correctness.  Two of these five writing traits fall within the general category of rhetorical knowledge.
• The five writing traits which were found to be weakest were: (1) sophistication of focus, (2) structure, order, (3) source, evidence integration, (4) critical thinking, and (5) language eloquence.  Two of these five writing traits fall within the general category of critical/rhetorical approach to analysis/evidence.
• This same pattern of strength in the general category of rhetorical knowledge, and weakness in the general category of critical, rhetorical approach to analysis, evidence emerged with both the US13 and US15 papers as well,

Graph 1: US12 Capstone Paper Assessment Results by Writing Trait

• Overall, the US12 papers were strongest in the rhetorical knowledge and language use general writing categories. 
• It is important to note that the differences in assessment scores as a function of the general writing category are rather small.

Graph 2: US12 Capstone Paper Assessment Results by Writing Category

• When the US12 paper assessment results are viewed side by side with the assessment results for Writing 39C and Humanities Core, the US12 papers appear to be stronger than the Writing 39C papers in all four of the general writing categories, and stronger than either of the other two courses in the structure/organization general writing category.

Graph 3: US 12, Writing 39C, and Humanities Core Assessment Results

Overall, the assessment of the US12 capstone papers suggests that students are meeting the writing expectations, are able to demonstrate achievement of the learning outcomes associated with fulfilling the lower-division writing requirement, and, in some instances, demonstrate a higher level of achievement than their counterparts.

Conclusions and Recommendations

The assessment of US12: Computer Games as Art, Culture, and Technology in its third year was concerned with gathering information about the students who participated in US12, the course, its content and assignments, and the students’ writing skills.

• The results of the data collected about the students demonstrated that students who decided to enroll in US12 were primarily Undecided/Undeclared or affiliated with the School of Information and Computer Science, possessed higher SAT Verbal and Math scores than all other non-FIP freshmen and were more likely to perform better academically in their first year at UCI.
• Over time, the US12 students reported making meaningful gains in both the content specific learning outcomes outlined on their course syllabi and the FIP-specific information literacy, writing, and research learning outcomes.  The students in US12 reported understanding different disciplinary approaches and perceived instruction as reinforcing the interdisciplinary nature of the course.
• The findings related to information literacy, writing, and research skills suggest that students perceived making gains in these areas, and that their writing assignments provided opportunities both for practice and appraisal of their skills.  
• The assessment of students’ writing skills confirmed that students in US12 were able to demonstrate that their writing reflects the shared expectations and learning outcomes for the lower-division writing requirement.
• The assessment of US12 capstone papers also suggests that US12 students were in some instances able to demonstrate the achievement of those skills at greater levels than their counterparts in Writing 39C and Humanities Core, confirming that US12 is an appropriate and successful alternative for fulfilling the second course of the lower-division writing requirement.
• Students are very satisfied with their overall experience in US12, perceive the course quality and instruction very positively, and are highly likely to recommend this course to other students.
• While the construction and instruction of interdisciplinary courses can be challenging and demanding, the benefits to student learning, both in terms of writing and developing an understanding of how different disciplines ask and answer questions, are many.