Student Outcomes at CSUS :
Perceptual and Objective Measures of Learning
I'm glad someone cares about how well we are
learning
instead of just how much.
--incoming first-year student
In the late 1980s the California State University system held several assessment
conferences for faculty and administrators, and, along with WASC, expected campuses to
develop and implement assessment initiatives to assess the effectiveness of their educational
programs. In 1993 the CSUS Academic Senate responded by developing a policy, which
the President approved, requiring departments to develop assessment plans to measure
student outcomes in the major. The 1992 revised General Education program also included
an assessment mandate. Faculty and administrators have attended assessment conferences
and meetings of the Assessment Forum of the American Association of Higher Education
to learn more about assessment.
When the WASC Steering Committee was presented with the opportunity to build
assessment into its Self-Study, the selection of the Student Outcomes theme was
particularly pertinent. However, when a proposal to have selected academic departments
measure student outcomes in the major was challenged, the Steering Committee moved
toward a more general assessment of educational outcomes. The Committee was aware that
an initiative to examine student outcomes would contribute to future curriculum changes,
and, in the long run, might have a positive effect on students, faculty, alumni, employers,
and the public. In Phase II student outcomes has been expanded to include more data.
The WASC Educational Standards, as well as the liberal arts tradition, require that
University graduates demonstrate competence in written and oral communication,
quantitative reasoning, critical thinking, and develop an appreciation of cultural diversity. A
subcommittee of the WASC Steering Committee examined instruments and studies
conducted at other universities designed to measure these competencies. Given the time
frame for the Self-Study, the subcommittee realized it would be impossible to develop
assessment tools unique to CSUS. Eventually the WASC Steering Committee agreed to
examine the performance of CSUS students on existing tests taken by graduating students
and to use opinion survey instruments developed by professional testing services that
measure student outcomes. Data pertaining to six, largely objective, outcomes are reviewed
in this chapter:
4.1 Thinking Skills: Students leaving CSUS think clearly and logically.
4.2 Information Skills: Students leaving CSUS find information and examine that information critically.
4.3 Communication Skills: Students leaving CSUS communicate effectively both orally and in writing.
4.4 Quantitative Skills: Students leaving CSUS reason quantitatively.
4.5 Pre-professional Skills: Students leaving CSUS are prepared to assume career responsibilities, meet credential requirements, and gain admission to graduate or professional schools.
4.6 Retention and Graduation: CSUS retains students who have not yet achieved a credential or degree and graduates students seeking a degree
4.1 Critical Thinking Skills - Perhaps there is no outcome more fundamental
to the purpose of a university than a commitment to produce graduates who are critical
thinkers. CSUS is no exception, and expects that Students leaving CSUS can think clearly
and logically.
Critical Thinking Skills Indicators:
Graduates and alumni have largely favorable perceptions of the critical thinking
skills developed during their years at CSUS. In short, students believed that the University
is doing a good job of teaching in this area. Across the five perceptual indicators of
thinking skills almost three out of four responses (73%) were positive. Interestingly, both
the top-ranked indicator in this category and the bottom-ranked indicator dealt with
"problem solving." Seventy-nine percent of the respondents to ACT Alumni Survey (ACT
Alum) responded positively to the stimulus "defining and solving problems," By contrast,
68% of the students responding to the ACT Graduating Seniors Survey (ACT Grad) rated
"developing problem-solving skills" as positive. The percentage of positive responses for
the remaining three indicators fell between 71% and 74%.
To measure critical thinking skills of CSUS students objectively, 600 students
were invited in Fall 1995 to take the Collegiate Assessment of Academic
Proficiency (CAAP) test of Critical Thinking published by the American College
Testing Program (ACT). This test measures the ability to clarify, analyze, evaluate, and
extend an argument. Fifty-six percent (338) of the students actually completed the test.
For the most part, CSUS students performed satisfactorily on the Critical
Thinking test. It should be noted, however, that the national norms for the critical thinking
test are based on sophomore level skills; students who completed the tests at CSUS were
enrolled in courses typically taken by seniors. Because of the low numbers of students
taking the test, caution must be exercised in drawing inferences from the results.
The Critical Thinking test score for the CSUS sample was 63.6. For comparison
purposes, the national mean was slightly less at 62.7. The standard deviation of the
national sample was 5.4. This means that roughly two-thirds of the national sample scored
between 57% and 68%.
The test mean for the 289 CSUS students whose primary language was English
was 64 compared with a mean of 59 for the 40 ESL students. Students majoring in the
health professions (n = 19) and home economics (n = 7) scored the best (mean = 66).
Students majoring in liberal studies (n = 6) and engineering (n = 32) scored slightly below
the mean.
Ethnically, only Asian students at CSUS scored substantially below the campus
and
national mean. The mean for Asian students was 58. However, Black, Filipino, and Native
American students also scored below the CSUS mean, as did ESL students.
4.2 Information Skills - Preparation for participation in the "information age"
is central to the purpose of any contemporary university. It is almost unthinkable that
students could leave CSUS without a modicum of expertise in accessing, evaluating and
applying information. Students leaving CSUS can locate information and evaluate that
information critically.
Information Skills Indicators:
Perceptual indicators of information skills indicate that most students and
graduates believed they received fairly good training in locating and critically examining
information. This finding held true across three different surveys with three different
populations: alumni, graduating seniors and continuing students.
Respondents to the surveys were most complimentary of training in research
skills. Seventy-nine percent of the respondents to the CASPER survey reported that the
curriculum in their major provided them with the research skills demanded by their
respective disciplines. The compliments extended to the full range of skills: locating,
accessing, screening, organizing, analyzing, drawing conclusions, and using information.
4.3 Communication Skills - It is hard to find a profession that does not
demand communication skills. This demand is particularly acute for those who rise to the
top of their profession. Nor is communication skill limited to professional settings.
Successful participation in society seems to demand written, oral, and interpersonal
communication skills. The University's desire, therefore, is thatStudents leaving CSUS
communicate effectively in both oral and written forms.
Communication Skills Indicators:
The list of indicators for communication skills yielded the largest discrepancy
between the top-rated and bottom-rated indicators. The variability among perceptual
indicators was amazing. The top rated survey item received an 84% positive rating,
compared to a 30% rating for the indicator at the bottom of the list. However, most
responses by students and former students were positive. Four out of five respondents
rated their CSUS education favorably when it came to providing verbal skills required by
their respective disciplines.
One clear finding from the perceptual data was that writing is more commonly
demanded by instructors than oral presentations. Seventy percent of the respondents to the
CASPER survey reported that most of their courses require written work. Yet only 30% of
the respondents to the same survey said that group presentations or individual presentations
were a requirement for successful completion of their courses. Consistent with this finding,
only 56% responded positively to a question regarding learning to speak more effectively.
Students were confident in their CSUS training in interpersonal communication
skills. The percentages of positive responses assigned to these indicators were "working in
groups" (72% positive), "listening and understanding what others have to say" (64%
positive), and learning to negotiate" (60% positive).
According to alumni, CSUS offers its best communication education in the area
of written communication. Eighty-four percent of the respondents to the ACT Alumni
Survey reported a major or moderate impact on their lives for the writing skills they
received at CSUS. On the ACT Graduating Senior survey, 65% reported progress toward
this outcome as a result of their CSUS experience.
The references to written communication skills so far in this chapter have been from
self-report or perceptual measures. CSUS has its own measure of writing skills, the
Writing Proficiency Exam (WPE). In order to graduate from CSUS, all CSUS
students must pass the WPE with a score of eight or better. The WPE is to be taken as
students begin the first semester of their junior year. Passage rates of the WPE are,
therefore, a gauge of the collective writing ability of the upper class student body at any
given time.
The graph below portrays the results of the WPE taken in March 1996. In the
interest of fairness and reliability, students who first scored a "7" on the first administration
are scored again by different faculty. On the second scoring the exam is assigned either a
"6," failing or an "8," passing. The distribution of scores for the March exam was roughly
normal, with most of the scores clustering one side or the other of the criterion for passing.
Fifty-six percent of those taking the exam received a passing score.
As a comparative measure of writing instruction at CSUS, scores from the WPE
work very well. The passage rate on the WPE for students taking English 1A at CSUS was
63%. The passage rate was higher only among those students who met their lower division
English requirement through Advanced Placement or at one of the UC campuses.
Considering the caliber of students who typically pass AP English or who begin their
college careers at UC campuses, the superior passage rate of these students was not
surprising.
When passage rates on the WPE are compared, other sources of lower division
writing instruction compare favorably to English 1A at CSUS. The passage rate for
students meeting their writing requirement at other CSU campuses was 57%. Students
attending community colleges at the time they took English 1A had a 53% passage rate.
Transfer students from outside the CSU, a UC, or from a community college system
passed the WPE at a 61% rate.
Sixty-one percent of the Caucasian students passed the WPE in March. Students
from other ethnic groups performed less successfully. Language, as expected, also
predicted passage rates. Students who met their freshman composition requirements
through various English as a second language (ESL) options passed at rates of 40% or
lower.
A second objective measurement of writing skill was the Collegiate Assessment
of Academic Proficiency (CAAP) test published by the American College Testing
Program (ACT). CAAP tests were selected because they measured more than minimum
competency in writing, and CSUS results could be compared with similar institutions. The
Writing Test was designed to measure skills most commonly taught in college-level writing
courses and required in most upper-division coursework. The test measures a student's
skills in formulating and supporting assertions about a given issue and in organizing and
connecting major ideas.
During Fall 1995, 600 students enrolled in advanced study courses were invited to
take the CAAP tests. However, only 288 completed the writing test. The CSUS sample of
students was comparable with national samples on all demographic elements except
ethnicity. The mean score on the CAAP Writing Test taken by CSUS students was 3.2
(standard deviation = 0.7, range 1 to 4.75)--a mean score identical to the national sample.
African American students (2.9), Filipino students (2.55), and Native American students
(2.61) scored below the mean.
Mean scores of English as a Second Language (ESL) students on the CAAP
Writing Test were lower than those of Native language speakers. The mean on the Writing
Test for the 258 students whose primary language was English was 3.26 compared with
the mean of 2.54 for the 28 ESL students who took the test. When comparing majors,
students scoring above the mean of 3.2 were from general studies (3.35, N=7), education
(3.45, N=24), and fine arts (3.32, N=10). The lowest scores were from students majoring
in criminal justice and social work (3.08, N=24), computer science (3.05, N=5),
engineering (2.94, N=13), and home economics (2.58, N=6).
Yet another objective measure of writing skill was the California Basic
Education Skills Test (CBEST) required of all students entering the primary and
secondary teaching professions. Students must pass the CBEST prior to entering the Basic
Teaching Credential Program. The CSUS passage rate for 8/95, 10/95 and 12/95 on the
writing portion of the test was 73%. The passage rate for the entire State of California on
the writing portion of the exam was 72%.
4.4 Quantitative Skills - Undeniably, the ability to think includes the ability
to measure, to count, to compare--to calculate. Success in work and life is best assured if
Students leaving CSUS can reason quantitatively.
Among the several surveys that measured student perceptions only one indicator
could be found that measured student opinions of quantitative skills taught at CSUS. The
ACT Graduating Seniors Survey asked respondents to rate "Understanding and applying
math concepts and statistical reasoning." Respondents were asked to rate the progress they
have made at CSUS toward attainment of the math and statistics outcome. The response
categories for this question were "very much," "much," "moderate (average)," "little," or
"none." Only 30% answered with "very much" or "much." CSUS students were not
confident in the quantitative education they received. It would have been helpful had
multiple indicators been available when drawing this conclusion. It is not know whether
these students completed their mathematics and/or statistics courses at CSUS.
Unlike the WPE which is available to measure writing proficiency, CSUS has no
test of mathematics proficiency prior to graduation. The campus does have a well-
developed system for evaluating mathematics competence of incoming students. Students
admitted to the University are required to have completed three years of mathematics in
high school. The Entry Level Mathematics (ELM) test, administered on campus at the
beginning of the student's first year, assesses student competence in mathematics at
entrance. Some students need developmental work in mathematics before they can enroll in
college level mathematics courses.
One objective measure of quantitative reasoning is available from the Graduate
Record Exam (GRE), an examination required by many graduate programs.
Unfortunately, CSUS students performed below national and state averages. The WASC
steering committee reported that 73% of the students who took the GRE scored at or below
the 50 th percentile on the quantitative reasoning component.
Another objective measure of quantitative reasoning is the pass rates of students
taking The California Basic Educational Skills Test (CBEST). During 1995
CSUS students on average scored below state averages on the math portion of the CBEST
with 422 CSUS students passing at a rate of 75.4%. The statewide passage rate was
78.7%.
For the WASC Self-Study, a mathematics knowledge exam was included in The
ACT-COS Graduating Seniors Survey. The exam was sent to all graduating students
in Spring 1995. The ten question exam is reproduced in its entirety below. The
mathematics assessment consisted of ten questions about mathematics knowledge. While
700 mathematics questionnaires were randomly included in the ACT-COS survey, only 56
(8%) were returned. The respondents answered an average of 4.5 questions correctly out
of the 10 (SD 1.8). The scores closely approximate a normal distribution. The performance
of the CSUS students was average, but there was only a very small number of
respondents. Thus, drawing conclusions based on the sample must be approached
cautiously.
Mathematics
Exam |
|
| The CSUS accreditation committee is interested in the level of math knowledge of our graduating seniors. We would like you to take a few minutes to answer the following questions by circling the letter of the most appropriate answer. This is an assessment of your present general math knowledge, so please answer to the best of your ability without using other resources. Your time and effort are much appreciated. | |
|
|
|
|
|
|
|
|
|
|
Taken together, the one perceptual measure and several objective measures of
CSUS quantitative student outcomes is not encouraging. Serious consideration should be
given to better measurement of student performance in the quantitative area. Of course, the
real issue is not measurement, it is student performance.
4.5 Pre-professional Skills - A university education serves a wide variety of
functions. Most CSUS students expect degree programs to prepare them both academically
and professionally, so that Students leaving CSUS are prepared to assume career
responsibilities, meet credential requirements, and gain admission to graduate or
professional schools.
Pre-professional Skills Indicators:
Among the many outcomes examined in this chapter, pre-professional skills
appeared to be the weakest. The mean percentage of positive responses to the many
perceptual indicators was only slightly more than half. Only about three in five of those
responding to graduating senior and alumni surveys believed that CSUS had "prepared
them well for a career." While one's job is often a matter of choice, only 59% of the
Alumni said their major was directly related to their first or current jobs. While the
professional glass is more than half full, an institution like CSUS should concentrate on
filling the glass a little more, especially in providing assistance and support for students
desirous of a career.
Even less satisfactory than professional preparation in the major was the
University's seeming inability to provide graduates with job-seeking skills. Only 31% said
that CSUS had contributed "very much" or "much" toward the outcome of "developing
effective job-seeking skills" (e.g. interviewing, resume construction). The 31% positive
response to this indicator was the lowest positive response among any of the indicators in
any of the outcomes summarized in this chapter. Departments should explore ways of
meeting the needs of students in developing job-seeking skills.
A higher scoring indicator in the area of pre-professional category was
preparation for further study. Alumni reported 64% positive impressions of professional
preparation. Graduating seniors registered a 62% positive figure.
Fortunately, when we are assessing professional preparation of CSUS students
we are not limited to perceptual data. Four of the standardized tests currently being taken by
CSUS students to meet credential requirements and to determine admission to graduate and
profession schools were reviewed.
The Graduate Record Examination (GRE) is a general test of verbal,
quantitative, and analytical reasoning designed to assess undergraduate achievement or the
qualifications of students for graduate study. The test does not and cannot measure all the
factors important in predicting success in graduate study or in assessing undergraduate
achievement. Although universities are discouraged from using the scores in aggregate
form, a review of score ranges at CSUS indicated that:
In 1997 the GRE will be adding new sections to the exam that will provide CSUS
with more outcome assessment information. The University has not established an
acceptable performance score to use in interpreting the GRE test scores for itself for
comparison with other institutions.
Since 1983, prospective teachers have been required to take the California
Basic Educational Skills Test (CBEST) to ensure an acceptable level of competence
for credentialed K-12 teachers. Data were provided to CSUS for its students and others
across the State of California, comparing scores by gender and ethnicity and those of
seniors to those with the BA, those with more than the BA, and those with the MA degree.
The overall first-time pass rate for the State on the CBEST was 68% and for
CSUS students was 64.7%. CSUS students met or exceeded the State rates in reading and
writing, but not mathematics. Those with more than the BA exceeded the State and CSUS
rates in reading, mathematics, and writing as did those with the MA. Black and Latino
students scored below the State and CSUS rates in reading, mathematics, and writing.
Other Latino students scored below the State and CSUS scores in reading. Males scored
below the norm in writing while females scored below the norm in reading and
mathematics.
In 1995 two test results were available for CSUS students seeking admission
to
medical schools. The Medical College Admission Test (MCAT) measures
proficiency in verbal reasoning and writing, and competency in biological and physical
sciences. Forty-two CSUS students completed the test with the following results:
On the MCAT the students from CSUS compared favorably with the national
sample except for writing competency. In August 1995 CSUS students scored in the 50th
percentile; in April of the same year, they scored in the 75th percentile.
Similar to the MCAT for prospective medical students is the Law School
Admission Test (LSAT). Scores on the LSAT are reported on a scale ranging from 120
to 180. Three sets of scores were reviewed for tests taken in December 1995, February
1996, and June 1966. On all three test dates applicants from CSUS scored, on average,
below the 50 th percentile. It is not known, however, whether this result is due to poor
preparation of students on the part of CSUS or whether the pool of applicants from
numbers of students participating.
4.6 Retention and Graduation - The expectation of degree-seeking students
who have been granted admission to CSUS is that within a reasonable period of time they
will leave with a diploma. Individual circumstances change, but for the most part it should
be the goal of CSUS faculty, staff and administration that CSUS retains students who
have not yet achieved a credential or degree and graduates students seeking a degree.
Over the nine-year period from 1986 to 1994, the average one-year retention rate
for new students has been about 77% for both freshman and transfer students. Over the
same time period, the two-year retention rate dropped to 65% for former CSUS freshmen
and to 70% for former transfer students. As the graph below illustrates, there has been a
slight trend upward in first year retention rates.
But it is not enough just to retain students for a number of years; the ultimate goal
is to graduate students. It appears that timely graduation, however, is simply not happening
for far too many CSUS students, particularly if timely graduation is defined in the
traditional sense of four years for an entering freshman and two years for a community
college transfer student.
Only 16% of entering freshman graduate with the expected four-year degree. The
percentage more than doubles to 34% after five years, but this is still only about a third of
the entering freshman class. After six years the percentage starts to level off at 42%,
reaching 48% after eight years.
Similarly, only 23% of transfer students graduate two years after transferring
from a community college or other institution. The percentage takes a big jump to 42% after
three years, but is still less than half of the population. After four years the percentage starts
to level off at 51%, eventually climbing to 57% after six years.
There could be several possible explanations for the length of "time to degree" at
CSUS. First this is a commuter campus. Many students cannot take a full load of classes
along with the jobs they hold to pay for their education. Second, the number of units
required to complete General Education requirements can be extensive for a student who
transfers and does not plan well. In some cases the "time to degree" may be because some
majors have large unit requirements. Whatever the reason, reducing length of "time to
degree" should be examined further.
§
Regarding student outcomes at CSUS, it appears that CSUS performed well in
training students to locate information and critically evaluate that information once found.
CSUS performed very well in educating students to communicate, particularly written
communication, but not as well in quantitative skills. Better measurement of quantitative
outcomes is needed. Students want more assistance in developing specific career skills.
Whether academicians like it or not, students see a university degree as the door to a career.
For most students that career does not, however, commence after the traditional four years
of higher education. For a variety of reasons a CSUS degree comes only after an extended
number of years for most students.