In 2008-2009, the University’s Office of Institutional Assessment and Studies initiated planning for the first institution-wide assessment of undergraduate research competency.
The University does not mandate that all undergraduates conduct or participate in research, although many degree programs do. Nearly all fourth-year students in 2012 reported that they were currently participating in or had done “a research project, creative activity, or research paper as part of coursework.” Half of fourth-years reported having completed a “significant research project” during their tenure at U.Va. A quarter had assisted faculty in research for credit, fewer for pay or as volunteers.
Definition
The Undergraduate Research Competency Assessment Committee defined research as:
the practice of carefully:
- formulating or addressing a question, problem or objective,
- analyzing it within a disciplinary or interdisciplinary framework,
- producing findings, conclusions, designs, or creative works, and
- clearly communicating and defending such to a critical audience.
Student Learning Outcomes
Students engaging in undergraduate research will:
- Approach the study from a particular disciplinary or interdisciplinary point of view
- Form a thoughtful research question or identify a problem to be solved
- Use a variety of sources
- Collect pertinent information
- Analyze data/evidence collected
- Draw logical and defensible conclusions
- Communicate clearly and effectively their findings and conclusion
- Defend their research to a critical audience.
Standards
As the first institution-wide assessment of research competency and without previous assessments to serve as precedent, the standards adopted in previous core competency assessments were applied:
- 25 percent of research projects would be highly competent (avg. total score=75% of possible points)
- 75 percent of research projects would be competent or better (50% of possible points)
- 90 percent of research projects would be minimally competent or better (33% of possible points)
Methodology
Acknowledging the variability in disciplinary or interdisciplinary frameworks for conducting research, two scoring rubrics were made available for assessment of student research papers or projects with permission to modify the rubrics to meet specific disciplinary priorities.
Throughout 2010-12, 332 research papers and projects, representing nine major disciplines among the six schools, were assessed: BIS/SCPS, Humanities/College, Interdisciplinary/College, Science/College, Social Science/College, Commerce, Education, Engineering, and Nursing.
Rubric results were analyzed across disciplines and by discipline. As some programs had altered the rubric, applying more or fewer outcomes or by modifying the scale, the scores were analyzed in terms of percent of possible points earned. Inter-rater reliability of ratings was low to moderate for the assessments, depending on the discipline.
Findings
As this was the first institution-wide assessment of undergraduate research competency, it was important to confirm that the scoring rubrics provided a reasonably valid measure of students’ competence, although the rubrics often needed to be tailored to the discipline.
Students’ research papers and projects met the standards for competence in research. Students were most able to approach the study of a subject or problem from a particular disciplinary or interdisciplinary point of view (e.g., “thinking like a…”), to collect pertinent data/information, and to communicate their findings clearly and effectively. Students were least able to analyze data or evidence and to draw logical and defensible conclusions. This result is not surprising as analysis and development of conclusions both require higher order thinking—the ability to weigh evidence and synthesize material.
2011-12 Committee Members
- Richard Handler (CLAS, Anthropology) – Co-Chair
- Wolcott, David (Provost) – Co-Chair
- Berne, Rosalyn (SEAS, STS)
- Brickhouse, Anna (CLAS, English)
- Culaty, Brian (Provost)
- Freedman, Paul (CLAS, Politics)
- Friberg, Elizabeth (NURS)
- Kelly, Luke (SED, Kinesiology)
- Marshall, Paxton (SEAS, ECE)
- Maxham, Trey (Assoc. Dean, SCC)
- Mills, Aaron (CLAS, Environmental Science)
- Palmer, Michael (TRC; CLAS, Chemistry)
- Plasket, Donna (SCPS, BIS)
- Pocanic, Dinko (CLAS, Physics)
- Rasbury, Michael (CLAS, Drama)
- Roettger, Betsy (SARC, Architecture)
- Russell, Lucy (Provost)
- Soule, Kathy Soule (Library)