Multiple Approaches for Measuring Student Learning

8 Jan

What evidence do you have that your students are learning?

In addition to student grades, evidence can include graduation and retention data, student capstone experiences and artifacts, or recent graduate employment or professional school attendance data.  Making improvements to your teaching based on this data is what Michael Reder refers to as an evidence-informed teaching approach.

Reder – Director of the Joy Schectman Mankoff Center for Teaching and Learning at Connecticut College – urges educators to measure something meaningful to them as a way to determine specifically if their teaching approaches are effective for student learners and learning.

 A specific example of something instructors measure at Connecticut College is the intellectual challenge and academic rigor of courses. Faculty participants who opt to participate in this assessment process ask their students to complete a “Supplemental Course Survey” at the end of the semester.  (Link to sample survey provided below.)  Analyzing data like this provides a snapshot in time about perceived academic rigor of a course.  Collecting individual course data every semester allows an instructor to track changes.  Drawing multiple course sections allows a unit to build a comparative data analysis to understand impacts of different instructor approaches, or of course-wide adoption of strategic teaching initiatives. 

Reder also offers a couple of cautions about measuring learning as part of reminding faculty to triangulate the data gathered so that our conclusions don’t rely on just one measure.  The first caution is to remember data doesn’t prove anything.  He advises that collecting data about student learning shouldn’t be about proof but rather about understanding.  As Reder notes, “The best way to dig deeper and make data multidimensional is to talk with students”. 

The second caution is to incorporate more than one measure.  A survey is one way to collect data about academic rigor.  Supplementary measurements could include qualitative analysis of samples of student work or quantitative analysis drawing on test scores.  Additionally, reflective peer review of a course syllabus and/or class session(s), or consultant-facilitated focus groups of students who took a course can yield helpful qualitative data about perceptions of teaching. 

Linda Nilson also advocates for incorporating at least one measure beyond end-of-term students ratings of teaching in order to understand student learning and perceptions of leanring.  In her recent article, “Measuring Student Learning to Document Faculty Teaching Effectiveness’,” Nilson describes some additional course level measures of student learning to consider employing.  Some of these measures may be more or less appropriate depending on a teacher’s discipline and on what the teacher is hoping to measure or understand. 

These measures may be direct or indirect, may occur only at the end of course, or take shape as combined pre- and post- course assessments of learning.  Some examples:

Indirect measures at the end of the course only.  These can take the forms of surveys that ask students to rate their perceived learning or knowledge gains at the end of the course.  The intellectual challenge and academic rigor survey described above would be an example of this.  Another commonly used template to consider is the Student Assessment of Learning Gains (www.salgsite.org).

Direct measure at the end of the course.  These would include assessments such as final exams, capstone assignments, and portfolios.  These are graded by the instructor against predetermined standards. Results can be compared between sections and semester-to-semester.

Indirect pre- and post- measures built into the beginning week to reappear again at the end of the course.  These can take the form of surveys asking students to rate their confidence in course skills and knowledge at the beginning of class and again at the end of the course.  Increases in the posttest scores over the pretest scores provide a gain score for students in each of the areas measured.  These scores can be compared semester-to-semester.  This is the type of measurement Teaching in Higher Education course instructors developed to ask students to rate their pre- and post-course confidence in their ability to design and teach college courses. 

Direct pre-and posttest on course concepts.  To do this an instructor would give a version of their final exam on the first day of class as an ungraded, diagnostic tool.  Another approach is to ask students on the first day of class to indicate their level of agreement with various statements about the course topic, including some commonly held misconceptions.  Ask students to and provide a brief sentence or two of explanation for their answer.  At the end of the semester distribute these to students and ask them to comment on their original answers and indicate any changes in thinking.  In addition, ask students to  provide evidence from the course to support their answer.  With either approach, increases over the ungraded pretest scores may be measured as percentage improvement or as a percentage of possible performance.  

Choosing appropriate multiple measures of your students’ learning positions you to make rational and informed decisions about your teaching approaches.

Resources Mentioned in Post:

Linda Nilson.  “Measuring Student Learning to Document Faculty Teaching Effectiveness.” To Improve the Academy 32 (2013): 287-300.  PDF available until 1 February 2014: http://www.scribd.com/doc/197408484/Nilson-Measuring-Student-Learning-to-Document-Faculty-Teaching-Effectiveness

Michael Reder.  ‘Cultivating a Culture of Evidence-Informed Decision Making about Teaching and Learning.” Workshop Session. Professional and Organizational Development Conference, 2013.

Reder.  “Supplemental Course Survey.” Connecticut College, 2013.  PDF available: http://www.scribd.com/doc/197408483/Intellectual-Challenge-and-Adademic-Rigor-Survey

Teaching in Higher Education Instructors.  “Pre-Course or Post-Course Confidence Survey.”  University of Minnesota Preparing Future Faculty Program, 2011.  PDF available: http://www.scribd.com/doc/197408482/Pre-Post-Confidence-Survey-GRAD-8101

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: