In major shift, AU begins adoption of holistic teaching evaluation system
While the data-driven Student Evaluations of Teaching will continue, they will be one component of a larger portfolio review
American University is moving away from relying entirely on the data-driven Student Evaluations of Teaching (SETs), the end-of-course survey for feedback on class experiences, to what many faculty members say is a more holistic approach.
AU will adopt a menu of portfolio options for course evaluation, encompassing everything from the SETs to new metrics like personal teaching statements, peer-to-peer evaluations and annotated syllabi.
The change requires adjustments from every teaching department in the University to overhaul both the way evaluations are conducted and how they are incorporated into staff management changes. These adjustments will improve both the reception of student voices and the quality of professor evaluations, said Kiho Kim, the executive director of AU’s Center for Teaching, Research and Learning.
“SETs tell us something about the classroom interaction, but it is just a small part of what we want to document,” Kim said. “So, we decided on the portfolio approach, which is, in essence, a best practice for documenting teaching.”
Karen Baehler, co-chair of the Beyond SETs Task Force, said that the University planned to move away from relying only on SETs this academic year, but the move was delayed due to the pandemic. Individual departments are now implementing the initiative in smaller steps, which helps professors, as the new requirements are significantly more rigorous on faculty, according to Kim.
Beyond SETs Task Force Co-Chair Max Friedman said that the initiative would include self-evaluation, which the group hopes will improve professor performance.
Kim said one of the greatest issues with the SETs, which necessitated more qualitative options, is that such data-heavy evaluations are inherently susceptible to all sorts of conscious and unconscious bias. Now, many universities in the U.S. are considering other options.
“We know that SETs measure some things, but it is a small part of what we believe is quality teaching — something that is rigorous and demanding of the students, and pushing of the students, which sometimes are at odds with how students want to evaluate their classroom experience,” Kim said.
Todd Eisenstadt, the chairman of the Faculty Senate’s Committee on Faculty Actions, said that professors want a more comprehensive way to have their teaching evaluated.
“They should be fair, but demanding,” Eisenstadt said. “The view has been that SETs don’t show how much engagement faculty have, they just show what the student says about them at the end. SETs don’t necessarily reward professors who go out of their way to make their classes challenging.”
The Beyond SETs Task Force was created after Cheryl Holcomb-McCoy, the dean of the School of Education, started a similar initiative within the school.
“How do you create an evaluation approach that relies on multiple measures, is evidence-based, is fair and unbiased, differentiates performance so there’s a sense of ways teachers can get better, [and] is relevant to all types of classes?” Holcomb-McCoy said. “We really wanted to … get all these pieces of the puzzle in our approach.”
In March 2019, the task force sent a memo to the Faculty Senate laying out three ways forward: the status quo, a full adoption of the new portfolio system without continued use of the SETs, or a combination of the old system and the new.
The Senate adopted the third option, and in April 2019, then-Provost Dan Meyers asked the deans and department chairs in every school to begin implementing the recommendations.
One of the most important parts of the new system, Baehler said, is that the departments can now choose which items from the menu of evaluation options they want to use.
According to the final task force report, those options are incredibly varied. They include: teaching statements; annotated syllabi; professional teaching development for professors through CTRL; examples of feedback given to students; written self-evaluations; peer assessments, including classroom observation, in-person and by video, and the review of course materials; student assessments, drawn from a student observer committee, focus groups, and the narrative portion of the SETS; and the old SETs themselves.
Those SETs are also changing. Karen Froslid Jones, the assistant provost of Institutional Research and Assessment, said that the survey instrument itself is now much more streamlined and accessible.
The rating system has switched over from 1-7 to 1-5, the open-ended questions are shortened and relegated to the end of the evaluation, and the overall numerical analysis is shifting from looking at the mean to the median score, she said. These changes are part of an effort to increase the response rate, which peaked when the University still used physical evaluations. According to data from the response rate reports from previous years, only 57 percent of SETs were filled out in fall 2019, and 55 percent were completed in the spring of 2019.
This low response rate means that the SETs are not very representative, which hinders the tenure consideration process, Froslid Jones said.
While the change to the SETs is independent of the work of the task force, Froslid Jones said, the adjustments dovetail nicely, with the shorter SETs being complemented by the variety of other evaluations.
Beyond general performance analysis, the SETs are vital for professors up for promotion, the renewal of a contract, a pay increase or tenure. They can also be used to trigger reviews of a professor’s teaching if students express significant concern.
Due to pandemic-related adjustments, most of the functions that SETs effect, like merit-based salary increases and retirement-fund matching, have been suspended, Eisenstadt said. This renders them less vital than normal, and more useful as a tool to address bad behavior than a reward for good teaching.
During a normal year, SETs are also vital for judging how a professor has evolved over time, he said.
“Getting a PhD teaches you how to research, not teach,” Eisenstadt said. “At AU, you get hired for both.”
Typically, teaching is a problem in a professor’s first couple years. When those same professors go up for tenure consideration, the committee wants to see an arc of improvement from their first few years. SETs, because of their quantitative bend, make that analysis easy.
“That means they won’t ever go away,” Eisenstadt said.
Kim said that adjuncts are exempt from the new system, while term faculty only need to update their portfolio for promotions, tenure consideration and similar actions.
In general, he said, faculty welcome the change. Some are wary of the ensuing workload, which is why so many schools lean heavily on SETs, but in general the more progressive style is popular across the board.
Kim and Holcomb-McCoy also said that students' voices stand to gain more representation in the learning environment, with changes like the student observer committee and focus groups institutionalizing feedback.
While student thoughts on the overhaul itself haven’t been officially surveyed yet, due to the delay caused by the pandemic, Holcomb-McCoy said that anecdotally, the changes seem to be quite popular among both students and faculty.
“You never stop learning how to be an effective teacher,” she said. “We wanted to take what we know about teaching and apply it to our own teaching in the School of Education. Makes sense, right? This is what we’re teaching our students to do, so we should be doing it too.”