A Guide to the Scholarship of Teaching and Learning

sotl word cloud

When adopting new, innovative teaching practices, you may eventually wonder about the effects of what you have done.

  • What has the impact been of the changes you have made?

  • Have there been effects on student satisfaction? Student engagement? Student learning?

  • Do the new learning activities serve all students equally well, or do they benefit some students more than others?

When you attempt to explore these questions through careful inquiry, you are engaging in what is often called the scholarship of teaching and learning or SoTL (which rhymes with yodel).  

What is SoTL?

SoTL is a systematic investigation of a teaching/learning issue that is shared for review, dissemination and possibly some action that changes what is done in the classroom.  Along with traditional educational research and the learning sciences, SoTL helps to provide the evidentiary basis for classroom teaching practice in higher education. It’s valuable because findings from these studies frequently have strong external validity, which is to say that they can be extended to other, similar classrooms and settings.

SoTL is a systematic investigation of a teaching/learning issue that is shared for review, dissemination and possibly some action that changes what is done in the classroom.

However, educational research is a specialized sub-field within social science research, built on a body of theory and existing research, with its own techniques, tools, and practices. In order to engage productively in SoTL, most instructors, even those with considerable research experience in their own fields, will require some guidance to conduct a valid and reliable study that yields information useful for advancing instructional practice.

These pages attempt to provide that guidance by describing some of the main building blocks of a successful, informative inquiry into questions surrounding innovative teaching practices. The guidance we give here is not comprehensive — some topics, like data analysis, or the development and testing of measurement tools, are beyond its scope — and it is necessarily general in nature. For more specific questions about actual research projects, please refer to our educational research and evaluation services.

History

The current Scholarship of Teaching & Learning movement stems from the foundational work in Ernest Boyer’s Scholarship Reconsidered (1990) and Lee Shulman’s essays in 1987 and 1999.  In response to a critique of Academia as privileging its research mission over its teaching mandate, Boyer outlined four types of overlapping scholarship: discovery, integration, application, and teaching.  In 1999, Lee Shulman, in his essay “Taking Teaching Seriously,” added “learning” to the scholarship of teaching and the name – the scholarship of teaching and learning” or simply “SoTL” – came into being.

The Carnegie Foundation and the American Association for Higher Education (until its demise), both of which had been responsible for starting this debate, funded several initiatives to promote SoTL nationally.  The Carnegie Academy for the Scholarship of Teaching and Learning (CASTL), the AAHE Summer Academy and several smaller efforts took up the initial call by providing training, expertise, and incentives to develop new kinds of scholarly efforts.  In 2002, the International Society for the Scholarship of Teaching and Learning emerged and it held its first conference in 2004. Dozens of refereed journals now publish SoTL either in disciplinary or multidisciplinary formats and many books on how to conduct SoTL have been published.    

Research questions and design

Research questions

Because research questions define a research project, this is the place to start your investigation. Different sorts of questions demand different types of research designs, methodologies, data collection methods, and analysis techniques.

Broadly speaking, your research questions should pose a unitary query using unambiguous terminology and they should be reasonably answerable given available resources.

What to avoid

  • Compound research questions. For example: "Do new learning spaces inspire faculty members to employ new active-learning techniques that enhance student motivation and learning?" This question contains several distinct strands that should be disentangled and listed as separate questions. 

  • Ambiguous terminology. For instance: “Are students more engaged after a service learning experience than they were before?” Student engagement is a popular concept in current educational discourse, but its meaning is not entirely clear. Do we mean affective engagement (improved feelings or emotions)? Cognitive engagement (increased intellectual interest)? Social engagement (greater interaction with peers or instructors around class issues)? Specifying a way of measuring engagement will often lead to greater precision in terminology. 

  • Overly ambitious questions. For example: “Does using a problem-based approach to teaching evolutionary theory in introductory biology courses cause greater student acceptance of evolution?” Causal questions are among the most difficult questions to answer because an affirmative answer must not only document change over time, but also justify attributing that change to the putative causal factor. This can be challenging, particularly if circumstances do not permit you to use a comparative research design that controls for extraneous factors.

Research design

The design of a study (or its methodology) refers to its overall structure and to the important components of the research contained within that structure.

A research design defines:

  • who or what is being studied

  • the framework within which the study's research questions will be addressed

  • the information to be gathered

  • whether there will be any manipulation of study conditions

  • what are the hypothesized relationships among the matters of concern in the study

One important distinction in research design has to do with whether a study involves a single group, or multiple groups. In educational research, these are likely to be a group or groups of instructors, or students, or classes. Studies that examine just one group are appropriate when what is being studied is a new and relatively unknown phenomenon. When that is the case, the study is likely to be exploratory and descriptive in nature, built around questions such as:

How do faculty members adapt their classes to employ new types of learning activities? What difficulties do they encounter? What benefits do they perceive?

Here, what is wanted is not a controlled experiment, but instead a detailed narrative that provides richness of context, which other faculty members can look to for guidance.

Studies that examine more than one group are appropriate when the objective of the research is to answer questions that call for comparative data, such as questions about change over time; about the association between two or more variables; or about cause and effect. The research designs appropriate for answering these questions are described in a detailed but accessible way in William Trochim’s Research Methods Knowledge Base.

 

Gathering data

Data-gathering (or measurement) can be done in many different ways, using different tools, depending on the type of data you wish to collect. One thing that is common to all forms of data collection is that they can be done well or done badly, and this can make a large difference to the success of a study. Even something as simple as a basic survey of student attitudes can provide high-quality data, but only if the survey is designed thoughtfully, so some effort to insure the quality of your measurement should be part of any educational investigation.

There are many ways of collecting data, each of which has advantages and disadvantages. Read more at the Web Center for Social Research Methods and in the Resources Tab.

Interviews

Simply speaking with people in a focused and intentional way can yield very useful information. Interviews excel at painting a richly contextualized portrait of a phenomenon from one person’s point of view, and they can range from tightly structured (using a predetermined set of questions in a specified order—an interview protocol--with each interviewee) to unstructured (a free-flowing conversation the direction of which is determined organically as the interview proceeds). Interviews generate large amounts of detailed data, which can be challenging to analyze.

Focus groups

Sometimes thought of as a subclass of interviews, focus groups involve speaking with several people in a group setting to generate detailed information from a variety of points of view. The interplay of perspectives can be quite revealing, particularly if the participants feel free to express both agreement and disagreement. Focus groups require skill to conduct successfully, because the group conversation must be guided and managed in a deft, tactful manner (Krueger & Casey 2014).

Surveys and questionnaires

Surveys are collections of questions that are usually delivered on paper or online to participants in a study. Due to the ease of constructing and implementing surveys, they are probably the most common measure used in educational research. However, it is easy to construct a survey that does not yield useful information because the questions are ambiguous, badly worded, or badly selected, or because of bias and error built into the response scales.

You can improve your survey design using the think-aloud technique, in which a few potential survey-takers are recruited to respond to a draft survey while speaking their thoughts aloud in the presence of a researcher who can use the feedback to improve the validity of the survey.

Tests, assignments, and other performance measures

Information about student learning is often provided by assignments associated with a class, or by standardized tests and other measures of student performance. When well-constructed, tests can yield important information about outcomes of great educational interest, but like surveys, tests that are developed in an informal way are often unreliable or lacking in other properties (such as consistency over time) that are necessary for them to produce data useful for educational studies (Thorndike 2005).

Class grades are usually compiled from a variety of assignments and are often used as indicators of student learning, but attention should be paid to their reliability and to the fact that they often have a distribution that is concentrated in a small area of the available scale.

Observations

Sometimes the phenomena you are interested in studying can be observed directly, such as the type and frequency of learning activities that occur in a class, the way an instructor moves through a classroom, or some aspects of student group interaction. When this is possible, observation has the advantage of providing large quantities of data that are not subject to the same biases that afflict self-reported data, although observers must be well-trained and aware of the biases they bring to the task.

Research, evaluation, and the IRB

After you have settled on research questions, a research design, and data collection methods, an important consideration is whether your investigation requires review by the University of Minnesota’s Institutional Review Board, or IRB.

The IRB reviews all studies at UMN that meet the federal definition of research to insure that participants in those studies are treated ethically. Broadly speaking, a study constitutes research if it is a systematic inquiry that involves human participants and is designed to produce generalizable knowledge.

Publication

Sharing your study and findings is at the heart of SoTL and creating a scholarly product is a key element of most academic inquiry.  

Journals

There are many peer reviewed outlets for SoTL studies.  Some journals are interdisciplinary and will consider work that is broadly applicable.  These include:

Many journals are discipline or subject based.  They are easy to locate using either the Kennesaw State University’s Teaching Journal Directory or the University of Central Florida’s SoTL Journal List.

Conferences

While many disciplines have teaching special interest groups (sigs) or threads, there are many conferences entirely or largely devoted to SoTL work.  For a complete list of those conferences, see the Kennesaw State University’s Teaching Conferences Directory.

Resources

Below are a few articles, books, and sites that might be helpful as you are planning your research project.  

Articles

Bartsch, R. A. (2013). Designing SoTL studies—part I: validity. New Directions for Teaching and Learning, 2013(136), 17-33.

Bartsch, R. A. (2013). Designing SoTL Studies—Part II: Practicality. New Directions for Teaching and Learning, 2013(136), 35-48.

Felten, P. (2013). Principles of good practice in SoTL. Teaching and Learning Inquiry: The ISSOTL Journal, 1(1), 121-125.

Hestenes, D., Wells, M. & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher 30, 141-151.

Klymkowsky MW, Underwood SM, Garvin-Doxas K. (2010). Biological Concepts Instrument (BCI): A diagnostic tool for revealing student thinking. arXiv.org. arXiv:1012.4501

Shulman, L. S. (1999).  “Taking teaching seriously.” Change. 31(4), 11-17.

Books

Bishop-Clark, C., & Dietz-Uhler, B. (2012). Engaging in the scholarship of teaching and learning: A guide to the process, and how to develop a project from start to finish. Stylus Publishing, LLC.

Boyer, E. L. (1990).  Scholarship reconsidered:  Priorities of the professoriate.  San Francisco, CA:  Jossey-Bass.

Dillman, D.A., Smyth, J.D. & Christian, L.M. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th Edition. Hoboken, NJ: John Wiley & Sons.

Krueger, R.A. & Casey, M.A. (2014). Focus groups: A practical guide for applied research. Thousand Oaks, CA: SAGE Publications.

McKinney, K. (2010). Enhancing learning through the scholarship of teaching and learning: The challenges and joys of juggling (Vol. 139). John Wiley & Sons.

McKinney, K. (Ed.). (2013). The scholarship of teaching and learning in and across the disciplines. Indiana University Press.

Pedhazur, E.J. & Schmelkin, L.P. (1991). Measurement, Design, and Analysis: An Integrated Approach. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Trochim, W.M.K. (2001). Research Methods Knowledge Base. Mason, OH: Thomson Learning.

Online Course

SoTL by Design - https://www.sotlbydesign.com/

Interactive Online Tool

Research Ethics Board User Guide on the Scholarship of Teaching and Learning Research - https://sites.google.com/view/sotl-user-guide/