brand osRMss banner unframed.png

Welcome! This website has all the materials I use for teaching research methods within the Psychology major at Pitt. I don't use a textbook. I'm making the entire course materials public, and calling this project Open Source Research Methods for the Social Sciences (osRMss). I hope that you find the content useful. I look forward to getting feedback and suggestions and to continually improve osRMss over time.

This content is made available in full through this Canvas site. In addition, a new feature is that the activities are now available with immediate feedback on both TopHat and H5P! If you haven't heard of H5P it is a way to create, share, and reuse interactive HTML5 content for teaching purposes. H5P content can be embedded in many of the most common learning management systems and it is open-source. A huge benefit of TopHat H5P over the Word Docs is that multiple choice questions have immediate feedback with explanations. Other sorts of questions often have feedback with links to videos. The TopHat and H5P versions were designed to be nearly identical so you have your choice. The activities on TopHat and H5P are meant to be read alongside Canvas, not on their own. The canvas pages are still necessary for explaining many aspects of the activities in TopHat and H5P.

	Canvas	TopHat NEW!	H5P NEW!
How to access content	Modules link on left	TopHat link on the left, or click here and then... select 'enter as a guest' If it asks you to Finish enrolling in Top Hat, do not click 'Go to Canvas'; instead just close that pop up by clicking x at the top right.	Here
Includes Lecture Videos?	Yes - links to Panopto videos.	No	No
Includes Activities?	Activities are in MS Word. Feedback is provided with answer keys.	Yes, with immediate feedback.	Yes, with immediate feedback.
Includes "Activity Info" (objectives) for Lectures and Activities and additional Instructions?	Yes	No	No
Includes Dowloadable files that to be used for assignments.	Yes	Yes	No
How to get materials into your own Learning Management System if you want to adopt or adapt.	You can download the Word documents. If you are interested in the videos, contact me. If you use canvas and you want to import the entire course, click here. Be in touch if you want to do this but are having difficulty importing the course.	Please be in contact with me. I think that usually the best way to do this is to make a new class in TopHat, then invite me as an instructor, which should allow me to copy all of the activities over to your class. Note: when assigning activities to students, make sure to select 1 or 2 attempts per question (not unlimited), and select 'provide feedback' and 'show correct answer'. If you don't do all three things the feedback will not show up correctly.	When looking at an activity, scroll down to the bottom. On the left hand side there is a "Reuse" button. This will allow you to download as an .h5p file and then upload to your own h5p.com account or h5p server. If you are interested in using many activities, be in contact and I will provide a zip file of all of them.

Sample Schedules NEW!

Why consider adopting some osRMss content? See example activities. Reading this section is the best way to quickly learn about what is unique about this course.

Why consider adopting some osRMss content? See example activities.

When teaching research methods to undergrads, I take the perspective that much of social science is aimed at trying to uncover causal relations, and though everyone has heard of 'correlation does not imply causation', in reality this is much more nuanced and complex than it initially seems. Most textbooks tell students to 'control' for variables, but take it for granted that students can grasp this concept quickly. In reality, understanding when, why, and how to control for variables is quite complex, and in fact, these details were not worked out clearly until the latter part of the 20th century and begging of the 21st! Other concepts that are critical to research methods such as mediation and selection effects also need to have causal interpretations. For a great reference on how modern theory of causality has changed statistics and research methods, see Judea Pearl's The Book of Why. For all these reasons, I focus extensively on causality.

Below I provide six examples of the sorts of activities that students do and skills that they learn. The first three are examples relating to causality, and the latter three are examples of other topics taught.

Example 1: Students learn to use causal diagrams to represent the claims made in a study. For example, they learn to translate the claims made in an abstract into a causal diagram.
"People often face outcomes of important events that are beyond their personal control, such as waiting for an acceptance letter, job offer, or medical test results. We suggest that when wanting and uncertainty are high and personal control is lacking, people may be more likely to help others, as if they can encourage fate’s favor by doing good deeds proactively. Four experiments support this karmic-investment hypothesis by showing that wanting an outcome over which one has little control increases donations of time and money (Experiments 1- 2), but not participation in other rewarding activities (Experiment 1b). At a job fair, it causes job-seekers who feel the process is outside (vs. within) their control to make more generous pledges (Experiment 3). We further demonstrate that karmic investments increase one’s optimism about a desired outcome (Experiment 4). Finally, we discuss the role of personal control and magical beliefs in this phenomenon." (Converse, Risen, & Carter, 2012)

Example 1: Students learn to use causal diagrams to represent the claims made in a study.

For example, they learn to translate the claims made in an abstract into a causal diagram.

from this abstract

create this diagram

"People often face outcomes of important events that are beyond their personal control, such as waiting for an acceptance letter, job offer, or medical test results. We suggest that when wanting and uncertainty are high and personal control is lacking, people may be more likely to help others, as if they can encourage fate’s favor by doing good deeds proactively. Four experiments support this karmic-investment hypothesis by showing that wanting an outcome over which one has little control increases donations of time and money (Experiments 1- 2), but not participation in other rewarding activities (Experiment 1b). At a job fair, it causes job-seekers who feel the process is outside (vs. within) their control to make more generous pledges (Experiment 3). We further demonstrate that karmic investments increase one’s optimism about a desired outcome (Experiment 4). Finally, we discuss the role of personal control and magical beliefs in this phenomenon." (Converse, Risen, & Carter, 2012)

Example 2: Students learn to translate graphs, regression results, and causal diagrams.

For example, given a set of data, if you were to graph the data or run this regression, figure out the causal diagram.

from this hypothetical graph

or these hypothetical

regression results

figure out the correct

causal diagram

0 Homepage Depression Scatterplot.png

Regression 1:

Depression ~ Anxiety

Predictor	B	p
Intercept	0	.964
Anxiety	.197	.008

Regression 2:

Depression ~ Anxiety + Insomnia

Predictor	B	p
Intercept	-1.5	<.001
Anxiety	-0.5	<.001
Insomnia	3.0	<.001

Which of these causal diagrams

could explain the data?

Intro Example Scatterplot Regression Diagram.png

Here's the answer!

Example 3: Students learn to identify types of variables in studies.

The two videos below are examples of informal studies, the kind done commonly in everyday life. These sorts of informal studies have many similarities with formal research studies (though also some differences, for example, usually it is just a single case rather than many bases being studied). As practice for identifying different sorts of variables in studies, watch the videos and try to answer the following questions:

identify the independent variable and dependent variable
identify some constants
identify some confounds
are there any 'irrelevant differences' - differences between the two conditions that you think could not possibly influence the outcome?
do you have a hypothesis about the mechanism for the difference in the car video?

More broadly, this class teaches students to conceptualize the roles that different sorts of variables play in experiments with causal diagrams. These roles include the mechanism, selection effect and design confounds, irrelevant differences, noise variables, and constants.

0 Homepage Types of Variables.png

Example 4: Students learn to choose the right statistical test to answer a given question.

Often when students learn statistics they get taught how to do low-level procedures like calculating a t statistic and p-value, but often miss out on higher level conceptual knowledge and skills. In this activity students get taught how to choose and then practice choosing the appropriate statistical test for a given question.

Question: You are studying the relationship between financial security and voting behavior. Specifically, participants rate how financially secure they feel on a 1 to 10 scale, and you want to use this to predict who they intend to vote for – they choose between the democratic candidate or the republican candidate.

What statistical test would you choose? Students are taught how to read charts like the one below to help them identify the correct statistical test. (Note: MIC stands for Metric / Interval / Continuous.)

0 Homepage Stat Test Chart.png

The answer is Logistic Regression. The IV (financial security) is MIC and the DV (voting choice) is binary.

Example 5: Students compare individual research articles to meta-analyses.

Something confusing for the public about science is that new studies often seem to contradict established findings. If this is misunderstood, it can lead to mistrust of science. The goal of this activity is to see how science gets built through multiple studies, and also to practice assessing the quality of evidence.

In this activity students pick a specific research question, for example, Does Eye Movement Desensitization and Reprocessing, EMDR, help improve PTSD symptoms? They find an article and answer a set of guided questions to assess the quality of the article:

Identify the IV and DV of the study
Whether it was an experiment / randomized controlled trial or an observational study
If you couldn't find a RCT, do you have ideas why?
If you thought the sample size was adequate
The participants, exclusion criteria, humans vs. animals
The journal and how reputable you thought the journal was
If you are concerned about attrition
Whether the study was blinded
Any other major concerns or limitations
In the end, they assess the article on the following matrix to assess the quality of evidence, somewhat similar to various approaches of assessing evidence, such as Cochrane's Grade approach.

Then students look for a meta-analysis from the Cochrane Library and discuss the following questions with their group:

What did the meta-analysis conclude - was it the same as what you concluded?
Was your study in the meta-analysis and what were the inclusion criteria for the meta-analysis
If it was excluded - why
Did you learn about any other issues about your study from the meta-analysis?
Do you agree with Cochrane's conclusion?

Example 6: Students learn to identify the factors that make scientific research newsworthy or not.

There is a ton of research covered in the news, of varying quality. The goal goal of this assignment is to help students quickly identify what content they should trust and be focusing on and what they should ignore based on a wide variety of factors relating to different types of validity.

Take a look at this NYTimes article covering this study on a particular form of exercise for building muscle. Do you think that this research is newsworthy? Why or why not?

NYTimes article

Here is my assessment and how I teach the class to think about newsworthiness.

How might osRMss be helpful for you?

New to teaching research methods?
- osRMss provides ready-to-use lectures, homework, and class activities
- or you can adapt these materials however you see fit
Want some inspiration for teaching research methods? Want to change the objectives of your course?
- I find it can be super helpful to see how other people teach, and it is better to see an entire semester of content rather than just one activity here or there that is decontextualized. I hope that both the style of teaching (active) and content (emphasizing the relationships between causality, research design, and statistics) might be inspirational.
- For another research methods course that I found inspirational, see Morton Ann Gernsbacher's Open Access Research Methods.
Want to use more active-learning exercises in teaching?
- If you are interested in switching from a more lecture-oriented to a more active-learning style of teaching, osRMss provides tons of small-group activities. Personally, I have found that the more active-learning I do and the less lecturing, the more I like teaching! (That said, many of the activities can be done individually.)
Want to see examples of me teaching?
- I find it really hard to use someone else's lecture slides because usually there is a lot of extra content that is not built into the slides. All of my lectures are recorded because they are used as 'flipped' homework assignments. Most of my classes are also recorded, so you can see how I introduce activities and explain answers. (These were mainly recorded when I was teaching virtually, but it works the same way in the classroom.)

What sorts of students and settings is osRMss aimed for?

How I am currently using osRMss content
- This class was designed for teaching a large (e.g., 140) classes of undergraduates, most commonly juniors who are intending to become psychology majors, and who have taken one introductory statistics class. Because I teach large courses, the grading is entirely multiple choice and credit/no credit; I make a lot of use of giving answer keys after assignments and going through answer keys rather than providing individual feedback.
- I use these materials to teach both a more 'traditional' course (mainly lecture during class time with homework activities) as well as a 'flipped' course in which students watch video lectures at home and do some other activities at home and then work in small groups in class. See above for suggested schedules for the two types of courses.

Which disciplines are this class designed for?
- Though I teach research methods for students majoring in Psychology, I have designed this class to be broadly applicable to the social or behavioral sciences - anything that studies the behavior of humans. For example, many of the examples come from behavioral medicine. There is a large focus on observational vs. experimental (randomized control trial) methods. I believe that many components of this course could be directly used in other disciplines or easily adapted with different examples.
- I find that pre-med students often get a lot out of the class; the material is especially relevant to the MCATs section on critical analysis and reasoning skills.
- I also think that a lot of this material is relevant to courses on critical thinking.
- osRMss does not provide many resources for teaching a class on qualitative methods. The main focus is quantitative, not qualitative research.
Is this class designed primarily for students going into research?
- No! The main objective for the class is to improve critical thinking skills. Only about 3% of psychology students go on to get a PhD in Psychology nationally and this is true at Pitt as well. And a fairly small percent go on to do an honors thesis at Pitt . So this course is not designed only for students going on to a career in research. At the same time, I suspect that many of the skills taught in this course (e.g., designing surveys, doing some basic data interpretation even if informally, really understanding causality vs. correlation, understanding what it means to control for a variable, interpreting null results) will be useful in many jobs and in life more broadly. My focus on causality and statistics at a qualitative level is my way of helping students develop these critical thinking skills.
Is statistics knowledge a prerequisite?
- There is a lot of material in this class aimed to develop a conceptual understanding of statistics. Having one introductory statistics class is important to be prepared for many of these materials. Very few of my students have been taught regression or 2x2 ANOVA beforehand, but assumes only that students have been introduced to bivariate statistics. That said, many students still struggle with basic statistics concepts after one introductory course, so this course assumes a fairly minimal amount of statistical knowledge. Instead of trying to reteach statistics, this class focuses much more on the conceptual ideas behind statistics such as what it means to statistically control for variables rather than the mathematical and computational aspects.
This material seems really hard - I don't think my students can do it!
- I have only taught with these materials at the University of Pittsburgh. The class is challenging, but most students do well. I think that often we shy away from teaching concepts we think might be hard (e.g., mediation, moderation/interaction), but I have found that these actually really aren't that hard for students. Furthermore, I believe that if given scaffolding, students will rise to the challenge. I would be happy to chat about this concern (or others) and how you might think about adapting the materials in ways that would work for your students.
Could these materials be used for small classes?
- I think that these materials would work even better for small class sizes than large. In small classes having teaching assistants would not be necessary, and it might be possible to provide students or groups with more feedback on assignments.
Could these materials be used for even larger classes?
- When I teach with 140 students, I like to have three teaching assistants, other undergrads who previously took the course, walk around during the activities and help the small groups past challenges and model how to engage in discussion. I do think that these activities, even the in-class small-group work could be used for larger classes but it would be very helpful to have more teaching assistants, and to have a classroom with enough aisles so that teaching assistants can talk with the groups.
Could these materials be used for graduate students?
- I have used some of materials, particularly on regression, for guest lecturing in a graduate statistics class. I have also have had graduate students sit in on class, either as teaching assistants, or to learn how to teach research methods, and they say that they have learned a lot. Because these materials teach research methods in a fairly different way (focusing on causality), I think that many of these materials could also be used for a graduate course. But also some of these materials would probably be too basic, so it would make sense to pick and choose.
Can I use single activities or do the activities build on each other?
- Almost all of the content in this class is designed to build off of other content.
- There are some activities that are fairly independent, or primarily just build off of the lecture that a student is supposed to watch for homework the same day.
- That said, much of the content builds off of concepts introduced much earlier in the semester. Just as some examples, I introduce power, effect size, and false positives early in the semester, and come back to these repeatedly. I teach about regression, controlling for variables, and causality early in the semester, and these concepts show up in many later activities. Many later activities were built around examples specifically designed to come back to concepts introduced earlier.
- In sum, don't assume you can easily incorporate a single activity into your class - you might be able to but might not. However, you may still find the activities as a useful starting point to adapt for your purposes.

Copyright, Citing, and Branding

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. This license gives you lots of flexibility to share and adapt the materials provided that you give attribution, do not use them for commercial purposes, and use the same license if you share them.
You can cite this work in APA style with the following:

Rottman, B. M. (Year, Month Date of retrieval). Open Source Research Methods for the Social Sciences. https://canvas.pitt.edu/courses/124970

Another way to give credit is by incorporating our branding. Right now the brand is new so I have not embedded it in any of the videos, presentations, or documents. However, it would be great if you could incorporate it when using these materials, and especially if giving presentations so that others can learn about it. Download all the logos here. Contact me with any questions! Here are examples of the brand:

brand osRMss sign framed.png

Acknowledgements

Some of these materials were created in collaboration with undergraduate and or graduate students. In those cases, the names of those students are listed in the materials. More broadly, I want to give a huge thanks to all the grad and undergrad teaching assistants and other instructors who have provided feedback on these materials, and to all the students who have provided feedback and granted me patience while modifying the materials over time.

Questions, Concerns, and Suggestions

Tests: If you are interested to see my tests, email me.
Can't find something? If you feel like there should be something available (e.g., a video of a class, powerpoint slides for a lecture, answer key for a homework) and you can't find it, email me - I probably just forgot to include it!
Please contact me! If you are using any of the materials I would love to hear from you! Email me, Ben Rottman, at rottman@pitt.edu.
More about me: Check out my website to learn more about me and my lab at the University of Pittsburgh.

This material is based upon work supported by the National Science Foundation under Grant Numbers 1651330, and 1430439, and two grants from from the University of Pittsburgh's dB-SERC program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Table of Contents