An incorrect answer that is provided as an option in a question (such as a multiple choice question) is called a distractor.
When writing a question, you should always keep track of:
- question type
- the exact text of the question / instructions
- the exact text of the correct answer(s) (when applicable) or otherwise designate the correct answer
- the exact text of the distractors (when applicable)
- the exact text of any feedback statements you want to provide learners based on their answer
- which learning objective(s) the question aligns to
- any images or media you need to include in the question
Multiple choice questions should have at least three distractors when at all possible.
True/False questions are just multiple choice questions with only one distractor. In almost all cases, they're not the best choice of question type. For one thing, if a learner guesses, they have a 50% chance of being correct. And in cases where you allow more than one attempt on the quiz, they then don't have to go back to the material and learn why their answer was incorrect and figure out what the correct answer is; they then know what the answer is because it's the only other option.
Think carefully about your distractors: are there situations in which they would actually be a correct answer to the question? You may need to list assumptions in the question so that the learners don't become confused by these edge cases. For example, "For the purposes of this question, assume that the patient is not hypoglycemic."
Always allow yourself to be creative about your use of the question types. For example, the Ordering question type was meant to ask learners to put things in order: in numerical order, in chronological order, in order of size, in alphabetical order etc. But you can just as easily give learners a bank of words and ask them to assemble a sentence using them.
Be extremely precise in your wording. Autogradable questions are designed to allow asynchronous assessment, so if someone is confused about what you are asking, you may not be immediately available to answer their question. Clarity in the language you use can prevent some (though not all) confusion.
Format your questions and instructions the same way each time so that learners don't have to spend extra effort understanding different instructions meaning the same thing. For example, for any question in which the learner may choose more than one answer, I always add "Select all that apply." to the end of the question, in exactly those words.
Have someone else proofread your assessment questions after you've completed a good draft. (More than one someone, if possible!) They might misinterpret something you wrote, allowing you to change how you worded it to prevent that same misinterpretation from a learner later on. Or, they may have an extremely difficult time answering a question you thought was easy, despite their being an expert on the subject.
Sometimes you can assess a more complex skill by giving a series of linked questions. While Canvas does not allow you to ask a different next question based on a learner's response to the first question, other platforms (such as Qualtrics) do allow for this sort of logic branching. Even so, you will sometimes benefit from asking a few questions that "go together."
Always consider whether you can and should follow up a question with a second question asking the learner about their reasoning on the first question. For example: "1) Which is the best choice of tool for this purpose? 2) Why did you choose the tool you did? Choose the statement(s) that factored into your reasoning." It has been shown that asking a learner to reflect on their reasoning improves their learning, even if they must choose their reasoning from a predetermined list of options instead of writing a paragraph about it in their own words.
Don't give away the answer to a previous question later in the same quiz. (Except for uncommon situations where you've decided that learners should not be allowed to go back and change their answers once they've progressed past a question.)
Provide detailed feedback on incorrect answers - explain why the answer is incorrect and provide suggestions for resources to review on that topic. Do be aware that if you are allowing learners to take the quiz again after receiving feedback, you may want to avoid giving feedback that makes it obvious which answer is correct.
If your answer choices have a meaningful order (for example, numbers 15, 150, 1500, 15000), try to list them in this logical order to avoid meaningless difficulty.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright 2022, University of Pittsburgh. Some rights reserved.
Developed by Sera Thornton of the University of Pittsburgh, University Center for Teaching and Learning.