Online Assesment Resources

Objective vs Subjective question types

Before you start using online assessment, it is important to consider what you want to achieve with the assessment. This will help you choose the tools and methods that will help you to achieve your assessment goals. A common misconception about online assessment is that your test will be automatically graded. Reducing marking time is also the most common incentive for starting to use online assessment. The fact is that only some question types are marked automatically by an online assessment platform such as an LMS. This is where objective vs subjective questioning plays an important role. The difference between these two question types are:

Objective question types require students to select a response from a list of alternatives provided to them or to supply a word or short phrase. Only one correct answer is identified. Subjective question types allows a student to organise an original answer, using their own words. These answers are usually longer (such as a paragraph or a couple of sentences).

Therefore, if the purpose for using online assessment is to reduce marking time and to have an online assessment program mark for you, you need to make use of objective question types. Subjective question types are not marked automatically by any platform. You will always need to mark subjective question types yourself, usually the marking can be alleviated by creating a rubric.

Home page

Why use online assessment

Methods and implementation ideas

Objective vs Subjective question types

Before you start using online assessment, it is important to consider what you want to achieve with the assessment. This will help you choose the tools and methods that will help you to achieve your assessment goals. A common misconception about online assessment is that your test will be automatically graded. Reducing marking time is also the most common incentive for starting to use online assessment. The fact is that only some question types are marked automatically by an online assessment platform such as an LMS. This is where objective vs subjective questioning plays an important role. The difference between these two question types are:

Objective question types require students to select a response from a list of alternatives provided to them or to supply a word or short phrase. Only one correct answer is identified. Subjective question types allows a student to organise an original answer, using their own words. These answers are usually longer (such as a paragraph or a couple of sentences).

Therefore, if the purpose for using online assessment is to reduce marking time and to have an online assessment program mark for you, you need to make use of objective question types. Subjective question types are not marked automatically by any platform. You will always need to mark subjective question types yourself, usually the marking can be alleviated by creating a rubric.

Objective question types

The most well-known and commonly used objective question type, is a multiple choice question which is a question with a number of options of which a student must select the correct or most correct option. There are, however, typically many other objective question types available to use on an online assessment platform such as Respondus or an LMS, which allow you to use a wide spectrum of different types of questioning in your test. In addition, online assessment provides several options for multiple choice questions that are not necessarily practical or feasible in a written format. All of the objective question types shown in this section will be automatically marked.

Multiple choice questions (MCQ)

A simple example of a multiple choice question is the following:

Which holiday is celebrated annually on 24 September in South Africa?

a) Heritage day *
b) Women’s day
c) Worker’s day
d) Boss’ day

You can do more with a MCQ online, however. For instance, consider using an image as part of the question, or a short video or audio clip (see examples here). Multiple choice questions can also be developed to test higher-order thinking.

Multiple response questions

A multiple response question is similar to a multiple choice question but instead of having to choose one option from a list, a student is required to choose more than one of the options. A simple example of a multiple response question is the following:

Which of the following elements are used to form water?

a) Carbon
b) Oxygen*
c) Chlorine
d) Helium
e) Hydrogen*
f) Nitrogen

As with a MCQ, you can incorporate images, videos or audio clips as part of your multiple response questions.

Matching questions

Matching questions are similar to match the column questions sometimes also used in written assessments. A simple example of a matching question is the following:

Match the capital cities in the column on the left, to the countries in the column on the right:

A matching question can be more difficult if there are some extra options in one column that do not have a match in the other column, such as in the following example:

Match the capital cities in the column on the left, to the countries in the column on the right:

In addition, matching questions do not only have to be questions with single words in each column. Instead, a single word or phrase can be used in one column to describe an explanation or description in another column. Another option is using images in one column and short descriptions in the other column. Matching questions are often used to test not only knowledge recall but also application of knowledge. Here are some examples of different types of matching questions. It is important to note that the assessment program (both Blackboard and Questionmark) automatically shuffles one of the two columns so that each students who completes an assessment sees a different order.

Ranking question

A ranking question requires a student to rank steps of a process or order of events in the correct sequence. An example of a simple ranking question is:

Select the correct order of the following significant events in world history:

Instead of one or two words, a short phrase or paragraph can also be used in the left column of a ranking question (to describe a step in a process for instance) in order to make the question more complex. It is important to note that the assessment program (both Blackboard and Questionmark) automatically shuffles the column on the left so that each students who completes an assessment sees a different order.

Select a blank question

A select a blank question is a longer question, usually in a paragraph format, with multiple answers. This question type is not commonly available on all online assessment platforms, but it is available on Questionmark, for example. An example of a simple select a blank question is:

Human Resources Management consists of various processes within an organisation. The process aimed at attracting applicants that match job criteria is known as [Recruitment/ Selection/ Hiring/ Remuneration Management/ Training and Development/ Performance Management/ Employee relations]. The aim of [Recruitment/ Selection/ Performance Management/ Remuneration Management/ Employee Relations] is to evaluate performance in order to reward good performance and address poor performance. [Recruitment/ Selection/ Hiring/ Remuneration Management/ Training and Development/ Performance Management/ Employee relations] is the process that involves determining salaries, incentives, and fringe benefits.

With this question students are provided with a list of possible answers for each ‘blank space’ in the question. This is an alternative to the better-known fill in the blank question. The advantage of this question, however, is that grading the correct answer is not influenced by incorrect spelling or synonyms of the correct answer. With a fill in the blank question students can easily make a spelling or typing error and will then not receive a mark as the system only recognises key words as identified by the lecturer as correct answers. Students also sometimes use synonyms for the identified key words, which is also not recognised by the system. The select a blank question eliminates these potential issues when a list of potential options is provided for each blank space in the question.

Hotspot question

A hotspot question allows a student to click somewhere on an image to indicate an answer. The system automatically gives a mark if the student clicks within a predetermined area but receives a zero if he/she clicks outside of this area. An example of a hotspot question is:

Refer to the image of a heart below. Click on the aorta:

Students who clicks anywhere in the green marked area will receive a mark and students who click outside of the area will receive zero. Students do not see the green marked area as part of the question, the image in the example above is to illustrate how the question is set up on the system (Blackboard/ Questionmark). As the example illustrates, this question type works well in modules where students need to visually be able to distinguish between certain areas such as in Physiology or Biology. However, it can also be used creatively in other types of modules as in these examples. It is important to note that this question type should be used with caution if you have students in your class who are visually impaired as the software on their computers will typically not read an image.

Drag and drop question

Drag and drop questions allow a student to drag and drop a number of labels to appropriate spaces on an image. A student can either get a single mark for placing all the labels at the correct spaces on the image or can get a mark for each label placed correctly. A simple example of a drag and drop question is:

Drag and drop the labels at the bottom of the image to the corresponding countries on the image. Place the label anywhere on its corresponding country:

Students do not see the colour-coded marked areas as part of the question, the image and labels in the example above is to illustrate how the question is set up on the system. Drag and drop questions also work well for an image of text such as matching a step in a process with a description. It is important to note that the labels at the bottom of a drag and drop questions should be short (preferably one or two words) as it must fit in next to each other at the bottom of the image. This question type should be used with caution if you have students in your class who are visually impaired as the software on their computers typically cannot read an image. This question type is not commonly available on all online assessment platforms but it is available on Questionmark.

Developing good multiple choice questions

It is important that a multiple-choice question is constructed well, before focusing on constructing questions that test higher order thinking. A multiple-choice question consists of a question (sometimes also referred to as the stem) and a number of options, of which only one is correct. Both the question and the options need to be constructed well to ensure the quality of a multiple-choice question. Since multiple-choice questions are often re-used, it is worth setting up good quality questions from the start. These questions can often be adapted to build a large question bank over time. The following guidelines are useful to consider in setting up multiple-choice questions.

***Table fits here (see last page of this document)***

Other guidelines to consider:

Avoid ‘all of the above’ and ‘none of the above’ options – It is often an easy option to eliminate. For instance, if students can eliminate one of the other options then they can also immediately eliminate ‘all of the above’. If you do want to use these options in some of your questions, ensure that it is sometimes also the correct option. Do not only use it to ensure that you have a certain number of options (e.g. 4 options or 5 options).
Statements that are correct but that do not answer the question can be strong distractors.
Avoid ‘always’ and ‘never’ options – these are easy options to rule out as extreme statements like this are often incorrect. Few things are universally true or false.
Do not create distractors that are so similar to the correct choice that it confuses students who really do know the answer. Distractors should be clearly different from the correct answer and not only in how it is phrased or what is emphasised.
Ensure that your options are mutually exclusive (that they do not overlap) – An example of overlapping options is:

How long does an annual plant generally live?

a) It dies after a year

b) It lives for many years

c) It needs to be replanted each year

d) It lives for 3 months

In this example, options a and c are not mutually exclusive. If a plant dies after a year it implies that it needs to be replanted every year. If this is the correct answer (as in this example) it technically means that both answers are correct. If both a and c were incorrect, it would have meant that a student would be able to eliminate both of these options and would only have two options left to consider.

Keep your options as short as possible – often, the more difficult a multiple-choice question becomes (see next section on creating questions that test higher order thinking) the longer the stem becomes. It is best practice to keep the options as short as possible to avoid the question becoming unwieldly.

You can use the below checklist (which includes the guidelines discussed in this section) before setting up multiple-choice questions. This checklist can also be used to assure the quality of your exiting question bank:

1. There is no irrelevant material in the question (stem) ☐

2. The question is asked in clear straightforward language ☐

3. The question is stated positively (or if not the negative part of the question is highlighted/ emboldened/ underlined so as to sufficiently stand out) ☐

4. As much of the question wording as possible is in the stem and not repeated in the options ☐

5. There are no grammatical clues in the question ☐

6. ‘All of the above’ or ‘none of the above’ options are avoided as far as possible (if not they are feasible options) ☐

7. ‘Always’ and ‘never’ options are avoided ☐

8. Distractors are not so similar to the correct option that it would confuse a student who knows the correct answer ☐

9. All options are mutually exclusive (none of the options overlap with any of the other options) ☒

10. Options are not unnecessarily long ☐

Bloom’s taxonomy

Developing multiple choice questions that test higher order thinking

Multiple-choice questions are the most commonly used objective assessment question type which means that it is a question type with a pre-determined single correct answer. While the usefulness of multiple choice questions in teaching and learning is undisputed, there is a common misconception that multiple-choice questions can only test the lower levels of Bloom’s Taxonomy such as remembering and understanding. It may be true that multiple choice tests should not be the sole assessment method in any module and that multiple-choice questions are not sufficient to test certain skills (such as writing and communication skills for instance), however, it is certainly not true that multiple-choice questions cannot be used to test higher-order thinking. It is important that you first ensure that you set up a good question before you focus on testing higher-order thinking.

In this section, four strategies for creating multiple-choice questions that test higher order thinking skills (at least at an application level of Bloom’s Taxonomy) is provided. Each strategy is explained with an example. These strategies are not discipline-specific.

Strategy 1: Two-tier multiple-choice question

In a two-tier multiple-choice question, the first tier pertains to a knowledge or comprehension statement and the second tier facilitates testing of higher-order thinking. It is in effect asking a lower-order question and then asking students to provide a reason (which requires higher-order thinking). Two-tier multiple-choice questions can also be seen as scaffolded questions.

Example:

Suppose you are given two clay balls of equal size and shape. The two clay balls also weigh the same. One ball is flattened into a pancake shaped piece. Which of these statements is correct?

The pancake shaped piece weighs more than the ball
The two pieces still weigh the same
The ball weighs more than the pancake shaped piece

Give a reason for your answer:

a) the flattened piece covers a larger area.
b) the ball pushes down more on one spot.
c) when something is flattened it loses weight.
d) clay has not been added or taken away.
e) when something is flattened it gains weight.
f) a and c
g) a and e

Strategy 2: Use real-world scenarios/ case studies

One of the best ways to promote and assess higher-order thinking is to use scenario-based questions. This changes a question from requiring simple recall to, at the very least, requiring the application of knowledge. This strategy particularly, can be used in most disciplines. You can also ask more than one multiple-choice question based on a single scenario/ case study.

Example

“Because of rapidly rising national defence expenditures, it is anticipated that Country A will experience a price inflation unless measures are taken to restrict the growth of aggregate private demand. Specifically, the government is considering either (1) increasing personal income tax rates or (2) introducing a very tight monetary policy.” If the government of Country A wishes to minimize the adverse effect of its anti-inflationary policies on economic growth, which one of the following policies should it use?

The tight money policy because it restricts consumption expenditures more than investment.
The tight money policy, since the tax increase would restrict consumption expenditures.
The personal income tax increase since it restricts consumption expenditures more than investment.
Either the tight money policy or the personal income tax rate increase since both depress investment equally.

Strategy 3: Analysis of visuals

Visuals as part of the question (stem) of a multiple-choice question, such as diagrams or graphs require higher-order thinking to solve. Interpreting a graph, for instance, in most instances require analysis as the student needs to understand the ‘whole’ as well as how individual parts function within the whole. Although graphs and diagrams may not form part of all disciplines, other visuals can also be used as part of a question to test higher-order thinking. For example, cartoons that depict a situation or certain behaviour can be used.

Without other data, which conclusion can you make from reviewing Figure 17?

The average American uses more drugs than citizens of any country except the United Kingdom
The average Mexican or Chilean consumes fewer drugs than citizens from other countries.
Americans are more likely than residents of other countries to use new drugs.
Japanese have regulations that make it very difficult to obtain new drugs

Strategy 4: Multi-logical thinking

A good strategy for testing higher-order thinking, is to develop questions that require multi-logical thinking. These are questions with more than one premise, from which a student must make the correct conclusion. It thus requires a student to take several facts and aspects in account before making a decision. In the example given, the student is required to both diagnose the symptoms and decide on what the best way would be to break the news to a parent.

Example

Tim’s second grade teacher is concerned because of the following observations about Tim’s behaviour in class:

Withdraws from peers on the playground and during group work
Often confuses syllables in words (ex: says mazagine instead of magazine)
Often confuses b and d, p and q, etc. when writing or recognizing letters
Loses his place when reading

The teacher has arranged a meeting with Tim’s mother to discuss these concerns. Which of the following statements is best for the teacher to say to Tim’s mother?

Tim needs extra practice reading and writing problematic letters and words at home at least 30 minutes per day.
Please discuss the importance of schoolwork to Tim so that he will increase his efforts in classwork.
These are possible symptoms of dyslexia so I would like to refer him to a specialist for diagnosis.
Please adjust Tim’s diet because he is most likely showing symptoms of ADHD due to food allergies.

Feedback to students

Using assessment results effectively

Item analysis

Considerations and challenges in the SA context

Challenge 1: Initial time investment

One of the biggest challenges of starting with online assessment for the first time is the time that it takes to design assessments for online delivery, to formulate assessment criteria and/ or rubrics, and to develop an overall assessment strategy for your module that includes new online assessments. The initial time investment required to start using online assessment is often underestimated.

How do I address this challenge?

It is important to plan for it. You will need to spend some extra time initially when you want to incorporate online assessment activities into your overall assessment strategy for a course for the first time. You need to make provisioning for the time it will take to plan your activities, compile your assessment criteria, and develop assessment activities.
Do not over extend yourself by incorporating too many new types of assessment activities at once. Often lecturers are excited to try different online assessment tools. You will need to spend some time planning and compiling assessments for each new tool you want to use and you may become despondent when you realise how much time it will take. Rather make a commitment to try one new tool each year and build on previous years’ knowledge and experiences.
Ensure that the time you spend initially is truly an investment by building up a question bank of questions that you can re-use and expand on year after year. If you create 20 questions on a topic, for instance, the first year that your students are completing online quizzes, you can add 10 more the following year and have a question bank of 30 questions. You also need to remember that although you spend some extra time to compile your questions initially, you will save time eventually when these questions are automatically graded.

Challenge 2: Students’ lack of computer/ technical skills

A common misconception is that all of our students are tech savvy and ready to use technology in their modules. Many of our students (especially first years) have not been exposed to computers or educational technology in school and need to master these tools for the first time at university.

How do I address this challenge?

Most universities have how-to guides for students on how to use the various online assessment tools that are available. It is important to make sure that your students have access to these documents and that they know where to find them. Spend some time in class by showing them where to find resources on how to use the online assessment tools they’ll be exposed to in your course. If proper how-to resources are not available on an institutional level, consider creating some step-by-step guides for your students that will show them exactly how to access and complete their online assessments in your course.
A ‘practice run’ is recommended before your students complete an online assessment for the first time. It can notably reduce queries from students while the assessment is open. A practice run is just a quick example assessment that does not count towards a semester mark but that allows students to familiarise themselves with the online assessment tool before having to complete their assessment.

Challenge 3: Cheating / student collaboration

If your students are completing an online assessment in an uncontrolled environment, it is possible that they may help each other or work together on the assessment. This may be undesirable for some types of assessment such as examinations or other summative assessments. Therefore, it is recommended that these types of assessments need to take place in a controlled environment such as a computer laboratory. However, for informal quizzes designed to help students to learn and grasp content it is not necessarily a problem if they collaborate on assessment tasks, as research shows that this still facilitates learning. There are also certain measures that you can put in place to limit meaningless copying of each other’s work though:

How do I address this challenge?

A question bank is an option. You can set up a question bank with 20 questions, for example, but each student only sees 10 questions. This means that students will not complete exactly the same assessment. A question bank must be used with caution, however, as you need to put measures in place to ensure that each test is of a similar difficulty level. To do this you need to group questions together that are on a similar difficulty level and then set up your assessment so that each student receives the same number of ‘easy’, ‘medium’, and/ or ‘difficult’ questions.
You can also randomise your test questions and options with most online testing platforms. This option is possible with a question bank but also if all of your students are completing the same test. If you, for instance, have a test consisting of 20 questions the option in which the questions are displayed can be randomised as well as the order in which the answer options are displayed.
The most effective way to deal with this challenge is having an appropriate time limit. Do not give your students too much time to complete an assessment. Especially if your test consists of objective assessment question types where they have to select a correct response without having to type an answer. They do not necessarily need as much time for objective test items as they would with a written test. Of course, the complexity of the questions you ask also need to be considered and even objective assessment question types can require students to reason and think before they’ll be able to answer, but ensure that the time limit of the assessment is appropriate for the number and types of questions you ask.
Another important measure to combat meaningless copying between students, is proper question design. The questions that you ask for an online assessment that will be completed in an uncontrolled environment cannot simply be questions for which students will easily find the answer in their textbooks or class notes. Informal online assessments, which contribute to an overall semester mark, are a great way to teach students to apply concepts to real-life scenarios.

Challenge 4: Dealing with student queries during and after an online assessment

Many lecturers who use online assessment for the first time underestimate the number of student queries they may receive. This is especially challenging when an assessment is completed in an uncontrolled environment. You may get queries from students who experienced internet connection problems, who were unable to complete the test before the deadline for various reasons, or who were uncertain on how to use the online assessment tool. The challenge not only lies in dealing with the number of queries but also in discerning when a query is valid or invalid.

How do I address this challenge?

The most effective way to reduce student queries during and after an assessment is proper communication before the assessment. Make sure that your students know exactly when they are expected to complete the assessment, that they are aware of any how-to or other information documents they need to consult prior to completing the assessment, and that the assessment criteria are clearly defined so that they know exactly what is expected of them. Also, ensure that students know in which instances they will not get a second chance to complete the assessment and be consistent in the way in which you deal with queries. It is a good idea to explain beforehand (preferably in writing via the LMS or in the study guide) to students that no late submissions will be accepted.
A ‘practice run’ is recommended to not only ensure that students are familiar with the assessment tool but also to reduce queries.

Challenge 5: Access to devices and the internet

Many South African students struggle financially and may not have access to their own personal mobile devices (such as laptops or even smartphones). Additionally, the high cost of data is a well-known challenge in the South African context. It is important to keep this in mind when students are required to complete any online activities, including online assessment.

How do I address this challenge?

Many students are dependent on institutional infrastructure, such as computer laboratories and Wi-Fi on campus to complete online assessment activities. Assessment activities that are open for an extended period (such as a week) allow students enough time to access campus resources. Of course, not all assessments are designed to be opened for an extended period. In such cases, it is recommended that, where possible, an appropriate venue is booked so that students will have access to required resources to complete their activities.
In addition to providing students with enough time to complete assessment activities, it is important to also give them enough advance notice of when and how online assessment activities in your course will be completed. This way students are afforded an opportunity to plan exactly when they will need to access which resources on campus if they do not have access to devices and or data at home.
Use graphics and videos, that require large quantities of data, to download sparingly. Online assessment offers opportunities for innovative assessment design such as the inclusion of a video instead of a written case study, for instance. However, the cost of data is prohibitive. There are several free software programs available to minimise video sizes Format Factory is one example. Another example is to insert a video into a PowerPoint presentation and to save the presentation as a video with a lower quality.

Institutional guidelines and rules

Guideline
Avoid unnecessary and irrelevant material in the question to avoid confusion and misdirecting the student.	Paul Muldoon, an Irish post-modern poet who uses experimental and playful language, uses which poetic genre in “Why Brownlee Left”? a. Sonnet b. Elegy c. Narrative poem d. Dramatic monologue e. Haiku	Paul Muldoon uses which poetic genre in “Why Brownlee Left”? a. Sonnet b. Elegy c. Narrative poem d. Dramatic monologue e. Haiku
Use clear and straightforward language in the question. Avoid making the question a reading comprehension rather than an assessment of the subject matter.	As the level of fertility approaches its nadir, what is the most likely ramification for the citizenry of a developing nation? a. A decrease in the labour force participation rate of women b. A dispersing effect on population concentration c. A downward trend in the youth dependency ratio d. A broader base in the population pyramid e. An increased infant mortality rate	A major decline in fertility in a developing nation is likely to produce which of the following? a. A decrease in the labour forces participation rate of women b. A dispersing effect on population concentration c. A downward trend in the youth dependency ratio d. A broader base in the population pyramid e. An increased infant mortality rate
Use negatives sparingly. If negatives must be used (as you may sometimes want to test knowledge about exceptions to a rule), capitalize, underscore, embolden or highlight it.	Weak question: Which of the following is not a symptom of osteoporosis? a. Decreased bone density b. Frequent bone fractures c. Raised body temperature d. Lower back pain	Which of the following is NOT a symptom of osteoporosis? a. Decreased bone density b. Raised body temperature c. Hair loss d. Painful joints
Put as much of the question wording in the stem as possible, rather than duplicating material in the options, which is difficult to read and makes the question look unwieldly.	Theorists of pluralism have asserted which of the following? a. The maintenance of democracy requires a large middle class. b. The maintenance of democracy requires autonomous centres of countervailing power. c. The maintenance of democracy requires the existence of a multiplicity of religious groups. d. The maintenance of democracy requires a predominantly urban population. e. The maintenance of democracy requires the separation of governmental powers.	Theorists of pluralism have asserted that the maintenance of democracy requires which of the following? a. A large middle class b. Autonomous centres of countervailing power c. The existence of a multiplicity of religious groups d. A predominantly urban population e. The separation of governmental powers
Avoid giving grammatical clues to the correct answer. As the example of the poor question shows, the question ends with the word ‘an’. Since only one of the options start with a vowel, it is the only plausible answer.	A fertile area in the desert in which the water table reaches the ground surface is called an a. Mirage b. Oasis c. Water hole d. Polder	What is a fertile area in the desert in which the water table reaches the ground surface called? a. Mirage b. Oasis c. Water hole d. Polder

Home page

Why use online assessment

Methods and implementation ideas

Objective vs Subjective question types

Before you start using online assessment, it is important to consider what you want to achieve with the assessment. This will help you choose the tools and methods that will help you to achieve your assessment goals. A common misconception about online assessment is that your test will be automatically graded. Reducing marking time is also the most common incentive for starting to use online assessment. The fact is that only some question types are marked automatically by an online assessment platform such as an LMS. This is where objective vs subjective questioning plays an important role. The difference between these two question types are:

Objective question types require students to select a response from a list of alternatives provided to them or to supply a word or short phrase. Only one correct answer is identified. Subjective question types allows a student to organise an original answer, using their own words. These answers are usually longer (such as a paragraph or a couple of sentences).

Therefore, if the purpose for using online assessment is to reduce marking time and to have an online assessment program mark for you, you need to make use of objective question types. Subjective question types are not marked automatically by any platform. You will always need to mark subjective question types yourself, usually the marking can be alleviated by creating a rubric.

Objective question types

The most well-known and commonly used objective question type, is a multiple choice question which is a question with a number of options of which a student must select the correct or most correct option. There are, however, typically many other objective question types available to use on an online assessment platform such as Respondus or an LMS, which allow you to use a wide spectrum of different types of questioning in your test. In addition, online assessment provides several options for multiple choice questions that are not necessarily practical or feasible in a written format. All of the objective question types shown in this section will be automatically marked.

Multiple choice questions (MCQ)

A simple example of a multiple choice question is the following:

Which holiday is celebrated annually on 24 September in South Africa?

a) Heritage day *
b) Women’s day
c) Worker’s day
d) Boss’ day

You can do more with a MCQ online, however. For instance, consider using an image as part of the question, or a short video or audio clip (see examples here). Multiple choice questions can also be developed to test higher-order thinking.

Multiple response questions

A multiple response question is similar to a multiple choice question but instead of having to choose one option from a list, a student is required to choose more than one of the options. A simple example of a multiple response question is the following:

Which of the following elements are used to form water?

a) Carbon
b) Oxygen*
c) Chlorine
d) Helium
e) Hydrogen*
f) Nitrogen

As with a MCQ, you can incorporate images, videos or audio clips as part of your multiple response questions.

Matching questions

Matching questions are similar to match the column questions sometimes also used in written assessments. A simple example of a matching question is the following:

Match the capital cities in the column on the left, to the countries in the column on the right:

A matching question can be more difficult if there are some extra options in one column that do not have a match in the other column, such as in the following example:

Match the capital cities in the column on the left, to the countries in the column on the right:

In addition, matching questions do not only have to be questions with single words in each column. Instead, a single word or phrase can be used in one column to describe an explanation or description in another column. Another option is using images in one column and short descriptions in the other column. Matching questions are often used to test not only knowledge recall but also application of knowledge. Here are some examples of different types of matching questions. It is important to note that the assessment program (both Blackboard and Questionmark) automatically shuffles one of the two columns so that each students who completes an assessment sees a different order.

Ranking question

A ranking question requires a student to rank steps of a process or order of events in the correct sequence. An example of a simple ranking question is:

Select the correct order of the following significant events in world history:

Instead of one or two words, a short phrase or paragraph can also be used in the left column of a ranking question (to describe a step in a process for instance) in order to make the question more complex. It is important to note that the assessment program (both Blackboard and Questionmark) automatically shuffles the column on the left so that each students who completes an assessment sees a different order.

Select a blank question

A select a blank question is a longer question, usually in a paragraph format, with multiple answers. This question type is not commonly available on all online assessment platforms, but it is available on Questionmark, for example. An example of a simple select a blank question is:

Human Resources Management consists of various processes within an organisation. The process aimed at attracting applicants that match job criteria is known as [Recruitment/ Selection/ Hiring/ Remuneration Management/ Training and Development/ Performance Management/ Employee relations]. The aim of [Recruitment/ Selection/ Performance Management/ Remuneration Management/ Employee Relations] is to evaluate performance in order to reward good performance and address poor performance. [Recruitment/ Selection/ Hiring/ Remuneration Management/ Training and Development/ Performance Management/ Employee relations] is the process that involves determining salaries, incentives, and fringe benefits.

With this question students are provided with a list of possible answers for each ‘blank space’ in the question. This is an alternative to the better-known fill in the blank question. The advantage of this question, however, is that grading the correct answer is not influenced by incorrect spelling or synonyms of the correct answer. With a fill in the blank question students can easily make a spelling or typing error and will then not receive a mark as the system only recognises key words as identified by the lecturer as correct answers. Students also sometimes use synonyms for the identified key words, which is also not recognised by the system. The select a blank question eliminates these potential issues when a list of potential options is provided for each blank space in the question.

Hotspot question

A hotspot question allows a student to click somewhere on an image to indicate an answer. The system automatically gives a mark if the student clicks within a predetermined area but receives a zero if he/she clicks outside of this area. An example of a hotspot question is:

Refer to the image of a heart below. Click on the aorta:

Students who clicks anywhere in the green marked area will receive a mark and students who click outside of the area will receive zero. Students do not see the green marked area as part of the question, the image in the example above is to illustrate how the question is set up on the system (Blackboard/ Questionmark). As the example illustrates, this question type works well in modules where students need to visually be able to distinguish between certain areas such as in Physiology or Biology. However, it can also be used creatively in other types of modules as in these examples. It is important to note that this question type should be used with caution if you have students in your class who are visually impaired as the software on their computers will typically not read an image.

Drag and drop question

Drag and drop questions allow a student to drag and drop a number of labels to appropriate spaces on an image. A student can either get a single mark for placing all the labels at the correct spaces on the image or can get a mark for each label placed correctly. A simple example of a drag and drop question is:

Drag and drop the labels at the bottom of the image to the corresponding countries on the image. Place the label anywhere on its corresponding country:

Students do not see the colour-coded marked areas as part of the question, the image and labels in the example above is to illustrate how the question is set up on the system. Drag and drop questions also work well for an image of text such as matching a step in a process with a description. It is important to note that the labels at the bottom of a drag and drop questions should be short (preferably one or two words) as it must fit in next to each other at the bottom of the image. This question type should be used with caution if you have students in your class who are visually impaired as the software on their computers typically cannot read an image. This question type is not commonly available on all online assessment platforms but it is available on Questionmark.

Developing good multiple choice questions

It is important that a multiple-choice question is constructed well, before focusing on constructing questions that test higher order thinking. A multiple-choice question consists of a question (sometimes also referred to as the stem) and a number of options, of which only one is correct. Both the question and the options need to be constructed well to ensure the quality of a multiple-choice question. Since multiple-choice questions are often re-used, it is worth setting up good quality questions from the start. These questions can often be adapted to build a large question bank over time. The following guidelines are useful to consider in setting up multiple-choice questions.

***Table fits here (see last page of this document)***

Other guidelines to consider:

Avoid ‘all of the above’ and ‘none of the above’ options – It is often an easy option to eliminate. For instance, if students can eliminate one of the other options then they can also immediately eliminate ‘all of the above’. If you do want to use these options in some of your questions, ensure that it is sometimes also the correct option. Do not only use it to ensure that you have a certain number of options (e.g. 4 options or 5 options).
Statements that are correct but that do not answer the question can be strong distractors.
Avoid ‘always’ and ‘never’ options – these are easy options to rule out as extreme statements like this are often incorrect. Few things are universally true or false.
Do not create distractors that are so similar to the correct choice that it confuses students who really do know the answer. Distractors should be clearly different from the correct answer and not only in how it is phrased or what is emphasised.
Ensure that your options are mutually exclusive (that they do not overlap) – An example of overlapping options is:

How long does an annual plant generally live?

a) It dies after a year

b) It lives for many years

c) It needs to be replanted each year

d) It lives for 3 months

In this example, options a and c are not mutually exclusive. If a plant dies after a year it implies that it needs to be replanted every year. If this is the correct answer (as in this example) it technically means that both answers are correct. If both a and c were incorrect, it would have meant that a student would be able to eliminate both of these options and would only have two options left to consider.

Keep your options as short as possible – often, the more difficult a multiple-choice question becomes (see next section on creating questions that test higher order thinking) the longer the stem becomes. It is best practice to keep the options as short as possible to avoid the question becoming unwieldly.

You can use the below checklist (which includes the guidelines discussed in this section) before setting up multiple-choice questions. This checklist can also be used to assure the quality of your exiting question bank:

1. There is no irrelevant material in the question (stem) ☐

2. The question is asked in clear straightforward language ☐

3. The question is stated positively (or if not the negative part of the question is highlighted/ emboldened/ underlined so as to sufficiently stand out) ☐

4. As much of the question wording as possible is in the stem and not repeated in the options ☐

5. There are no grammatical clues in the question ☐

6. ‘All of the above’ or ‘none of the above’ options are avoided as far as possible (if not they are feasible options) ☐

7. ‘Always’ and ‘never’ options are avoided ☐

8. Distractors are not so similar to the correct option that it would confuse a student who knows the correct answer ☐

9. All options are mutually exclusive (none of the options overlap with any of the other options) ☒

10. Options are not unnecessarily long ☐

Bloom’s taxonomy

Developing multiple choice questions that test higher order thinking

Multiple-choice questions are the most commonly used objective assessment question type which means that it is a question type with a pre-determined single correct answer. While the usefulness of multiple choice questions in teaching and learning is undisputed, there is a common misconception that multiple-choice questions can only test the lower levels of Bloom’s Taxonomy such as remembering and understanding. It may be true that multiple choice tests should not be the sole assessment method in any module and that multiple-choice questions are not sufficient to test certain skills (such as writing and communication skills for instance), however, it is certainly not true that multiple-choice questions cannot be used to test higher-order thinking. It is important that you first ensure that you set up a good question before you focus on testing higher-order thinking.

In this section, four strategies for creating multiple-choice questions that test higher order thinking skills (at least at an application level of Bloom’s Taxonomy) is provided. Each strategy is explained with an example. These strategies are not discipline-specific.

Strategy 1: Two-tier multiple-choice question

In a two-tier multiple-choice question, the first tier pertains to a knowledge or comprehension statement and the second tier facilitates testing of higher-order thinking. It is in effect asking a lower-order question and then asking students to provide a reason (which requires higher-order thinking). Two-tier multiple-choice questions can also be seen as scaffolded questions.

Example:

Suppose you are given two clay balls of equal size and shape. The two clay balls also weigh the same. One ball is flattened into a pancake shaped piece. Which of these statements is correct?

The pancake shaped piece weighs more than the ball
The two pieces still weigh the same
The ball weighs more than the pancake shaped piece

Give a reason for your answer:

a) the flattened piece covers a larger area.
b) the ball pushes down more on one spot.
c) when something is flattened it loses weight.
d) clay has not been added or taken away.
e) when something is flattened it gains weight.
f) a and c
g) a and e

Strategy 2: Use real-world scenarios/ case studies

One of the best ways to promote and assess higher-order thinking is to use scenario-based questions. This changes a question from requiring simple recall to, at the very least, requiring the application of knowledge. This strategy particularly, can be used in most disciplines. You can also ask more than one multiple-choice question based on a single scenario/ case study.

Example

“Because of rapidly rising national defence expenditures, it is anticipated that Country A will experience a price inflation unless measures are taken to restrict the growth of aggregate private demand. Specifically, the government is considering either (1) increasing personal income tax rates or (2) introducing a very tight monetary policy.” If the government of Country A wishes to minimize the adverse effect of its anti-inflationary policies on economic growth, which one of the following policies should it use?

The tight money policy because it restricts consumption expenditures more than investment.
The tight money policy, since the tax increase would restrict consumption expenditures.
The personal income tax increase since it restricts consumption expenditures more than investment.
Either the tight money policy or the personal income tax rate increase since both depress investment equally.

Strategy 3: Analysis of visuals

Visuals as part of the question (stem) of a multiple-choice question, such as diagrams or graphs require higher-order thinking to solve. Interpreting a graph, for instance, in most instances require analysis as the student needs to understand the ‘whole’ as well as how individual parts function within the whole. Although graphs and diagrams may not form part of all disciplines, other visuals can also be used as part of a question to test higher-order thinking. For example, cartoons that depict a situation or certain behaviour can be used.

Without other data, which conclusion can you make from reviewing Figure 17?

The average American uses more drugs than citizens of any country except the United Kingdom
The average Mexican or Chilean consumes fewer drugs than citizens from other countries.
Americans are more likely than residents of other countries to use new drugs.
Japanese have regulations that make it very difficult to obtain new drugs

Strategy 4: Multi-logical thinking

A good strategy for testing higher-order thinking, is to develop questions that require multi-logical thinking. These are questions with more than one premise, from which a student must make the correct conclusion. It thus requires a student to take several facts and aspects in account before making a decision. In the example given, the student is required to both diagnose the symptoms and decide on what the best way would be to break the news to a parent.

Example

Tim’s second grade teacher is concerned because of the following observations about Tim’s behaviour in class:

Withdraws from peers on the playground and during group work
Often confuses syllables in words (ex: says mazagine instead of magazine)
Often confuses b and d, p and q, etc. when writing or recognizing letters
Loses his place when reading

The teacher has arranged a meeting with Tim’s mother to discuss these concerns. Which of the following statements is best for the teacher to say to Tim’s mother?

Tim needs extra practice reading and writing problematic letters and words at home at least 30 minutes per day.
Please discuss the importance of schoolwork to Tim so that he will increase his efforts in classwork.
These are possible symptoms of dyslexia so I would like to refer him to a specialist for diagnosis.
Please adjust Tim’s diet because he is most likely showing symptoms of ADHD due to food allergies.

Feedback to students

Using assessment results effectively

Item analysis

Considerations and challenges in the SA context

Challenge 1: Initial time investment

One of the biggest challenges of starting with online assessment for the first time is the time that it takes to design assessments for online delivery, to formulate assessment criteria and/ or rubrics, and to develop an overall assessment strategy for your module that includes new online assessments. The initial time investment required to start using online assessment is often underestimated.

How do I address this challenge?

It is important to plan for it. You will need to spend some extra time initially when you want to incorporate online assessment activities into your overall assessment strategy for a course for the first time. You need to make provisioning for the time it will take to plan your activities, compile your assessment criteria, and develop assessment activities.
Do not over extend yourself by incorporating too many new types of assessment activities at once. Often lecturers are excited to try different online assessment tools. You will need to spend some time planning and compiling assessments for each new tool you want to use and you may become despondent when you realise how much time it will take. Rather make a commitment to try one new tool each year and build on previous years’ knowledge and experiences.
Ensure that the time you spend initially is truly an investment by building up a question bank of questions that you can re-use and expand on year after year. If you create 20 questions on a topic, for instance, the first year that your students are completing online quizzes, you can add 10 more the following year and have a question bank of 30 questions. You also need to remember that although you spend some extra time to compile your questions initially, you will save time eventually when these questions are automatically graded.

Challenge 2: Students’ lack of computer/ technical skills

A common misconception is that all of our students are tech savvy and ready to use technology in their modules. Many of our students (especially first years) have not been exposed to computers or educational technology in school and need to master these tools for the first time at university.

How do I address this challenge?

Most universities have how-to guides for students on how to use the various online assessment tools that are available. It is important to make sure that your students have access to these documents and that they know where to find them. Spend some time in class by showing them where to find resources on how to use the online assessment tools they’ll be exposed to in your course. If proper how-to resources are not available on an institutional level, consider creating some step-by-step guides for your students that will show them exactly how to access and complete their online assessments in your course.
A ‘practice run’ is recommended before your students complete an online assessment for the first time. It can notably reduce queries from students while the assessment is open. A practice run is just a quick example assessment that does not count towards a semester mark but that allows students to familiarise themselves with the online assessment tool before having to complete their assessment.

Challenge 3: Cheating / student collaboration

If your students are completing an online assessment in an uncontrolled environment, it is possible that they may help each other or work together on the assessment. This may be undesirable for some types of assessment such as examinations or other summative assessments. Therefore, it is recommended that these types of assessments need to take place in a controlled environment such as a computer laboratory. However, for informal quizzes designed to help students to learn and grasp content it is not necessarily a problem if they collaborate on assessment tasks, as research shows that this still facilitates learning. There are also certain measures that you can put in place to limit meaningless copying of each other’s work though:

How do I address this challenge?

A question bank is an option. You can set up a question bank with 20 questions, for example, but each student only sees 10 questions. This means that students will not complete exactly the same assessment. A question bank must be used with caution, however, as you need to put measures in place to ensure that each test is of a similar difficulty level. To do this you need to group questions together that are on a similar difficulty level and then set up your assessment so that each student receives the same number of ‘easy’, ‘medium’, and/ or ‘difficult’ questions.
You can also randomise your test questions and options with most online testing platforms. This option is possible with a question bank but also if all of your students are completing the same test. If you, for instance, have a test consisting of 20 questions the option in which the questions are displayed can be randomised as well as the order in which the answer options are displayed.
The most effective way to deal with this challenge is having an appropriate time limit. Do not give your students too much time to complete an assessment. Especially if your test consists of objective assessment question types where they have to select a correct response without having to type an answer. They do not necessarily need as much time for objective test items as they would with a written test. Of course, the complexity of the questions you ask also need to be considered and even objective assessment question types can require students to reason and think before they’ll be able to answer, but ensure that the time limit of the assessment is appropriate for the number and types of questions you ask.
Another important measure to combat meaningless copying between students, is proper question design. The questions that you ask for an online assessment that will be completed in an uncontrolled environment cannot simply be questions for which students will easily find the answer in their textbooks or class notes. Informal online assessments, which contribute to an overall semester mark, are a great way to teach students to apply concepts to real-life scenarios.

Challenge 4: Dealing with student queries during and after an online assessment

Many lecturers who use online assessment for the first time underestimate the number of student queries they may receive. This is especially challenging when an assessment is completed in an uncontrolled environment. You may get queries from students who experienced internet connection problems, who were unable to complete the test before the deadline for various reasons, or who were uncertain on how to use the online assessment tool. The challenge not only lies in dealing with the number of queries but also in discerning when a query is valid or invalid.

How do I address this challenge?

The most effective way to reduce student queries during and after an assessment is proper communication before the assessment. Make sure that your students know exactly when they are expected to complete the assessment, that they are aware of any how-to or other information documents they need to consult prior to completing the assessment, and that the assessment criteria are clearly defined so that they know exactly what is expected of them. Also, ensure that students know in which instances they will not get a second chance to complete the assessment and be consistent in the way in which you deal with queries. It is a good idea to explain beforehand (preferably in writing via the LMS or in the study guide) to students that no late submissions will be accepted.
A ‘practice run’ is recommended to not only ensure that students are familiar with the assessment tool but also to reduce queries.

Challenge 5: Access to devices and the internet

Many South African students struggle financially and may not have access to their own personal mobile devices (such as laptops or even smartphones). Additionally, the high cost of data is a well-known challenge in the South African context. It is important to keep this in mind when students are required to complete any online activities, including online assessment.

How do I address this challenge?

Many students are dependent on institutional infrastructure, such as computer laboratories and Wi-Fi on campus to complete online assessment activities. Assessment activities that are open for an extended period (such as a week) allow students enough time to access campus resources. Of course, not all assessments are designed to be opened for an extended period. In such cases, it is recommended that, where possible, an appropriate venue is booked so that students will have access to required resources to complete their activities.
In addition to providing students with enough time to complete assessment activities, it is important to also give them enough advance notice of when and how online assessment activities in your course will be completed. This way students are afforded an opportunity to plan exactly when they will need to access which resources on campus if they do not have access to devices and or data at home.
Use graphics and videos, that require large quantities of data, to download sparingly. Online assessment offers opportunities for innovative assessment design such as the inclusion of a video instead of a written case study, for instance. However, the cost of data is prohibitive. There are several free software programs available to minimise video sizes Format Factory is one example. Another example is to insert a video into a PowerPoint presentation and to save the presentation as a video with a lower quality.

Online Assesment Resources

Why use online assement?

Objective vs Subjective question types

Objective Question Types

Objective vs Subjective question types

Home page

Why use online assessment

Methods and implementation ideas

Objective vs Subjective question types

Objective question types

Developing good multiple choice questions

Bloom’s taxonomy

Developing multiple choice questions that test higher order thinking

Feedback to students

Using assessment results effectively

Item analysis

Considerations and challenges in the SA context

Institutional guidelines and rules

Further reading and other resources

Home page

Why use online assessment

Methods and implementation ideas

Objective vs Subjective question types

Objective question types

Developing good multiple choice questions

Bloom’s taxonomy

Developing multiple choice questions that test higher order thinking

Feedback to students

Using assessment results effectively

Item analysis

Considerations and challenges in the SA context

Institutional guidelines and rules

Further reading and other resources