Implementing a Parsons-style Exam Problem

This guide describes the approach taken to write the problem demo-uni-finalexam-unscramble-lines, which is a variant of a Parsons Problem.

If you cannot access this problem you need to request read access to our "Demo University" content via uni-support@groklearning.com.

What is a Parsons Problem?

A Parsons Problem asks a student to re-arrange lines of code to implement the required program behaviour. There are several variants - a good summary of Parsons Problems can be found in A Review of Research on Parsons Problems by Du et al..

Implementation

The variant described here makes some design choices, either for ease of implementation or for pedagogical reasons.
  • For simplicity, the editor allows students to edit the lines of code.
  • All lines are provided without indentation, and the student is required to indent the each line as appropriate (this is mostly relevant for languages with whitespace semantics, such as Python, but could also be required on the basis of good style).
  • To avoid hinting from the syntax highlighting (e.g., about indentation or loop constructs), a plain-text editor is used.

Problem Description Content

The problem content should include a description of the desired behaviour. It could include examples of expected outcomes. To prevent students editing or running code, examples should be use the readonly and norun flags.

```norun;readonly<br>import csv<br>import itertools<br>```

The lines of code that the student should rearrange can be presented in the content panel as a reference, either as a single code block or as individual lines. Either way, use the readonly and norun flags as above.

Guidelines

It's useful to include some specific instructions to the student about how to complete the problem in Grok. E.g.,

> ## Guidelines 
> The lines of the function have been provided for you in the editor panel. Cut and paste the lines to rearrange them into the correct order, and indent them appropriately.
> * To submit, click the Mark button.
> * You can submit as many times as you like, but you will be assessed on you last submission.
> * If you encounter problems with the editor you can reset the editor file.

which is displayed as

Editor Workspace

The intended outcome here is to provide a text editor containing the lines to unscramble, plus optionally some read-only lines, as shown here:

Disable Running

We currently don't have plain-text problem types. Instead, you can prevent the Run operation from working by creating the default file ( program.py in the case of a Python problem) and making it hidden. For extra security, make it explicitly undeletable, unrenamable, and read-only. Even though the run operation is disabled, it is good practice to put some content into the file, such as an innocuous print statement.

Main submission file

By convention, the file into which the student enters their answers is called submission.txt. The .txt suffix instructs the editor not to use any syntax highlighting. This file should contain the scrambled lines that the user is required to rearrange.

Read-only lines

If you want to include some lines in the submission file that the student can't edit, you can use the text-marker API (see Read only and other text markers). In the example here, the imports are fixed at the top of the page. In the file settings the File Metadata (note, the Show advanced options? flag must be clicked at in the admin page for this to be visible) is set as shown:

This specifies that the first three lines (lines 0 to 2, inclusive) are read-only (the default option for text marker regions). The extra options prevent the user from selecting anywhere on the these lines, which avoids the student accidentally adding text to them before or after the read-only characters.

Solution

Even if you don't intend to make solutions available to students, it's worth adding at least one version of a working solution to the problem, since this makes validation of any tests much easier.

Testing

Acknowledge submission

Students should receive positive confirmation that they have submitted something.

If this is the only test, you can configure the common options to always pass by default:

Then include a single validation test that simply displays a confirmation message:

Line order

Manual marking can be made easier by checking that lines are in the correct order (even if indentation is wrong). This can be checked with an Assess test to prevent the student getting feedback during the exam.

  • Test Case Label (e.g.): line-order
  • Test Case Type: Assess
  • "On Pass" Message: Checking line order.
  • "On Fail" Message/"On Error" Message: (Automatic "Regular" message generated from On-Pass message)
  • Files:
    1. This program dumps the lines of submission.txt to stdout.
      • File Type Normal In
      • File Path: program.py
      • File Content:
        import sys
        filename = sys.argv[1] if len(sys.argv)>1 else 'submission.txt'
        with open(filename) as s:
          for line in s:
            line = line.strip()
            if line!="" and line[0]!='#':
              print(line)
        			
    2. Reference output.
      • File Type: stdout
      • File Content: The correct solution to the problem as outputted by the above "program.py" script
  • Test Options: If the common options are Always Pass, override the Test Options for this test to be
    • Driver: Python 3 IO
    • Output Checker: Differ

Example Outcomes

Manual marking can also be assisted by making sure that the submission functions as expected, using a few example situations.

  • Test Case Label (e.g.): example-outcomes
  • Test Case Type: Assess
  • "On Pass" Message: Checking example outcomes.
  • "On Fail" Message/"On Error" Message: (Automatic "Regular" message generated from On-Pass message)
  • Files:
    1. This program dumps the output for the different test parameters, run through the code in submission.txt, to stdout.
      • File Type Normal In
      • File Path: program.py
      • File Content:
        import sys
        filename = sys.argv[1] if len(sys.argv)>1 else 'submission.txt'
        
        with open(filename) as f:
          exec(f.read())
        
        # This is just an example for this particular problem
        # Write something specific to your own problem here.
        # Note that the 'superheroes.csv' must also be included in the
        # test workspace
        for s in [('superheroes.csv', 1970)]:
          print(f">>> race_list('{s[0]}', {s[1]})")
          print(race_list(s[0], s[1]))	
    2. Reference output.
      • File Type: stdout
      • File Content: The correct solution to the problem as outputted by the above "program.py" script.
  • Test Options: If the common options are Always Pass, override the Test Options for this test to be
    • Driver: Python 3 IO
    • Output Checker: Differ