Writing micro:bit & Arduino test cases
Note: this article mostly talks about the micro:bit marker, but the Arduino one works in almost the same way. The main difference being the various parts of the board that can be interacted with.
The micro:bit and Arduino markers use a Python script to run a simulated interaction with the student's program. The main idea is that the script can trigger events on the device (i.e. press buttons, toggle pins, set sensor values, etc) and check that the desired state changes (e.g. pin voltage, display, etc) happen within the correct amount of time.
The marker interacts with the simulator by exchanging JSON messages. A set of helper functions are available to the test script for generating these messages and waiting for messages back from the simulator.
Adding new test cases
The language for the problem should be set to "micro:bit MicroPython", which will mean that the default test driver is also set to "micro:bit MicroPython". It needs to also use the "Output JSON Checker". Each test case should have a single file of type "microbit test script", which is a Python script containing commands and expect.
Writing test scripts
Waiting for things to happen
The most important thing your test script does is wait for the device to get to the correct state. This is done using the expect function. Note that these functions are imported automatically for you.
# Tap the A button
tap('A')
# Ensure that Image.HAPPY is on the screen at some point in the next 100ms.
expect(ms(100), display(Image.HAPPY))
The expect function takes the following arguments:
- ticks -- number of device ticks to wait. On the micro:bit, a tick is 6ms, but use the ms() function to calculate the number of ticks.
- what -- the thing to wait for specified as a five-tuple of parameters. Use one of the helper methods (such as display()) to generate this five-tuple.
- message='' -- the message to display if this condition is not met.
- details=True -- whether or not to show additional information about why the condition was met (e.g. for display(), will show what was on the display when the timer ran out).
- recovery=None -- a hook for customising the error message with problem-specific information. See below for more information.
In many cases you'll only want to use the first two or three arguments.
Note: the simulators are not perfectly timing accurate, so you'll always want to leave a bit of timing slack in your test scripts. A simple example is that updating the display takes three ticks (18ms), but the simulator could be anywhere in that three-tick cycle when the display.show was initiated, so you have to wait for the next cycle to start. Additionally it takes a tick to register a button was_pressed, so it can be up to six ticks (36ms) to register a button press. Generally, try to avoid writing problems that need more than 50-100ms of precision for timing.
Some notes about expect:
- It stops waiting and succeeds if the condition becomes true at any point during the wait time, even if it immediately changes afterwards.
- By default it will generate additional information about the failure, relevant to the particular type of check. e.g. for display(), after timeout it will append to the message to say what was on the screen at the time of timeout, or for pitch(), what frequency was currently playing. You'll quite often want to turn this off using details=False.
There are some other variations on expect:
- expect_hold(wait_ticks, hold_ticks, what, message='', details=True)
This works like regular expect as it first waits wait_ticks for the what to become true, at which point it ensures that the condition stays true for the next hold_ticks ticks. - expect_not(ticks, what, message='', details=True)
This ensures that the what is initially not true, then ensures that it stays not true for the ticks. - expect_scroll(scroll_text, message=None, details=True, initial_ticks=None)
Special helper for detecting scrolling text. By default it waits 2500ms for each letter to appear, but you can customise the time to wait for the first letter using the initial_ticks argument.
Grok staff: see https://groklearning.atlassian.net/browse/GI-484 for discussion on improvements to these functions.
Expect helpers (micro:bit)
- display(leds)
Takes an array of LED brightnesses and verifies that's what's on the screen. There are a set of predefined constants matching the built-in micro:bit images, available as Image.FOO.
To check if the screen is blank, use Image.BLANK.
As with other helpers, you can also use the special value ANYTHING to check that at least one pixel is non-zero.
You can also built your own LED array, with 25 brightnesses between 0 and 9 (inclusive), and negative values to skip checking that pixel. i.e. to see that the top-left pixel is set, but ignore everything else, use display([9] + [-1] * 24). - display_char(c)
Checks that the specified letter is on the screen, but possibly with another letter following to the right (e.g. because of display.scroll). - music(freqs)
Checks that the last N tones played by the micro:bit match the list of frequencies. Use the predefined Tunes.FOO to match the built-in tunes in the music module. - pin_analog(pin, v)
Checks that the specified pin number has the specified analog (PWM) value. - pin_digital(pin, mode)
Checks that the specified pin number has the specified digital pin mode. (0 = low, 1 = high, 2 = pwm, 3 = floating. see GpioPinState for more) - pitch(pin, frequency)
Checks that the specified pin is currently in PWM mode with the specified frequency. - pixel(x, y, brightness)
Checks that a single pixel is set to the specified brightness. - radio_tx(msg, channel=7, base=0x75626974, prefix=0, data_rate=1)
Checks that the last frame transmitted matches the specified parameters.
Use msg=ANYTHING to check that any message was transmitted. Use msg=None to check that no message has been transmitted.
Note: expect() will repeatedly call this until it times out, so expect(ms(1000), radio_tx('foo')) waits until the message is transmitted (even if it had not been already transmitted when expect() was first called).
Controlling device inputs (micro:bit)
To control the inputs of the micro:bit use the following functions:
- accelerometer(x, y, z)
Sets the values for the three axes of the accelerometer (i.e. exactly the value that will be returned by accelerometer.get_x() on the micro:bit). - button(id, state)
Sets the state of the specified button. e.g. button('a', 1) to hold down the button. Note: you often want to use tap(), see below. - compass(x, y z)
Sets the values for the three axes of the compass (as for the accelerometer, corresponds to compass.get_x()) - drive_pin(pin, voltage)
Sets the pin number to the specified voltage, or use the strings 'high' and 'low' for 3.3 (or 5 on Arduino) and 0. - gesture(name)
Simulates the specified gesture name (e.g. 'shake'). - heading(degrees)
Sets the compass values to cause the computed heading to be the specified value in degrees. - radio_rx(msg, channel=7, base=0x75626974, prefix=0, data_rate=1)
Simulates receiving a radio message (i.e. the simulated code will see this in the next call to radio.receive()). Note the message is always a string message, there's currently no way to simulate a raw message. - random(next, repeat=-1)
Ensures that calls to random.randint() returns the specified value. Optionally for only the next repeat calls. - random_choice(count, result)
Forces a random choice to return a particular value. Ensures that random.choice() is called with a list of count items, and returns result. - tap(id)
Simulates a press and release of the specified button, and ensures that the timing is correct so that it will be detected. - temperature(celcius)
Sets the simulated temperature of the device (i.e. what temperature() will return). -
moisture_level(pin, level)
Sets the moisture level of the moisture probe on pin (i.e. what read_analog will return).
Expect helpers (Arduino)
- pin_digital
- pin_pwm
- pitch
- rgb
- serial_output
Controlling device inputs (Arduino)
- accelerometer
- button
- drive_mux
- drive_pin
- gesture
- joystick
- light_sensor
- microphone
- random
- random_choice
- serial_input
- serial_input_raw
- slider
- tap
- temperature
Useful constants (Arduino)
Whereas the micro:bit marker uses numbered pins, named buttons, and the strings 'high' and 'low, the Arduino marker has a set of constants for passing to the expect and input methods.
- SWITCH_1
- SWITCH_2
- SWITCH_3
- SWITCH_4
- SWITCH_LEFT
- SWITCH_UP
- SWITCH_RIGHT
- SWITCH_DOWN
- HIGH
- LOW
- PRESSED
- RELEASED
- PIN_RGB_R
- PIN_RGB_G
- PIN_RGB_B
- PIN_BUZZER
- JOYSTICK_DOWN
- JOYSTICK_LEFT
- JOYSTICK_UP
- JOYSTICK_RIGHT
- DEGREES_C
- DEGREES_F
- MODE_LOW
- MODE_HIGH
- MODE_PWM
Other methods
- fail(reason='', reason_details='', system_reason='')
Fail the test immediately. - sleep(ticks)
Allow ticks to pass. Use ms(n) to calculate ticks. For example, if the program being tested needs to show something on the screen within 200ms, wait at least 1000 ms but no more than 1500ms, then clear it, you would write:
expect(ms(200), display(Image.HEART))
sleep(ms(1000))
expect(ms(500), display(Image.BLANK))
Tips for writing tests
- Each test should only test one feature of the program (i.e. if the program needs to show something on the screen for 500ms after a button was tapped), don't write a single test that does all that. Write individual tests:
- Check that something appears on the screen after the button goes down (i.e. tap('a'), expect(ms(2000), display(ANYTHING)) )
- Check that it's the right thing (i.e. tap('a'), expect(ms(2000), display(Image.HEART)) )
- Check that it's the right thing straight away (i.e. tap('a'), expect(ms(100), display(Image.HEART)) )
- Check that it stays on the screen (i.e. tap('a'), expect_hold(ms(100), ms(500), display(Image.HEART)) )
- Check that it stays on the screen for the right amount of time and then clears (i.e. tap('a'), expect_hold(ms(100), ms(100), display(Image.HEART)), expect(ms(100), display(Image.BLANK)) )
- Don't use button('a', 1) followed immediately by button('a', 0). Use tap('a') instead. However, in order to verify correct use of is_pressed, was_pressed, it may be useful to leave the button pushed.
- It's much easier if your questions use longer time intervals (i.e. play a tone for 500ms rather than for 100ms). (it's very hard to do precise timing for less than 50-100ms).
- It's simpler not to repeat previous test cases in subsequent tests, but you may need to use expects to ensure that the timing is synchronised. (i.e. using expect to wait for something to appear on the display then sleeping, rather than jsut assuming the thing is already on the display which might mean you might be sleeping for the wrong amount of time).
- Use for loops in the test script to avoid repeating code.
- Also use loops to verify that the program's behaviour is repeatable.
Recovery tests
Sometimes, instead of just detecting whether the program has done the exact right thing, you may want to detect common mistakes and handle them with custom error messages. This is what the recovery parameter to expect is for.
The recovery parameter can be an iterable of (output, message) tuples, where output is an alternate parameter to the what helper, and message replaces the default message. So if you wanted to fail if the program didn't have Image.HEART on the screen, but provide an alternate message specifically for HAPPY and SAD, you could write:
expect(ms(100), display(Image.HEART), 'Expected a heart on the screen after pressing the A button', recovery=((Image.HAPPY, 'Only show the happy face when the B button is pressed.'), (Image.SAD, 'It appears that you didn\'t detect the button press.'),))
Alternatively, the recovery parameter can be a function that you can use to return a custom message. It is passed a single parameter which you can use to check if the output matches a particular value. e.g. if you're using the display helper, then the argument to the recovery function will be a function that returns true if a specified thing is on the display. The example above could be written as:
def recover(c):
if c(Image.HAPPY):
return 'Only show the happy face when the B button is pressed.'
elif c(Image.SAD):
return 'It appears that you didn\'t detect the button press.'
expect(ms(100), display(Image.HEART), 'Expected a heart on the screen after pressing the A button')
You can also write your own custom expect handlers.