How to write a test case to lint against PEP-8 conformance

For Python problems, you can have a test case that does linting against PEP-8.

Our PEP-8 linting is currently performed by flake8, with the pep8-naming extension installed. We run the following command in the root directory of the students submission:

flake8 --jobs=1 --ignore=E121,E123,E125,E126,E127,E128,E129,E221,E222,E223,E224,E225,E131,E133,E301,E302,E303,E304,E731,F401,F403,W2,W3 .

If you would like to have a test case that does PEP-8 linting, we'd recommend making this a SUGGEST test case. If a SUGGEST test cases fails, it does not cause the submission to fail; the output is shown to the student, but the next test case is continued on to. This has the effect of it being like a "warning".

To setup a PEP-8 SUGGEST test case, first enabled advanced mode via the checkbox at the top of the problem authoring screen:

Then, add a new test case as the first test case to the problem. Give it a useful label, something like "pep8". Change the Test Case Type field from its default value of VALIDATE to SUGGEST. Add a useful description in the "On pass" message field, something like "Checking your submission against PEP8.". 

Lastly, at the bottom of this test case, expand the "Test Options" pane. Select "Python 3 Linter" as the "Driver" and "Output OK checker" as the "Output Checker".

These override the problem-default Driver and Output Checker for this test case.

Save the problem and try it out in preview mode. When SUGGEST test cases pass, they display as a green info icon instead of a green tick.

When SUGGEST test cases fail, they display as an orange warning icon instead of as a red cross.