How to write a test case to lint against PEP-8 conformance
For Python problems, you can have a test case that does linting against PEP-8.
Our PEP-8 linting is currently performed by flake8, with the pep8-naming extension installed. We run the following command in the root directory of the students submission:
flake8 --jobs=1 --ignore=E121,E123,E125,E126,E127,E128,E129,E221,E222,E223,E224,E225,E131,E133,E301,E302,E303,E304,E731,F401,F403,W2,W3 .
If you would like to have a test case that does PEP-8 linting, we'd recommend making this a SUGGEST test case. If a SUGGEST test cases fails, it does not cause the submission to fail; the output is shown to the student, but the next test case is continued on to. This has the effect of it being like a "warning".
To setup a PEP-8 SUGGEST test case, first enabled advanced mode via the checkbox at the top of the problem authoring screen:
Then, add a new test case as the first test case to the problem. Give it a useful label, something like "pep8". Change the Test Case Type field from its default value of VALIDATE to SUGGEST. Add a useful description in the "On pass" message field, something like "Checking your submission against PEP8.".
Lastly, at the bottom of this test case, expand the "Test Options" pane. Select "Python 3 Linter" as the "Driver" and "Output OK checker" as the "Output Checker".
These override the problem-default Driver and Output Checker for this test case.
Save the problem and try it out in preview mode. When SUGGEST test cases pass, they display as a green info icon instead of a green tick.
When SUGGEST test cases fail, they display as an orange warning icon instead of as a red cross.