Error handling and testing solutions

According to the part of the course you are following, we will review two kinds of tests:

  • Part A testing with asserts: moved to SoftPython

  • Part B testing with unittest: read this notebook

Testing with Unittest

NOTE: Testing with Unittest is only done in PART B of this course

Is there anything better than assertfor testing? assert can be a quick way to check but doesn’t tell us exactly which is the wrong number in the list returned by even_number(5). Luckily, Python offers us a better option, which is a complete testing framework called unittest. We will use unittest because it is the standard one, but if you’re doing other projects you might consider using better ones like pytest (note it can also execute tests made with unittest, so if your visualstudio code for some reason doesn’t work with unittest, you can try setting pytest as test framework)

So let’s give unittest a try. Suppose you have a file called file_test.py like this:

[2]:
import unittest

def even_numbers(n):
    """
    Return a list of the first n even numbers

    Zero is considered to be the first even number.

    >>> even_numbers(5)
    [0,2,4,6,8]
    """
    r = [2 * x for x in range(n)]
    r[n // 2] = 3   # <-- evil bug, puts number '3' in the middle
    return r

class MyTest(unittest.TestCase):

    def test_long_list(self):
        self.assertEqual(even_numbers(5),[0,2,4,6,8])


We won’t explain what class mean (for classes see the book chapter), the important thing to notice is the method definition:

def test_long_list(self):
    self.assertEqual(even_numbers(5),[0,2,4,6,8])

In particular:

  • method is declared like a function, and begins with 'test_' word

  • method takes self as parameter

  • self.assertEqual(even_numbers(5),[0,2,4,6,8]) executes the assertion. Other assertions could be self.assertTrue(some_condition) or self.assertFalse(some_condition)

Running tests

To run the tests, enter the following command in the terminal:

python -m unittest file_test

!!!!! WARNING: In the call above, DON’T append the extension .py to file_test !!!!!!

!!!!! WARNING: Still, on the hard-disk the file MUST be named with a .py at the end, like file_test.py!!!!!!

You should see an output like the following:

[3]:
jupman.show_run(MyTest)
F
======================================================================
FAIL: test_long_list (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython-input-2-ec89a972a892>", line 19, in test_long_list
    self.assertEqual(even_numbers(5),[0,2,4,6,8])
AssertionError: Lists differ: [0, 2, 3, 6, 8] != [0, 2, 4, 6, 8]

First differing element 2:
3
4

- [0, 2, 3, 6, 8]
?        ^

+ [0, 2, 4, 6, 8]
?        ^


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

Now you can see a nice display of where the error is, exactly in the middle of the list!

When tests don’t run

When -m unittest does not work and you keep seeing absurd errors like Python not finding a module and you are getting desperate (especially because Python has unittest included by default, there is no need to install it! ), try putting the following code at the very end of the file you are editing:

unittest.main()

Then simply run your file with:

python file_test.py

In this case it should REALLY work. If it still doesn’t, call the Ghostbusters. Or, better, the IndentationBusters, you’re likely having tabs mixed with spaces mixed with very bad luck.

Adding tests

How can we add (good) tests? Since best ones are usually short, it would be better starting small boundary cases. For example like n=1 , which according to function documentation should produce a list containing zero:

[9]:
class MyTest(unittest.TestCase):

    def test_one_element(self):
        self.assertEqual(even_numbers(1),[0])

    def test_long_list(self):
        self.assertEqual(even_numbers(5),[0,2,4,6,8])

Let’s call again the command:

python -m unittest file_test
[5]:
jupman.show_run(MyTest)
FF
======================================================================
FAIL: test_long_list (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython-input-4-306d9f1c7777>", line 7, in test_long_list
    self.assertEqual(even_numbers(5),[0,2,4,6,8])
AssertionError: Lists differ: [0, 2, 3, 6, 8] != [0, 2, 4, 6, 8]

First differing element 2:
3
4

- [0, 2, 3, 6, 8]
?        ^

+ [0, 2, 4, 6, 8]
?        ^


======================================================================
FAIL: test_one_element (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython-input-4-306d9f1c7777>", line 4, in test_one_element
    self.assertEqual(even_numbers(1),[0])
AssertionError: Lists differ: [3] != [0]

First differing element 0:
3
0

- [3]
+ [0]

----------------------------------------------------------------------
Ran 2 tests in 0.003s

FAILED (failures=2)

From the tests we can now see there is clearly something wrong with the number 3 that keeps popping up, making both tests fail. You can see immediately which tests have failed by looking at the first two FF at the top of the output. Let’s fix the code by removing the buggy line:

[10]:
def even_numbers(n):
    """
    Return a list of the first n even numbers

    Zero is considered to be the first even number.

    >>> even_numbers(5)
    [0,2,4,6,8]
    """
    r = [2 * x for x in range(n)]
    # NOW WE COMMENTED THE BUGGY LINE  r[n // 2] = 3   # <-- evil bug, puts number '3' in the middle
    return r

And call yet again the command:

python -m unittest file_test
[7]:
jupman.show_run(MyTest)
..
----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK

Wonderful, all the two tests have passed and we got rid of the bug.

WARNING: DON’T DUPLICATE TEST CLASS NAMES AND/OR METHODS!

In the following, you will be asked to add tests. Just add NEW methods with NEW names to the EXISTING class MyTest !

Exercise: boundary cases

Think about other boundary cases, and try to add corresponding tests.

  • Can we ever have an empty list?

  • Can n be equal to zero? Add a test inside MyTest class for its expected result.

  • Can n be negative? In this case the function text tells us nothing about the expected behaviour, so we might choose it now: either the function raises an error, or it gives a back something, like i.e. list of even negative numbers. Try to modify even_numbers and add a relative test inside MyTest class for expecting even negative numbers (starting from zero).

Exercise: expecting assertions

What if user passes us a float like 3.5 instead of an integer? If you try to run even_numbers(3.5) you will discover it works anyway, but we might decide to be picky and not accept inputs other than integers. Try to modify even_numbers to make so that when input is not of type int, raises TypeError (to check for type, you can write type(n) == int).

To test for it, add following test inside MyTest class :

def test_type(self):

    with self.assertRaises(TypeError):
        even_numbers(3.5)

The with block tells Python to expect the code inside the with block to raise the exception TypeError:

  • If even_numbers(3.5) actually raises TypeError exception, nothing happens

  • If even_numbers(3.5) does not raise TypeError exception, with raises AssertionError

After you completed previous task, consider when the input is the float 4.0: in this case it might make sense to still accept it, so modify even_numbers accordingly and write a test for it.

Exercise: good tests

What difference is there between the following two test classes? Which one is better for testing?

class MyTest(unittest.TestCase):

    def test_one_element(self):
        self.assertEqual(even_numbers(1),[0])

    def test_long_list(self):
        self.assertEqual(even_numbers(5),[0,2,4,6,8])

and

class MyTest(unittest.TestCase):

    def test_stuff(self):
        self.assertEqual(even_numbers(1),[0])
        self.assertEqual(even_numbers(5),[0,2,4,6,8])

Running unittests in Visual Studio Code

You can run and debug tests in Visual Studio Code, which is very handy. First, you need to set it up.

  1. Hit Control-Shift-P (on Mac: Command-Shift-P) and type Python: Configure Tests

vscode 1 4292234

  1. Select unittest:

vscode 2 2341234123

  1. Select . root directory (we assume tests are in the folder that you’ve opened):

vscode 3 3142434

  1. Select *test*.py Python files containing the word 'test':

vscode 4 92383283

Hopefully, on the currently opened test file new labels should appear above class and test methods, like in the following example. Try to click on them:

vscode 5 8232114

In the bottom bar, you should see a recap of tests run (right side of the picture):

vscode 6 2348324332

TROUBLESHOOTING

If you encounter problems running tests and have Anaconda, sometimes an easy solution can be just closing Visual Studio Code and running it from the Anaconda Navigator. You can also try updating it.

Running tests by console does not work:

  • remember to SAVE the files before executing tests: in Windows, a file appears as not saved when its filename in the tab is written in italics; on Linux, you might see a dot to the right of the filename

Run Test label does not show up in code:

  • if you see red squiggles in the code, most probably syntax is not correct and thus no test will get discovered ! If this is the case, fix the syntax error, SAVE, and then tell Visual Studio to discover test.

  • you might also try Right click->Run current Test File.

  • try selecting another testing framework , try pytest, which is also capable to discover and execute unittests.

  • if you are really out of luck with the editor, there is always the option of running tests from the console.

Spend time using the console !!!!

During exams VSCode testing might not work, so please be prepared to use the console

Functional programming

In functional programming, functions behave as mathematical ones so they always take some parameter and return new data without ever changing the input. They say functional programming is easier to test. Why?

Immutable data structures: all data structures are (or are meant to be) immutable -> no code can ever tweak your data, so other developers just cannot (should not) be able to inadvertently change your data.

Simpler parallel computing: point above is particularly inmportant in parallel computation, when the system can schedule thread executions differently each time you run the program: this implies that when you have multiple threads it can be very very hard to reproduce a bug where a thread wrongly changes a data which is supposed to be exclusively managed by another one: it might fail in one run and succeed in another just because the system scheduled differently the code execution! Functional programming frameworks like Spark solve these problems very nicely.

Easier to reason about code: it is much easier to reason about functions, as we can use standard equational reasoning on input/outputs as traditionally done in algebra. To understand what we’re talking about, you can see these slides: Visual functional programming (will talk more about it in class)

Show solution