Testing music21

Music21 works only because the code has been extensively tested so that new changes are confirmed not to break existing code. Time spent testing always slows down coding in the short run, but long experience has shown that it saves coding time in the long run. Contributions to music21 without tests will always be rejected, so as developers, it’s important that code be well tested.

All functions/methods/etc. need documentation and at least one test. Oh, and that test must pass.

Module-level testing

Every music21 module should have exactly one class called Test which should run the tests for the module. (If the module is getting very big, it is possible to have this class reside in a different module, so long as it is imported into the module that it tests).

Make the Test class be a subclass of unittest and each of the test methods within it must begin with test, for instance testTupletsEndProperly.

import unittest

class Test(unittest.TestCase):

    def testNotesHaveStemDirection(self):
        from music21 import note
        n = note.Note('C#')
        self.assertTrue(hasattr(n, 'stemDirection'))
        n.stemDirection = 'down'
        self.assertEqual(n.stemDirection, 'down')

The Test class must not produce any output if the tests succeed. They must not create new windows, lose the focus from the current window, play music, etc. They must not rely on modules that are not part of the music21 ecosystem (i.e., scipy).

If you need to create a second test that generates external output, create a second unittest.TestCase called TestExternal. Tests should run quickly: what “quickly” means depends on the complexity of what is being contributed; a major new module can add a few seconds to the test suite. A tiny addition, should add milliseconds at most. If your test adds 10 seconds to running, that’s 10 seconds that everyone will have to wait when running test. If you need to have a test that takes a lot of time, create a second Test class called TestSlow.

End your file so that if the module is run directly, it calls these tests.

if __name__ == "__main__":
    import music21
    music21.mainTest(Test)

 .
 ----------------------------------------------------------------------
 Ran 1 test in 0.001s

 OK

instead of music21.mainTest(Test) you can write music21.mainTest(Test, verbose=True) or while debugging music21.mainTest(Test, runTest='testNotesHaveStemDirection') to run a single test. Or to run a single test, you can pass it from the command line:

python3 note.py testNotesHaveStemDirection

You can also call the tests for the module with:

python3 -m music21.note

Documentation Testing

One great feature Python has is the ability to write tests within the documentation itself (“doctest”). We use a lot of these to demonstrate expected behavior in calling a method or function and also to test that the expected behavior works. However, doctests do take up space in the documentation, so do not write anything that can’t be read (use the module-level testing instead).

Doc tests are written like so:

def flipStemDirection(n):
    '''
    Takes in a note with a `.stemDirection` of 'up' or 'down'
    and transforms it to the other.  Does nothing if the `stemDirection` is not defined.

    Notes without stem direction of `up` or `down` are left alone:

    >>> n = note.Note()
    >>> n.stemDirection
    'unspecified'
    >>> flipStemDirection(n)
    >>> n.stemDirection
    'unspecified'

    Here we flip an `up` to `down` and back again.

    >>> n.stemDirection = 'up'
    >>> flipStemDirection(n)
    >>> n.stemDirection
    'down'
    >>> flipStemDirection(n)
    >>> n.stemDirection
    'up'

    Objects without `.stemDirection` raise a Music21Exception

    >>> r = note.Rest()
    >>> flipStemDirection(r)
    Traceback (most recent call last):
    music21.exceptions21.Music21Exception: flipStemDirection only works on Notes
    '''
    try:
        sd = n.stemDirection
    except AttributeError:
        raise exceptions21.Music21Exception("flipStemDirection only works on Notes")

    if sd == 'up':
        n.stemDirection = 'down'
    elif sd == 'down':
        n.stemDirection = 'up'

Generally in a doctest, the full path to the function (assuming from music21 import *) must be given, so if this function were in a hypothetical stem.py file, it would need to be called as stem.flipStemDirection. The reason for this is that we’re writing tests and docs at the same time, so we want to document usage as our users (who are suggested to run from music21 import *) will need to run the routine.

All doctests start with a clean slate of variables, so variables created in a prior doctest will not be available in another.

Running all tests

Once you’ve made sure your contributions to a module work, you’ll want to run all the tests in the system.

The simplest way of running all the tests for music21 is to run:

python3 test/multiprocessTest.py

This will use (n - 1) of your n-processor system to run all of the tests and will print a lot of output along the way. The tests run in reverse-alphabetical order (with some slow tests boosted to the beginning). All failed tests will appear near the bottom of the output.

A few modules are not tested in multiprocessTest because they do not play well with multiprocessing or are in obscure subdirectories. They do not need to be tested on a general basis, but before a major release they should be tested with:

python3 test/singleCoreAll.py

Before a release, this should be run with every version of Python that is supported (and preferably on both Mac and PC). This can be a pain, so fortunately we have…

Continuous Integration Tests

Upon any commit or pull request, continuous integration testing on Github runs. See:

https://github.com/cuthbertLab/music21/actions

and hopefully it’s running Green at least for the master branch. GitHub runs the ciMain() function of test/singleCoreAll which is the same as the main() function except it exits with 0 or -1 or whatever the UNIX convention for failure is (I always forget–that’s why I coded it). Thanks to the great folks at GitHub, the risks of accepting pull requests is lower (though not zero, especially in cases where there are changes to external output or to parsing files outside the corpus). Although the amount of time to run varies depending on the load of GitHub, keep a look out for changes that significantly slow down the system – they may indicate an inefficient algorithm.

All pull requests must pass GitHub to be accepted. It doesn’t matter that “that bug was probably there before” – it’s time to fix it or to get help fixing it.

Code Coverage Tests

One of the systems running on GitHub will always run slower than the rest. This system is running coverage to measure how much of the music21 code is being run. For the results see:

https://coveralls.io/github/cuthbertLab/music21

While 100% code coverage is impossible, increasing the coverage percentage with each new contribution is essential. We’re at 93% code coverage and get the green badge of pride on our README.md file, and we like that!

If something should be impossible to trigger, but the code is being maintained for some reason, you can add “# pragma: no cover” to the line or to the fork, but this is generally a no-no.

Testing the User’s Guide

Most new contributions will not change the documentation, but occasionally it will. To make sure that the documentation is correct, run testDocumentation.py (under Python 3 only) in the documentation directory (outside the music21 source code directory). If there are any changes, check to make sure they make sense (maybe there are some new files in the corpus that is changing the count somewhere?) and if they do, rerun the notebook with that documentation (line by line!) and make sure that the new output makes better sense than the previous.

This can take some time, so it does not need to be done for every commit.

Conclusion

The music21 project relies on great tests, both to ensure everything is working properly and also to save time for those who will support your code in the future. One of the main things we can always use contributors to is writing tests for code that is under tested, so please feel free to grab some code from Coveralls and contribute tests for it. It may sound like a small contribution, but it’s one that will be greatly appreciated.