secml.testing

CUnitTest

class secml.testing.c_unittest.CUnitTest(methodName='runTest')[source]

Bases: unittest.case.TestCase

Superclass for unittests.

Provides a wrapper of unittests.TestCase in addition to:

  • integrated logging functionalities (see CLog)

  • integrated timing functionalities (see CTimer)

  • addition assertion methods from numpy.testing

  • skip, skipif, importorskip functions from pytest

Attributes
logger

Logger for current object.

Methods

__call__(self, *args, **kwds)

Call self as a function.

addCleanup(self, function, *args, **kwargs)

Add a function, with arguments, to be called when the test is completed.

addTypeEqualityFunc(self, typeobj, function)

Add a type specific assertEqual style function to compare a type.

assertAlmostEqual(self, first, second[, …])

Fail if the two objects are unequal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the between the two objects is more than the given delta.

assertCountEqual(self, first, second[, msg])

An unordered sequence comparison asserting that the same elements, regardless of order.

assertDictContainsSubset(self, subset, …)

Checks whether dictionary is a superset of subset.

assertEqual(self, first, second[, msg])

Fail if the two objects are unequal as determined by the ‘==’ operator.

assertFalse(self, expr[, msg])

Check that the expression is false.

assertGreater(self, a, b[, msg])

Just like self.assertTrue(a > b), but with a nicer default message.

assertGreaterEqual(self, a, b[, msg])

Just like self.assertTrue(a >= b), but with a nicer default message.

assertIn(self, member, container[, msg])

Just like self.assertTrue(a in b), but with a nicer default message.

assertIs(self, expr1, expr2[, msg])

Just like self.assertTrue(a is b), but with a nicer default message.

assertIsInstance(self, obj, cls[, msg])

Same as self.assertTrue(isinstance(obj, cls)), with a nicer default message.

assertIsNone(self, obj[, msg])

Same as self.assertTrue(obj is None), with a nicer default message.

assertIsNot(self, expr1, expr2[, msg])

Just like self.assertTrue(a is not b), but with a nicer default message.

assertIsNotNone(self, obj[, msg])

Included for symmetry with assertIsNone.

assertIsSubDtype(actual, desired)

Raises an AssertionError if the first dtype is NOT a typecode lower/equal in type hierarchy.

assertLess(self, a, b[, msg])

Just like self.assertTrue(a < b), but with a nicer default message.

assertLessEqual(self, a, b[, msg])

Just like self.assertTrue(a <= b), but with a nicer default message.

assertListEqual(self, list1, list2[, msg])

A list-specific equality assertion.

assertLogs(self[, logger, level])

Fail unless a log message of level level or higher is emitted on logger_name or its children.

assertMultiLineEqual(self, first, second[, msg])

Assert that two multi-line strings are equal.

assertNotAlmostEqual(self, first, second[, …])

Fail if the two objects are equal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the between the two objects is less than the given delta.

assertNotEqual(self, first, second[, msg])

Fail if the two objects are equal as determined by the ‘!=’ operator.

assertNotIn(self, member, container[, msg])

Just like self.assertTrue(a not in b), but with a nicer default message.

assertNotIsInstance(self, obj, cls[, msg])

Included for symmetry with assertIsInstance.

assertNotRegex(self, text, unexpected_regex)

Fail the test if the text matches the regular expression.

assertRaises(self, expected_exception, …)

Fail unless an exception of class expected_exception is raised by the callable when invoked with specified positional and keyword arguments.

assertRaisesRegex(self, expected_exception, …)

Asserts that the message in a raised exception matches a regex.

assertRegex(self, text, expected_regex[, msg])

Fail the test unless the text matches the regular expression.

assertSequenceEqual(self, seq1, seq2[, msg, …])

An equality assertion for ordered sequences (like lists and tuples).

assertSetEqual(self, set1, set2[, msg])

A set-specific equality assertion.

assertTrue(self, expr[, msg])

Check that the expression is true.

assertTupleEqual(self, tuple1, tuple2[, msg])

A tuple-specific equality assertion.

assertWarns(self, expected_warning, *args, …)

Fail unless a warning of class warnClass is triggered by the callable when invoked with specified positional and keyword arguments.

assertWarnsRegex(self, expected_warning, …)

Asserts that the message in a triggered warning matches a regexp.

assert_allclose(self, actual, desired[, …])

Raises an AssertionError if two objects are not equal up to desired tolerance.

assert_approx_equal(self, actual, desired[, …])

Raises an AssertionError if two items are not equal up to significant digits.

assert_array_almost_equal(self, x, y[, …])

Raises an AssertionError if two objects are not equal up to desired precision.

assert_array_almost_equal_nulp(self, x, y[, …])

Compare two arrays relatively to their spacing.

assert_array_equal(self, x, y[, err_msg, …])

Raises an AssertionError if two array_like objects are not equal.

assert_array_less(self, x, y[, err_msg, verbose])

Raises an AssertionError if two array_like objects are not ordered by less than.

assert_array_max_ulp(self, a, b[, maxulp, dtype])

Check that all items of arrays differ in at most N Units in the Last Place.

debug(self)

Run the test without collecting errors in a TestResult

doCleanups(self)

Execute all cleanup functions.

fail(self[, msg])

Fail immediately, with the given message.

failureException

alias of builtins.AssertionError

main

alias of unittest.main.TestProgram

setUp(self)

Hook method for setting up the test fixture before exercising it.

setUpClass()

Hook method for setting up class fixture before running tests in the class.

shortDescription(self)

Returns a one-line description of the test, or None if no description has been provided.

skipTest(self, reason)

Skip this test.

subTest(self[, msg])

Return a context manager that will return the enclosed block of code in a subtest identified by the optional message and keyword parameters.

tearDown(self)

Hook method for deconstructing the test fixture after testing it.

tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

timer(self)

Returns a CTimer to be used as context manager.

assertAlmostEquals

assertDictEqual

assertEquals

assertNotAlmostEquals

assertNotEquals

assertNotRegexpMatches

assertRaisesRegexp

assertRegexpMatches

assert_

countTestCases

defaultTestResult

failIf

failIfAlmostEqual

failIfEqual

failUnless

failUnlessAlmostEqual

failUnlessEqual

failUnlessRaises

id

importskip

run

skip

skipif

static assertIsSubDtype(actual, desired)[source]

Raises an AssertionError if the first dtype is NOT a typecode lower/equal in type hierarchy.

Parameters
actualdtype or str

dtype or string representing a typecode.

desireddtype or str

dtype or string representing a typecode.

assert_allclose(self, actual, desired, rtol=1e-06, atol=0, equal_nan=True, err_msg='', verbose=True)[source]

Raises an AssertionError if two objects are not equal up to desired tolerance.

The test is equivalent to allclose(actual, desired, rtol, atol) (note that allclose has different default values). It compares the difference between actual and desired to atol + rtol * abs(desired).

New in version 1.5.0.

Parameters
actualarray_like

Array obtained.

desiredarray_like

Array desired.

rtolfloat, optional

Relative tolerance.

atolfloat, optional

Absolute tolerance.

equal_nanbool, optional.

If True, NaNs will compare equal.

err_msgstr, optional

The error message to be printed in case of failure.

verbosebool, optional

If True, the conflicting values are appended to the error message.

Raises
AssertionError

If actual and desired are not equal up to specified precision.

Examples

>>> x = [1e-5, 1e-3, 1e-1]
>>> y = np.arccos(np.cos(x))
>>> np.testing.assert_allclose(x, y, rtol=1e-5, atol=0)
assert_approx_equal(self, actual, desired, significant=6, err_msg='', verbose=True)[source]

Raises an AssertionError if two items are not equal up to significant digits.

Note

It is recommended to use one of assert_allclose, assert_array_almost_equal_nulp or assert_array_max_ulp instead of this function for more consistent floating point comparisons.

Given two numbers, check that they are approximately equal. Approximately equal is defined as the number of significant digits that agree.

Parameters
actualscalar

The object to check.

desiredscalar

The expected object.

significantint, optional

Desired precision, default is 7.

err_msgstr, optional

The error message to be printed in case of failure.

verbosebool, optional

If True, the conflicting values are appended to the error message.

Raises
AssertionError

If actual and desired are not equal up to specified precision.

See also

assert_allclose

Compare two array_like objects for equality with desired relative and/or absolute precision.

assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal

Examples

>>> np.testing.assert_approx_equal(0.12345677777777e-20, 0.1234567e-20)
>>> np.testing.assert_approx_equal(0.12345670e-20, 0.12345671e-20,
...                                significant=8)
>>> np.testing.assert_approx_equal(0.12345670e-20, 0.12345672e-20,
...                                significant=8)
Traceback (most recent call last):
    ...
AssertionError:
Items are not equal to 8 significant digits:
 ACTUAL: 1.234567e-21
 DESIRED: 1.2345672e-21

the evaluated condition that raises the exception is

>>> abs(0.12345670e-20/1e-21 - 0.12345672e-20/1e-21) >= 10**-(8-1)
True
assert_array_almost_equal(self, x, y, decimal=6, err_msg='', verbose=True)[source]

Raises an AssertionError if two objects are not equal up to desired precision.

Note

It is recommended to use one of assert_allclose, assert_array_almost_equal_nulp or assert_array_max_ulp instead of this function for more consistent floating point comparisons.

The test verifies identical shapes and that the elements of actual and desired satisfy.

abs(desired-actual) < 1.5 * 10**(-decimal)

That is a looser test than originally documented, but agrees with what the actual implementation did up to rounding vagaries. An exception is raised at shape mismatch or conflicting values. In contrast to the standard usage in numpy, NaNs are compared like numbers, no assertion is raised if both objects have NaNs in the same positions.

Parameters
xarray_like

The actual object to check.

yarray_like

The desired, expected object.

decimalint, optional

Desired precision, default is 6.

err_msgstr, optional

The error message to be printed in case of failure.

verbosebool, optional

If True, the conflicting values are appended to the error message.

Raises
AssertionError

If actual and desired are not equal up to specified precision.

See also

assert_allclose

Compare two array_like objects for equality with desired relative and/or absolute precision.

assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal

Examples

the first assert does not raise an exception

>>> np.testing.assert_array_almost_equal([1.0,2.333,np.nan],
...                                      [1.0,2.333,np.nan])
>>> np.testing.assert_array_almost_equal([1.0,2.33333,np.nan],
...                                      [1.0,2.33339,np.nan], decimal=5)
Traceback (most recent call last):
    ...
AssertionError:
Arrays are not almost equal to 5 decimals
Mismatch: 33.3%
Max absolute difference: 6.e-05
Max relative difference: 2.57136612e-05
 x: array([1.     , 2.33333,     nan])
 y: array([1.     , 2.33339,     nan])
>>> np.testing.assert_array_almost_equal([1.0,2.33333,np.nan],
...                                      [1.0,2.33333, 5], decimal=5)
Traceback (most recent call last):
    ...
AssertionError:
Arrays are not almost equal to 5 decimals
x and y nan location mismatch:
 x: array([1.     , 2.33333,     nan])
 y: array([1.     , 2.33333, 5.     ])
assert_array_almost_equal_nulp(self, x, y, nulp=1)[source]

Compare two arrays relatively to their spacing.

This is a relatively robust method to compare two arrays whose amplitude is variable.

Parameters
x, yarray_like

Input arrays.

nulpint, optional

The maximum number of unit in the last place for tolerance (see Notes). Default is 1.

Returns
None
Raises
AssertionError

If the spacing between x and y for one or more elements is larger than nulp.

See also

assert_array_max_ulp

Check that all items of arrays differ in at most N Units in the Last Place.

spacing

Return the distance between x and the nearest adjacent number.

Notes

An assertion is raised if the following condition is not met:

abs(x - y) <= nulps * spacing(maximum(abs(x), abs(y)))

Examples

>>> x = np.array([1., 1e-10, 1e-20])
>>> eps = np.finfo(x.dtype).eps
>>> np.testing.assert_array_almost_equal_nulp(x, x*eps/2 + x)
>>> np.testing.assert_array_almost_equal_nulp(x, x*eps + x)
Traceback (most recent call last):
  ...
AssertionError: X and Y are not equal to 1 ULP (max is 2)
assert_array_equal(self, x, y, err_msg='', verbose=True)[source]

Raises an AssertionError if two array_like objects are not equal.

Given two array_like objects, check that the shape is equal and all elements of these objects are equal (but see the Notes for the special handling of a scalar). An exception is raised at shape mismatch or conflicting values. In contrast to the standard usage in numpy, NaNs are compared like numbers, no assertion is raised if both objects have NaNs in the same positions.

The usual caution for verifying equality with floating point numbers is advised.

Parameters
xarray_like

The actual object to check.

yarray_like

The desired, expected object.

err_msgstr, optional

The error message to be printed in case of failure.

verbosebool, optional

If True, the conflicting values are appended to the error message.

Raises
AssertionError

If actual and desired objects are not equal.

See also

assert_allclose

Compare two array_like objects for equality with desired relative and/or absolute precision.

assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal

Notes

When one of x and y is a scalar and the other is array_like, the function checks that each element of the array_like object is equal to the scalar.

Examples

The first assert does not raise an exception:

>>> np.testing.assert_array_equal([1.0,2.33333,np.nan],
...                               [np.exp(0),2.33333, np.nan])

Assert fails with numerical imprecision with floats:

>>> np.testing.assert_array_equal([1.0,np.pi,np.nan],
...                               [1, np.sqrt(np.pi)**2, np.nan])
Traceback (most recent call last):
    ...
AssertionError:
Arrays are not equal
Mismatch: 33.3%
Max absolute difference: 4.4408921e-16
Max relative difference: 1.41357986e-16
 x: array([1.      , 3.141593,      nan])
 y: array([1.      , 3.141593,      nan])

Use assert_allclose or one of the nulp (number of floating point values) functions for these cases instead:

>>> np.testing.assert_allclose([1.0,np.pi,np.nan],
...                            [1, np.sqrt(np.pi)**2, np.nan],
...                            rtol=1e-10, atol=0)

As mentioned in the Notes section, assert_array_equal has special handling for scalars. Here the test checks that each value in x is 3:

>>> x = np.full((2, 5), fill_value=3)
>>> np.testing.assert_array_equal(x, 3)
assert_array_less(self, x, y, err_msg='', verbose=True)[source]

Raises an AssertionError if two array_like objects are not ordered by less than.

Given two array_like objects, check that the shape is equal and all elements of the first object are strictly smaller than those of the second object. An exception is raised at shape mismatch or incorrectly ordered values. Shape mismatch does not raise if an object has zero dimension. In contrast to the standard usage in numpy, NaNs are compared, no assertion is raised if both objects have NaNs in the same positions.

Parameters
xarray_like

The smaller object to check.

yarray_like

The larger object to compare.

err_msgstring

The error message to be printed in case of failure.

verbosebool

If True, the conflicting values are appended to the error message.

Raises
AssertionError

If actual and desired objects are not equal.

See also

assert_array_equal

tests objects for equality

assert_array_almost_equal

test objects for equality up to precision

Examples

>>> np.testing.assert_array_less([1.0, 1.0, np.nan], [1.1, 2.0, np.nan])
>>> np.testing.assert_array_less([1.0, 1.0, np.nan], [1, 2.0, np.nan])
Traceback (most recent call last):
    ...
AssertionError:
Arrays are not less-ordered
Mismatch: 33.3%
Max absolute difference: 1.
Max relative difference: 0.5
 x: array([ 1.,  1., nan])
 y: array([ 1.,  2., nan])
>>> np.testing.assert_array_less([1.0, 4.0], 3)
Traceback (most recent call last):
    ...
AssertionError:
Arrays are not less-ordered
Mismatch: 50%
Max absolute difference: 2.
Max relative difference: 0.66666667
 x: array([1., 4.])
 y: array(3)
>>> np.testing.assert_array_less([1.0, 2.0, 3.0], [4])
Traceback (most recent call last):
    ...
AssertionError:
Arrays are not less-ordered
(shapes (3,), (1,) mismatch)
 x: array([1., 2., 3.])
 y: array([4])
assert_array_max_ulp(self, a, b, maxulp=1, dtype=None)[source]

Check that all items of arrays differ in at most N Units in the Last Place.

Parameters
a, barray_like

Input arrays to be compared.

maxulpint, optional

The maximum number of units in the last place that elements of a and b can differ. Default is 1.

dtypedtype, optional

Data-type to convert a and b to if given. Default is None.

Returns
retndarray

Array containing number of representable floating point numbers between items in a and b.

Raises
AssertionError

If one or more elements differ by more than maxulp.

See also

assert_array_almost_equal_nulp

Compare two arrays relatively to their spacing.

Examples

>>> a = np.linspace(0., 1., 100)
>>> res = np.testing.assert_array_max_ulp(a, np.arcsin(np.sin(a)))
importskip
property logger

Logger for current object.

main

alias of unittest.main.TestProgram

classmethod setUpClass()[source]

Hook method for setting up class fixture before running tests in the class.

skip
skipif
timer(self)[source]

Returns a CTimer to be used as context manager.