OPENCV_PYTEST_FILTER fails to filter as expected; runs too many python tests

OPENCV_PYTEST_FILTER fails to filter as expected due to a logic bug. It runs too many tests due to an errant wildcard.
For example, OPENCV_PYTEST_FILTER=test_cudaa runs 4 tests. I expect it to run 0 tests.

System information (version)

  • OpenCV => 4.5.1
  • Operating System / Platform => Microsoft Windows [Version 10.0.19042.746]
  • Compiler => VS2019 v16.8.4 Community

Detailed description

Too many tests are run due to an errant wildcard in the test.py python test runner. There is also a minor output bug.

tests_pattern = os.environ.get(‘OPENCV_PYTEST_FILTER’, ‘test_’) + ‘*.py’
if tests_pattern != ‘test_*py’:
print(‘Tests filter: {}’.format(tests_pattern))

Line 37 has an errant wildcard. It should not be present. Instead, the user should include a wildcard in OPENCV_PYTEST_FILTER if they want such behavior. Line 38 has the minor display bug. The coder intended to check for the default case but failed to include the .

It is easily possible to fix the python code to support both the intended default case and expectant wildcard behavior. Then OPENCV_PYTEST_FILTER would match the wildcard behavior of --gtest_filter. FYI: to achieve full test case name filtering, Python 3.7 is needed and use of unittest.testloader.testNamePatterns.

Steps to reproduce

  1. Setup Windows and Python 2.7
  2. Build a release version of OpenCV 4.5.1
  3. set env var OPENCV_PYTEST_FILTER=test_cudaa
  4. c:\python27\python opencv\modules\ts\misc\run.py . -a -t python2

Result

4 test found and 4 test run

...
Discovering python tests from: C:\repos-nobackup\opencv_contrib\modules\cudaarithm\misc\python\test
    found 4 tests
...
test_arithmetic (test_cudaarithm.cudaarithm_test) ... ok
test_convolution (test_cudaarithm.cudaarithm_test) ... ERROR
test_cudaarithm (test_cudaarithm.cudaarithm_test) ... ok
test_logical (test_cudaarithm.cudaarithm_test) ... ok

Expected

No tests found and no tests run.

Issue submission checklist

  • I report the issue, it’s not a question
  • I checked the problem with documentation, FAQ, open issues,
    forum.opencv.org, Stack Overflow, etc and have not found solution
  • I updated to latest OpenCV version and the issue is still there
  • There is reproducer code and related data files: videos, images, onnx, etc

1 possible answer(s) on “OPENCV_PYTEST_FILTER fails to filter as expected; runs too many python tests

  1. tests_pattern = os.environ.get('OPENCV_PYTEST_FILTER', 'test_*') + '.py'
    if tests_pattern != 'test_*.py':
        print('Tests filter: {}'.format(tests_pattern))

    The results in what I consider expected behavior

    • OPENCV_PYTEST_FILTER=test_cudaa no tests are run
    • OPENCV_PYTEST_FILTER=test_cudaa* 4 tests are run
    • undefined OPENCV_PYTEST_FILTER all tests are run