Modification of tests for one method causes failures for some others

🐛 Bug

#48230 was closed because it couldn’t be reproduced again after #50578,
but it still persists if tested with the current code in the master branch.

Some index_fill tests are failing if a test-input of another operator (namely, __rpow__) in method_tests is removed,
but passes if that test-input is added back to method_tests, or if all test-inputs for __rpow__ are removed.

To Reproduce

Steps to reproduce the behavior:

  1. Remove the following line from method_tests in common_methods_invocations.py:

    (‘__rpow__’, torch.rand(S, S, S) + 1e-3, (3.14,), ‘constant’, (True, ‘aten::pow’)),

    .

  2. Run the following tests:
    TestAutogradDeviceTypeCPU.test_index_fill_variable_dim_cpu
    TestAutogradDeviceTypeCPU.test_index_fill_variable_dim_neg0_cpu
    TestAutogradDeviceTypeCUDA.test_index_fill_variable_dim_cuda
    TestAutogradDeviceTypeCUDA.test_index_fill_variable_dim_neg0_cuda

  3. Tests fail with the same error as in #48230:

    raise RuntimeError(msg)
RuntimeError: Jacobian mismatch for output 0 with respect to input 1,
numerical:tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.0000, 1.0000, 1.0000,
         1.0000, 1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]])
analytical:tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 2., 2., 2.,
         2., 2., 0., 0., 0., 0., 0.]])

Expected behavior

Modifying a test for an operator should not affect another operator.

Environment

Locally, and on CI.

PyTorch version: 1.9.0a0+git5095332
Is debug build: True
CUDA used to build PyTorch: None

OS: Ubuntu 18.04.1 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.6 (64-bit runtime)

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.9.0a0+git5095332

Additional context

This issue might block removing old tests and porting method tests to OpInfos.
@nikitaved‘s suggestion fixes the issue I stumbled upon, but it doesn’t seem to address the root-cause of this issue,
which might manifest again in the future.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @mruberry @VitalyFedyunin @walterddr

1 possible answer(s) on “Modification of tests for one method causes failures for some others