Forbid to use pytest.mark.xfail without a raises= argument

[sobolevn]

Today I got hit by a xfail problem. My module was decorated with pytest.mark.xfail, because I wanted to test an explicit test failure in my plugin: https://github.com/dry-python/returns/blob/2223f03a2dbb96d4cd273fb41ad5c26331d9eef8/tests/test_contrib/test_hypothesis/test_laws/test_wrong_custom_type_with_init.py#L10

But, under some condition, I've noticed that sometimes it was failing due to an unknown reason.
It was really hard to find, because this test was passing for a year now.

So, my idea is to:

• Allow pytest.mark.xfailure(raises=ExceptionType)
• Forbid pytest.mark.xfailure or ``pytest.mark.xfailure()`

Docs: https://docs.pytest.org/en/6.2.x/skipping.html#raises-parameter

It hides errors 😞

[djmattyg007]

If you’re testing know failure conditions, it sounds like you should be using pytest.raises().

[sobolevn]

No, I should not 🙂

Please, take a look at my sample:

pytestmark = pytest.mark.xfail

_ValueType = TypeVar('_ValueType')
_NewValueType = TypeVar('_NewValueType')


class _Wrapper(
    BaseContainer,
    SupportsKind1['_Wrapper', _ValueType],
    mappable.Mappable1[_ValueType],
):
    _inner_value: _ValueType

    def __init__(self, inner_value: _ValueType) -> None:
        super().__init__(inner_value)

    def map(
        self,
        function: Callable[[_ValueType], _NewValueType],
    ) -> '_Wrapper[_NewValueType]':
        return _Wrapper(
            'wrong-{0}'.format(function(self._inner_value)),  # type: ignore
        )


check_all_laws(_Wrapper, use_init=True)

There's no place for pytest.raises.

[m-burst]

In any scenario, if we expect a test to fail, it is best to pin how exactly we expect it to fail, so that it will be noticed if something changes