yield-tests and fixtures: should they have a future?

[holger krekel]

Hi testing folks, hi Jason,

i am looking at some recent pytest issues and would like to simplify
pytest's internal fixture handling. One obstacle/complication are
yield-tests, i.e. the style of producing tests with a generator::

def test_gen(self):
for x in range(10):
yield check, x

This currently produces 10 test items, calling the "check" function with the
respective parameter. This by itself is not a big deal to support.
However, some people expect fixtures/setup_function/method functions to execute
before the generator does and this mixes the collection with the runtest phase.
Unfortunately, nose also supports this notion although i am wondering how
nose2 is going to deal with it as Jason also plans to separate collection
from running.

So i am thinking about dropping fixture/setup support for yield-tests in pytest
but OTOH i'd like to keep backward and nose compatibility. As far as pytest
is concerned, it has many others means of parametrization independently
from yield, see e. g. http://pytest.org/latest/fixture.html#fixture-parametrize
and http://pytest.org/latest/parametrize.html and more and more people
are starting to use those (pytest does not document or recommend yield
for 1-2 years now).

If anyone has any input/thoughts on this, please shoot.

best,
holger

[Ronny Pfannschmidt]

Hi all,

My oppinion on yield Tests is,
that they should be turned into part of reporting extensions
instead of their current place at running/collection

i would like to have more than one report for functional/acceptance
tests anyway
preferably in a way that allows parts to fail but still run the complete
test

with that in place a yield test would be just a loop running the check
of all items without propagating single item failures to the outside

also there is another upcoming task
yield tests based on the the new async api


best,
Ronny

Post by holger krekel
Hi testing folks, hi Jason,
i am looking at some recent pytest issues and would like to simplify
pytest's internal fixture handling. One obstacle/complication are
yield check, x
This currently produces 10 test items, calling the "check" function with the
respective parameter. This by itself is not a big deal to support.
However, some people expect fixtures/setup_function/method functions to execute
before the generator does and this mixes the collection with the runtest phase.
Unfortunately, nose also supports this notion although i am wondering how
nose2 is going to deal with it as Jason also plans to separate collection
from running.
So i am thinking about dropping fixture/setup support for yield-tests in pytest
but OTOH i'd like to keep backward and nose compatibility. As far as pytest
is concerned, it has many others means of parametrization independently
from yield, see e. g. http://pytest.org/latest/fixture.html#fixture-parametrize
and http://pytest.org/latest/parametrize.html and more and more people
are starting to use those (pytest does not document or recommend yield
for 1-2 years now).
If anyone has any input/thoughts on this, please shoot.
best,
holger
_______________________________________________
testing-in-python mailing list
http://lists.idyll.org/listinfo/testing-in-python

[holger krekel]

Hi Ronny,

Post by Ronny Pfannschmidt
Hi all,
My oppinion on yield Tests is,
that they should be turned into part of reporting extensions
instead of their current place at running/collection
i would like to have more than one report for functional/acceptance
tests anyway preferably in a way that allows parts to fail but still
run the complete test
with that in place a yield test would be just a loop running the check
of all items without propagating single item failures to the outside

Well, one of the main reasons people used "yield" is to get separate progress
dots, and to allow some callables to fail without disrupting the whole sequence.
This means that the running of single callables needs to be reported separately.
Let's put detailed discussions of this to #pylib rather, it's going to get
pytest-specific. I am not sure yet, also taking Jason's confirmation
into account, supporting yield is worth much. Especially since pytest already
has extensive means for parametrization.

Post by Ronny Pfannschmidt
also there is another upcoming task
yield tests based on the the new async api

that's not materialized yet i think.

holger

Post by Ronny Pfannschmidt
best,
Ronny

Post by holger krekel
Hi testing folks, hi Jason,
i am looking at some recent pytest issues and would like to simplify
pytest's internal fixture handling. One obstacle/complication are
yield check, x
This currently produces 10 test items, calling the "check" function with the
respective parameter. This by itself is not a big deal to support.
However, some people expect fixtures/setup_function/method functions to execute
before the generator does and this mixes the collection with the runtest phase.
Unfortunately, nose also supports this notion although i am wondering how
nose2 is going to deal with it as Jason also plans to separate collection
from running.
So i am thinking about dropping fixture/setup support for yield-tests in pytest
but OTOH i'd like to keep backward and nose compatibility. As far as pytest
is concerned, it has many others means of parametrization independently
from yield, see e. g. http://pytest.org/latest/fixture.html#fixture-parametrize
and http://pytest.org/latest/parametrize.html and more and more people
are starting to use those (pytest does not document or recommend yield
for 1-2 years now).
If anyone has any input/thoughts on this, please shoot.
best,
holger
_______________________________________________
testing-in-python mailing list
http://lists.idyll.org/listinfo/testing-in-python

_______________________________________________
py-dev mailing list
http://codespeak.net/mailman/listinfo/py-dev

[jason pellerin]

Hi folks & Holger,

Post by holger krekel
However, some people expect fixtures/setup_function/method functions to execute
before the generator does and this mixes the collection with the runtest phase.
Unfortunately, nose also supports this notion although i am wondering how
nose2 is going to deal with it as Jason also plans to separate collection
from running.

nose2 does separate collection from running and breaks backwards
compatibility with nose's generator tests. This can be pretty surprising in
some cases, if you're using generator tests in a test class and you expect
to be able to use instance attributes set in setUp in the generator
function -- you can't. That's currently filed as a bug against nose2, but
I'm coming around to thinking that it's just the way the design has to
work. So I guess that would be my opinion -- there's no need to preserve
compatibility with nose.

JP