Asynchronous fixtures

[jml]

I'd like something that's almost exactly like fixtures, except that I want to have _setUp and any cleanups I add return Deferreds.

I also therefore want to be able to useFixture on those asynchronous fixtures.

Left to my own devices, I think I would achieve this by factoring out the addCleanup logic that's already in Twisted's testing framework, and then create a parallel implementation of fixtures, along with an adapter that takes regular synchronous fixtures and makes them return Deferreds.

However, that's not optimal. I'd like something that works for various asynchronous abstractions, not just Twisted, and I'd like to have some means of avoiding interface skew.

(Was going to file on Launchpad but lost my 2FA token)

[jml]

Stupid sketch below.

Notes:

• if we encapsulate inlineCallbacks, returnValue, and maybeDeferred behind some object, we could have thing that works for more than just Twisted
  • maybe even we could use that for the normal Fixture?
• untested
• I chose useAsyncFixture rather than useFixture, but didn't rename any other methods. This feels right, but I can't say why.

class AsyncFixture(object):
    """Sketch of an asynchronous fixture."""

    def addCleanup(self, cleanup, *args, **kwargs):
        self._cleanups.push(maybeDeferred, cleanup, *args, **kwargs)

    def addDetail(self, name, content_object):
        self._details[name] = content_object

    @inlineCallbacks
    def cleanUp(self, raise_first=True):
        errors = []
        for function, args, kwargs in self._cleanups:
            try:
                yield function(*args, **kwargs)
            except Exception:
                errors.append(sys.exc_info())
        if errors:
            if 1 == len(errors):
                error = errors[0]
                reraise(error[0], error[1], error[2])
            else:
                raise MultipleExceptions(*errors)

        try:
            return self._cleanups(raise_errors=raise_first)
        finally:
            self._remove_state()

    def _clear_cleanups(self):
        # XXX: Change this. CallMany assumes asynchrony.
        self._cleanups = CallMany()
        self._details = {}
        self._detail_sources = []

    def _remove_state(self):
        self._cleanups = None
        self._details = None
        self._detail_sources = None

    def getDetails(self):
        result = dict(self._details)
        for source in self._detail_sources:
            combine_details(source.getDetails(), result)
        return result

    @inlineCallbacks
    def setUp(self):
        self._clear_cleanups()
        try:
            yield maybeDeferred(self._setUp)
        except:
            err = sys.exc_info()
            details = {}
            if gather_details is not None:
                # Materialise all details since we're about to cleanup.
                gather_details(self.getDetails(), details)
            else:
                details = self.getDetails()
            errors = [err] + self.cleanUp(raise_first=False)
            try:
                raise SetupError(details)
            except SetupError:
                errors.append(sys.exc_info())
            if issubclass(err[0], Exception):
                raise MultipleExceptions(*errors)
            else:
                six.reraise(*err)

    def _setUp(self):
        """Template method for subclasses to override.

        Can return Deferred.
        """

    @inlineCallbacks
    def reset(self):
        yield self.cleanUp()
        yield self.setUp()

    @inlineCallbacks
    def useAsyncFixture(self, fixture):
        try:
            yield fixture.setUp()
        except MultipleExceptions as e:
            if e.args[-1][0] is SetupError:
                combine_details(e.args[-1][1].args[0], self._details)
            raise
        except:
            # The child failed to come up and didn't raise MultipleExceptions
            # which we can understand... capture any details it has (copying
            # the content, it may go away anytime).
            if gather_details is not None:
                gather_details(fixture.getDetails(), self._details)
            raise
        else:
            self.addCleanup(fixture.cleanUp)
            # Calls to getDetails while this fixture is setup will return
            # details from the child fixture.
            self._detail_sources.append(fixture)
            returnValue(fixture)

[jml]

The bit that I don't really get is how this would integrate with the useFixture testtools without using inheritance. Best idea I've come up is that the run_tests_with helper sets an asyncFixture attribute on the test case.

[rbtcollins]

Have you looked at Effect?

[jml]

Yes.

[glyph]

There are now 3 things that one might wish to take into account: Deferreds, asyncio.Futures, and Effects; so having some Fixture-level interface that allows for portability to all three would be ideal. (Or perhaps you could depend on https://pypi.python.org/pypi/deferred and we could start putting that kind of integration logic there, and then have Twisted depend on it …

[jml]

👍 for shifting deferred out of Twisted ala constantly.

As for where the code for this effect thingummy lives, I'm going to postpone that decision until I have some idea of how it should look.

My hunch is that writing a single thing in effect and then providing various dispatchers might be the way to go.

Or, as outlined above, have an ExecutionMethod type that provides inlineCallbacks and returnValue for Twisted, do and do_return for effect, etc.

To me, the interesting question is how it would integrate with unit testing framework. At the moment, testtools.TestCase has a useFixture method which using the current, synchronous Fixture interface.

Within a test, f = useFixture(SomeFixture()) is more or less equivalent to:

f = SomeFixture()
f.setUp()
self.addCleanup(f.cleanUp)

If you had asynchronous code, it'd look like this:

f = AsyncFixture()  # this doesn't actually exist for real anywhere
d = f.setUp()
d.addBoth(lambda _: self.addCleanup(f.cleanUp))  # maybe addCallback?
...

But, that assumes your addCleanup is already geared to handle whatever asynchrony f.cleanUp returns.

For Twisted support, rather than subclassing, testtools has a really nice compositional abstraction for controlling how tests are run called RunTests. e.g.

class TwistedTests(TestCase):
    run_tests_with = AsynchronousDeferredRunTest()

    def test_foo(self):
        d = deferLater(5)
        d.addCallback(lambda _: 20)
        return d.addCallback(self.assertEqual, 20)

Currently, the RunTest object is unavailable to test code, and doesn't modify any environments that test code can access (modulo shenanigans). We could change that, maybe. I mean, it's arguably a bug that TestCase.addCleanup schedules synchronously on tests that use ADRT. Still, the right interface isn't clear to me. Play & input are required.

For other test frameworks that dwell in the benighted lands of reuse-by-inheritance, they would need some other way saying "this test uses this (deferred|effect|asyncio|vanilla) fixture".

[rbtcollins]

What about https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with ? Seems like it has some bearing too.

[rbtcollins]

I think ideally we'd have something that can handle PEP0492 async context managers in all versions of Python (this might require some clever code); but then

Fixtures can define glue around PEP-0492 as a model
A Twisted -> PEP-0492 adapter takes care of deferred using functions

etc

[jml]

Sure, that makes sense.

However, I still don't have a clear picture of how this should integrate with testtools (or indeed any test framework that tries to support fixtures).

Or rather, I have a rough picture (new method useAsyncFixture and delegate all cleanup handling to the runtest object), but I'd like more feedback on it.

[cjwatson]

This isn't an actual solution, but I wanted to point to some possible workaround strategies in case anyone else runs across this. I found myself needing an asynchronous fixture in Launchpad a while back, and resorted to this:

https://git.launchpad.net/launchpad/tree/lib/lp/testing/keyserver/inprocess.py?id=1445a2883c

It required a couple of workarounds, and is definitely not optimal. But it does more or less work.