hspec's `around` doesn't wrap shrunk quickcheck tests

[ChickenProp]

At first I thought this was a bug in hspec, I opened an issue at hspec/hspec#899. But as things currently stand I'm not sure how hspec could fix it without changes to quickcheck. So I'm posting here too to try to get more eyes.

Short version is that hspec has an around function that lets you wrap every run of a quickcheck test. But if the test fails and shrinks, it doesn't wrap the shrunk attempts. For example:

  let hook act = do
        putStrLn "before"
        act
        putStrLn "after"
  around_ hook $ it "xxx" $ do
    let shr = \case
          2 -> [1, 0]
          _ -> []
    forAllShrink getSize shr $ \someInt -> ioProperty $ do
      putStrLn $ "Testing " <> show someInt
      someInt `shouldNotBe` 2

before     
Testing 0
after
before
Testing 1
after
before
Testing 2
after
Testing 1
Testing 0

We have two successful tests that get wrapped, a failing test that gets wrapped, then two shrinks of the failing test that don't get wrapped.

The way this works is that hspec examples can be of type a -> Property, and hooks can provide the a. a -> Property is essentially a -> QCGen -> Int -> Rose Result, where the root Result is the result of the first test and children are the results of any shrinks. (Even if shrinking is unnecessary, every shrink is in the tree, though not actually evaluated.) But that type can't do what we want, since we have a single a to construct the whole tree. Currently it essentially calls hook $ \a -> reduceRose (r a), which means the function passed to the hook

• Does the IO necessary to get the root Result of the tree
• Does the IO necessary to get the IO necessary to get any immediate child Results of the tree
• Does not get any child Results of the tree.

Which gives us the behavior observed, that the hook wraps the initial test but not any shrinks.

I don't see any way to get around this with the types as they are. Switching to QCGen -> Int -> Rose (a -> IO Result) might work, but it feels like that would need large changes to both quickcheck and hspec, and I can't rule out that it would fail for some other reason.

Any suggestions?

[MaximilianAlgehed]

I'm not sure I understand what's going on here from the point of view of QuickCheck. As far as I can tell, hspec is poking around in QuickCheck internals in a way that's incorrect w.r.t. the implementation of QuickCheck. What's the bug / missing feature in QuickCheck that you're asking for here?

[Rewbert]

This is something that should be addressed, but as @MaximilianAlgehed says, the fault really lies with hspec. hspecs around is implemented on hspecs side using internals of QC in a way they were not intended.

I believe QuickCheck should perhaps offer an API to do this, so that hspec and family can move out of the internals. But on the topic of this specific issue, there is nothing that QuickCheck can do.

[ChickenProp]

The problem is that as far as I can tell, there's currently no way for hspec to make around work as it's intended to, without some changes to quickcheck - it's not just that it's using the internals wrong, it's that the internals don't make it possible. So I'm hoping for some change to quickcheck that would allow it to work, but I admittedly don't know what that would be.

(Actually, what I'm really hoping for is to learn that I'm wrong, and hspec can make it work already. But failing that...)

It's been a while since I looked at this, but I guess one question would be: is there some way for quickcheck itself to implement hooks? That is, some way to take what would currently be written as

withConn :: (Conn -> IO a) -> IO a
QC.forAll getSize $ \someInt ->
  QC.ioProperty $ withConn $ \someConn ->
    ...

but express it as something like

withConn :: (Conn -> IO a) -> IO a
QC.around withConn $ QC.forAll getSize $ ...

instead? I don't see a way to do it without major changes to quickcheck, but maybe I'm missing something.