Fail-fasting test runner

[vemv]

Context

"Fail-fast" is a commonly wanted feature. Some alternative test runners provide a form of it, although they may be a bit framework-ish. I like that utils.test is literally a collection of utils for clojure.test, assuming nothing.

It seems great to have a reusable defn that can be invoked from any project (especially from a repl) regardless of its setup.

Proposal

Bundle a helper such as:

(defn run-tests [& namespaces]
  (let [summary (-> (->> namespaces
                         (reduce (bound-fn [r n]
                                   (let [{:keys [fail error]
                                          :as v} (clojure.test/test-ns n)
                                         failed? (some pos? [fail error])
                                         ret (merge-with + r v)]
                                     (cond-> ret
                                       failed? reduced)))
                                 clojure.test/*initial-report-counters*))
                    (assoc :type :summary))]
    (clojure.test/do-report summary)
    summary))

(defn run-all-tests
  "Like `clojure.test/run-all-tests` but fails-fast."
  []
  (->> (all-ns)
       (filter (fn [n]
                 (->> n
                      ns-publics
                      vals
                      (some (fn [var-ref]
                              {:pre [(var? var-ref)]}
                              (-> var-ref meta :test))))))
       (sort-by pr-str)
       (reverse) ;; unit.* first
       (apply run-tests)))

Then one can do (clojure.tools.namespace.repl/refresh :after 'my/run-all-tests).

This could be further integrated with formatting-stack's test runner, with the project template, etc

...the snippet has worked well for me over the last month, also I had a similar but more intrincate incarnation of it based off CircleCI's test runner over 2020.

Thoughts?

cc/ @thumbnail

[thumbnail]

This seems like a fine addition, personally I often use a snippet like this:

(defn refresh-and-run-tests
  [substr]
  (refresh)
  (let [test-vars (->> ((requiring-resolve 'eftest.runner/find-tests) refresh-dirs)
                       (filter #(str/includes? % substr)))]
    ((requiring-resolve 'eftest.runner/run-tests) test-vars {:report (requiring-resolve 'eftest.report.pretty/report)
                                                             :fail-fast?   true
                                                             :multithread? :namespaces})))

Because of how we name tests / testing namespaces it allows me to target broader or more specific tests across multiple namespaces, it also uses eftest (or kaocha, depending on the specific project).

Although it'd be complex to support any configuration like this; so probably relying on clj.test is a fine assumption.

[vemv]

Cheers, I'll see if I can contribute this one day of these

(the bulk of the work being decent test coverage)