|
2 | 2 |
|
3 | 3 | [](https://pkg.go.dev/github.com/efficientgo/e2e) |
4 | 4 |
|
5 | | -Robust framework for running complex workload scenarios in isolation, using Go; for integration, e2e tests, benchmarks and more! |
| 5 | +Go Module providing robust framework for running complex workload scenarios in isolation, using Go and Docker. For integration, e2e tests, benchmarks and more! 💪 |
6 | 6 |
|
7 | | -## What is it? |
| 7 | +## What are the goals? |
8 | 8 |
|
9 | | -`e2e` is a Go module which implements a fully featured e2e suite allowing utilizing `go test` for setting hermetic up complex microservice testing scenarios using `docker`. |
| 9 | +* Ability to schedule isolated processes programmatically from single process on single machine. |
| 10 | +* Focus on cluster workloads, cloud native services and microservices. |
| 11 | +* Developer scenarios in mind e.g preserving scenario readability, Go unit test integration. |
| 12 | +* Metric monitoring as the first citizen. Assert on Prometheus metric values during test scenarios or check overall performance characteristics. |
10 | 13 |
|
11 | | -```go mdox-gen-exec="sh -c 'tail -n +6 e2e/doc.go'" |
12 | | -// This module is a fully featured e2e suite allowing utilizing `go test` for setting hermetic up complex microservice integration testing scenarios using docker. |
13 | | -// Example usages: |
14 | | -// * https://github.com/cortexproject/cortex/tree/master/integration |
15 | | -// * https://github.com/thanos-io/thanos/tree/master/test/e2e |
16 | | -// |
17 | | -// Check github.com/efficientgo/tools/e2e/db for common DBs services you can run out of the box. |
18 | | -``` |
| 14 | +## Usage Models |
19 | 15 |
|
20 | | -Credits: |
| 16 | +There are three main use cases envisioned for this Go module: |
21 | 17 |
|
22 | | -* [Cortex Team](https://github.com/cortexproject/cortex/tree/f639b1855c9f0c9564113709a6bce2996d151ec7/integration) |
23 | | -* Initial Authors: [@pracucci](https://github.com/pracucci), [@bwplotka](https://github.com/bwplotka) |
| 18 | +* *Unit test use* ([see example](examples/thanos/unittest_test.go)). Use `e2e` in unit tests to quickly run complex test scenarios involving many container services. This was the main reason we created this module. You can check usage of it in [Cortex](https://github.com/cortexproject/cortex/tree/main/integration) and [Thanos](https://github.com/thanos-io/thanos/tree/main/test/e2e) projects. |
| 19 | +* *Standalone use* ([see example](examples/thanos/standalone.go)). Use `e2e` to run setups in interactive mode where you spin up workloads as you want *programmatically* and poke with it on your own using your browser or other tools. No longer need to deploy full Kubernetes or external machines. |
| 20 | +* *Benchmark use* ([see example](examples/thanos/benchmark_test.go)). Use `e2e` in local Go benchmarks when your code depends on external services with ease. |
| 21 | + |
| 22 | +### Getting Started |
| 23 | + |
| 24 | +Let's go through an example leveraging `go test` flow: |
| 25 | + |
| 26 | +1. Implement the workload by embedding`e2e.Runnable` or `*e2e.InstrumentedRunnable`. Or you can use existing ones in [e2edb](db/) package. For example implementing Thanos Querier with our desired configuration could look like this: |
| 27 | + |
| 28 | + ```go mdox-exec="sed -n '47,64p' examples/thanos/standalone.go" |
| 29 | + func newThanosSidecar(env e2e.Environment, name string, prom e2e.Linkable) *e2e.InstrumentedRunnable { |
| 30 | + ports := map[string]int{ |
| 31 | + "http": 9090, |
| 32 | + "grpc": 9091, |
| 33 | + } |
| 34 | + return e2e.NewInstrumentedRunnable(env, name, ports, "http", e2e.StartOptions{ |
| 35 | + Image: "quay.io/thanos/thanos:v0.21.1", |
| 36 | + Command: e2e.NewCommand("sidecar", e2e.BuildArgs(map[string]string{ |
| 37 | + "--debug.name": name, |
| 38 | + "--grpc-address": fmt.Sprintf(":%d", ports["grpc"]), |
| 39 | + "--http-address": fmt.Sprintf(":%d", ports["http"]), |
| 40 | + "--prometheus.url": "http://" + prom.InternalEndpoint(e2edb.AccessPortName), |
| 41 | + "--log.level": "info", |
| 42 | + })...), |
| 43 | + Readiness: e2e.NewHTTPReadinessProbe("http", "/-/ready", 200, 200), |
| 44 | + User: strconv.Itoa(os.Getuid()), |
| 45 | + }) |
| 46 | + } |
| 47 | + ``` |
| 48 | + |
| 49 | +2. Implement test. Start by creating environment. Currently `e2e` supports Docker environment only. Use unique name for all your tests. It's recommended to keep it stable so resources are consistently cleaned. |
| 50 | + |
| 51 | + ```go mdox-exec="sed -n '22,26p' examples/thanos/unittest_test.go" |
| 52 | + // Start isolated environment with given ref. |
| 53 | + e, err := e2e.NewDockerEnvironment("e2e_example") |
| 54 | + testutil.Ok(t, err) |
| 55 | + // Make sure resources (e.g docker containers, network, dir) are cleaned. |
| 56 | + t.Cleanup(e.Close) |
| 57 | + ``` |
| 58 | + |
| 59 | +3. Program your scenario as you want. You can start, wait for their readiness, stop, check their metrics and use their network endpoints from both unit test (`Endpoint`) as well as within each workload (`InternalEndpoint`). You can also access workload directory. There is a shared directory across all workloads. Check `Dir` and `InternalDir` runnable methods. |
| 60 | + |
| 61 | + ```go mdox-exec="sed -n '28,87p' examples/thanos/unittest_test.go" |
| 62 | + // Create structs for Prometheus containers scraping itself. |
| 63 | + p1, err := e2edb.NewPrometheus(e, "prometheus-1") |
| 64 | + testutil.Ok(t, err) |
| 65 | + s1 := newThanosSidecar(e, "sidecar-1", p1) |
| 66 | + |
| 67 | + p2, err := e2edb.NewPrometheus(e, "prometheus-2") |
| 68 | + testutil.Ok(t, err) |
| 69 | + s2 := newThanosSidecar(e, "sidecar-2", p2) |
| 70 | + |
| 71 | + // Create Thanos Query container. We can point the peer network addresses of both Prometheus instance |
| 72 | + // using InternalEndpoint methods, even before they started. |
| 73 | + t1 := newThanosQuerier(e, "query-1", s1.InternalEndpoint("grpc"), s2.InternalEndpoint("grpc")) |
| 74 | + |
| 75 | + // Start them. |
| 76 | + testutil.Ok(t, e2e.StartAndWaitReady(p1, s1, p2, s2, t1)) |
| 77 | + |
| 78 | + // To ensure query should have access we can check its Prometheus metric using WaitSumMetrics method. Since the metric we are looking for |
| 79 | + // only appears after init, we add option to wait for it. |
| 80 | + testutil.Ok(t, t1.WaitSumMetricsWithOptions(e2e.Equals(2), []string{"thanos_store_nodes_grpc_connections"}, e2e.WaitMissingMetrics())) |
| 81 | + |
| 82 | + // To ensure Prometheus scraped already something ensure number of scrapes. |
| 83 | + testutil.Ok(t, p1.WaitSumMetrics(e2e.Greater(50), "prometheus_tsdb_head_samples_appended_total")) |
| 84 | + testutil.Ok(t, p2.WaitSumMetrics(e2e.Greater(50), "prometheus_tsdb_head_samples_appended_total")) |
| 85 | + |
| 86 | + // We can now query Thanos Querier directly from here, using it's host address thanks to Endpoint method. |
| 87 | + a, err := api.NewClient(api.Config{Address: "http://" + t1.Endpoint("http")}) |
| 88 | + testutil.Ok(t, err) |
| 89 | + |
| 90 | + { |
| 91 | + now := model.Now() |
| 92 | + v, w, err := v1.NewAPI(a).Query(context.Background(), "up{}", now.Time()) |
| 93 | + testutil.Ok(t, err) |
| 94 | + testutil.Equals(t, 0, len(w)) |
| 95 | + testutil.Equals( |
| 96 | + t, |
| 97 | + fmt.Sprintf(`up{instance="%v", job="myself", prometheus="prometheus-1"} => 1 @[%v] |
| 98 | + up{instance="%v", job="myself", prometheus="prometheus-2"} => 1 @[%v]`, p1.InternalEndpoint(e2edb.AccessPortName), now, p2.InternalEndpoint(e2edb.AccessPortName), now), |
| 99 | + v.String(), |
| 100 | + ) |
| 101 | + } |
| 102 | + |
| 103 | + // Stop first Prometheus and sidecar. |
| 104 | + testutil.Ok(t, s1.Stop()) |
| 105 | + testutil.Ok(t, p1.Stop()) |
| 106 | + |
| 107 | + // Wait a bit until Thanos drops connection to stopped Prometheus. |
| 108 | + testutil.Ok(t, t1.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"thanos_store_nodes_grpc_connections"}, e2e.WaitMissingMetrics())) |
| 109 | + |
| 110 | + { |
| 111 | + now := model.Now() |
| 112 | + v, w, err := v1.NewAPI(a).Query(context.Background(), "up{}", now.Time()) |
| 113 | + testutil.Ok(t, err) |
| 114 | + testutil.Equals(t, 0, len(w)) |
| 115 | + testutil.Equals( |
| 116 | + t, |
| 117 | + fmt.Sprintf(`up{instance="%v", job="myself", prometheus="prometheus-2"} => 1 @[%v]`, p2.InternalEndpoint(e2edb.AccessPortName), now), |
| 118 | + v.String(), |
| 119 | + ) |
| 120 | + } |
| 121 | + } |
| 122 | + ``` |
| 123 | + |
| 124 | +## Credits |
| 125 | + |
| 126 | +* Initial Authors: [@pracucci](https://github.com/pracucci), [@bwplotka](https://github.com/bwplotka), [@pstibrany](https://github.com/pstibrany) |
| 127 | +* [Cortex Team](https://github.com/cortexproject/cortex/tree/f639b1855c9f0c9564113709a6bce2996d151ec7/integration) hosting previous form of this module initially. |
0 commit comments