"Push Multiple" target?

[dougthor42]

Would there be any desire for a "push multiple" target? We have a variety of oci_push targets that we query for, build in parallel, and then bazel run sequentially. For example:

# bash code from memory - please ignore all errors
target_file="$(mktemp)"
bazel query 'kind("rule", "oci_push")' > target_file
bazel build --target_pattern_file="${target_file}"
while read target; do
  bazel run "${target}"
done <"${target_file}"

However the push is done sequentially and is now becoming a bottleneck for us.

I see that rules_oci uses crane under the hood, and from what I can tell crane doesn't support pushing multiple. But it seems like it might be possible to use gnu parallel or similar within the templated push shell script, yeah?

Theoretical API:

oci_push_multiple(
   name = "some_name",
   push_rules = [
       ":oci_push_rule_1",
       "//path/to:my_oci_push_rule_2",
       "//other/path/to:my_oci_push_rule_3",
   ],
)

Each oci_push rule generates an output script, so I think that oci_push_multiple could essentially be a genrule that gobbles up all those scripts.

Perhaps something like:

def oci_push_multiple(name, push_rules):

    native.genrule(
        name = ...,
        srcs = push_rules,
        out = ["push_multiple.sh"],
        cmd = """
            echo 'SCRIPTS=(' > $@
            for f in $(SRCS); do
                echo "  $$f" >> $@
            done
            echo 'printf "%s\n" "${SCRIPTS[@]" | xargs -n1 -P8 bash -c' >> $@
        """,
        executable = True,
    )

The cmd attribute would make something like this shell script:

SCRIPTS=(
  oci_push_rule_1.sh
  path/to/my_oci_push_rule_2.sh
  other/path/to/my_oci_push_rule_3
)
printf "%s\n" "${SCRIPTS[@]" | xargs -n1 -P8 bash -c

We could even use gnu parallel, but that would require adding that as a bazel dep (http_archive, probably).

I'm obviously hand-waving away a good amount of detail, but does that sound like a reasonable start and something that would be useful to people?

[dougthor42]

I just realized that I've forgotten about https://github.com/keith/rules_multirun. That might be a solution here, I'll see if it works for us.

[Aseeef]

@dougthor42 this is something I was looking into as well. Did multirun work for you?

[dougthor42]

I haven't had a chance to try it out yet 🫤