sjsonnet 1.0.0 release bug

[stephenamar-db]

comment here if there are any blockers or must-have.

[stephenamar-db]

0.5.x branch benchmark run:

[1-114] Benchmark                                                                  (path)  Mode  Cnt   Score   Error  Units
[1-114] RegressionBenchmark.main             bench/resources/bug_suite/assertions.jsonnet  avgt        0.274          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.01.jsonnet  avgt        0.085          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.02.jsonnet  avgt       34.902          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.03.jsonnet  avgt       15.216          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.04.jsonnet  avgt       41.941          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.06.jsonnet  avgt        0.418          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.07.jsonnet  avgt        3.674          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.08.jsonnet  avgt        0.071          ms/op
[1-114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.09.jsonnet  avgt        0.137          ms/op
[1-114] RegressionBenchmark.main         bench/resources/cpp_suite/gen_big_object.jsonnet  avgt        1.041          ms/op
[1-114] RegressionBenchmark.main      bench/resources/cpp_suite/large_string_join.jsonnet  avgt        1.709          ms/op
[1-114] RegressionBenchmark.main  bench/resources/cpp_suite/large_string_template.jsonnet  avgt        4.121          ms/op
[1-114] RegressionBenchmark.main             bench/resources/cpp_suite/realistic1.jsonnet  avgt        3.480          ms/op
[1-114] RegressionBenchmark.main             bench/resources/cpp_suite/realistic2.jsonnet  avgt       84.599          ms/op
[1-114] RegressionBenchmark.main                  bench/resources/go_suite/base64.jsonnet  avgt        0.910          ms/op
[1-114] RegressionBenchmark.main            bench/resources/go_suite/base64Decode.jsonnet  avgt        0.698          ms/op
[1-114] RegressionBenchmark.main       bench/resources/go_suite/base64DecodeBytes.jsonnet  avgt       13.266          ms/op
[1-114] RegressionBenchmark.main       bench/resources/go_suite/base64_byte_array.jsonnet  avgt        1.263          ms/op
[1-114] RegressionBenchmark.main              bench/resources/go_suite/comparison.jsonnet  avgt        6.625          ms/op
[1-114] RegressionBenchmark.main             bench/resources/go_suite/comparison2.jsonnet  avgt       43.999          ms/op
[1-114] RegressionBenchmark.main        bench/resources/go_suite/escapeStringJson.jsonnet  avgt        0.064          ms/op
[1-114] RegressionBenchmark.main                   bench/resources/go_suite/foldl.jsonnet  avgt       11.136          ms/op
[1-114] RegressionBenchmark.main             bench/resources/go_suite/lstripChars.jsonnet  avgt        0.730          ms/op
[1-114] RegressionBenchmark.main          bench/resources/go_suite/manifestJsonEx.jsonnet  avgt        0.085          ms/op
[1-114] RegressionBenchmark.main          bench/resources/go_suite/manifestTomlEx.jsonnet  avgt        0.107          ms/op
[1-114] RegressionBenchmark.main         bench/resources/go_suite/manifestYamlDoc.jsonnet  avgt        0.088          ms/op
[1-114] RegressionBenchmark.main                  bench/resources/go_suite/member.jsonnet  avgt        0.583          ms/op
[1-114] RegressionBenchmark.main                bench/resources/go_suite/parseInt.jsonnet  avgt        0.064          ms/op
[1-114] RegressionBenchmark.main                 bench/resources/go_suite/reverse.jsonnet  avgt       15.909          ms/op
[1-114] RegressionBenchmark.main             bench/resources/go_suite/rstripChars.jsonnet  avgt        0.697          ms/op
[1-114] RegressionBenchmark.main              bench/resources/go_suite/stripChars.jsonnet  avgt        0.805          ms/op
[1-114] RegressionBenchmark.main                  bench/resources/go_suite/substr.jsonnet  avgt        0.223          ms/op

[stephenamar-db]

Master branch benchmark run:

[1-113] Benchmark                                                                  (path)  Mode  Cnt   Score   Error  Units
[1-113] RegressionBenchmark.main             bench/resources/bug_suite/assertions.jsonnet  avgt        0.264          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.01.jsonnet  avgt        0.065          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.02.jsonnet  avgt       35.279          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.03.jsonnet  avgt       15.370          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.04.jsonnet  avgt       42.711          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.06.jsonnet  avgt        0.382          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.07.jsonnet  avgt        3.693          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.08.jsonnet  avgt        0.050          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.09.jsonnet  avgt        0.054          ms/op
[1-113] RegressionBenchmark.main         bench/resources/cpp_suite/gen_big_object.jsonnet  avgt        1.013          ms/op
[1-113] RegressionBenchmark.main      bench/resources/cpp_suite/large_string_join.jsonnet  avgt        1.648          ms/op
[1-113] RegressionBenchmark.main  bench/resources/cpp_suite/large_string_template.jsonnet  avgt        4.041          ms/op
[1-113] RegressionBenchmark.main             bench/resources/cpp_suite/realistic1.jsonnet  avgt        3.464          ms/op
[1-113] RegressionBenchmark.main             bench/resources/cpp_suite/realistic2.jsonnet  avgt       79.863          ms/op
[1-113] RegressionBenchmark.main                  bench/resources/go_suite/base64.jsonnet  avgt        0.861          ms/op
[1-113] RegressionBenchmark.main            bench/resources/go_suite/base64Decode.jsonnet  avgt        0.677          ms/op
[1-113] RegressionBenchmark.main       bench/resources/go_suite/base64DecodeBytes.jsonnet  avgt       13.179          ms/op
[1-113] RegressionBenchmark.main       bench/resources/go_suite/base64_byte_array.jsonnet  avgt        1.338          ms/op
[1-113] RegressionBenchmark.main              bench/resources/go_suite/comparison.jsonnet  avgt        9.052          ms/op
[1-113] RegressionBenchmark.main             bench/resources/go_suite/comparison2.jsonnet  avgt       64.319          ms/op
[1-113] RegressionBenchmark.main        bench/resources/go_suite/escapeStringJson.jsonnet  avgt        0.044          ms/op
[1-113] RegressionBenchmark.main                   bench/resources/go_suite/foldl.jsonnet  avgt       10.897          ms/op
[1-113] RegressionBenchmark.main             bench/resources/go_suite/lstripChars.jsonnet  avgt        0.663          ms/op
[1-113] RegressionBenchmark.main          bench/resources/go_suite/manifestJsonEx.jsonnet  avgt        0.068          ms/op
[1-113] RegressionBenchmark.main          bench/resources/go_suite/manifestTomlEx.jsonnet  avgt        0.080          ms/op
[1-113] RegressionBenchmark.main         bench/resources/go_suite/manifestYamlDoc.jsonnet  avgt        0.065          ms/op
[1-113] RegressionBenchmark.main                  bench/resources/go_suite/member.jsonnet  avgt        0.450          ms/op
[1-113] RegressionBenchmark.main                bench/resources/go_suite/parseInt.jsonnet  avgt        0.045          ms/op
[1-113] RegressionBenchmark.main                 bench/resources/go_suite/reverse.jsonnet  avgt       15.722          ms/op
[1-113] RegressionBenchmark.main             bench/resources/go_suite/rstripChars.jsonnet  avgt        0.679          ms/op
[1-113] RegressionBenchmark.main              bench/resources/go_suite/stripChars.jsonnet  avgt        0.657          ms/op
[1-113] RegressionBenchmark.main                  bench/resources/go_suite/substr.jsonnet  avgt        0.229          ms/op

[stephenamar-db]

Most of them are pretty much equivalent. Definitely an improvement for stripChars benchmark.

Strangely, bench/resources/go_suite/comparison.jsonnet and bench/resources/go_suite/comparison2.jsonnet have regressed significantly. I'm going to take look.

[He-Pin]

https://jmh.morethan.io/

May help for easy viewing the bench result

And we can optimize the array buffer performance too.

[stephenamar-db]

the problem is not with comparison. it's with array creation/concat:

comparison.jsonnet:

local long_array = std.range(1, 1000000);
long_array + [1] < long_array + [2]

I've tested std.range and the statement long_array < long_array - no difference.

[stephenamar-db]

Did some bisect
Reverting 9e9bf61 seems to fix the regression.

[He-Pin]

@stephenamar-db I think we can set a newScopes.sizeHint(scopes.length * 4) to see if the performance improves. I don't think reverting that will improve the performance, or we can try something like this to skip the ensure size.

case (spec @ ForSpec(_, name, expr)) :: rest =>
    var totalSize = 0
    var i = 0
    while (i < scopes.length) {
      val s = scopes(i)
      visitExpr(expr)(s) match {
        case a: Val.Arr => totalSize += a.length
        case r =>
          Error.fail(
            "In comprehension, can only iterate over array, not " + r.prettyName,
            spec
          )
      }
      i += 1
    }
    val newScopes = new Array[ValScope](totalSize)
    var idx = 0
    i = 0
    while (i < scopes.length) {
      val s = scopes(i)
      val a = visitExpr(expr)(s).asInstanceOf[Val.Arr]
      val lazyArr = a.asLazyArray
      var j = 0
      while (j < lazyArr.length) {
        newScopes(idx) = s.extendSimple(lazyArr(j))
        idx += 1
        j += 1
      }
      i += 1
    }
    visitComp(rest, newScopes)

[stephenamar-db]

Sure. do you know how to run the benchmark?

[He-Pin]

./mill bench.runJmh , but it costs 2 hours， does it support selective bench?

[stephenamar-db]

There’s a runRegressions. It accepts the path to the regression test

…

On Tue, Dec 9, 2025 at 7:46 PM He-Pin(kerr) ***@***.***> wrote: *He-Pin* left a comment (databricks/sjsonnet#558) <#558 (comment)> ./mill bench.runJmh , but it costs 2 hours， does it support selective bench? — Reply to this email directly, view it on GitHub <#558 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDKJ7I2CKVEUEDSHR5KIJOL4A6JRZAVCNFSM6AAAAACNWRCDMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMZVGIZTCNBYGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[He-Pin]

on main:

[113] 
[113] Benchmark                                                                  (path)  Mode  Cnt   Score   Error  Units
[113] RegressionBenchmark.main             bench/resources/bug_suite/assertions.jsonnet  avgt        0.316          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.01.jsonnet  avgt        0.070          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.02.jsonnet  avgt       41.670          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.03.jsonnet  avgt       12.969          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.04.jsonnet  avgt       31.149          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.06.jsonnet  avgt        0.437          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.07.jsonnet  avgt        2.859          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.08.jsonnet  avgt        0.051          ms/op
[113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.09.jsonnet  avgt        0.060          ms/op
[113] RegressionBenchmark.main         bench/resources/cpp_suite/gen_big_object.jsonnet  avgt        0.933          ms/op
[113] RegressionBenchmark.main      bench/resources/cpp_suite/large_string_join.jsonnet  avgt        2.194          ms/op
[113] RegressionBenchmark.main  bench/resources/cpp_suite/large_string_template.jsonnet  avgt        3.143          ms/op
[113] RegressionBenchmark.main             bench/resources/cpp_suite/realistic1.jsonnet  avgt        3.052          ms/op
[113] RegressionBenchmark.main             bench/resources/cpp_suite/realistic2.jsonnet  avgt       70.855          ms/op
[113] RegressionBenchmark.main                  bench/resources/go_suite/base64.jsonnet  avgt        0.828          ms/op
[113] RegressionBenchmark.main            bench/resources/go_suite/base64Decode.jsonnet  avgt        0.634          ms/op
[113] RegressionBenchmark.main       bench/resources/go_suite/base64DecodeBytes.jsonnet  avgt        9.249          ms/op
[113] RegressionBenchmark.main       bench/resources/go_suite/base64_byte_array.jsonnet  avgt        1.464          ms/op
[113] RegressionBenchmark.main              bench/resources/go_suite/comparison.jsonnet  avgt       25.669          ms/op
[113] RegressionBenchmark.main             bench/resources/go_suite/comparison2.jsonnet  avgt       78.312          ms/op
[113] RegressionBenchmark.main        bench/resources/go_suite/escapeStringJson.jsonnet  avgt        0.046          ms/op
[113] RegressionBenchmark.main                   bench/resources/go_suite/foldl.jsonnet  avgt        9.194          ms/op
[113] RegressionBenchmark.main             bench/resources/go_suite/lstripChars.jsonnet  avgt        0.630          ms/op
[113] RegressionBenchmark.main          bench/resources/go_suite/manifestJsonEx.jsonnet  avgt        0.067          ms/op
[113] RegressionBenchmark.main          bench/resources/go_suite/manifestTomlEx.jsonnet  avgt        0.083          ms/op
[113] RegressionBenchmark.main         bench/resources/go_suite/manifestYamlDoc.jsonnet  avgt        0.070          ms/op
[113] RegressionBenchmark.main                  bench/resources/go_suite/member.jsonnet  avgt        0.673          ms/op
[113] RegressionBenchmark.main                bench/resources/go_suite/parseInt.jsonnet  avgt        0.045          ms/op
[113] RegressionBenchmark.main                 bench/resources/go_suite/reverse.jsonnet  avgt       11.370          ms/op
[113] RegressionBenchmark.main             bench/resources/go_suite/rstripChars.jsonnet  avgt        0.625          ms/op
[113] RegressionBenchmark.main              bench/resources/go_suite/stripChars.jsonnet  avgt        0.599          ms/op
[113] RegressionBenchmark.main                  bench/resources/go_suite/substr.jsonnet  avgt        0.171          ms/op
[113/113] ============================== __.runRegressions ============================== 410s

on 0.5.5-fix

[114] Benchmark                                                                  (path)  Mode  Cnt   Score   Error  Units
[114] RegressionBenchmark.main             bench/resources/bug_suite/assertions.jsonnet  avgt        0.309          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.01.jsonnet  avgt        0.097          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.02.jsonnet  avgt       41.078          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.03.jsonnet  avgt       12.961          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.04.jsonnet  avgt       31.405          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.06.jsonnet  avgt        0.470          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.07.jsonnet  avgt        2.930          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.08.jsonnet  avgt        0.083          ms/op
[114] RegressionBenchmark.main               bench/resources/cpp_suite/bench.09.jsonnet  avgt        0.147          ms/op
[114] RegressionBenchmark.main         bench/resources/cpp_suite/gen_big_object.jsonnet  avgt        0.963          ms/op
[114] RegressionBenchmark.main      bench/resources/cpp_suite/large_string_join.jsonnet  avgt        2.323          ms/op
[114] RegressionBenchmark.main  bench/resources/cpp_suite/large_string_template.jsonnet  avgt        3.096          ms/op
[114] RegressionBenchmark.main             bench/resources/cpp_suite/realistic1.jsonnet  avgt        3.181          ms/op
[114] RegressionBenchmark.main             bench/resources/cpp_suite/realistic2.jsonnet  avgt       73.071          ms/op
[114] RegressionBenchmark.main                  bench/resources/go_suite/base64.jsonnet  avgt        0.867          ms/op
[114] RegressionBenchmark.main            bench/resources/go_suite/base64Decode.jsonnet  avgt        0.686          ms/op
[114] RegressionBenchmark.main       bench/resources/go_suite/base64DecodeBytes.jsonnet  avgt        9.580          ms/op
[114] RegressionBenchmark.main       bench/resources/go_suite/base64_byte_array.jsonnet  avgt        1.505          ms/op
[114] RegressionBenchmark.main              bench/resources/go_suite/comparison.jsonnet  avgt       25.653          ms/op
[114] RegressionBenchmark.main             bench/resources/go_suite/comparison2.jsonnet  avgt       77.781          ms/op
[114] RegressionBenchmark.main        bench/resources/go_suite/escapeStringJson.jsonnet  avgt        0.071          ms/op
[114] RegressionBenchmark.main                   bench/resources/go_suite/foldl.jsonnet  avgt        9.213          ms/op
[114] RegressionBenchmark.main             bench/resources/go_suite/lstripChars.jsonnet  avgt        0.680          ms/op
[114] RegressionBenchmark.main          bench/resources/go_suite/manifestJsonEx.jsonnet  avgt        0.093          ms/op
[114] RegressionBenchmark.main          bench/resources/go_suite/manifestTomlEx.jsonnet  avgt        0.110          ms/op
[114] RegressionBenchmark.main         bench/resources/go_suite/manifestYamlDoc.jsonnet  avgt        0.094          ms/op
[114] RegressionBenchmark.main                  bench/resources/go_suite/member.jsonnet  avgt        0.713          ms/op
[114] RegressionBenchmark.main                bench/resources/go_suite/parseInt.jsonnet  avgt        0.070          ms/op
[114] RegressionBenchmark.main                 bench/resources/go_suite/reverse.jsonnet  avgt       11.185          ms/op
[114] RegressionBenchmark.main             bench/resources/go_suite/rstripChars.jsonnet  avgt        0.656          ms/op
[114] RegressionBenchmark.main              bench/resources/go_suite/stripChars.jsonnet  avgt        0.765          ms/op
[114] RegressionBenchmark.main                  bench/resources/go_suite/substr.jsonnet  avgt        0.231          ms/op
[114/114] ============================== __.runRegressions ============================== 418s

[He-Pin]

@stephenamar-db I think the performance seems good on main.

[stephenamar-db]

ship it