Errors with docker test script

[tetlowgm]

I setup an Ubuntu 22.04 host to do docker tests and I'm getting the following errors when run against rbsec/sslscan@master. @jtesta Do you have any insights into what might be going on here?

Thanks.

Running all tests...
Test #1 passed.
Test #2 passed.
Test #3 passed.
Test #4 passed.
Test #5 passed.
Test #6 passed.
Test #7 passed.
Test #8 passed.
Test #9 skipped.
Test #10 skipped.
Test #11 passed.
Test #12 passed.
Test #13 FAILED.

--- docker_test/expected_output/test_13.txt	2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_13.txt	2025-01-26 00:26:58.671104306 +0000
@@ -6,8 +6,8 @@
   SSL/TLS Protocols:
 SSLv2     disabled
 SSLv3     disabled
-TLSv1.0   enabled
-TLSv1.1   enabled
+TLSv1.0   disabled
+TLSv1.1   disabled
 TLSv1.2   enabled
 TLSv1.3   enabled

@@ -15,7 +15,7 @@
 Server supports TLS Fallback SCSV

   TLS renegotiation:
-Secure session renegotiation supported
+Session renegotiation not supported

   TLS Compression:
 Compression disabled
@@ -23,8 +23,6 @@
   Heartbleed:
 TLSv1.3 not vulnerable to heartbleed
 TLSv1.2 not vulnerable to heartbleed
-TLSv1.1 not vulnerable to heartbleed
-TLSv1.0 not vulnerable to heartbleed

   Supported Server Cipher(s):
 Preferred TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve 25519 DHE 253
@@ -49,24 +47,13 @@
 Accepted  TLSv1.2  128 bits  AES128-CCM
 Accepted  TLSv1.2  256 bits  AES256-SHA
 Accepted  TLSv1.2  128 bits  AES128-SHA
-Preferred TLSv1.1  256 bits  ECDHE-RSA-AES256-SHA          Curve 25519 DHE 253
-Accepted  TLSv1.1  256 bits  DHE-RSA-AES256-SHA            DHE 2048 bits
-Accepted  TLSv1.1  128 bits  ECDHE-RSA-AES128-SHA          Curve 25519 DHE 253
-Accepted  TLSv1.1  128 bits  DHE-RSA-AES128-SHA            DHE 2048 bits
-Accepted  TLSv1.1  256 bits  AES256-SHA
-Accepted  TLSv1.1  128 bits  AES128-SHA
-Preferred TLSv1.0  256 bits  ECDHE-RSA-AES256-SHA          Curve 25519 DHE 253
-Accepted  TLSv1.0  256 bits  DHE-RSA-AES256-SHA            DHE 2048 bits
-Accepted  TLSv1.0  128 bits  ECDHE-RSA-AES128-SHA          Curve 25519 DHE 253
-Accepted  TLSv1.0  128 bits  DHE-RSA-AES128-SHA            DHE 2048 bits
-Accepted  TLSv1.0  256 bits  AES256-SHA
-Accepted  TLSv1.0  128 bits  AES128-SHA

   Server Key Exchange Group(s):
 TLSv1.3  128 bits  secp256r1 (NIST P-256)
 TLSv1.3  192 bits  secp384r1 (NIST P-384)
 TLSv1.3  260 bits  secp521r1 (NIST P-521)
 TLSv1.3  128 bits  x25519
+TLSv1.3  224 bits  x448
 TLSv1.3  112 bits  ffdhe2048
 TLSv1.3  128 bits  ffdhe3072
 TLSv1.3  150 bits  ffdhe4096
@@ -76,6 +63,7 @@
 TLSv1.2  192 bits  secp384r1 (NIST P-384)
 TLSv1.2  260 bits  secp521r1 (NIST P-521)
 TLSv1.2  128 bits  x25519
+TLSv1.2  224 bits  x448

   SSL Certificate:
 Signature Algorithm: sha256WithRSAEncryption

Test #14 FAILED.

--- docker_test/expected_output/test_14.txt	2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_14.txt	2025-01-26 00:27:01.675104757 +0000
@@ -25,21 +25,21 @@
 TLSv1.2 not vulnerable to heartbleed

   Supported Server Cipher(s):
-Preferred TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve P-521 DHE 521
-Accepted  TLSv1.3  256 bits  TLS_AES_256_GCM_SHA384        Curve P-521 DHE 521
-Accepted  TLSv1.3  256 bits  TLS_CHACHA20_POLY1305_SHA256  Curve P-521 DHE 521
-Accepted  TLSv1.3  128 bits  TLS_AES_128_CCM_SHA256        Curve P-521 DHE 521
-Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve P-521 DHE 521
+Preferred TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve 448 DHE 448
+Accepted  TLSv1.3  256 bits  TLS_AES_256_GCM_SHA384        Curve 448 DHE 448
+Accepted  TLSv1.3  256 bits  TLS_CHACHA20_POLY1305_SHA256  Curve 448 DHE 448
+Accepted  TLSv1.3  128 bits  TLS_AES_128_CCM_SHA256        Curve 448 DHE 448
+Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve 448 DHE 448
 Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-GCM-SHA384     DHE 8192 bits
-Accepted  TLSv1.2  256 bits  ECDHE-RSA-CHACHA20-POLY1305   Curve P-521 DHE 521
+Accepted  TLSv1.2  256 bits  ECDHE-RSA-CHACHA20-POLY1305   Curve 448 DHE 448
 Accepted  TLSv1.2  256 bits  DHE-RSA-CHACHA20-POLY1305     DHE 8192 bits
 Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-CCM            DHE 8192 bits
-Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve P-521 DHE 521
+Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve 448 DHE 448
 Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-GCM-SHA256     DHE 8192 bits
 Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-CCM            DHE 8192 bits
-Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve P-521 DHE 521
+Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve 448 DHE 448
 Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA            DHE 8192 bits
-Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve P-521 DHE 521
+Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve 448 DHE 448
 Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA            DHE 8192 bits
 Accepted  TLSv1.2  256 bits  AES256-GCM-SHA384
 Accepted  TLSv1.2  256 bits  AES256-CCM
@@ -50,8 +50,10 @@

   Server Key Exchange Group(s):
 TLSv1.3  260 bits  secp521r1 (NIST P-521)
+TLSv1.3  224 bits  x448
 TLSv1.3  192 bits  ffdhe8192
 TLSv1.2  260 bits  secp521r1 (NIST P-521)
+TLSv1.2  224 bits  x448

   SSL Certificate:
 Signature Algorithm: sha256WithRSAEncryption

Test #15 FAILED.

--- docker_test/expected_output/test_15.txt	2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_15.txt	2025-01-26 00:27:03.063104961 +0000
@@ -6,8 +6,8 @@
   SSL/TLS Protocols:
 SSLv2     disabled
 SSLv3     disabled
-TLSv1.0   enabled
-TLSv1.1   enabled
+TLSv1.0   disabled
+TLSv1.1   disabled
 TLSv1.2   enabled
 TLSv1.3   enabled

@@ -15,7 +15,7 @@
 Server supports TLS Fallback SCSV

   TLS renegotiation:
-Secure session renegotiation supported
+Session renegotiation not supported

   TLS Compression:
 Compression disabled
@@ -23,8 +23,6 @@
   Heartbleed:
 TLSv1.3 not vulnerable to heartbleed
 TLSv1.2 not vulnerable to heartbleed
-TLSv1.1 not vulnerable to heartbleed
-TLSv1.0 not vulnerable to heartbleed

   Supported Server Cipher(s):
 Preferred TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve 25519 DHE 253
@@ -38,16 +36,13 @@
 Accepted  TLSv1.2  128 bits  ECDHE-ECDSA-AES128-CCM        Curve 25519 DHE 253
 Accepted  TLSv1.2  256 bits  ECDHE-ECDSA-AES256-SHA        Curve 25519 DHE 253
 Accepted  TLSv1.2  128 bits  ECDHE-ECDSA-AES128-SHA        Curve 25519 DHE 253
-Preferred TLSv1.1  256 bits  ECDHE-ECDSA-AES256-SHA        Curve 25519 DHE 253
-Accepted  TLSv1.1  128 bits  ECDHE-ECDSA-AES128-SHA        Curve 25519 DHE 253
-Preferred TLSv1.0  256 bits  ECDHE-ECDSA-AES256-SHA        Curve 25519 DHE 253
-Accepted  TLSv1.0  128 bits  ECDHE-ECDSA-AES128-SHA        Curve 25519 DHE 253

   Server Key Exchange Group(s):
 TLSv1.3  128 bits  secp256r1 (NIST P-256)
 TLSv1.3  192 bits  secp384r1 (NIST P-384)
 TLSv1.3  260 bits  secp521r1 (NIST P-521)
 TLSv1.3  128 bits  x25519
+TLSv1.3  224 bits  x448
 TLSv1.3  112 bits  ffdhe2048
 TLSv1.3  128 bits  ffdhe3072
 TLSv1.3  150 bits  ffdhe4096
@@ -57,6 +52,7 @@
 TLSv1.2  192 bits  secp384r1 (NIST P-384)
 TLSv1.2  260 bits  secp521r1 (NIST P-521)
 TLSv1.2  128 bits  x25519
+TLSv1.2  224 bits  x448

   SSL Certificate:
 Signature Algorithm: sha256WithRSAEncryption

Test #16 passed.
Test #17 passed.
Test #18 FAILED.

--- docker_test/expected_output/test_18.txt	2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_18.txt	2025-01-26 00:27:06.343105429 +0000
@@ -33,6 +33,7 @@
 TLSv1.2  192 bits  secp384r1 (NIST P-384)
 TLSv1.2  260 bits  secp521r1 (NIST P-521)
 TLSv1.2  128 bits  x25519
+TLSv1.2  224 bits  x448

   SSL Certificate:
 Signature Algorithm: sha256WithRSAEncryption



!! SOME TESTS FAILED !!

[jtesta]

Generally, the Docker tests will spin up containers for your newly- built sslscan to run against. The output will be diff'ed against the expected output, and any variation will cause a failure. When new functionality is added, the expected results may change, and hence, the tests must be updated. Of course, failures can also be the result of new bugs introduced by the code update. In this case, since you're only removing supposedly dead code from the code base, the expected results should all be the same. Test #13 failed (among other reasons) because TLS v1.0 and v1.1 are enabled on the test endpoint, but your sslscan build thinks they're disabled. Also, strangely, your sslscan build is detecting that x448 is enabled in TLS v1.2 and v1.3, even though the master branch does not.

[tetlowgm]

My point was this was against the HEAD of the master branch without any local changes at all. My expectation would be there are no failed tests, but when I run it, the output I'm seeing is what I've put in this issue. Any pointers on where to look at this? Am I the only one seeing this?

For reference, I'm using an Ubuntu 22.04 aarch64 VM running on my Mac. I don't think that should make a difference, but I thought I would mention.

[jtesta]

I just cloned the master branch into a fresh directory, built it with `make -f Makefile static`, then ran `./docker_test.sh` and got all passing tests. Just so we make extra sure, can you try the same? (i.e.: clone into a fresh directory just to make sure the master HEAD is being built & tested.)

For reference, I'm using an Ubuntu 22.04 aarch64 VM running on my Mac.

This would be my second suspicion. When you run `docker_test.sh` on a new machine, it builds a Docker test image named "sslscan-test" with a couple versions of OpenSSL and GnuTLS; the server executables in this image are what the locally-built sslscan is tested against. I've only built this image on x86_64 machines, but perhaps the resulting image is different on aarch64! Hence, the problem may be with the test image itself. What git branch of yours is resulting in these failures? I'll try running tests on your branch with my x86_64 test image and see what happens.

[jtesta]

I spun up an aarch64 machine in AWS and found that some of the tests fail. So this isn't a problem on your end. Interestingly, the tests fail for different reasons in AWS than they do for you!

Until this is fixed, I suppose I'll simply run the Docker tests on your PRs as they come in. Please ping me, and I'll do my best to be quick about it!