HTTP server tests broken on master in network sandbox

[Al2Klimov]

Describe the bug

88% tests passed, 25 tests failed out of 209

To Reproduce

with import <nixpkgs> {}; icinga2.overrideAttrs (old: {
#cmakeFlags = old.cmakeFlags ++ ["-DICINGA2_UNITY_BUILD=OFF"];
#buildInputs = old.buildInputs ++ [protobuf];
#doCheck = false;
src = fetchFromGitHub {
          owner = "Icinga";
          repo = "icinga2";
          #rev = "28c3d8193cae59cea6cad3de5d21ce5a3549edb2"; # otel
          #hash = "sha256-SsOr8vo/kNWFIWdFLWzwRhppPPWLgJq2LXek8TTCxLs=";
#rev = "v2.15.0";
#hash = "sha256-IuK6kD2OBAm7bKtPcMnplwNSGyGUcX3UWiwm8Vry1is=";
#rev = "4b4d15ce727571aa5e6f603546e6236b3131fb0d"; # master
#hash = "sha256-WKUIktUUqQYPLKKUGs6DVUv/8WVIBL3kOUUpX8cgpAU=";
#rev = "a2dc35031c029a076bad43797525ac2211bd8efe"; # bad
#hash = "sha256-eHE5iYYM86/JkaJgXo2FUMgnyZRgg6iopW8FXMHdugU=";
#rev = "d1582766eca12d1c0e2ad03b7707bc93066328d6"; # good
#hash = "sha256-lmiWHCBrCXTeXVbqVxa+loi/ZUc74Jzgu5N4WHUsSJ8=";
#rev = "c2c9b6b159b517805bf40f99d4c904548e315f7d"; # bad
#hash = "sha256-cO/ZMu4mhIMDP1BFPuvpjakuag/80GX3yiqSZ4/KPaM=";
#rev = "b4681b10eca47ccf626b887a207b927e6dbf60e2"; # bad
#hash = "sha256-j0D9A5ZKbC7oyNXx7EOQPB3QgMvOqqqDyScZgcNnrZQ=";
#rev = "0c2fd00383cab7d64b401fcc8ab20fe26222405e"; # good
#hash = "sha256-Ey9QgczRM0vqlHQZv10T8LRcCTagE7BzXYeNNyThuH0=";
#rev = "bb75d7301282e7c034abbef78cdacc87234915a7"; # good
#hash = "sha256-yPrlHBaqfRHdsVTXRp2vv5wi+J5V2bXFpcVUXGPNVKo=";
rev = "7373f36cc5651017acff9bb0f13c8cb4b494b885"; # bad
hash = "sha256-c8Q317GBL4inFeiE4xlf7ufIn6GtuyRFPdja9z5/Km0=";
#rev = "4782ea8a75482a0c2d1521e96a4f3149e2bb79fc"; # good
#hash = "sha256-pc/gqhTsU9Z6blXZ+UjQWpDkgcqjtVqDczc43cZkVmU=";
        };
})

Expected behavior

Tests work when being run offline.

Screenshots

Total Test time (real) =  12.60 sec

The following tests FAILED:
        175 - base-remote_httpmessage/request_parse (Failed)
        176 - base-remote_httpmessage/request_params (Failed)
        177 - base-remote_httpmessage/response_clear (Failed)
        178 - base-remote_httpmessage/response_flush_nothrow (Failed)
        179 - base-remote_httpmessage/response_flush_throw (Failed)
        180 - base-remote_httpmessage/response_write_empty (Failed)
        181 - base-remote_httpmessage/response_write_fixed (Failed)
        182 - base-remote_httpmessage/response_write_chunked (Failed)
        183 - base-remote_httpmessage/response_sendjsonbody (Failed)
        184 - base-remote_httpmessage/response_sendjsonerror (Failed)
        185 - base-remote_httpmessage/response_sendfile (Failed)
        186 - base-remote_httpserverconnection/expect_100_continue (Failed)
        187 - base-remote_httpserverconnection/bad_request (Failed)
        188 - base-remote_httpserverconnection/error_access_control (Failed)
        189 - base-remote_httpserverconnection/error_accept_header (Failed)
        190 - base-remote_httpserverconnection/authenticate_cn (Failed)
        191 - base-remote_httpserverconnection/authenticate_passwd (Failed)
        192 - base-remote_httpserverconnection/authenticate_error_wronguser (Failed)
        193 - base-remote_httpserverconnection/authenticate_error_wrongpasswd (Failed)
        194 - base-remote_httpserverconnection/reuse_connection (Failed)
        195 - base-remote_httpserverconnection/wg_abort (Failed)
        196 - base-remote_httpserverconnection/client_shutdown (Failed)
        197 - base-remote_httpserverconnection/handler_throw_error (Failed)
        198 - base-remote_httpserverconnection/handler_throw_streaming (Failed)
        199 - base-remote_httpserverconnection/liveness_disconnect (Failed)
Errors while running CTest

Your Environment

I use NixOS btw.

Additional context

I was testing #10685.

[jschmidt-icinga]

If I understand this right, this happens because NixOS builds/tests in a sandbox with disabled network? If that's the case maybe even tests that setup local network connections might fail. Could you override the ctest command to do something like ctest -LE http instead? I'll see that in the future we have an additional test label for all tests that require network (because more will be added) so they can also be conveniently excluded in that way.

[Al2Klimov]

In contrast, the AI suggested me skipping the individual test if binding to 127.0.0.1:0 fails.

[jschmidt-icinga]

I don't think we should skip tests automatically depending on the runtime environment. If anything we could add a build option ICINGA2_NO_NETWORK_TESTS or something like that that disables the tests at build-time. I don't really see the reason though if there's an easy way to exclude all network tests at runtime via a ctest option.

From looking it up it seems Nix has ctestFlags where you could add the -LE http much in the same way as cmakeFlags in your snippet above.

[Al2Klimov]

it seems Nix has ctestFlags where you could add the -LE http

Good idea! Sounds better than patching our list of tests as of Icinga 2.16...

CC @dasJ @Conni2461 @helsinki-Jo

[julianbrost]

If I understand this right, this happens because NixOS builds/tests in a sandbox with disabled network? If that's the case maybe even tests that setup local network connections might fail.

Yes, this isn't about being offline but rather about how NixOS sandboxes these processes. I just tried it, the tests work just fine offline, all they need is a working loopback interface (the test process binds the HTTP server to a random port on localhost and then connects back to itself to run the tests). How exactly does NixOS implement that sandbox? Is there a way we could make the tests work there? Or is this a strict "no TCP whatsoever" and there's pretty much no chance to get this working?

In contrast, the AI suggested me skipping the individual test if binding to 127.0.0.1:0 fails.

Is that just speculation or do you have test output you didn't share in this issue that suggests that the socket bind is the operation that is failing?

[jschmidt-icinga]

Is there a way we could make the tests work there?

Aside from making real network work in the NixOS sandbox, the ideal would be to not even require a loopback interface.

My initial idea for testing the HTTP stuff was to add a mock stream type specifically for testing¹, which would have made many of the tests much easier and would have allowed to accurately simulate different types of connection failures.

However that would have meant templating all the classes and lots of (at the time) unnecessarily general code (lots of std::visit()). At this point, with #10692, we're almost at the point where we don't have to touch production code anymore to add such a mock stream to the Stream variant, maybe give it a more general name and use it consistently in new code. The existing HTTP unit tests would be easy to migrate.

Maybe not something to delay 2.16 for, but still IMHO worth considering.

Footnotes

1. I think boost::beast already has a stream like that, but I'm not sure if it would be enough for our purposes. ↩

[Al2Klimov]

How exactly does NixOS implement that sandbox?

PING @oxzi

skipping the individual test if binding to 127.0.0.1:0 fails.

Is that just speculation

As you said yourself, you had a working loopback interface and tests passed offline. Obviously, without a working loopback interface you can't bind(2) to 127.0.0.0/8 as there's no such local address then.

[Al2Klimov]

@jschmidt-icinga FYI, Gentoo is also affected, at least if one explicitly uses FEATURES="network-sandbox": FEATURES="test network-sandbox" emerge -av1 =net-analyzer/icinga2-9999

The following tests FAILED:
	 80 - base_tlsutility/create_verify_ca (Failed)
	 81 - base_tlsutility/create_verify_leaf_certs (Failed)
	183 - remote_httpserverconnection/expect_100_continue (Failed) http
	184 - remote_httpserverconnection/bad_request (Failed)  http
	185 - remote_httpserverconnection/error_access_control (Failed) http
	186 - remote_httpserverconnection/error_accept_header (Failed) http
	187 - remote_httpserverconnection/authenticate_cn (Failed) http
	188 - remote_httpserverconnection/authenticate_passwd (Failed) http
	189 - remote_httpserverconnection/authenticate_error_wronguser (Failed) http
	190 - remote_httpserverconnection/authenticate_error_wrongpasswd (Failed) http
	191 - remote_httpserverconnection/reuse_connection (Failed) http
	192 - remote_httpserverconnection/wg_abort (Failed)     http
	193 - remote_httpserverconnection/client_shutdown (Failed) http
	194 - remote_httpserverconnection/handler_throw_error (Failed) http
	195 - remote_httpserverconnection/handler_throw_streaming (Failed) http
	196 - remote_httpserverconnection/liveness_disconnect (Failed) http
	197 - remote_httpmessage/request_parse (Failed)         http
	198 - remote_httpmessage/request_params (Failed)        http
	199 - remote_httpmessage/response_clear (Failed)        http
	200 - remote_httpmessage/response_flush_nothrow (Failed) http
	201 - remote_httpmessage/response_flush_throw (Failed)  http
	202 - remote_httpmessage/response_write_empty (Failed)  http
	203 - remote_httpmessage/response_write_fixed (Failed)  http
	204 - remote_httpmessage/response_write_chunked (Failed) http
	205 - remote_httpmessage/response_sendjsonbody (Failed) http
	206 - remote_httpmessage/response_sendjsonerror (Failed) http
	207 - remote_httpmessage/response_sendfile (Failed)     http
Errors while running CTest

[Al2Klimov]

... or if someone just enabled the tests: FEATURES="test"

[jschmidt-icinga]

80 - base_tlsutility/create_verify_ca (Failed)
81 - base_tlsutility/create_verify_leaf_certs (Failed)

Interesting. These two don't do anything with the loopback interface or networking at all. But they use the CertificateFixture, same as all the http tests, which do lots of filesystem stuff. I'll try this on my own system to see if rerunning the build with ctest -V will output anything useful.

[jschmidt-icinga]

Update: I could not reproduce this with my own Gentoo system. On @Al2Klimov's test system, which was a livecd chroot environment, the reason for the failing tests was, as I already expected, certificate-related. With CTEST_OUTPUT_ON_FAILURE in /etc/portage/make.conf I get the following:

/var/tmp/portage/net-analyzer/icinga2-9999/work/icinga2-9999/lib/base/tlsutility.cpp(531): fatal error: in "MakeX509CSR": /var/tmp/portage/net-analyzer/icinga2-9999/work/icinga2-9999/lib/base/tlsutility.cpp(531): Throw in function MakeX509CSR
Dynamic exception type: boost::wrapexcept<icinga::openssl_error>
std::exception::what: std::exception
[Context] = [icinga::errinfo_openssl_error_*] = 2147483650
[boost::errinfo_file_name_*] = /var/tmp/portage/net-analyzer/icinga2-9999/work/icinga2-9999_build/test/data/ca//ca.key
[boost::errinfo_api_function_*] = BIO_new_file

I don't really know what could be the exact issue here, but since it is unrelated to the network sandbox, I'd stop here. We can keep this in mind and investigate further in a separate issue if this pops up anywhere else.

[jschmidt-icinga]

#10767 adds a network label to the remaining network dependent tests that didn't have it before, so a simple ctest -LE network should be enough in a sandboxed context that doesn't allow binding to localhost.

(Also the same PR should fix the parallel tests on Gentoo which were also discussed in this issue.)