go tests fail on 32-bit systems

[rogers0]

two tests pass on 64-bit systems, but fail on 32-bit systems, such as i386, armhf.

error log:

=== RUN   Test_AsString
    Test_AsString: conversion_test.go:143: 
        	Error Trace:	conversion_test.go:143
        	Error:      	Not equal: 
        	            	expected: "6273346999"
        	            	actual  : "6.273346999e+09"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-6273346999
        	            	+6.273346999e+09
        	Test:       	Test_AsString
--- FAIL: Test_AsString (0.00s)


=== RUN   TestNewCollection
--- FAIL: TestNewCollection (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x804a78c]

goroutine 21 [running]:
testing.tRunner.func1.1(0x83569a0, 0x863c860)
	/usr/lib/go-1.14/src/testing/testing.go:940 +0x27c
testing.tRunner.func1(0x8ce01e0)
	/usr/lib/go-1.14/src/testing/testing.go:943 +0x349
panic(0x83569a0, 0x863c860)
	/usr/lib/go-1.14/src/runtime/panic.go:969 +0x122
runtime/internal/atomic.Xadd64(0x8c94f84, 0x1, 0x0, 0xffffffff, 0xf)
	/usr/lib/go-1.14/src/runtime/internal/atomic/asm_386.s:105 +0xc
github.com/viant/toolbox/data.(*CompactedSlice).Add(0x8c94f60, 0x8c4ce70)
	/build/1st/golang-github-viant-toolbox-0.32.2/obj-i686-linux-gnu/src/github.com/viant/toolbox/data/compacted.go:146 +0x72
github.com/viant/toolbox/data.TestNewCollection(0x8ce01e0)
	/build/1st/golang-github-viant-toolbox-0.32.2/obj-i686-linux-gnu/src/github.com/viant/toolbox/data/compacted_test.go:14 +0x30d
testing.tRunner(0x8ce01e0, 0x83b8410)
	/usr/lib/go-1.14/src/testing/testing.go:991 +0xb4
created by testing.(*T).Run
	/usr/lib/go-1.14/src/testing/testing.go:1042 +0x2ad
FAIL	github.com/viant/toolbox/data	0.032s

Reference:

• https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/golang-github-viant-toolbox_0.32.2-1.rbuild.log.gz
• https://tests.reproducible-builds.org/debian/rbuild/unstable/armhf/golang-github-viant-toolbox_0.32.2-1.rbuild.log.gz

[adranwit]

I've patched converter test,
However, with the second issue, it looks like using int64 data type cause panic,
If there is a way to detect test architecture I can disable that test ?

var initSize = len(s.fieldNames)
if initSize < len(data) {
	initSize = len(data)
}

line 146# atomic.AddInt64(&s.size, 1)

[rogers0]

for detecting CPU, maybe you can refer:

• https://github.com/klauspost/cpuid

[rogers0]

tried latest 0.32.3 on i386, and there're two fails:

# github.com/viant/toolbox_test [github.com/viant/toolbox.test]
src/github.com/viant/toolbox/converter_test.go:409:9: constant 1668069210749 overflows int
src/github.com/viant/toolbox/converter_test.go:521:8: constant 1668069210749 overflows int
FAIL    github.com/viant/toolbox [build failed]
# github.com/viant/toolbox/data/udf [github.com/viant/toolbox/data/udf.test]
src/github.com/viant/toolbox/data/udf/conversion_test.go:142:17: constant 6273346999 overflows int
FAIL    github.com/viant/toolbox/data/udf [build failed]
# github.com/viant/toolbox/storage_test [github.com/viant/toolbox/storage.test]
src/github.com/viant/toolbox/storage/file_info_test.go:13:3: constant 2147484141 overflows int
src/github.com/viant/toolbox/storage/file_info_test.go:14:3: constant 2147484159 overflows int
src/github.com/viant/toolbox/storage/file_info_test.go:15:3: constant 2147484128 overflows int
FAIL    github.com/viant/toolbox/storage [build failed]

another one:

=== RUN   TestCompactedSlice_SortedRange
--- FAIL: TestCompactedSlice_SortedRange (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x804a78c]

goroutine 8 [running]:
testing.tRunner.func1.1(0x83569a0, 0x863c860)
        /usr/lib/go-1.14/src/testing/testing.go:940 +0x27c
testing.tRunner.func1(0xa4a6320)
        /usr/lib/go-1.14/src/testing/testing.go:943 +0x349
panic(0x83569a0, 0x863c860)
        /usr/lib/go-1.14/src/runtime/panic.go:969 +0x122
runtime/internal/atomic.Xadd64(0xa466fe4, 0x1, 0x0, 0xfffffff, 0x0)
        /usr/lib/go-1.14/src/runtime/internal/atomic/asm_386.s:105 +0xc
github.com/viant/toolbox/data.(*CompactedSlice).Add(0xa466fc0, 0xa413540)
        /build/golang-github-viant-toolbox-0.32.3/obj-i686-linux-gnu/src/github.com/viant/toolbox/data/compacted.go:146 +0x72
github.com/viant/toolbox/data.TestCompactedSlice_SortedRange(0xa4a6320)
        /build/golang-github-viant-toolbox-0.32.3/obj-i686-linux-gnu/src/github.com/viant/toolbox/data/compacted_test.go:211 +0x12a6 
testing.tRunner(0xa4a6320, 0x83b8404)
        /usr/lib/go-1.14/src/testing/testing.go:991 +0xb4
created by testing.(*T).Run
        /usr/lib/go-1.14/src/testing/testing.go:1042 +0x2ad
FAIL    github.com/viant/toolbox/data   0.145s
FAIL    github.com/viant/toolbox/data/udf [build failed]

[adranwit]

Updated test with skip for non 64 bit arch

[rogers0]

@adranwit Thanks for your update!

For latest 0.33.0, the result is:

• https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/golang-github-viant-toolbox_0.33.0-1.rbuild.log.gz
• https://tests.reproducible-builds.org/debian/rbuild/unstable/armhf/golang-github-viant-toolbox_0.33.0-1.rbuild.log.gz

# github.com/viant/toolbox/data/udf [github.com/viant/toolbox/data/udf.test]
src/github.com/viant/toolbox/data/udf/conversion_test.go:151:17: constant 2323232323223 overflows int
# github.com/viant/toolbox/storage_test [github.com/viant/toolbox/storage.test]
src/github.com/viant/toolbox/storage/file_info_test.go:13:3: constant 2147484141 overflows int
src/github.com/viant/toolbox/storage/file_info_test.go:14:3: constant 2147484159 overflows int
src/github.com/viant/toolbox/storage/file_info_test.go:15:3: constant 2147484128 overflows int
# github.com/viant/toolbox_test [github.com/viant/toolbox.test]
src/github.com/viant/toolbox/converter_test.go:527:8: constant 1668069210749 overflows int

another one:

=== RUN   TestNewCollection
--- FAIL: TestNewCollection (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x804a78c]

goroutine 6 [running]:
testing.tRunner.func1.1(0x83594e0, 0x8642880)
	/usr/lib/go-1.14/src/testing/testing.go:940 +0x27c
testing.tRunner.func1(0x94ac1e0)
	/usr/lib/go-1.14/src/testing/testing.go:943 +0x349
panic(0x83594e0, 0x8642880)
	/usr/lib/go-1.14/src/runtime/panic.go:969 +0x122
runtime/internal/atomic.Xadd64(0x9428fb4, 0x1, 0x0, 0xffffffff, 0xffffff)
	/usr/lib/go-1.14/src/runtime/internal/atomic/asm_386.s:105 +0xc
github.com/viant/toolbox/data.(*CompactedSlice).Add(0x9428f90, 0x9451e70)
	/build/1st/golang-github-viant-toolbox-0.33.0/obj-i686-linux-gnu/src/github.com/viant/toolbox/data/compacted.go:146 +0x72
github.com/viant/toolbox/data.TestNewCollection(0x94ac1e0)
	/build/1st/golang-github-viant-toolbox-0.33.0/obj-i686-linux-gnu/src/github.com/viant/toolbox/data/compacted_test.go:17 +0x31b
testing.tRunner(0x94ac1e0, 0x83bbe50)
	/usr/lib/go-1.14/src/testing/testing.go:991 +0xb4
created by testing.(*T).Run
	/usr/lib/go-1.14/src/testing/testing.go:1042 +0x2ad
FAIL	github.com/viant/toolbox/data	0.036s

[adranwit]

Looks like 64 detection, I've added to the test does not work,
if cpuid.CPU.Family < 64 {
return
}

could you please print me the following on the machine the test failed ?

fmt.Println("Name:", cpuid.CPU.BrandName)
fmt.Println("PhysicalCores:", cpuid.CPU.PhysicalCores)
fmt.Println("ThreadsPerCore:", cpuid.CPU.ThreadsPerCore)
fmt.Println("LogicalCores:", cpuid.CPU.LogicalCores)
fmt.Println("Family", cpuid.CPU.Family, "Model:", cpuid.CPU.Model)
fmt.Println("Features:", cpuid.CPU.Features)
fmt.Println("Cacheline bytes:", cpuid.CPU.CacheLine)
fmt.Println("L1 Data Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L1 Instruction Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L2 Cache:", cpuid.CPU.Cache.L2, "bytes")
fmt.Println("L3 Cache:", cpuid.CPU.Cache.L3, "bytes")

[rogers0]

For debian, I used the patch:
https://salsa.debian.org/go-team/packages/golang-github-viant-toolbox/-/blob/debian/sid/debian/patches/0001-Skip-some-tests-fail-at-git-pbuilder-environment.patch

This most important part is:

-	if cpuid.CPU.CacheLine < 64 {
-		return
+	if (32 << uintptr(^uintptr(0)>>63)) < 64 {
+		t.Skip()

[rogers0]

For debian, I used the patch:
https://salsa.debian.org/go-team/packages/golang-github-viant-toolbox/-/blob/debian/sid/debian/patches/0001-Skip-some-tests-fail-at-git-pbuilder-environment.patch

Let me clarify. I do not mean you should merge this patch to upstream repository.
I just show you how I workaround current errors from reproducible builds.
Thanks for your working!

[adranwit]

Thanks for the suggestion, updated tests

[adranwit]

Have you run test with the latest code ?

[rogers0]

Sorry for the late reply.
The test code still build fails under 32-bit systems, so I have to disable it.

• https://salsa.debian.org/go-team/packages/golang-github-viant-toolbox/-/blob/debian/sid/debian/rules#L17

But I still think some code need to be updated:

• https://salsa.debian.org/go-team/packages/golang-github-viant-toolbox/-/blob/debian/sid/debian/patches/0002-Skip-some-tests-fail-at-git-pbuilder-environment.patch

I'll send you a PR later.

[adranwit]

Removed 32 bit test

[rogers0]

Except PR #39
there're a few other issues.
(since I set ignore all 32-bit go test errors, so currently reproducible builds all "passed")

• https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/golang-github-viant-toolbox.html

All current log based on latest, + PR #39 patch:

• test fails on i386 (x86 32-bit)

=== RUN   TestConverter_AsInt
    TestConverter_AsInt: converter_test.go:702: 
        	Error Trace:	converter_test.go:702
        	Error:      	Should be true
        	Test:       	TestConverter_AsInt
--- FAIL: TestConverter_AsInt (0.00s)

https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/golang-github-viant-toolbox_0.33.0-11.rbuild.log.gz

• panic: runtime error on i386 (x86 32-bit) and armhf (arm 32-bit)

=== RUN   TestLogger
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x804ab9b]

goroutine 165 [running]:
runtime/internal/atomic.Load64(0xa86b2ec, 0xbfc1ad5d, 0xd905ff2)
	/usr/lib/go-1.14/src/runtime/internal/atomic/asm_386.s:194 +0xb
github.com/viant/toolbox.(*LogStream).isFrequencyFlushNeeded(0xa86b2c0, 0x0)
	/build/1st/golang-github-viant-toolbox-0.33.0/obj-i686-linux-gnu/src/github.com/viant/toolbox/file_logger.go:138 +0x69
github.com/viant/toolbox.(*LogStream).manageWritesInBatch(0xa86b2c0)
	/build/1st/golang-github-viant-toolbox-0.33.0/obj-i686-linux-gnu/src/github.com/viant/toolbox/file_logger.go:166 +0x23b
github.com/viant/toolbox.(*FileLogger).NewLogStream.func1(0xa86b2c0)
	/build/1st/golang-github-viant-toolbox-0.33.0/obj-i686-linux-gnu/src/github.com/viant/toolbox/file_logger.go:219 +0x21
created by github.com/viant/toolbox.(*FileLogger).NewLogStream
	/build/1st/golang-github-viant-toolbox-0.33.0/obj-i686-linux-gnu/src/github.com/viant/toolbox/file_logger.go:218 +0xfe
FAIL	github.com/viant/toolbox	0.239s

https://tests.reproducible-builds.org/debian/rbuild/unstable/armhf/golang-github-viant-toolbox_0.33.0-11.rbuild.log.gz

• build error on 32-bit systems:

# github.com/viant/toolbox/data/udf [github.com/viant/toolbox/data/udf.test]
src/github.com/viant/toolbox/data/udf/conversion_test.go:142:17: constant 2323232323223 overflows int
# github.com/viant/toolbox/storage_test [github.com/viant/toolbox/storage.test]
src/github.com/viant/toolbox/storage/file_info_test.go:13:3: constant 2147484141 overflows int
src/github.com/viant/toolbox/storage/file_info_test.go:14:3: constant 2147484159 overflows int
src/github.com/viant/toolbox/storage/file_info_test.go:15:3: constant 2147484128 overflows int

https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/golang-github-viant-toolbox_0.33.0-11.rbuild.log.gz

Thank you!