contrib/rootfs-builder: Support timestamps and xz compression#598
Merged
liangchenye merged 6 commits intoopencontainers:masterfrom Mar 14, 2018
Merged
Conversation
wking
changed the title
Contributor
Author
|
This needs more work for symlink and |
"busybox --list" includes "busybox" itself. Exclude that item from the list of symlinks. Signed-off-by: Alban Crequy <alban@kinvolk.io> Signed-off-by: W. Trevor King <wking@tremily.us>
The rootfs needs to have /proc, /dev, /sys pre-created because the runtime might not be able to create those, specially in user namespaces. Signed-off-by: Alban Crequy <alban@kinvolk.io> And sort them alphabetically. Signed-off-by: W. Trevor King <wking@tremily.us>
This saves some unnecessary network traffic for folks who already know what they want. Also guard against redundant configuration, like: $ DATE=20170907 STAGE3=stage3-amd64-20170907.tar.bz2 get-stage3.sh Signed-off-by: W. Trevor King <wking@tremily.us>
I'm not sure when the change happened (sometime since ff5e578, contrib/rootfs-builder: Support multiple architectures, 2017-09-20, opencontainers#479), but Gentoo is now using timestamps for some amd64 stages: $ curl http://distfiles.gentoo.org/releases/amd64/autobuilds/latest-stage3.txt # Latest as of Fri, 09 Mar 2018 15:00:02 +0000 # ts=1520607602 20180308T214502Z/stage3-amd64-20180308T214502Z.tar.xz 188851072 20180308T214502Z/hardened/stage3-amd64-hardened-20180308T214502Z.tar.xz 174759140 20180308T214502Z/hardened/stage3-amd64-hardened+nomultilib-20180308T214502Z.tar.xz 167639136 20180301T214503Z/hardened/stage3-amd64-hardened-selinux-20180301T214503Z.tar.xz 187319664 20180308T214502Z/stage3-amd64-nomultilib-20180308T214502Z.tar.xz 181866296 20180303/systemd/stage3-amd64-systemd-20180303.tar.bz2 281873162 20180225/uclibc/stage3-amd64-uclibc-hardened-20180225.tar.bz2 57147392 20180225/uclibc/stage3-amd64-uclibc-vanilla-20180225.tar.bz2 144490332 ... Signed-off-by: W. Trevor King <wking@tremily.us>
I'm not sure when the change happened (sometime since ff5e578, contrib/rootfs-builder: Support multiple architectures, 2017-09-20, opencontainers#479), but Gentoo is now using xz compression for some amd64 stages: $ curl http://distfiles.gentoo.org/releases/amd64/autobuilds/latest-stage3.txt # Latest as of Fri, 09 Mar 2018 15:00:02 +0000 # ts=1520607602 20180308T214502Z/stage3-amd64-20180308T214502Z.tar.xz 188851072 20180308T214502Z/hardened/stage3-amd64-hardened-20180308T214502Z.tar.xz 174759140 20180308T214502Z/hardened/stage3-amd64-hardened+nomultilib-20180308T214502Z.tar.xz 167639136 20180301T214503Z/hardened/stage3-amd64-hardened-selinux-20180301T214503Z.tar.xz 187319664 20180308T214502Z/stage3-amd64-nomultilib-20180308T214502Z.tar.xz 181866296 20180303/systemd/stage3-amd64-systemd-20180303.tar.bz2 281873162 20180225/uclibc/stage3-amd64-uclibc-hardened-20180225.tar.bz2 57147392 20180225/uclibc/stage3-amd64-uclibc-vanilla-20180225.tar.bz2 144490332 20180308T214502Z/stage3-x32-20180308T214502Z.tar.xz 205285160 20180308T214502Z/hardened/stage4-amd64-hardened+minimal-20180308T214502Z.tar.bz2 261776436 20180308T214502Z/hardened/stage4-amd64-hardened+minimal-nomultilib-20180308T214502Z.tar.bz2 251821245 20180308T214502Z/stage4-amd64-minimal-20180308T214502Z.tar.bz2 280918565 20180308T214502Z/stage4-amd64-minimal-nomultilib-20180308T214502Z.tar.bz2 268835860 20180303/systemd/stage4-amd64-systemd-20180303.tar.bz2 622067712 This commit adapts to that change, allowing us to continue to pull the vanilla stage3-amd64-20180308T214502Z.tar.xz. Signed-off-by: W. Trevor King <wking@tremily.us>
wking
force-pushed
the
rootfs-builder-compression
branch
from
62d839e to
94e38cf
Compare
wking
changed the title
wking
force-pushed
the
rootfs-builder-compression
branch
from
94e38cf to
a7f94a2
Compare
Rebuild for BusyBox 1.28.0 (the old roots had v1.25.1). This
removes bin/catv and adds:
* /bin/arch
* /bin/factor
* /bin/fallocate
* /bin/fsfreeze
* /bin/hexedit
* /bin/ipneigh
* /bin/link
* /bin/logread
* /bin/lsscsi
* /bin/minips
* /bin/netcat
* /bin/nl
* /bin/nproc
* /bin/nuke
* /bin/partprobe
* /bin/paste
* /bin/resume
* /bin/run-init
* /bin/setfattr
* /bin/setpriv
* /bin/shred
* /bin/ssl_client
* /bin/svc
* /bin/w
* /bin/xxd
* /dev/
* /proc/
* /sys/
These were extracted with from
stage3-i486-20180307T214502Z.tar.xz and
stage3-amd64-20180308T214502Z.tar.xz. The stage3s had SHA512
hashs of [1,2] and valid signature by [3].
Generated with:
$ cd contrib/rootfs-builder
$ make ARCHES='i486 amd64'
$ cp -f rootfs-i486.tar.gz ../../rootfs-386.tar.gz
$ cp -f rootfs-amd64.tar.gz ../../rootfs-amd64.tar.gz
[1]: 31fb95e03f82d8d88113ec14fde8cd33d2837a2207a6a8d1fe7ea1c41f1f1c34a324a232d289a348117be248bd3a18725a7d8bbbafbabfbed5b689d1821c4ee9 stage3-amd64-20180308T214502Z.tar.xz
8a201c154ba5d886697137516ccaaba40df64297d2ccf63946bae9539649118185e8d5fcf33ea1b967bd77ee8988c57d0ffde16f795b20de52480177b8672947 stage3-i486-20180307T214502Z.tar.xz
[2]: Gentoo Linux Release Engineering (Automated Weekly Release Key) <releng@gentoo.org>
Primary key fingerprint: 13EB BDBE DE7A 1277 5DFD B1BA BB57 2E0E 2D18 2910
Signed-off-by: W. Trevor King <wking@tremily.us>
Contributor
Author
|
Ok, I've cherry-picked @alban's 0453de2 and 5f15151 from #597 and added some more commits to improve |
Member
|
It fails in my test, in the latest version, it does have commands like 'arping/su/mount/passwd/umount/' . |
Contributor
Author
|
On Mon, Mar 12, 2018 at 10:05:25AM +0000, 梁辰晔 (Liang Chenye) wrote:
It fails in my test, in the latest version…
Travis is happy with it [1], although I'm not sure those tests care
about the root tarballs. Can you provide details about the failure
you're seeing?
… it does have commands like 'arping/su/mount/passwd/umount/' .
It has those:
$ sha256sum rootfs-amd64.tar.gz
197f5faf022dd5e5d25336c5721f62b1714f9ed5c44055d6f4ab828ae11edf5d rootfs-amd64.tar.gz
$ tar -tf rootfs-amd64.tar.gz | grep '/arping\|/su$\|/mount$\|/passwd\|umount'
./bin/arping
./bin/mount
./bin/passwd
./bin/su
./bin/umount
./etc/passwd
[1]: https://travis-ci.org/opencontainers/runtime-tools/builds/351488881
|
Member
|
@wking the rootfs-amd64.tar.gz works good. I think it is caused by the Makefile: The commands before 'arping' are all symbolic links, but 'arping' is not. We can change it to |
Contributor
|
@liangchenye did you forget the flag for the |
Member
|
@alban thanks, so it should be |
Contributor
Author
Do you know where your |
Member
|
I checked again, still failed. Once I 'tar xvf' this file, there are only a few symlink files and most of them are not point to 'busybox'. |
Contributor
Author
|
On Tue, Mar 13, 2018 at 12:06:44PM +0000, 梁辰晔 (Liang Chenye) wrote:
I downloaded this file:
http://distfiles.gentoo.org/releases/amd64/autobuilds/20180311T214502Z/stage3-amd64-20180311T214502Z.tar.xz
Its sha256sum is `af8…424 stage3-amd64-20180311T214502Z.tar.xz`.
Once I 'tar xvf' this file, there are only a few symlink files and
most of them are not point to 'busybox'.
That is the Gentoo stage3 pulled down by get-stage3.sh using this [1]
Makefile target. The make rule should drop it into
downloads/stage3-amd64-20180311T214502Z.tar.xz with a symlink
downloads/stage3-amd64-current.tar pointing at
downloads/stage3-amd64-20180311T214502Z.tar.xz. That's not the rootfs
tarball though.
The next step in building the rootfs tarball is to verify the
signature and unpack anything matching the rootfs-files wildcards into
rootfs/${ARCH}/bin/busybox [2]. Then we create symlinks for every
command supported by that particular BusyBox binary [3]. And finally
we tar it up into rootfs-${ARCH}.tar.gz [4].
So instead of looking at the Gentoo stage3 in downloads/, you should
build and look at the final rootfs-${ARCH}.tar.gz. Steps to reproduce
the tarballs I commit here are in the a7f94a2 commit message after
“Generated with”.
[1]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L27-L29
[2]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L8-L19
[3]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L21-L25
[4]: https://github.com/opencontainers/runtime-tools/blob/a7f94a268d2cbb5be522c433200aa44bd5f4d825/contrib/rootfs-builder/Makefile#L5-L6
|
Member
Contributor
|
The branch works for me. Here is the logs of my test: |
Member
|
@alban I found that in my test, |
Member
|
LGTM |
Contributor
Author
What version are you using? |
Member
|
GNU Make 3.81 on Ubuntu 14.04. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I'm not sure when the change happened (sometime since ff5e578, #479), but Gentoo is now using timestamps and xz compression for their amd64 stage3:
This commit adapts to that change, allowing us to continue to pull the vanilla
stage3-amd64-20180308T214502Z.tar.xz.