Skip to content

Conversation

@quaresmajose
Copy link
Member

On the fisrt build step we run bitbake for setscene only tasks and obvious this one is the only one that use the reads and use the sstate-cache.
The second step will be a 0% hit of available because everything has already been accelerated in the first step. In this second step everything will be build from scratch and the local sstate-cache populadted with the artifacts.

This will improve and significantly reduce the load generated on the sstate-mirror http server by our CI builds.

@quaresmajose quaresmajose requested a review from a team October 25, 2024 10:50
Copy link
Contributor

@angolini angolini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are 2 typos on the commit log
fisrt
populadted

The description is confusing to me, maybe, instead of saying "first step" and "second step", say the command? Because there are other bitbake steps, so which you mean by first?

@quaresmajose quaresmajose force-pushed the sstate-mirror-improvements branch from 68bf732 to f6991d9 Compare October 29, 2024 17:41
@quaresmajose
Copy link
Member Author

quaresmajose commented Oct 29, 2024

I have updated the commit, hope it is more clear now

    lmp/bb-build: only use the sstate-cache mirror when it is necessary
    
    Our build can be divided into two steps:
    1 - build only using the sstate-cache
    2 - build everything else missing
    
    On the step [1] we run bitbake for setscene only tasks
    and obvious this one is the only one that uses and reads the
    sstate-cache mirror.
    
    The step [2] will have 0% hit of cache available because everything
    has already been accelerated in the step [1]. In the step [2]
    everything will be build from scratch and the local sstate-cache
    will be populated with the new artifacts.
    
    This change will improve and significantly reduce the load generated on
    the sstate-mirror http server by our CI builds.

@quaresmajose quaresmajose force-pushed the sstate-mirror-improvements branch from f6991d9 to 1da0a40 Compare October 29, 2024 17:43
@ricardosalveti
Copy link
Member

@quaresmajose I understand the reduced load on the server side, but did you measure if this also have any impact on the build time during the CI run?

@quaresmajose
Copy link
Member Author

I am testing this on the scarthgap factory but I didn't measure the time.
The time should be very similar because all the bitbake steps are the same except for the initial step where we ask the http server if it has the artifact xyz. When we are compiling the code this ask to http server, at the beginning of the build, is not needed and the answer given by the server is already known and is no.

@quaresmajose
Copy link
Member Author

In theory the build time should be about 2 minutes less, which is the time we waste asking the http server.

Copy link
Member

@ricardosalveti ricardosalveti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ndechesne
Copy link

btw, in our CI, don't we have access to the sstate mirror through NFS by any chance?

@quaresmajose
Copy link
Member Author

btw, in our CI, don't we have access to the sstate mirror through NFS by any chance?

The internal sstate mirror of the factory (every factory has its own) works through NFS but the public only in HTTPS

Copy link
Contributor

@igoropaniuk igoropaniuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@quaresmajose quaresmajose force-pushed the sstate-mirror-improvements branch from 1da0a40 to dcccd92 Compare March 31, 2025 17:33
Our build can be divided into two steps:
1 - build only using the sstate-cache
2 - build everything else missing

On the step [1] we run bitbake for setscene only tasks
and obvious this one is the only one that uses and reads the
sstate-cache mirror.

The step [2] will have 0% hit of cache available because everything
has already been accelerated in the step [1]. In the step [2]
everything will be build from scratch and the local sstate-cache
will be populated with the new artifacts.

This change will improve and significantly reduce the load generated on
the sstate-mirror http server by our CI builds.

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
@quaresmajose quaresmajose force-pushed the sstate-mirror-improvements branch from dcccd92 to 3808366 Compare June 23, 2025 15:23
@ricardosalveti
Copy link
Member

@quaresmajose this one was already approved but pending merge, anything blocking us from merging this now?

@quaresmajose
Copy link
Member Author

@quaresmajose this one was already approved but pending merge, anything blocking us from merging this now?

No, it can be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants