downloader improvements during outages

[eriknordmark]

If we have 100% packet loss on one interface, then we still try to use it for every POST to zedcloud. Since we are sending info messages for the device, apps, volumes, content-trees, and blobs, it means that EVE can be far behind in terms of reporting things to zedcloud. The first commit addresses that by using the test status from nim.

As we retry the state transitions seen by the controller are confusing, since the error is cleared when a retry starts. With these changes we don't clear it until the retry has succedded. Also, the log messages include that 1) the operation will be retried in N seconds, and 2) "retry N" has started

Each commit can be reviewed separately.

[rvs]

Oooph -- this is a bit complex -- need more time to properly review this.

In the meantime -- let it test!

[gkodali-zededa]

LGTM

[giggsoff]

Interfaces don't seem to return from an error state during verify. I add modifications for adam: lf-edge/adam#88 and apply diff to your PR and it works fine:

diff --git a/pkg/pillar/zedcloud/send.go b/pkg/pillar/zedcloud/send.go
index db5d05d3a..9c45436d4 100644
--- a/pkg/pillar/zedcloud/send.go
+++ b/pkg/pillar/zedcloud/send.go
@@ -196,10 +196,17 @@ func VerifyAllIntf(ctx *ZedCloudContext,
                case types.SenderStatusRefused, types.SenderStatusCertInvalid:
                        remoteTemporaryFailure = true
                }
-               if resp != nil &&
-                       (resp.StatusCode >= http.StatusInternalServerError &&
-                               resp.StatusCode <= http.StatusNetworkAuthenticationRequired) {
-                       remoteTemporaryFailure = true
+               if resp != nil {
+                       if resp.StatusCode >= http.StatusInternalServerError &&
+                               resp.StatusCode <= http.StatusNetworkAuthenticationRequired {
+                               remoteTemporaryFailure = true
+                       }
+                       if resp.StatusCode == http.StatusForbidden {
+                               log.Tracef("VerifyAllIntf: Zedcloud reachable via interface %s", intf)
+                               intfStatusMap.RecordSuccess(intf)
+                               intfSuccessCount++
+                               continue
+                       }
                }
                if err != nil {
                        log.Errorf("Zedcloud un-reachable via interface %s: %s",

[eriknordmark]

Interfaces don't seem to return from an error state during verify. I add modifications for adam: lf-edge/adam#88 and apply diff to your PR and it works fine:

Thanks for explaining this.
My PR tried to optimize it so that when there is a choice, skip the failed interfaces.
But since nim only updates the failed/succeeded every 5 minutes and as you discovered, we can potentially have false notions of failed during the onboarding/registration, I need to have the code in send.go try the failed ones if there are no other choices. So added a commit for that and let's see if it passes.

[rvs]

kicking off Eden

[eriknordmark]

@rvs do you still need more time to review?

[rvs]

LGTM! Thanks for being patient @eriknordmark !

Lets merge this