Resource Processor no longer processing requests

[TonyWildish-BH]

Not sure how to classify this one. The symptoms are:

• our TRE was working fine until this morning. Now, any and all interactions to create/delete, start/stop, or enable/disable resources get stuck in 'pending'.
• The service bus is receiving the requests, I see them accumulating in the workspacequeue. All messages are Active, none are in the dead-letter queue.
• The Resource Processor is healthy, running one instance, and that instance appears to be well configured:
  • It can resolve the address of the SB to it's private IP address.
  • It can connect to the SB on port 443 (but not 5761 or 5762, in case that matters).
  • The RP docker container has all the right environment variables set, the full list from the cloud-init.yaml in the codebase.
  • The only environment variable which is not set is RP_BUNDLE_key_store_id, this is empty. I don't know if this is normal or not.
• The docker container logs show repeated failures to connect to the SB. Screenshots below.
• The managed identity the container is running with is correct, and still has all the correct rights to read/send-to the SB.
• Killing & restarting the VMSS instance doesn't help.

I've tried restarting the API, restarting the VMSS instance, running make tre-stop / make tre-start, but nothing makes any difference.

Our production TRE is running version 0.21.0, but I've also stood up a new TRE this morning from the head of this repo, and it behaves identically. Yesterday, both the production SDE and another dev SDE I booted were working fine.

The fact that it's happening on two TREs today, where it didn't happen on two TREs yesterday, points at the tenancy or subscription, but I'm not aware of anything that could change there that would cause this problem.

So, I don't believe this is a bug in the TRE, but whatever's happening, I need to get my TRE running. Any suggestions?

[marrobi]

@TonyWildish-BH

The error looks like the managed identity isn't available on the resource processor. Maybe check it exists:

If you log on to the resource processor and try az login --identity --username <client_id> with the user assigned managed identity ID

[TonyWildish-BH]

the ID exists, and has the correct rights.

From the resource processor instance, az login --identity --client-id <client-id> worked.

It rejects --username as no longer accepted:

Passing the managed identity ID with --username is no longer supported. Use --client-id, --object-id or --resource-id instead

Is this supposed to work inside the RP docker container? There it fails because it can't connect to the metadata server, 169.254.169.254.

[marrobi]

Hmm, the python uses it here:

AzureTRE/resource_processor/vmss_porter/runner.py

Lines 31 to 37 in e76bde4

	async def default_credentials(msi_id):
	"""
	Context manager which yields the default credentials.
	"""
	credential = DefaultAzureCredential(managed_identity_client_id=msi_id) if msi_id else DefaultAzureCredential()
	yield credential
	await credential.close()

if look at env on the resource processor can you see VMSS_MSI_ID and ARM_USE_MSI what are they set to?

[TonyWildish-BH]

all the environment variables passed from the cloud-init.yaml are correctly set in the container. In particular, ARM_USE_MSI is true, and VMSS_MSI_ID is set to the right value.

[marrobi]

So on my deployment I can login using Azure CLI in Docker. Albeit is an older version of the cli so use username:

[marrobi]

Can you check the az cli version too:

root@1fa4231dcd70:/app# az version
{
  "azure-cli": "2.67.0",
  "azure-cli-core": "2.67.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}

[TonyWildish-BH]

hmm, not sure how it rejected --username earlier, I must have done something wrong. It's accepting it now, but giving an error:

My az version is identical to yours.

[marrobi]

Can you try curl to the metadata endpoint as per https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=linux

[marrobi]

curl -s -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance?api-version=2025-04-07"

Returns a load of JSON for me, inside Docker. If doesn't work, check on the host too.

Also docker version:

adminuser@vmss-rp-porter-tre011200000M:~$ docker version
Client: Docker Engine - Community
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:28 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 12:07:28 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[TonyWildish-BH]

curl works on the host but not in the container. No output, no error.

My docker version is identical to yours.

$ docker version
Client: Docker Engine - Community
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:28 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 12:07:28 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[marrobi]

Hmm, I wonder if its the docker image build.

I'll try rebuild and test, but might not be tonight although will try.

Suggest you open an MS support case, for VM - and the fact cant curl the metadata endpoint inside a container. I don't think it's a TRE code thing, so much be an OS/Docker thing.

The docker base image is: python:3.12-slim-bookworm

[marrobi]

Was faster than expected, just done a fresh Docker build, runs fine. Will reimage the OS too.

[marrobi]

@TonyWildish-BH just done a re-image of the VMMS, run a new build of the Docker container and everything is working as it should.

Are there any subscription level policies that might cause this? I'm out of ideas.

Maybe try a vanilla VM outside of the TRE, install Docker and see if that works?

[marrobi]

One other thing, is try docker inspect <container>

Check the network mode is bridge... and has IPs etc:

 "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "MacAddress": "e2:0b:02:80:a8:70",
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "de24484d24ebf2b3e411265cca7513fa0887620920217c64e256b79a98db95bc",
                    "EndpointID": "4898ff822a3773ba91869019dd622b416de45e464ffb0ca685181679ad6d0d0e",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": null
                }
            }

[TonyWildish-BH]

no subscription level policies.

vanilla VM outside the TRE works fine, I can start a Docker container and fetch the metadata from there.

My docker inspect output looks exactly like yours.

What happens when the resource processor VM boots, does it pull in startup stuff from somewhere? Could that have changed behind the scenes?