Resource processor disk full

[JC-wk]

Describe the bug
vmss resource processor disk (64GB) is full
write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/717/fs/cnab/app/.cnab/app/runtimes/porter-runtime: no space left on device

adminuser@vmss-rp-porter-00000X:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        62G   61G  1.3G  99% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           774M  1.1M  773M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs        128K   40K   84K  33% /sys/firmware/efi/efivars
/dev/sda15      105M  6.1M   99M   6% /boot/efi
/dev/sdb1       7.8G   28K  7.4G   1% /mnt
tmpfs           387M  4.0K  387M   1% /run/user/1000

caused by docker images filling the disk

adminuser@vmss-rp-porter-axdev25052800000X:~$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          33        1         61.81GB   1.434GB (2%)
Containers      1         1         5.423MB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

Steps to reproduce

1. observe error deploying resource
2. check resource processor disk space df -h
3. run hourly cron doesn't clean up any space

Azure TRE release version (e.g. v0.14.0 or main):
main

Deployed Azure TRE components - click the (i) in the UI:
latest

Workaround

Run docker image prune -a -f manually or add it to the cron script sudo nano /etc/cron.hourly/docker-prune

[CodeVishal-17]

Thanks for the detailed repro and diagnostics — this makes the root cause very clear.

It looks like the resource processor VM is accumulating container images indefinitely, and the existing hourly cleanup isn’t reclaiming image space. Since a single active container is using ~62GB of images, this eventually fills the 64GB OS disk.

A more robust fix might be one (or a combination) of:

Enforcing periodic docker image prune -a (or equivalent containerd GC) via systemd/cron with logging

Adding a disk-usage guardrail (fail early or trigger cleanup at a threshold, e.g. 80%)

Increasing the default VMSS OS disk size to reduce blast radius

Documenting disk requirements explicitly for resource processors

I’m happy to open a PR if a cleanup or guardrail-based fix sounds reasonable.

[JC-wk]

Thanks for the detailed repro and diagnostics — this makes the root cause very clear.

It looks like the resource processor VM is accumulating container images indefinitely, and the existing hourly cleanup isn’t reclaiming image space. Since a single active container is using ~62GB of images, this eventually fills the 64GB OS disk.

A more robust fix might be one (or a combination) of:

Enforcing periodic docker image prune -a (or equivalent containerd GC) via systemd/cron with logging

Adding a disk-usage guardrail (fail early or trigger cleanup at a threshold, e.g. 80%)

Increasing the default VMSS OS disk size to reduce blast radius

Documenting disk requirements explicitly for resource processors

I’m happy to open a PR if a cleanup or guardrail-based fix sounds reasonable.

I'm currently testing with docker image prune -a. Do you have an AzureTRE to test with? I think it's going to be difficult to contribute on this repo without one.

[CodeVishal-17]

I don’t have an AzureTRE environment deployed yet, but I do have an Azure subscription if setting one up is recommended.

Before I go down that path, I wanted to check whether you’d prefer:

• a smaller, low-risk change (docs / guardrails), or
• a fully tested fix that requires a live AzureTRE environment

[marrobi]

@CodeVishal-17 we only want PRs that have validated. If you have no TRE environment. Unfortunately you have no way to do that.

[marrobi]

@tamirkamara could #4827 have anything to do with this? As in the base layers not being reused across images.

[CodeVishal-17]

@marrobi Thanks for clarifying — that makes sense.

In that case, I’ll step back from making code changes here since I don’t currently have a TRE environment to validate against. I appreciate the guidance and the quick responses.