Redesign airlock to reduce number of storage accounts used

[jonnyry]

The airlock uses a large number of storage accounts. The number of accounts (particularly the workspace ones) are a bind on scalability, and also increase the overall cost.

Is it possible to consolidate some of these accounts, and use containers to segregate data instead?

---

Per account:

• Private endpoint $7.30/account/month
• Defender scanning $10/account/month

E.g. TRE with 10 workspaces = 6 core airlock accounts, 50 workspace airlock accounts

Airlock storage accounts - core

Name	Description
st + airlockp + <TRE_ID>	Airlock Processor
st + alexapp + <TRE_ID>	Airlock Export Approved
st + alimblocked + <TRE_ID>	Airlock Import Blocked
st + alimex + <TRE_ID>	Airlock Import External
st + alimip + <TRE_ID>	Airlock Import In Progress
st + alimrej + <TRE_ID>	Airlock Import Rejected

Airlock storage accounts - per workspace

Name	Description
st + alexblocked + ws + <WS_ID>	Airlock Export Blocked
st + alexint + ws + <WS_ID>	Airlock Export Internal
st + alexip + ws + <WS_ID>	Airlock Export In Progress
st + alexrej + ws + <WS_ID>	Airlock Export Rejected
st + alimapp + ws + <WS_ID>	Airlock Import Approved

[TonyWildish-BH]

It's not clear to me that the rej SAs serve any purpose at all. If a file is rejected, why is there any need to keep it at all? What happens to it after it goes to the rej SA, does it just sit there gathering dust until the workspace is destroyed? Can't we just get rid of that one altogether?

[jonnyry]

Indeed - or if necessary to keep could there be a rejected container, rather than a whole seperate storage account?

In fact could some of the accounts that share the same networking be consolidated, and containers used instead of seperate accounts... e.g.

I believe RBAC can be applied at the container level.

[marrobi]

Rejected was there in case of false positives, the fact that occasionally there might be a need to go retrieve the data.

I'm trying to recall why they were separate accounts, I agree if the share the networking and RBAC can be done at a container level, that might work.

@LizaShak (or even @eladiw ), can you remember?

[West-P]

I believe RBAC can be applied at the container level.

To do RBAC at the container level the storage accounts need Hierarchical namespace Enabled and the storage account needs to be a Data Lake Storage Gen2 I believe.

[tamirkamara]

1. What is possible now might not have been at the time it was created.
2. Enabling HNS on the storage accounts might have implications on events and subscription possibilities
3. I remember something around repeat defender scans. If we copy (move) files between containers on the same account it might trigger a repeat scan. Maybe at the time we could only turn on defender on the account level and maybe that has changed.
4. Networking wise I think looking at the core vnet isn't enough as I think different subnets have different access, so need to confirm what merging accounts will do to network security.

p.s. I haven't checked this now, this is just thoughts that I had.

[matthew-a-dunlap]

Our AzureTRE installation is considering disabling airlock for a workspace when it is not needed to cut costs. Ideally this would not need to be managed by our TRE Admin.

What are the technical limitations on AzureTRE doing this automatically someday, only creating these storage accounts when needed?

[marrobi]

@matthew-a-dunlap as in spinning them up when needed? Hmm, there is infra that needs to be deployed, but its an option, how would you implement it? Storage accounts dont cost, its the data in them. Some people have also been discussing moving the airlock accounts to a shared model.

[matthew-a-dunlap]

@marrobi The motivation for considering disabling/enabling airlock are mostly that the private endpoint costs add up when you are considering hundreds of research workspaces. Any solution that makes the workspaces leaner is one we'd be interested in.

I am unsure of implementation at this point, I mostly wanted to test the waters about whether we are off-base on some fundamentals about this. We are in early stages of talking through options.

[marrobi]

@matthew-a-dunlap hmm, didbt realy think about the $0.01 an hour base cost. Eveb if centralised, would likely need private endpoints, as need a way to restrict access based on traffic source.

Creting PEs on demand might be an option, but DNS caching etc is likely to get messy.

[marrobi]

Could look at https://learn.microsoft.com/en-us/azure/storage/blobs/storage-auth-abac-attributes#subnet, but would be using service endpoints, not private link, whick may raise security questions. Traffic is restricted by source, abd stays on the Azure network, but the storage account has a public IP.

[marrobi]

Looking more, the docs at https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview:

Say:
"Microsoft recommends use of Azure Private Link and private endpoints for secure and private access to services hosted on the Azure platform. Azure Private Link deploys a network interface into a virtual network of your choosing for Azure services such as Azure Storage or Azure SQL. For more information, see Azure Private Link and What is a private endpoint?."

So I really think we need to stick to private endpoint. The storage account limit is an issue, so propose we look at using this: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-auth-abac-examples?tabs=portal-visual-editor#example-allow-access-to-a-container-only-from-a-specific-private-endpoint . This doesn't resolve the fact we need multiple private endpoints, but does reduce the storage account quota issues.

Thoughts?