feat: migrate to miden VM 0.21 and miden crypto 0.22

[PhilippGackstatter]

Changes

• Updates dependencies:
  • Miden VM crates to 0.21
  • Miden Crypto to 0.22
  • Removes unused miden-air, winterfell, winter-rand-utils, winter-air
• Deactivates the agglayer crate to avoid merge conflicts with work done in the agglayer branch.
• Reverses StackInputs and StackOutputs since this changed in the VM.
• The definition of "tracked" in a PartialSmt has changed which is why a PartialAccountTree test needed an update (upsert_state_commitments_fails_on_untracked_key).
• Removes const from some AccountId methods due to as_canonical_u64 not being const (which replaces the previously const fn as_int).
• Attempts to establish consistent usage of RATE0, RATE1, CAPACITY as the hasher state. Previously the order of the rate words was not documented (RATE, RATE) and after hashing we sometimes used PERM, PERM, PERM which doesn't say much. Not yet fully converted in this PR -> done as a follow-up.
• Optimized advice stack in a few cases to avoid a swap when hashing such as swapping NOTE_ATTACHMENT and NOTE_METADATA_HEADER.
• The input and output notes commitment could avoid a swapw in the kernel if we re-arranged how the hash is defined (though this isn't done in this PR, and I'm not fully convinced yet it's worth doing).
• deny.toml
  • Upgrades toml to version 1 to fix the cargo deny check that complained about two versions being used.
  • Remove ISC license that is not encountered in dependencies.
  • Add "CC0-1.0" license used by tiny-keccak.
  • Removes some unused skip-tree entries.

Migration

• See Change stack ordering through unified LE convention and sponge state remapping miden-vm#2547 for the changes at the VM level.
• The order of [prefix, suffix] was changed to [suffix, prefix] for all slot IDs and account IDs (and the newly introduced asset ID). Note that slot IDs can still be pushed with push.SLOT_NAME[0..2]. See the diff of docs/src/protocol_library.md with the previous version for what has changed in miden::protocol.
• Word layouts must be reversed on the VM stack, e.g. TokenMetadata changes from stack [token_symbol, decimals, max_supply, token_supply] to [token_supply, max_supply, decimals, token_symbol].

Follow-Ups

Done in #2512 except for the agglayer ones.

• All TODO(bele). Some of these are agglayer-related.
• Refactor account ID to be defined as suffix = digest0 and prefix = digest1 where digest is the hash of seed and the commitments so it is more consistent with the half-word layout.
• Reorder tx summary on the stack so it can be hashed without swapping and adv.insert_hqword can be used.
• Rename the Falcon512Rpo variants in Signature, AuthScheme, AuthSecretKey to Falcon512Poseidon2.
• Couple smaller improvements post-migration to optimize stack layout and consistency.

Thanks to @Al-Kindi-0 and @huitseeker for their prior migration work that was a great reference, and the support in DMs!

[bobbinth]

All looks good! Thank you!

[partylikeits1983]

I only took a close look at MASM code just to familiarize myself with the upcoming changes. Only found one small issue, where there were two swapw swapw opcodes back to back. Otherwise everything looks really good.

Regarding the two back to back swapw swapw opcodes, I checked out your branch, and applied the suggestions I left, and all tests passed.

[mmagician]

I selectively reviewed a few kernel, protocol and standards files, the changes seem to be similar in nature across the other changed files from briefly skimming over them. So not a full review, but LGTM ✅

[mmagician]

nothing for this PR, but I'd like to move away from this hand-coded hashing, and either wrap these in helper procedures here or move them to the poseidon2 module.
Ideally, we would do something like:

const BLOCK_DATA_NUM_WORDS = 10

proc your_favorite_proc
    push.BLOCK_DATA_NUM_WORDS
    # => [num_words]
    exec.poseidon2::hash_advice_words
end

this would be less error prone, and waaay easier to follow and reason about. If you think this is a valid suggestion, let's make an issue for this.

[PhilippGackstatter]

I think the code could be simplified and made a bit more explicit with something like:

exec.poseidon2::init_no_padding
repeat.BLOCK_DATA_NUM_DOUBLE_WORDS
    adv_pipe exec.poseidon2::permute
end
exec.poseidon2::squeeze_digest

The main reason I can think of to keep using this instead of a helper is that it would not require any looping/branching, though I haven't benchmarked how many cycles this saves.

We could encapsulate the above in a hash_advice_words procedure with a loop, but we wouldn't save that many lines of code and readability would probably be about the same, so I'm not sure what is better.

[mmagician]

Useful across the board:

    push.ACCT_DETAILS_NUM_WORDS
    # => [num_words]
    exec.poseidon2::hash_advice_words

[mmagician]

We could benefit from similar helpers here

[mmagician]

For a follow-up issue/PR: this would be useful in a lot of other places:

• https://github.com/0xMiden/miden-base/blob/e85700616ecadf8f48600776297d250e51e02bbd/crates/miden-protocol/asm/protocol/active_account.masm#L515
• https://github.com/0xMiden/miden-base/blob/e85700616ecadf8f48600776297d250e51e02bbd/crates/miden-protocol/asm/protocol/active_account.masm#L547
• https://github.com/0xMiden/miden-base/blob/e85700616ecadf8f48600776297d250e51e02bbd/crates/miden-protocol/asm/shared_utils/util/asset.masm#L177
• etc. (search for push.0.0 in the codebase, most of them construct ACCOUNT_ID_KEY)

[PhilippGackstatter]

I think the ones you listed and the ones I can find aren't actually account ID keys but asset keys. They happen to have the same layout, but I would say they have a semantic difference. The first two entries in an account ID key are just unnamed zeros, but the first two entries in an asset key are the asset ID limbs, which are explicitly set to 0 for fungible assets. So I think account::create_id_key doesn't need to be exposed, because, at least currently, miden::protocol doesn't construct account ID keys and user code should probably rarely need to do that.

I think it would be useful to expose a miden::protocol::asset::create_fungible_key procedure for the purpose of creating asset keys (that internally happens to work the same way as create_id_key).

Opened #2541.

[mmagician]

this usage pattern is quite more involved, but I think here in the user-facing protocol lib it would be especially important to move away from hand-crafting the hasher states

[PhilippGackstatter]

I tried to replace this using poseidon2::prepare_hasher_state, but for subtle reasons this only works for hash_elements_with_state but not for initializing hasher states for pipe_double_words_to_memory, otherwise this would've been simplified some.

It sounds like what you want would be basically a single procedure that takes care of all of that. I guess that would be ideal and will require changes in miden::core. I'm not yet sure if we can have procedures for all our use cases, but this requires a bit more in-depth analysis. Will open an issue.