Non-blocking I/O without cancellation

[badeend]

Hi! The concurrency explainer contains an example on how to implement non-blocking IO.

• write() calls stream.write, forwarding the caller's buffer.
• If stream.write returns that it successfully copied some bytes without
  blocking, write() returns success.
• Otherwise, to avoid blocking:
  • write() calls [stream.cancel-write] to regain ownership of the
    caller's buffer.
  • If select() has not indicated that this file descriptor is ready,
    write() starts a zero-length write and returns EWOULDBLOCK.
  • Otherwise, to avoid the potential infinite loop:
  • write() copies the contents of the caller's buffer into an
    internal buffer, starts a new stream.write to complete in the
    background using the internal buffer, and then returns success.
  • The above logic implicitly waits for this background stream.write
    to complete before the file descriptor is considered ready again.

But it also says:

Cancellation is cooperative, (...) allowing the subtask to continue executing for an arbitrary amount of time

Is there a contradiction here?

If the other end of a stream doesn't respect cancellation immediately (which is allowed) or doesn't support cancellation at all (also allowed) the guest will still end up being blocked.

[alexcrichton]

Personally I see this as a consequence of implementing a readiness-based I/O system on a completion-based I/O system. Some compromises will have to be made somewhere, and this is a case where that shows up. If a stream doesn't actually support cancellation then, yes, it'll cause nonblocking I/O to actually be blocking. In practice though I'd expect this to be rare for "expected to work" nonblocking I/O (e.g. TCP/UDP). For file, however, in Wasmtime today cancellation doesn't actually cancel so it means that cancelling a read on a file would actually just block waiting for the I/O to finish anyway.

[lukewagner]

@badeend Great observation! Indeed that was written before async became an effect type and thus the assumption was that a synchronous stream.cancel-write would be used and that that would be fine in practice because, as @alexcrichton mentioned, stream/future read/write cancellation should basically never block b/c, when the other end is wasm, cancellation is spec-defined to complete immediately and when the other end is host, until we have a host doing Overlapped I/O or io_uring, there should be no reason that the host can't cancel the read/write immediately as well. However, that's obviously not a super-satisfying answer.

So one optimization idea that I mentioned a while back (when we first discussed this non-blocking issue) seems like it might also be the fix to avoid this trapping corner case. The idea is to add {stream,future}.try-{read,write} built-ins that work like normal {stream,future}.{read,write} except that, when they returned BLOCKED, instead of continuing the read/write in the background (thereby requiring cancellation), they leave the stream/future in the initial state (with buffer ownership returned to the caller).

I'm not sure when we should do this (maybe as part of a 0.3.x follow-up batch that includes stream.splice), but it seems like a good idea to do before 1.0 so that this is a non-issue.

[syntactically]

Is the expectation that the stream.try-{read,write} primitives would both (a) be guaranteed to make progress eventually, if checked repeatedly, and (b) be guaranteed to never suspend? This is a bit difficult for our planned p3 I/O model in hyperlight-wasm, because we are/were planning for a purely completion-based interface: all of the driver state for e.g. a stream provided by the host lives outside a virtual machine boundary, with a relatively slow and more-or-less-inherently async call required to get access to it from the guest. This means that for e.g. the TCP recv case, even finding out whether there are packets available on the NIC/host network stack requires an async/suspending call. We could of course have an extra buffering layer in the wasm runtime when these primitives are used, but I think it would remove the possibility of 0-copy, specifically when these are used.

[badeend]

@syntactically You may also be interested in this thread, which discusses the try- operations in more detail.

[lukewagner]

@badeend Ah, thanks for finding the link, and sorry for forgetting that you were in fact the one who first mentioned this idea!

@syntactically My guess is that stream.try-read would not be expected to make progress on its own and, rather, it would be the async 0-length stream.read that would tell the host, if it's using a completion-based API, to go ahead and read into a host buffer and signal the 0-length read as completed once there is something in the buffer (b/c how else would the 0-length read ever complete and allow the guest to make progress?).

For writes, I think there are two ways it could go, depending on whether the host wants to buffer or not. If the host buffers, then stream.try-write would succeed by writing into a buffer, but fail if the buffer was full which you would then wait on via async 0-length stream.write. If the host does not buffer but is completion-based, I would think that stream.try-write is allowed to always fail and 0-length writes are (already) allowed to always succeed, in which case, according to the rules, must be handled by the guest by buffering the write in guest memory, flushing asynchronously via stream.write.

"Buffering" may not sound great, but if stream.try-{read,write} are being used for fundamentally non-blocking/readiness-based source-code APIs, then I think buffering has to happen somewhere, and if the host is using completion-based APIs, we're effectively just moving this buffering from the kernel to the user-space host or the guest wasm.

[syntactically]

You may also be interested in #441 (comment) thread, which discusses the try- operations in more detail.

Thanks! I was looking for that thread but somehow couldn't find it.

If the host does not buffer but is completion-based, I would think that stream.try-write is allowed to always fail and 0-length writes are (already) allowed to always succeed, in which case, according to the rules, must be handled by the guest by buffering the write in guest memory, flushing asynchronously via stream.write.

Ah, that makes more sense. I was expecting this case to be banned if try-write were to be added as its own thing, but if the intention is to keep it then everything's fine :)

rather, it would be the async 0-length stream.read that would tell the host, if it's using a completion-based API, to go ahead and read into a host buffer and signal the 0-length read as completed once there is something in the buffer

From my reading of the concurrency section, the same approach as in the write case (where the guest detects that 0-length reads are not actually signalling readiness and does buffering itself, because try-read always fails after read) would be possible here too?

"Buffering" may not sound great, but if stream.try-{read,write} are being used for fundamentally non-blocking/readiness-based source-code APIs, then I think buffering has to happen somewhere, and if the host is using completion-based APIs, we're effectively just moving this buffering from the kernel to the user-space host or the guest wasm.

I agree with this. I think having it in the guest wasm makes a lot of sense, since that's what can hopefully actually know whether it needs it. I worry with host-side buffering that it might be easy to get into a state where buffering was happening but not needed (e.g. some early-on call did a 0-length write but then later switched to a readiness-based approach on the same fd).

[badeend]

Another related issue is this one. The wasi:sockets UDP interface doesn't use streams, but it runs into essentially the same problem in that: wasi-libc wants to implement non-blocking UDP sends, but with the current idiomatic interface this is generally impossible to do efficiently.
At the time of writing, the consensus in that ticket is leaning toward preemptively copying every buffer. One possible way to avoid this would be to also add "try" variants in WIT here.

It's not directly related to streams, but I'd like to converge the readiness-based I/O story for WASI APIs to whatever the C/M ends up doing.

---

I'm not sure when we should do this (maybe as part of a 0.3.x follow-up batch that includes stream.splice), but it seems like a good idea to do before 1.0 so that this is a non-issue.

I'm not sure about the timing either. Now that p3 is reaching the stage where implementations are being built, three closely related issues have appeared within a week: 1, 2, and 3.

Maybe this is just recency bias, but it does make it feel like 0.3.0 won't have a clear story for readiness-based libraries right out of the gate. Reading across these threads, the current guidance for guests seems to be either:

• issue a blocking cancellation and hope it won't trap, or
• defensively copy the buffer, in places where that wasn't necessary in 0.2.x

Some language ecosystems (e.g. Python) are already planning to skip 0.2 entirely because of the churn in WASI. If proper support for readiness-based guests (which realistically covers most existing libraries) is postponed to a later 0.3.x release, that may add another round of churn for early adopters. Streams are a core part of the value proposition for p3, so getting them (ahem) "wrong" the first time could be more harmful than last-minute adjustments. I dislike last-minute changes as much as the next person, but maybe something worth considering.

Anyhow, regardless of timelines:

---

Various possibilities were discussed earlier. Perhaps we could start with a minimal version:

• Add stream.try-{read,write} built-ins.
  • Guaranteed never to block.
  • No async canonopt required.
  • No manual cancellation needed.
• If the operation cannot be performed immediately, they return canceled instead of blocked. The buffer is immediately returned to the caller, with no data copied.
• Once try-write returns canceled, the guest may not call it again until it has issued (and awaited!) a successful regular non-0-sized write; otherwise, a trap occurs. This ensures forward progress for all combinations of {readiness,completion}-based producer <-> {readiness,completion}-based consumer.

The net result of this protocol:

• if either the producer or the consumer is completion-based -> everything is fine.
• if both the producer and the consumer are readiness-based -> the C/M places responsibility of providing a buffer onto the producer.

[jellevandenhooff]

I'm not sure about the timing either. Now that p3 is reaching the stage where implementations are being built, three closely related issues have appeared within a week: WebAssembly/WASI#897, #617, and #618.

Maybe this is just recency bias, but it does make it feel like 0.3.0 won't have a clear story for readiness-based libraries right out of the gate.

It’s painful but I think extremely valuable. Implementing guests and hosts efficiently is what makes wasip3 impactful. I don’t quite know if having all languages go through wasi-libc was the plan all along or not, but with that path it may make modifying the spec less painful than it was before; if languages don’t think about wasip3 themselves and you can rebase on a new wasi-libc.

[syntactically]

The net result of this protocol:

• if either the producer or the consumer is completion-based -> everything is fine.
• if both the producer and the consumer are readiness-based -> the C/M places responsibility of providing a buffer onto the producer.

This summary confused me from the perspective of a host implementing read/writes for host-provided streams, until I realised that it makes sense when looking at streams between two components. The case of a host trying to implement a stream via a zerocopy completion-based I/O API doesn't fit incredibly neatly into this taxonomy, because there's never a time that it's safe to signal readiness if the underlying completion-based I/O API doesn't have some sense of readiness/returning fast enough that it's safe to wait for it, so the guest still needs to buffer and send off a true async (allowed to suspend, and to be slow/impossible to cancel) call (but arguably for a different reason than in the readiness-based consumer case). However, I think the same rule for detecting when buffering is needed still works in this case.

The similar issue in WebAssembly/WASI#897 is interesting to me, because it implies that the same issue of wanting to be readiness-based occurs outside of streams where the 0-byte read/write convention doesn't work. It would be nice not to avoid having multiple conventions for readiness; I wonder if the convention suggested there, of having try-write return a future that indicates readiness would also work for try-{read,write} on streams.

It feels like this is getting at something quite fundamental. I wonder if there will be more use cases for this "readiness-style" of async outside of primitive I/O.

[lukewagner]

@badeend Personally, I buy the reasoning for adding try variants in 0.3.0. If we added try as an optional immediate (with a new opcode in the binary format to minimize churn) as opposed to new built-in, that might be both less of a pain, and I think semantically try ends up being a boolean parameter to the existing read/write paths as opposed to something wholly new, so altogether it might not be a huge lift to include in 0.3.0, but I'd be interested to hear what others think.

My impression is that WASI#897, while also about "cancellation", is a distinct issue (for reasons I just mentioned in that issue) and that there's no component-model fix suggested there (other than perhaps to make streams more expressive in some additive, later 0.3.x release so that UDP could use streams just like TCP). But it's definitely good to juxtapose these issue and ask the question of whether there's some single underlying issue.

[alexcrichton]

In talking about this some more with @lukewagner and @dicej today one thing I wanted to write down here as well is that personally I don't think we can avoid cancellations entirely. With the goal of giving affordance to completion-based systems it seems that no matter what variants of try-* are added or what wasi-libc does there will always eventually be a case of wasi-libc having open file descriptors with in-progress I/O operations on them. For example today it's required when 0-length reads/writes don't work, and with try-* it's required to determine readiness when the try-* blocks. In this situation close(fd) is fundamentally a non-asynchronous function and somehow needs to be handled. That leaves two possible implementations:

1. close performs a blocking cancel
2. close deallocates what it can but allows the full closure to proceed in the background (somehow)

With (1) although try-* might make it so libc read and write functions don't ever use a blocking cancel it'd still be in libc via close. With (2) I'd personally be worried about getting resources into weird states where close cleans up the libc-based fd but doesn't actually clean up the underlying host resource. For example files would remain open, ports would remain bound, etc. This means that subsequent operations aren't guaranteed that the previous resources are actually released, which I've seen is often a constraint when using close for example (e.g. not being able to delete an open file on Windows, ports not being able to be re-bound for the next test, etc).

Personally I feel that (2) here is hazardous enough that I'd like to avoid it. Thus assuming libc were instead to take on (1) as an approach, issuing blocking cancellations in close, then it's still not actually possible to do non-blocking I/O without cancellations. Additions of try-* functions would mean that read and write don't have to do a blocking cancellation, yes, but it would mean that a theoretical high-level goal of never needing blocking cancellation would remain unfulfilled.

From a concrete perspective we could consider the impact of using wasi-libc in a synchronous component model function context. In such a context it's not possible to call blocking functions, but it is possible to call async functions (e.g. issue I/O ops). As-is today it would mean that read and write libc functions (nonblocking) could not be used with a sync component model function because of their issuance of blocking cancellation. That would change with try-* where read and write would be valid, but close would continue to be invalid. However we could also change wasi-libc to use an async cancel + block on cancel instead of a blocking cancel. This would mean that guests would only dynamically trap if cancellation blocked and wouldn't trap if dynamically cancellation all worked out.

Overall, to me there's a spectrum of possibilities here:

• Do nothing - read, write, and close cannot be used reliably in sync function contexts.
• Use async cancel - read, write, and close would work "most of the time" in a sync function context, but this is not a hard guarantee
• Add try-* functions - read and write would be guaranteed to work in a sync function context, but close would not.
• Add try-* and use async cancel - this requires effectively a background thread to perform the "real cleanup" and means that host/OS resources are kept alive beyond a call to close if necessary.

Overall, to me, none of these are particularly satisfying. I don't find that any of these necessarily neatly solves the problems at hand here. I agree with @syntactically that this touches on the fundamental nature of completion-vs-readiness-based I/O, and I basically think there's only so much that can be done with what we have.

W.r.t. timelines -- I agree we want 0.3.0 to be solid. I think it's best to first determine what the end state we want is, and then we can work backwards from there how exactly to ship and implement things.

---

So, overall, I'm left with -- what's the fundamental problem we're trying to solve? Is it to remove all blocking calls to cancellation in libc? Is it to only remove cancellation in read and write? Is it to remove blocking cancellation from the component model entirely? Basically I feel like I need more clarity on this part to better evaluate/discuss possible options here.

[jellevandenhooff]

I think the primary goal should be to make c/python/ruby/rust on top of wasi-libc behave as expected and for language maintainers to be happy relying on wasi-libc. I think that means select/O_NONBLOCK guest code today does not introduce new pauses, buffers, or requires stack switching. But perhaps that is too much; maybe making some blocking cancellation calls is okay; maybe relying on sync lowering is okay also.

A host that uses epoll behind the scenes and promises instant cancellations of read/connect/etc. will behave like libc-on-linux today with no API changes, but that introduces hidden assumptions and I don't think another component could implement that. It seems to me now that either you sacrifice guest behavior (by requiring completion IO); host freedom of implementation (by requiring readiness IO); or API complexity (by providing both readiness and completion IO).

I do not have a good sense for what language teams might want and/or would be happy accepting behavior-wise. My perhaps uniformed opinion that completion IO is still rare, the performance benefits are small for wasm today, and if performance were to become a problem in the future an extra API would be an acceptable cost then. At the same time, I don't know how to implement readiness IO on the component model today in a clean way without having try- methods everywhere; and then the read(0) like @alexcrichton mentions would have to not block closing, and, well, ugh.

[badeend]

what's the fundamental problem we're trying to solve?

My take:

• Support all combinations of {readiness,completion}-based producers <-> {readiness,completion}-based consumers for both hosts and guests
• Non-blocking APIs should reliably not block
• Avoid redundant memory copies
• Keep cancellation cooperative and async, for all the reasons outlined in (A)synchronous cancellation guarantees #618

Essentially, I want to have my cake and eat it too, but I actually think this is all achievable if we're willing to tolerate that close() indeed can block (as it already does).

---

Even outside WASI, blocking during close() is standard:

• Sockets with SO_LINGER: The socket(7) man page states that close() won't return until queued messages are sent or the linger timeout expires
• Filesystem descriptors: Linux's "close-to-open cache consistency" for NFS causes close() to block until pending changes are written, ensuring subsequent open() calls see the latest data
• The POSIX docs mention two other cases, but those are not directly relevant to current-day WASI.

I think its OK and inevitable for close() to block in wasi-libc; it does so in POSIX too. Though people tend to blissfully ignore this corner case. There isn't much they can do about it either ¯\(ツ)/¯

---

1. close performs a blocking cancel

Actually, close() should wait on in progress I/O rather than cancel it. After a successful write/send(), callers should be able to assume it is safe to close() without data loss.

2. close deallocates what it can but allows the full closure to proceed in the background (somehow)
   (..) Personally I feel that (2) here is hazardous enough that I'd like to avoid it

Agree. If guests want to prevent traps from a blocking close(), they should handle it themselves on a background task (somehow)

---

Alex' third option is exactly what I had in mind:

Add try-* functions - read and write would be guaranteed to work in a sync function context, but close would not.

---

Is it to remove blocking cancellation from the component model entirely?

No, quite the contrary. I think the removal of blocking cancellation was mixed in with this discussion because it is currently the only way to emulate non-blocking IO. If the C/M provides native support for readiness-based operations through e.g. try options, I don't see async cancellation as a concern.

---

Is it to remove all blocking calls to cancellation in libc?

Not a standalone goal. However, with try options in place, the need for cancellation in libc naturally disappears.

---

Is it to only remove cancellation in read and write?

Those are the most critical cases. The goal is to remove blocking from any API that claims to be non-blocking.