Updating toolchain requirements in the phases process

[tlively]

We briefly discussed this at the 7/7/2020 video meeting and decided to take the discussion to GitHub.

Currently the phases process recommends that toolchains implement proposals in phase 3 and requires that features be implemented by at least one toolchain in order to enter phase 4. There are a few issues with this requirement that might warrant changes to the process document.

1. The toolchain requirements are introduced too late in the process. Since WebAssembly is meant to be a compilation target, its design working well for toolchains is just as important as its design working well for engines. But the phase process is biased toward incorporating engine feedback over toolchain feedback. Entering phase 3 requires an engine implementation, but a toolchain implementation is not required until entering phase 4, when a proposal's momentum makes larger changes unpleasant. Concretely, @RossTate has suggested in Process requirement for sufficient demand proposals#68 that we add a phase 3 entry requirement that some user base has plans to use the proposal. Similar changes might be to require that some toolchain has started prototyping a use of the proposal or to require that some toolchain has sketched a design for how it will use the proposal.

2. It is unclear whether an entire proposal needs to be implemented by toolchains. For example, bulk memory recently reached phase 4 despite there being no known users of the table.fill instruction. Should the toolchain requirement have prevented that phase advancement? If not, should it be amended to prevent similar situations in the future?

3. It is unclear what constitutes a toolchain. For example, is it sufficient for Binaryen to support optimizing binaries that use a feature, even if no compilers produce it? (IMO, no). Is it sufficient for postprocessing tool like wasm-bindgen to insert uses of a feature even if the core compiler for the language does not use that feature? (IMO, yes). Should we clarify this to avoid confusion in the future?

What do folks think about these issues and potential changes?

[fitzgen]

It is unclear whether an entire proposal needs to be implemented by toolchains. For example, bulk memory recently reached phase 4 despite there being no known users of the table.fill instruction. Should the toolchain requirement have prevented that phase advancement? If not, should it be amended to prevent similar situations in the future?

(Nitpick: table.fill ended up as part of reference types, rather than bulk memory, because of the init value operand.)

IMO: If there is no toolchain that is using or intends to use an instruction, then that instruction should be dropped from the proposal, and then the proposal should advance without that instruction (assuming there are other bits of value in the proposal). If a toolchain intends to use an instruction but hasn't implemented that support yet, then the proposal should not advance to the next phase until that support is implemented.

In the specific case of table.fill and the ref types proposal, this would have resulted in us dropping the table.fill instruction and advancing the proposal to the next phase.

It is unclear what constitutes a toolchain. For example, is it sufficient for Binaryen to support optimizing binaries that use a feature, even if no compilers produce it? (IMO, no). Is it sufficient for postprocessing tool like wasm-bindgen to insert uses of a feature even if the core compiler for the language does not use that feature? (IMO, yes). Should we clarify this to avoid confusion in the future?

Agreed on both counts: "toolchain" should focus on tools people use to create wasm binaries, not on the libraries used by those tools under the covers. For example, emscripten counts but not binaryen, wasm-bindgen counts but not walrus, etc.

We might even want to explicitly list which toolchains are considered valid for the process, since there are so few right now, and adding new ones to the list should be easy as they pop up.

[binji]

In the specific case of table.fill and the ref types proposal, this would have resulted in us dropping the table.fill instruction and advancing the proposal to the next phase.

I agree for the most part. One issue is that it's not hard to imagine that we will support this feature in the future. So we'd probably give it an opcode, but not include it in this proposal (as we've done for some SIMD insructions). The issue then is that we're increasing our feature-testing burden. That said, this is an issue we already have and have some plans to tackle, so maybe it's OK. But it would likely make the problem worse.

We might even want to explicitly list which toolchains are considered valid for the process, since there are so few right now, and adding new ones to the list should be easy as they pop up.

If we make a toolchain list, I'd like these to be ones that we consider to be "definitely-allowed", but not an exhaustive list.

[tlively]

I'm not sure we need to list toolchains. If the maintainers of a non-toy toolchain pop up out of nowhere and say they're successfully using a proposal, that's good enough for me even if we had no idea they were working on WebAssembly support.

[RossTate]

My sense from the discussion is that there are three separate issues at hand: the role of (and definition of) the toolchain in the process, the evaluation of the utility of the proposal, and the coverage of a proposal's components.

Regarding "toolchain", here's a rough stab at major breaking the development chain for a technology like WebAssembly into components:

1. Generation: creating WebAssembly modules from various "surface-level" languages
2. Transformation: operating on WebAssembly modules, e.g. static linking, wasm-to-wasm optimization
3. Execution: actually executing WebAssembly modules, e.g. dynamic linking, engines, embeddings, C/JS API
4. Retrospection: understanding why WebAssembly programs behave as they do, e.g. debugging (both at the wasm level and at the source-code level), tracing

All of these are quite important, but my impression is that the current phase process places significant emphasis on the Execution component, little emphasis on Generation, possibly including Transformation, and no emphasis on Retrospection.

What's particularly concerning to me is how little Generation is a part of the process. Without Generation:

• you cannot evaluate how practical a design is to generate
• you cannot determine if the design actually achieves its purpose
• you cannot identify interfering factors
• you cannot assess which components are actually useful/necessary
• you cannot analyze usage patterns for insights into how to better fit the design to its uses
• you cannot compare design choices empirically
• you cannot test engine implementations (or any other aspect of the development chain) at any reasonable scale

Having Generation be one of the last requirements leads to a lot of surprises after a lot of effort has been invested, making it painful to incorporate lessons to be gained from those surprises, and at the same time leaving us with little information with which to make decisions about those designs.

I would suggest that Phase 2's responsibility should be to demonstrate that the design produced by Phase 1 is beneficial. This involves having one (or more) generators, which in turn have generated a corpus of realistic programs (not just test cases), which in turn execute on some engine. If the purpose of the proposal is some new functionality, then the corpus should cover the use cases that were intended to be served by that functionality (these use cases should be produced by Phase 1). If the purpose of the proposal is efficiency, then there should be experiments demonstrating the expected improvement to efficiency. This corpus and experimentation should also be reviewed for insights on how to revise the proposal. For example, we can analyze how frequently instructions are being used, or explore whether they are being used in particular patterns that maybe could be compressed for more compact binaries. The point is, if Phase 3 is about incorporating the proposal broadly into the WebAssembly ecosystem, then Phase 2 should be demonstrating that that effort is worthwhile and that appropriate empirical reflection on the design has been done before committing that effort.

[tlively]

I agree that design feedback from code generators is important, and I broadly agree with your description of the phases, but I don't want to overstate the current problem. Most proposals that are conspicuously missing toolchain feedback are still in phase 1 or 2, so the process is already mostly working as intended, even if there is already a lot of momentum that makes large design changes unpopular. This issue is mostly about calibrating and documenting our expectations for what kind of feedback proposals should be seeking out in the near future. I also want to acknowledge that it is more difficult to be motivated to implement a proposal in toolchains if there is not yet an engine implementation available for end-to-end testing, so I wouldn't say we should entirely turn things over and require toolchain feedback before engine feedback.

[RossTate]

Good points.

[lars-t-hansen]

In the specific case of table.fill and the ref types proposal, this would have resulted in us dropping the table.fill instruction and advancing the proposal to the next phase.

I agree for the most part. One issue is that it's not hard to imagine that we will support this feature in the future. So we'd probably give it an opcode, but not include it in this proposal (as we've done for some SIMD insructions).

I'm likely nitpicking, but: Another aspect is that some instructions may not be suitable for use by a generator or what I would normally think of as a toolchain, but may still be necessary to support other code. Maybe this comes down to terminology, is libc a "toolchain" even if it is not something we run to generate or transform wasm code, and is memory.grow (no doubt invoked by a __builtin phrase) thus "used by a toolchain"? Is libc part of the toolchain because it supports C/C++? Even if the answers to those questions are affirmative, the criterion for an instruction's acceptance should probably be open to the fact that it can be useful in limited circumstances even if it does not meet the "generator" or "toolchain" requirement, or it requires a fairly broad definition of "toolchain".

[tlively]

Good point, @lars-t-hansen. I totally agree that instructions used by low-level runtime libraries like compiler-rt should count as used by a toolchain. In that case the code generator does not necessarily produce the instructions, but it still depends on their existence and use to produce functioning code.

Your mention of builtin functions raises another nitpicky question, which is whether it counts for a code generator to support producing an instruction from a builtin function, but not produce that instruction from any other code sequence and not use that builtin function in any runtime library. I would say that probably shouldn't count, since it doesn't demonstrate that the instruction is useful for anything, even though it is nominally supported.

[binji]

Your mention of builtin functions raises another nitpicky question, which is whether it counts for a code generator to support producing an instruction from a builtin function, but not produce that instruction from any other code sequence and not use that builtin function in any runtime library.

@tlively I think we shouldn't have a hard rule about this. With SIMD I think it's expected that some instructions will only be used in intrinsics and written directly by developers, and that's OK. I don't think the spirit of the toolchain requirement is violated in that case.