Java applets, Flash, and Silverlight were famously examples of people being overconfident in software isolation. There's been an active effort to kill off browser plugins, and wasm (and asm.js and Native Client before it) is the result of an attempt to accomplish the same goal in an actually sound manner.
Dalvik on Android is not a software isolation mechanism. Each app has its own Dalvik VM using traditional UNIX processes and user accounts for isolation, which are in turn powered by hardware isolation. (I'm betting they did this because they knew that JVM software isolation had been a disaster in practice, although in part this also means apps can link native code libraries without changing the security model at all.)
The JVM architecture had and has proven safety characteristics. The WASM machine model is more constrained in order to make proof easier. The JVM is more complex but is intrinsically safe all the same. Sun just had more dedicated sources to work through the proof.
The promise of the JVM failed not because of design, but because of buggy implementations. The drive for features and performance brought risk. And WASM is heading down the same path. WASM can add GC with an unimpeachable proof of design safety, but it would be irrelevant.
The issue isn't design or architecture. The issue is that the motivation for adding GC and other features to WASM is principally for performance. Version 1.0 of WASM will be the last specification that made security paramount. Everything after that will be a managed retreat as more and more features and code are placed outside the sandbox in the endless pursuit of performance.
To reiterate: Java applets weren't insecure because the JVM was insecure. Java applets were insecure because the vast majority of the implementation of the environment--and particularly the most complex aspects--existed outside the confines of the sandbox. The more feature rich and performant the sandbox environment then necessarily the more code and complexity must exist outside the sandbox.
ActiveX, Silverlight, and (worst of all) Flash were far more insecure, both in design and implementation. But that's a distinction without a difference; their relative inferiority didn't make Java applets more viable.
WASM will be insecure for all the same reasons: JITs and GCs are tremendously complex beasts, with implementations least amenable to verification methods relative to most other software projects. Rust's borrow checker is worthless for ensuring memory ordering and life cycle invariants of machine objects. DOM implementations have similarly become tremendously complex, yet the focus is on how to expose these implementations and their interfaces in entirely novel and brittle ways, prioritizing performance above all else. The most safe alternative and likely sufficient for 80% of use cases would be a message passing interface, afterall. Instead, design proposals are focused on finding the thinnest possible abstraction over direct addressing of DOM object references from within the VM. Thus the prioritization of GC.
Java has had about 20 years to get the implementation right. In my day job I still run Java desktop code because that's how server remote consoles work and they've been slow to switch to HTML5, so there was obviously a market for getting Java applets to work soundly until very recently. If your design results in buggy implementations despite 20 years of work, it's a buggy design. If your "proven safety characteristics" are that it worked on paper and not any implementation actually secured anything in the real world, it's a useless definition of "proven."
I do not think that "JITs and GCs are tremendously complex beasts" is a meaningful argument against wasm, because plain old JavaScript on the web has JITs and GCs, and is overall a much more secure environment to run than Java. (For instance, approximately every enterprise disables Java for security reasons except for people with a business need for it, and approximately no enterprise disables JavaScript for security reasons. I'm not claiming JS is foolproof, just that it's been much much more secure in practice.) You might note that Sing#, where this subthread started, had a GC. So does Inferno's virtual machine. If you think GCs make things too complex for software fault isolation to work, do you believe Singularity and Inferno are insecure?
I don't claim to be an expert in this, but I think there's a coherent reason why Java's design failed: Java attempted to do isolation at the language level (and inside a language-specific bytecode), which is a richer interface. It should be entirely possible to develop a high-performance interface for software fault isolation as long as it's a small enough interface to successfully secure, and my sense is that that's where wasm is going.
One important advantage of wasm over Java, Flash, and Silverlight is that it can learn from their failures.
Message-passing doesn't need to be slow either - just write your messages in a high-performance but securely parseable format like Cap'n Proto. We know how to do such things now; we didn't 20 years ago. The state of the world keeps advancing.
"The promise of the JVM failed not because of design, but because of buggy implementations. "
Well said. That happened everywhere from desktop to server to embedded. However, safety-critical side of embedded further supported your point by building real-time, safe implementations of JVM (or subset) designed for certification. There were also companies using Ada to get systematic protections against errors with one, Praxis, doing semi-automated proofs with their SPARK language. A JVM implemented in Ada, SPARK, and (where necessary) C/C++ might have been much safer.
"JITs and GCs are tremendously complex beasts, with implementations least amenable to verification methods relative to most other software projects. "
There's actually verified JIT's and GC's they can draw on. I doubt they will, though. History shows they'll go with a non-verified design followed by penetrate and patch.
"Rust's borrow checker is worthless for ensuring memory ordering and life cycle invariants of machine objects."
I've long pushed Abstract, State Machines (or languages based on them) to do this kind of stuff better. The work on memory models can be ported to something like Asmeta. The algorithms can be checked against them by solvers. Then, equivalent software and/or hardware comes out the code generator with more analysis/tests in case it messed up. I got excited seeing Galois was using ASM's for hardware/software verification recently. They'd be great for checking security of interpreters against software and hardware level issues.
"The most safe alternative and likely sufficient for 80% of use cases would be a message passing interface, afterall."
I haven't updated myself yet on advances in typing for that stuff. Pony's method for type checking might help here since it uses a capability-secure, actor model. Wallaroo also uses it for a high-performance database. So, it's not a slouch either.
ART on Android 5 and 6 is as much runtime as any other programming language with a AOT compilation model.
And as of Android 7, there are multiple execution modes. An hand optimized interpreter written in Assembly, a JIT compiler with PGO feedback, and an AOT compiler that takes the JIT PGO data to generate a proper executable on when the device is charging and idle.
Traditional native code is also pretty much clamped down in recent versions of Android via SELinux, seccomp and white list of shared objects.
Google doesn't want you to do more than just implementing native methods, high performance 3D graphics, real time audio or importing "legacy" libraries.
I don't consider WASM that sound because it still allows for internal data corruption of modules written in unsafe languages, instead of supporting proper memory tagging like SPARC and the upcoming ARM architecture.
ART still translates Dalvik byte code. If you separate "Dalvik the interpreter implementation" from "Dalvik the specification", you see that Android is still fundamentally based on Dalvik.
I think the parent comment's point about Android application isolation being enforced by hardware rather than software still stands.
Xerox Parc workstations with microcoded CPUs, IBM and Unisys language environments with JIT at installation time, Oberon, Java with ART on Android 5/6, .NET on Windows 8.x/UWP are certainly not using a VM.
It is not my fault that many devs still don't get the point between a VM, a language runtime, and how multiple implementations relate to a single language specification.
You know you managed to list examples that are literally case studies of VMs in "Virtual Machines: Versatile Platforms for Systems and Processes", right?
Going through your list:
Xerox PARC worstations: these didn't really run smalltalk in microcode, but ran an interpreter written Data General Nova asm, with the microcode "emulator task" dispatching and executing Nova machine code. There were a few new instructions added for smalltalk, but that was stuff like bit blit instructions. All that being said, even if the smalltalk VM interpreter was pushed down into microcode, it'd still be (a component of) a VM.
The big iron JITing environments are quintessential hardware/software codesigned VMs.
Oberon the language you might have a point, but the secure loader/verifier/compiler in it's only implementation is absolutely a VM.
Java with ART is absolutely a VM, unless you're going to make the argument that HotSpot isn't a VM.
.Net UWP is absolutely a VM too. Yes, it's partially compiled before it reaches end users, but also includes the entirety of .Net core linked in for cases where you're dynamiclly adding new code to your're running process.
Essientially, I think you've come up with some weird definition of VM that doesn't match industry or academia, and then berating people who don't follow that non standard definition.
Far as language-based security, the first mainframe for businesses used a high-level language combined with a CPU that dynamically checked the programs. Still sold by Unisys but I doubt hardware checks still exist.
Both AS/400 and System/370 have a so called language environments.
And yes I call them mainframes, because it is as I always heard people referring to them during my Summer job back in the day, so the name stuck with me even it isn't correct.
Language environment in IBM parlance is closer to "ABI" than "VM", or "sandbox" in the rest of the world. It's a common set of idioms to allow two languages (generally ASM and a higher level language like C) to call eachother and interoperate.
Xerox Parc workstations, IBM and Unisys mainframes, UCSD Pascal, Oberon, Inferno, Java, .NET, Flash.
Or for more actual examples, Garmin Apps, watchOS bitcode, DEX on Android.