Notably, the C standard does have an undefined behavior annex, Annex J.2, since C99. It's not mentioned in the post, and this omission in tandem with repeated usage of "C and C++" at various points might suggest otherwise to some people.
And C has a UB study group and we are working on a technical rapport that in great detail describes how UB works in C, and a comprehensive list of sample code for all known UB.
Thw C standard has a list of all known UBs. We don't know if the list is complete, because the C standard makes anything not defined by omission UB, so its possible to discover new UB. The language is however pretty thoroughly explored.
No. It would require listing not just every place the standard states undefined behavior is the result, but also every corner case not covered by the spec, or that is under-specified. It’s possible to document known instances, but never to document all possible sources of undefined or unspecified behavior.
> We also will require future proposal authors to keep the annex updated if they add or remove undefined behavior. They are making undefined behavior an explicit topic to be discussed when reviewing a proposal.
It’s a poor name all around, because the spec already precisely defines boundaries of what is UB — “if this and this happens, the behavior is undefined”.
These are precisely defined situations, that in modern compilers’ interpretation, are forbidden from ever happening in any program under any circumstances.
UB is only a name for nebulous consequences of violating these very specific prohibitions.
The spec explicitly does not define the boundaries of all UB. Look at the note on the definition in 1.3.24 and the context of the surrounding definitions:
Undefined behavior may be expected when this International Standard omits any explicit definition of behavior...
Rather than having specific causes, UB is simply the label given to "stuff outside the standard with no requirements whatsoever". Some of this stuff is important and much more of it isn't. A small subset of that infinite universe of UB is the set of enumerated undefined behaviors you're talking about, where the standard declines to specify any requirements in certain situations. These are what the annex is trying to organize.
There's already a bunch of stuff in the standard which explicitly says, "if you do xyz, the behavior is undefined". Stuff doesn't become "defined" just because the standard points out that it's not defined.
I don’t know. Safe rust has 0 UB although if I recall correctly things are still a bit messy with integer overflow. I could imagine a world where we define annotations around “signed integer will never overflow” that you can add to hit paths but otherwise disallow optimizing around that UB and for all other UB require an explicit annotation acknowledging it or just a warning and a missed optimization instead.
Safe Rust has 0 UB, but the person you're responding to qualified that with:
> that must be enforced at non-zero runtime cost.
Some of Rust's behavior that permits this is not zero runtime costs. Bounds-checking, for example, has a non-zero cost: the bounds check!
Now, I greatly prefer Rust's approach: I'd rather have the marginal CPU cost: I value my time far higher, at least until the profiler speaks up, and in many cases the optimizer is pretty good at eliminating checks that I myself might look at and go "but I know $condition is true here!" — so too does the optimizer. (And these days, Godbolt makes testing that very simple.)
And in the worse case, there's unsafe{}, and it is at least explicitly labelled as such to the next reader.
> things are still a bit messy with integer overflow
Integer overflow is well-defined behavior, but the behavior depends on compile settings. (https://doc.rust-lang.org/book/ch03-02-data-types.html#integ...) I do hope that someday, the debug behavior becomes the behavior: IME most overflows are errors / the author did not intend for it to occur. And the "panic on overflow" behavior is removable by simply using one of the functions that specifies an overflow behavior, in which case then the author's intention is just explicitly stated.)
things are still a bit messy with integer overflow
No, they aren't messy, they're fully defined. Integer overflow in safe Rust works just like Java: the numbers wrap around. It's totally defined.
You can, optionally, also enable a runtime check for this wraparound. This is usually enabled for debug builds. There is of course a performance cost to doing this.
> they aren't messy, they're fully defined. Integer overflow in safe Rust works just like Java: the numbers wrap around.
That is both fully defined and messy. They are not mutually exclusive. It means there are integers where n+1 is less than n. That is messy. No integers that I learned about in math class work like that.
The only two non-messy well-defined behaviours are 1) bignums by default like in Python, but not suitable for a low level language like Rust; or, 2) trap on overflow, like Ada is supposed to do though it is usually shut off by a pragma. Both of those have significant runtime cost.
> No integers that I learned about in math class work like that.
Did you learn about modular arithmetic in math class?
> The only two non-messy well-defined behaviours are 1) bignums by default like in Python, but not suitable for a low level language like Rust; or, 2) trap on overflow, like Ada is supposed to do though it is usually shut off by a pragma. Both of those have significant runtime cost.
No, they're not the only "non-messy" behaviors. An example of where wraparound is the desired behavior is computing hash functions. An example of where saturating arithmetic is the desired behavior is processing audio samples.
> Did you learn about modular arithmetic in math class?
I knew some wiseacre would say that. No those aren't integers, they are equivalence classes of integers.
Yes there are times when wraparound is desirable, just like there are when addition mod 12 is desirable (when figuring out times of day). But you don't want the default behaviour of integers to give 8+5=1. If you want that, fine, but ask for it explicitly.
For example, in Ada, if you want modular arithmetic, just specify it in the variable declaration. That is the right way to do it. C++ gets it wrong in that unsigned overflow is modular, signed overflow is UB (so at least you can ask the compiler to signal an exception), but there is no option of unsigned arithmetic where overflow is an exception.
> those aren't integers, they are equivalence classes of integers.
And they form a ring and sometimes even a field. That's the least messy behavior of any arithmetic type usually implemented on a computer. The only messy part is division, which doesn't match modular division, but it's what you actually want, usually.
Most of the design problems around arithmetic types in programming languages are a result of people "wanting integers" (or "real numbers") rather than facing the reality of the machines they're programming for.
> trap on overflow, like Ada is supposed to do though it is usually shut off by a pragma
Sounds like people "usually" opt for "messy" behavior.
> For example, in Ada, if you want modular arithmetic, just specify it in the variable declaration. That is the right way to do it.
No disagreement there. Unfortunately, almost all non-niche programming languages get arithmetic wrong, ironically, because they're supposed to be the simplest types.
You have to choose from a range of possible behaviors that all have their use cases, and you have to be aware of the limitations of the actual type you're working with, which is not an integer but something finite. The language cannot make that choice for use. What the default is almost doesn't matter because choosing a particular behavior should be clear and simple.
It sounds like you're telling me Rust also gets it wrong. C and C++ at least allow the implementation to do the right thing (i.e. trap) on overflow for signed ints, though they mandate doing the wrong thing for unsigned. I'd call Rust, Ada, C, and C++ all niche languages these days though (the niche is low level system work). #1 on TIOBE is Python whose native arithmetic type is arbitrary precision, which is really the right thing to do.
Yes, modular arithmetic is convenient for computers, but it's not integer arithmetic! If you're using machine words to denote actual integers, and your program does something that causes an overflow, that is an error and signalling the error is far better than quietly giving the wrong answer. It's just like in Python where ints are arbitrary precision. They can still get too big for the implementation (i.e. the computer can run out of memory) but that is unquestionably an error condition. Machine integers are the same thing except the constraint is running out of bits in the machine word, rather than running out of memory. If that happens, it's also an error, unless you chose a datatype indicating a different intention.
This all seems obvious to me, but I've seen the same misunderstanding in other places before. I don't understand why it isn't obvious to everyone.
The key is "that must be enforced at non-zero runtime cost." Safe Rust is indeed generally free of UB, but that requires bounds checks at a minimum.
> I could imagine a world where we define annotations around “signed integer will never overflow” that you can add to hit paths but otherwise disallow optimizing around that UB
That's kind of what Rust does - overflow is not UB, but you can use unchecked_add/mul/div/sub to get UB-on-overflow if you explicitly want it.
There's lots of things beyond bounds checks that constitute UB and bounds checks are but 1 small item. Generally Rust avoids the runtime cost of that through a three prong strategy of idiomatic Rust amortizing the cost of a bounds check to 0 (e.g. efficient iterator implementations that don't need bounds checks), eliding the check altogether if the compiler can prove it's duplicate / unneeded for some reason, or providing unsafe Rust as an escape hatch when you absolutely need it. C++ already has something similar via `.at` vs `[]` although the shorter notation being the unsafer but faster option is debatable & likely the thing most people use by default. Explicit annotations are probably the better approach.
It's also got a great ecosystem. I love the assume crate to do these annotations instead of writing unsafe code explicitly:
assume!(unsafe: i < v.len())
Now you've explicitly written an assumption that will cause the compiler to elide the bounds check in release mode but still assert it in debug mode (vs just doing unsafe & using variants that bypass the bounds check).
> There's lots of things beyond bounds checks that constitute UB and bounds checks are but 1 small item.
Indeed, hence "at a minimum". The type system and choosing to avoid UB when defining some operations helps a lot as well, but those have no direct overhead so there's no runtime cost there. Bounds checks (and overflow checks, if those are ever added to release mode) are just the one thing that have a direct runtime cost when they aren't elided.
How does Rust as a language avoid the "UB" traps that are fundamentally just compiler optimizations? For example in the case where you have an array of 8 elements, you access the element at index x, then (in program order) test whether x is less than 8. It seems than an optimizing compiler with a known-bits analysis could conclude that x is always less than 8, and remove that test. I don't know much about languages, so I am interested in how that trap is a consequence of C and avoided by Rust.
Rust performs runtime bounds checks on array access unless the compiler can prove that the index is in bounds. As such, removing the check is safe, because it is actually true that the program will never reach that line with an out of bounds array.
This is in contrast to C, where the out of bounds access us merely undefined, so the compiler is allowed to have the program continue execution passed it.
The Rust compiler/stdlib inserts bounds checks for accesses that cannot be proven to be safe, so in this example you'd get a panic on the out-of-bounds access itself instead of hitting the assert afterwards.
If you use get_unchecked() to intentionally bypass those bounds checks, the assert is indeed removed, but that requires an unsafe block.
Speaking more generally, Rust gates UB behavior behind unsafe blocks, so it's much harder to unintentionally hit UB than in C.
Rust performs a runtime check to protect against out of bounds array accesses. In some cases those checks can be optimized out if the compiler can prove it's not needed.
Yes, and the compiler does that. That's one of the reasons that iterators are preferred over for-loops with manual access in Rust. While compilers can for very simple cases determine that you will always stay within the bounds (as in your example, where it can find at compile time that the length will always be 8 and no access outside happens) with iterators the compiler doesn't need to know the actual length and can always elide bound checks.
The Rust compiler is smarter than that. Even for classic subscript-type loops, it can often elide the bounds checks. There was a discussion on this in HN a few months back, with machine code excerpts. An iterator and an explicit loop with subscripts both generated the same machine instructions.
If the compiler can elide bounds checks in inner loops, that's usually enough. Bounds checks elsewhere rarely affect performance.
Pedantic note: compiler doesn’t elide bounds checks in code using iterators. They’re not there to begin with. Iterators in the standard library are implemented using unsafe code, which calls non-bounds-checking variants of functions getting elements from collections. There is nothing for the compiler left to do here regarding bounds-checking.
Safe Rust has no undefined behavior. Undefined behavior does not mean no crashing, it means that the semantics of the program are undefined.
Rust's semantics are to abort on a stack overflow. A language like C or C++ have no such semantics, they may abort or they may continue running and producing jibberish.
The fact that this program results in reading/writing an unmapped memory address means it’s doing an out-of-bounds access. It segfaults on macOS because the runtime/OS has allocated the stack such that the overflow results in a bad memory access, but that is a behavior of the runtime/OS/hardware, not the language.
I guarantee I could exploit this on a system that does not have virtual memory, or a runtime that does not have unmapped addresses at the end of the stack, to, say, manipulate the contents of another thread’s stack. Therefore, this behavior is undefined.
The language runtime can require that the OS & hardware always results in an exception on stack overflow (or, alternatively, compile in explicit checks for it). You running the program in an environment without that is, technically, just as wrong as running it on a system where integer addition does multiplication.
Now perhaps this means that there are real rust deployments that are "wrong", but that shouldn't include regular sane standard systems, and embedded users should know the tradeoffs.
.LBB3_1:
sub rsp, 4096
mov qword ptr [rsp], 0
cmp rsp, r11
jne .LBB3_1
That's a loop at the start of your 'main' that probes the stack specifically to ensure a segfault definitely happens if your array didn't fit on the stack.
> It segfaults on macOS because the runtime/OS has allocated the stack such that the overflow results in a bad memory access, but that is a behavior of the runtime/OS/hardware, not the language.
Stack overflows are checked in C on macOS not because of guard pages but because the compiler emits stack checks (with cookies). Probably the same is true here.
> I guarantee I could exploit this on a system that does not have virtual memory, or a runtime that does not have unmapped addresses at the end of the stack, to, say, manipulate the contents of another thread’s stack. Therefore, this behavior is undefined.
Software stack checking does not guarantee protection from stack overflows wreaking havoc. E.g., your thread could blow its stack, then get preempted before the stack checker can run.
Mandating guard pages/MPU protection would rule out targeting embedded platforms which lack sufficient hardware support.
What does preemption change here? Before the stack checker has finished, nothing else should hold a reference to any of the yet-unchecked stack. That's plenty trivial to ensure. (unless you mean preemption somehow breaking the stack checker itself, in which case, well, that's a broken stack checker and/or preemption, and should be fixed)
If you can't have hardware support, it's trivial for the compiler to do it in software - just an "if (stack_curr - stack_end < desired_size) abort();". I can't imagine a platform where there you cannot reasonably get a lower bound for the range of stack available. Worst-case, you ditch the architectural stack pointer and manage your own stack on the heap, if that's what you need to ensure correct Rust behavior on your funky platform (or accept the non-compliant compromise of no stack checking).
> What does preemption change here? Before the stack checker has finished, nothing else should hold a reference to any of the yet-unchecked stack.
If your thread overflows the stack, it could start writing into memory for which it does not hold a reference. If the thread is preempted before the stack checker can run (see below*) and detect the overflow, and another thread runs which accesses the now-corrupted memory, then you're hosed.
> just an "if (stack_curr - stack_end < desired_size) abort();"
That's not how the compiler-emitted stack checking works AFAIK (*I believe it uses canaries on the stack which are checked at certain points in code). But, I could see this solving the problem. Basically, for every instruction that manipulates the stack pointer (function calls, alloca's, and on some arch's interrupts use the current stack), the resulting address would need to be checked. That would be costly and require OS awareness, but I think it would be safe. Is this an option that the compiler provides? It would save me a lot of time debugging.*
Canaries are a separate unrelated thing solving a different problem - buffer overruns, i.e. writing out-of-bounds. (canaries are a best-effort thing and don't guarantee catching all such problems, and they're also useless for safe Rust where unchecked OOB indexing is not a thing; whereas stack overflow checking can be done precisely)
In my sibling comment showing the assembly that your Rust program generates, it is writing a "0" every 4096 bytes of the stack range that is intended to be later used as the buffer (this "0" is independent from the "0" in your "[0; N]"; it's just an arbitrary value to ensure that the page is writable). It does this, once, at the very start of the function, before everything else (i.e. before the variable "var" even exists, much less is accessible by anything or even initialized). This is effectively exactly the same as my "if (stack_curr - stack_end < desired_size) abort();", just implemented via guaranteed page faults. You can enable this on clang & gcc with -fstack-clash-protection where supported.
Indeed, stack checking can have overhead (so do other requirements Rust makes!), but in general it's not that large. If you don't have stack-allocated VLAs, it's a constant amount of machine code at the start of every function, checking that all possible stack usage the function may do is accessible. And on systems with guard pages (i.e. all of non-embedded) the overhead is trivially none for functions with frame size below 4096 bytes (or however big the guard range is; and for larger frame sizes the overhead of this check will be miniscule compared to whatever actually uses the massive amount of stack).
I don't know if it's technically UB or well defined. The crash is a SEGFAULT and not a panic/abort, but it's probably a SEGFAULT due to guard pages. Still, it's possible to evade guard pages so if you access var[X] such that X points to the heap, it's possible you're reading aliased memory which would be UB in safe Rust.
EDIT: Going to take it back. I'm unable to create a situation where I create a large stack array that doesn't result in an immediate stack overflow. I even tried nightly MaybeUninit::uninit_array but that crashed explicitly with a "fatal runtime error: stack overflow" so it seems like the standard library has improved reporting instead of the old SEGFAULT. So no UB.
Panics are not quite the same as an abort in Rust. Most notably a panic can be caught and execution can resume so as to gracefully terminate the application, but an abort is an immediate termination, a go to jail do not pass go kind of situation.
An out of bounds access in Rust will result in a panic but a stack overflow is an abort.
The stack guards would normally be setup by the system runtime (e.g. kernel in the case of the main thread stack, libc for thread stacks), not Rust's runtime. Likewise, stack probes that ensure stack operations don't skip guard pages are usually (always?) emitted by the compiler backend (e.g. GCC, LLVM), not Rust's instrumentation, per se.
In this sense Rust isn't doing anything different than any other typical C or C++ binary, except that automagically hijacking SIGSEGV (or any other signal) from non-application code as Rust does is normally frowned upon, especially when it's merely for aesthetics--i.e. printing a pretty message in-process before dying. Also, attempting to introspect current thread metadata from a signal handler gives me pause. I'm not familiar enough with Rust to track down the underlying implementation code. I presume it's using some POSIX threads interfaces, but POSIX threads interfaces aren't async-signal safe, and though SIGSEGV would normally be sent synchronously (sometimes permitting greater assumptions about the state of the thread), that doesn't mean the Rust runtime isn't technically relying on undefined behavior.
EDIT: To get the guard page range it's using pthread_self, pthread_getattr_np, pthread_attr_getstack, and friends, of which only pthread_self is async-signal safe. See https://github.com/rust-lang/rust/blob/411f34b/library/std/s...
I have no concrete evidence to believe the reliance isn't safe inpractice on the targeted platforms (OTOH, I could imagine the opposite), but it's a little ironic that it's depending on undefined behavior.
The runtime thing is the easy part. I was wondering about the stack probes, which require LLVM support. There's a comment in the sources that suggest it's still x86-only, but that may be outdated:
“
//! Finally it's worth noting that at the time of this writing LLVM only has
//! support for stack probes on x86 and x86_64. There's no support for stack
//! probes on any other architecture like ARM or PowerPC64. LLVM I'm sure would
//! be more than welcome to accept such a change!
”
I don't see where those methods are getting called from a Unix signal handler but the code is complex enough that it's easy to miss, especially perusing through github instead of vscode.
AFAICT those methods are called from `guard::current`. In turn, `guard::current` is used to initialize TLS data when a thread is spawned before a signal is generated (& right after the signal handler is installed): https://github.com/rust-lang/rust/blob/26907374b9478d84d766a...
It doesn't look like there's any UB behavior being relied upon but I could very easily be misreading. If I missed it, please give me some more pointers cause this should be a github issue if it's the case - calling non async-safe methods from a signal handler typically can result in a deadlock which is no bueno.
x86_64 macOS has tier 1 rust platform support, which I believe means that it's guaranteed that you get a crash on stack overflow and you can't evade stack protection in safe rust.
It's not possible on all platforms, hence the tiers.
Apparently ARM64 macOS has tier 2 rust platform support, which might mean that that this is not true there, but maybe safe rust has some different unrelated soundness issue on this platform.
I only have very surface knowledge about the tier stuff, so maybe someone can correct me.
The hard part of UB isn't where it's mentioned in the standard. The problem is when the standard says "in case A, X occurs, in case B, Y occurs" then somebody invents a situation where neither A nor B apply.
And even better, start reducing the list after it's complete, by pushing things like integer overflow into implementation defined; several uses of inc/decrement operators into compilation error; and etc.
I don't want integer overflow to be implementation defined though, I want it undefined so my compiler can optimize my code for the fact that I don't overflow my integers in the first place.
Implementation defined is not much better than undefined. It only requires documenting the behavior, so compilers could just go like "here are all the optimizations that we apply on each target platform for each set of compiler flags". It's not very helpful, you might as well look at the source code of the compiler.
That's still substantially better than undefined, where multiple runs of the same compiler version on the same source with the same settings are allowed to result in different behaviors. I suppose an implementation could define non-deterministic behavior, but that's unlikely and could be forbidden by the standard.
That said, my overall point is that optimization does not rely on undefined behavior at all. It's commonly argued that it does, but there are languages without undefined behavior that have working optimizers, so it's clearly false. Some optimizations for C (and C++) currently depend on undefined behavior, but there's nothing inherent about that dependence.
I love how in C++ there seems to be this security issue, and now hundreds of people try to come up with different ways on how to tackle the problem.
Everybody wants to help and spend their free time on it.
You don't see this in other languages so much. People just use those languages and "they" have to fix any flaws.
Also keep your bingo cards ready for people saying Rust does not have undefined behaviour.
I'm also not so sure Rust is even that safe. Yes it's memory safe but the languages with the most vulnerabilities is not C++. It's PHP, .NET, C, Java, JavaScript.
C++ is not an unsafe language. You can't make a language safe by adding an "unsafe" keyword end expect everything outside it to be safe. Else .NET and Java would be lower on the list than C++ because they do have bounds checks and they don't have raw pointers.
It's a bit of a false sense of security and a play of words this "unsafe" keyword in Rust.
> You don't see this in other languages so much. People just use those languages and "they" have to fix any flaws.
You absolutely do see that elsewhere, and that's why here on HN we have a different new JS framework every day.
The safety in Rust (and .NET, and Java) doesn't come from the `unsafe` keyword, it comes from the guarantees that you have when you don't use said keyword. Ownership, garbage collectors, and less undefined behaviour are some reasons why those languages are considered safer.
At the end, memory safety is a spectrum, and while you can still break Rust's (or even Java's) memory model, it's much harder to do so unintentionally. That's a win. Even if there are some undefined edge cases!
That sounds great, and that's the narrative you constantly hear. Java, .NET are memory safe.
BUT: they all have more vulnerabilities on average than C++.
So just saying, memory safety alone does almost nothing to make a language safe.
Why would Rust be so much safer than C++ if so almost all other languages also have memory safety but are on average less safe than C++? Meaning more vulnerabilities (so not only memory safety just to be very clear)
You have data to confirm your claims?
Moreover, vulnerabilities in languages themselves are not that important in comparison to vulnerabilities in software written in the specific language, at least how I understand this.
I don't have solid data but for "core" mature software written in c/c++ like browsers and Linux, I feel like I see far more high profile security bugs from the lack of memory safety rather than something like "Linux failed to enforce the existing permissions".
