To add to this part: "My main epiphany was that realtime applications do not exist. Rather they cannot exist."
Most of us think our applications are realtime (never mind the OS schduling cpu time for your code, so inherently not realtime) but this is only because it behaves as expected, while there is not too much load nor too much data. The moment you start having significant computational load or start having too much data (aka the cpu, memory or storage starts choking), or when you start to split data apart (aka distributed applications) then the problems with "realtime" applications starts to become obvious.
If something goes wrong, to fix the immediate problems would be to retry/rerun the simulation, but that might result in duplicating your intents, which results in the simulation and its state as invalid. Then you either (hopefully) have some mechanism to de-duplicate whatever happened more than once or you have to start you simulation from scratch after fixing the data manually.
Or just design the core of the system with Idempotency in-mind from the start so you have a good way to handle such scenarios. Embrace that most things can/should be handled in a eventually-consistent way. Very few things in life has to happen RIGHT NOW (good luck with that).
An interesting spin on realtime systems are computer games, where we refresh a view of the simulation every few milliseconds (aka 60fps). Every frame gets rendered based on the inputs (keyboard/mouse)from just-after the previous frame. So in effect, the current frame being drawn is just a reaction based on previous inputs. So let me ask you this: in a game engine like this, where would you assign the value of "now"? Right after the previous frame? When capturing user input? When rendering the new frame starts? When showing the new frame but before capturing the new inputs? Where is reality's "now" moment in a computer game? Let's ignore the delays between keyboard/mouse input and drawing from CPU/GPU to the physical screen and lets assume this is a single player FPS and not a multiplayer turn-based game. And lets ignore sound processing too - just think of the simplest of game loops - should we consider them as realtime? Ponder that.
> An interesting spin on realtime systems are computer games (...) Ponder that.
My younger self did that, quite a lot, thank you. I never felt comfortable with various approach of intermixing input handling, physics/game logic and rendering - whether to run them sequentially, decouple some or all of them, whether the physics should be "look-ahead" or "look-behind"... I ended up picking the pattern that leads to the most stable and deterministic behavior (look-behind physics in lockstep with input and on fixed update rate, rendering independent and done variable-rate), and stopped thinking about it.
So thanks for mentioning it in this context, it makes me more comfortable to hear from someone else that "now" in games is ill-defined on philosophical level, and I shouldn't have worried about it as much as I did.
Going a little bit further, there's the time between the frame displaying on the screen and it reaching your retina, then the time between that and it reaching your brain in the form of electro-chemical signals, then the time between that and your brain decoding it.
By the time you perceive anything, and even more so by the time you react to it, it's already in the past.
