There is something I need to add, it is a subtle but important point:
Non-determinism can be caused by
a) random seeds derived from hardware, such as seek times in a HDD controller, which is fed into pseudo random number (PRNG) generation. This is not a problem. For debugging, or comparison, it can make sense to switch it off, though.
b) data race conditions, which is a form of undefined behavior. This not only can dramatically change results of a program run, but also invalidates the program logic, in languages such as C and C++. This is what he blog post in "lockdownskeptics.org" suggests. For the application area and its consequences, this would be a major nightmare.
c) What I had forgotten is that parallel execution (for example in LAM/MPI, map/reduce or similar frameworks) is inherently non-deterministic and, in combination with properties of floating-point computation, can yield different but valid results.
Here an example:
A computation is carried out on five nodes and they return the values 1e10, 1e10, 1e-20, -1e10, -1e10, in random order. The final result is computed by summing these up.
Now, the order of computation could be:
((((1e10 + 1e10) + 1e-20) + -1e10) + -1e10)
or it could be:
(((1e10 + -1e10) + 1e-20) + (+1e10 + -1e10))
In the first case, the result would be zero,
in the second case, 1e-20, because of the finite
length of floating point representation.
_However_... if the numerical model or simulation or
whatever is stable, this should not lead to
a dramatic qualitative difference in the result
(otherwise, we have a stability problem with the model).
Finally, I want to cite one last paragraph from the post on lockdownskeptics.org:
> Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.
> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.
Non-determinism can be caused by
a) random seeds derived from hardware, such as seek times in a HDD controller, which is fed into pseudo random number (PRNG) generation. This is not a problem. For debugging, or comparison, it can make sense to switch it off, though.
b) data race conditions, which is a form of undefined behavior. This not only can dramatically change results of a program run, but also invalidates the program logic, in languages such as C and C++. This is what he blog post in "lockdownskeptics.org" suggests. For the application area and its consequences, this would be a major nightmare.
c) What I had forgotten is that parallel execution (for example in LAM/MPI, map/reduce or similar frameworks) is inherently non-deterministic and, in combination with properties of floating-point computation, can yield different but valid results.
Here an example:
A computation is carried out on five nodes and they return the values 1e10, 1e10, 1e-20, -1e10, -1e10, in random order. The final result is computed by summing these up.
Now, the order of computation could be:
((((1e10 + 1e10) + 1e-20) + -1e10) + -1e10)
or it could be:
(((1e10 + -1e10) + 1e-20) + (+1e10 + -1e10))
In the first case, the result would be zero, in the second case, 1e-20, because of the finite length of floating point representation.
_However_... if the numerical model or simulation or whatever is stable, this should not lead to a dramatic qualitative difference in the result (otherwise, we have a stability problem with the model).
Finally, I want to cite one last paragraph from the post on lockdownskeptics.org:
> Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.
> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.