I appreciate the author sharing his experience but I must admit that it was less exciting that I expected it to be. He basically used off-the-shelf hardware, ran an off-the-shelf program and then waited for a while. There's no obvious innovation, anybody who cares enough to break that record and has ~$20k lying around can do it.
If anything it got me a lot more curious about this y-cruncher program and all the fun optimizations it must implement.
The previous record, which was basically just corporate promotion [1], did use the exact same program.
For a sysadmin/hardware nerd this post was really interesting, although I would have probably have appreciated even more details (quick example, how many disks failed partially/totally during the process?).
Reminds me of my attempt to compute the number of Go positions, which similarly needed many TB of disk space and ended up generating 30 petabytes of disk IO.
For those that are interested in reciting digits of Pi (I would guess you can either memorise the digits or calculate as you go) the current record is held by a South African: https://www.youtube.com/watch?v=_mGjJMVKWcU
But pi defines its structure. By being seemingly random and irrational, it has created a specific ordered set of symbols and imbued that set with meaning because it has a referent: we call it pi.
It’s not random, but that doesn’t mean it is structured. And there’s absolutely no difference between the digits of pi and a completely random sequence for anyone trying to memorise them.
>> As luck would have it, a transformer next to my house blew and the power went out yet again. This means I had to restart from the last checkpoint yet again, losing another 2 weeks worth of work.
I remember a time back freshman year of collage... It was an extra credit assignment. Who could factor the largest number (or something like this) in the least time before the due date. My friend and I devised an algorithm, and launched it on my computer. I honestly don't remember anything other than the fact the god damn power transformer downtown blew and fucked up our tests because of a large scale power outage. I assume we'd have not even come in top #5 for that assignment (who knows), but it's just so frustrating.
Basically, OSs should have better fail recovery mechanics than are the default.
Running under EUMEL, L3, or KeyKOS would have enabled the program to continue from a recent checkpoint, without requiring any logic for this in the program itself.
Probably you're so familiar with modern computing environments — and unfamiliar with any others — that you take their drawbacks for granted, perhaps even assuming they are unavoidable.
This got me thinking, and I have a few weeks spare time - can you please recommend something that’s as fundamentally different as possible so that I can get way out of my comfort zone?
I can’t reply to your below comment I’m not sure why - but you seem experienced in alternative computing environments, whereas I’m mostly a HPC python / c++ developer that’s spent the last 10ish years doing deep learning and scientific computing - the newer environment doesn’t have to be practical at all, I’m interested to use it for a change in perspective
I wish I had a recommendation based on experience for one of these really strange operating systems like EUMEL, Guardian, OS/400, and L3. But I don't. I've used CP/M and MS-DOS, but those are just really limited, not really interesting. Although, with ZCPR and 4DOS, you could make them reasonably usable, it was like coming out of Plato's cave when I switched my primary operating environment from 4DOS to csh on Ultrix.
Squeak is a pretty different operating environment that isn't simply primitive. Oberon is another. They can both run as user processes on top of Linux, as well as on bare metal. Both of them are somewhat alien.
Are you comfortable with embedded development? If not, try Arduino. It starts out easy, since you program the boards in C++, but you have the opportunity to build things that will run for months on a AA battery with submicrosecond interrupt response time — because there's no OS. (It's routine for even programming novices to write their own interrupt handlers.) Arduino instantly gives you the ability to measure things on microsecond timescales, a thousand times faster than you can normally see. Modern boards like the Blue Pill have response latencies in the 100-nanosecond range when they're awake. That's the time it takes light to go 30 meters, as you're probably aware.
In retrocomputing land, VMS was the first OS I used that was really usable. The OpenVMS Hobbyist Program still exists, and it's actually possible to run old versions of Mozilla on it. F-83 was an interactive Forth IDE that provided higher-order programming, virtual memory, and multithreading under MS-DOS, in 1983 — without syntax or types. Turbo Pascal was also an IDE, in a way the first modern IDE, around the same time; the first versions ran on CP/M and MS-DOS. But I think that you kind of had to be grappling with the limitations of BASIC on those systems to appreciate that.
There are Pick systems that still have enthusiastic users: https://www.pickwiki.com/index.php/Pick_Operating_System but they don't sound appealing to me. Other systems with cult fanbases include FileMaker, HyperCard, and Lotus Agenda, which last I think you can run successfully under FreeDOS. Agenda is interesting in part because it's so alien. (It's easy to forget that it was normal at the time to have to use the program manual to figure out how to exit.)
There are a bunch of modern specialized development environments that can do strange things. Radare2 is an environment focused on reverse engineering. Emacs is focused on text editing, but for some reason it's also the main user interface for interactive proof assistants like Coq and Lean, which are shaping up to be pretty interesting. R is focused on statistics. Jupyter is sort of focused on data visualization, although not really. (Now I see you've been doing deep learning for 10 years, so I guess Jupyter is your best friend.) LibreOffice Calc is focused on rectangular arrays of mostly numerical data (although in many cases their most advanced users use Excel instead). You can develop applications in all of them.
How about math? It's one thing to invoke a Runge-Kutta integration method; it's another to be able to prove convergence bounds on it. And machine-checked formal proof is shaping up to be an interesting thing, like I said.
How about cryptography? That has the advantage that there are right answers and wrong answers, so you can test your code.
How about shaders? Shadertoy is accessible and super fun. Maybe that's too similar to HPC, but the shader parallelism model (similar to ispc) is pretty different from both AVX and MPI.
How about mobile development? SIGCHI papers are full of experimental user interface ideas to explore, and Android Studio is free and relatively usable, if clumsy. Have you seen Onyx Ashanti's Beatjazz?
In the neighborhood of beatjazz, there's livecoding. It's a thrill to get a nightclub full of people dancing to your code, and there are a bunch of different environments.
GNU Radio with an RTL-SDR makes it possible for you to run DSP algorithms on RF signals over a pretty wide frequency range, with applications in communications and sensing. Maybe if you've been doing HPC, DSP is already second nature, but if not it might be rewarding. And DSP has close connections to control theory and image processing, as well as the more obvious applications.
How about alternative programming paradigms? If you're comfortable in procedural and OO programming, how about extreme alternatives — answer-set programming like miniKANREN, constraint-logic programming (as supported by modern Prologs https://www.metalevel.at/prolog/clpz not just Mozart/Oz), Erlang-style fault-tolerance-focused programming, APL-style array programming (though maybe you're familiar enough with that to take it for granted), or Forth? How about strongly typed programming like Haskell, Rust, or OCaml? (And of course Haskell is purely functional, and OCaml is mostly so.)
And STM solvers like Z3 can easily solve problems now that were infeasible only a few years ago.
Also, wasm.
Or maybe try hacking together some games in Godot.
I don't know, myself I find that it's hard to avoid getting out of my comfort zone in some direction, just because the world is so big and my knowledge is so small. Deep learning is the out-of-my-comfort-zone programming thing I want to try next!
Holy moly. Over the past hour I've come back to your response and read it several times. There's so much to unpack there.
I started writing several replies but felt I wasn't able to give you the praise you deserved, but the passage of time compels me to respond so please know you have my full gratitude
STM is either a scanning tunneling microscope or software transactional memory, both of which are pretty interesting, but Z3 is neither.
Also, it occurred to me that I didn't even mention browsers, probably because I myself am so comfortable with them. The big software platforms right now are POSIX, Android, Java, browsers, and whatever Microsoft is doing now — it's enough effort to reuse code written for one environment for another that people often just rewrite it. Browsers have by far the best GUI library — not that the DOM is a great or even acceptable interface, but things like React and D3 are — and the major advantage that if you write stuff in a browser you can immediately show it to many people. The development tools are insane, and HTML5 supports most cellphone peripherals in a cross-platform way.
Same thing happened to me in college. I actually saw the transformer blow, it's kinda freaky (and that ozone smell is horrid, you only get to smell it when you weld or really screw the pooch).
Except I wasn't working on anything academically meaningful, more spiritual. I was the only one that could fit a keg in my trunk. Thankfully it was a day party in the spring.
For a prolonged power outage, the hibernate feature is probably more useful. And a UPS to perform it. But generally not impossible to survive such an event.
By "compressed Pi digits", does the author mean simply an efficient encoding of decimal digits in binary, or there is some way to compress Pi digits further? I thought they were incompressible like random.
Typically, that means storing two digits in a byte (halves storage size compared to a text string), 9 in each 32 bits (gains another 11%), 19 in each 64 bits (gains another 5%), or something similar (at this scale, I would guess it uses at least the ‘19 digits in each 64 bits’)
Idea is to not use “one digit per byte”, but to keep addressing individual digits cheap.
”I thought they were incompressible like random.”
They’re easily compressed, if you accept taking this program and it’s configuration file as a compressed version.
(And yes, the output of a pseudo-random number generator compresses extremely well, too)
That's the notion of kolmogorov complexity. The smallest program that can generate this output. Pi and I guess any other algebraic number, no matter how randomly distributed its digits are, are not that complex.
Use base pi and the shortest string of digits representing all of the digits of pi can be written as '1'.
This project seems pointless to me. There's no scientific value in knowing this many digits of pi. He used off the shelf components and off the shelf software. So there was no new engineering that advance the state of the art. At the end of the day the only thing of meaning that happened her is he used a bunch of electricity (and a corresponding CO2 release) for no socially valuable purpose.
Assuming 3kw and a PUE of 1.5 for cooling comes out to approx 20,000 kwh of power. Assuming a high-ish rate of $.13/kwh it comes out to around 2.5k for 6m. Not too bad.
If you cross byte boundaries, i.e. compactly pack the bits such that there are no wasted bits; then you have 4 bits per decimal digit, 50T digits becomes 200T bits == 25TB. Technically there is still some slack in there because each 4 bit block represents 16 values, and we only use 10 of those.
I got curious about the BOINC platform he mentioned. It reminds me of the SETI@home screensaver I used to run many years ago. I guess I'd just forgotten about it at some point, but I used to enjoy watching the data it was processing. It's pretty cool that there's another platform out there that lets you contribute to other programs.
Did he calculate the value in decimal? The record would probably be a lot easier to achieve if he'd gone with, say, base 14. Everybody does decimal, even though there's no theoretical advantage, and no practical advantage since nobody actually needs trillions of digits of pi.
Could probably save a good bit with committed use discounts.
Spot / Preemptible instances would not work, in fact before Emma did this calculation a lot of people thought this kind of thing wasn't possible on public cloud because of perceived instabilities in a multi-tenant system.
I'm trying to do some estimates using the AWS Pricing calculator.
You can't go with a single instance and a ton of EBS storage, because it caps out at 16TB of disk, and 19Gbit[0] of EBS bandwidth, even on an instance with 100Gbit networking.
So, depending on how you can allocate storage, you're probably going to need some kind of clustered filesystem like GlusterFS
It's also not clear how well the application can spread it's writes - if it's all focussed on writing one file at a time, we need the most throughput to a single node at a time.
Storage:
Option 1: "GlusterFS: Hope it spreads writes"
20x c5n.9xlarge (36x vcpu/96GB RAM/50Gbit NIC / 9.5Gbit EBS) + 16TB st1 HDD storage each
= $43k
Option 2: "GlusterFS: more EBS IO"
20x c5n.9xlarge (72x vcpu/192GB RAM/100Gbit NIC / 19Gbit EBS) + 16TB gp2 SSD storage each
= $89k
Option 3: "GlusterFS: hey local storage is faster/cheaper"
6x i3en.24xlarge (96x vcpu/768GB RAM/100Gbit NIC/ no EBS)
= $47k
I was wondering about insane ideas like using mdadm in RAID0 over NFS Mounts presented by (say) 281 t3.2xlarge instances each with 1x 1TB EBS volume. That comes out at around 62k for the storage instances.
Compute:
I don't know how important CPU vs Disk IO bandwidth is.
The instance the author is using has 4x 15 cores (60 cores total).
The most I can get with standard EC2 instances is 64 cores, but that has 25Gbit network, and the next down from that is 48 cores.
1x i3en.24xlarge (96x vcpu/768GB RAM/100Gbit NIC/ no EBS)
= $7.9k
There are bare metal instances with more cores/memory and up to 100Gbit networking[1], but I can't find any pricing on them.
All up, I think $51-55k/month using standard instances would probaably do the job.
[0] There are the bare metal instances mentioned in the link below that get up to 28Gbit EBS per instance, but again no details on pricing.
> You can't go with a single instance and a ton of EBS storage, because it caps out at 16TB of disk
I just used the calculator to price out a single instance without issue. Just type in nineteen 16TB EBS volumes (you’d create an LVM volume group for them if launched). I used to have EC2 instances (albeit not by choice, I inherited the bad architecture) with 42TB total of EBS volumes using LVM without issue.
I didn't realise they'd upped the maximum volume size from 1TB to 16TB, so thought the calculator was telling me it was capped at 16x EBS volumes per instance. The new calculator isn't helping things here[1] telling me that I can only assign 16TB to an instance.
So, given that - then the issue becomes is 25Gbit NIC / 19Gbit EBS bandwidth enough IO to at least equal the needs of the task.
On paper the total bandwidth of the author's disk controller was 24Gbit, but that will depend on how the output is spread and whether the EBS limit includes any overheads that aren't present in DAS.
Interestingly, if the requirements are mostly sequential, you can get better performance/$ going with throughput-optimised HDDs rather than gp2 SSD.
Applying striping in either case will ensure you saturate the per-instance EBS bandwidth limits.
I found it, but clicking it caused my computer to run out of available memory.
Edit: on a more serious note, a site[0] that tracks these records says:
> Downloading of digits is no longer available due to the massive bandwidth requirements. Your best bet is to directly contact one of the record holders and see if they still have a copy of the digits.
The compressed representation requires 44 TB of disk space.
Assuming the author has a typical home internet connection with about 5 Mbps upload rate, the transfer would take 2 years longer than it took to actually run the calculation in the first place!
It appears they have, the y-cruncher program he used does verification. Alexander Yee, who wrote the y-cruncher program (which has been used for the previous 4 world record calculations of digits of Pi), has accepted it and posted it on their site, along with screenshots of the output of the program.
One cool note per my reading here - prior record holder Emma Haruka Iwao was working for GCP as a developer advocate, and GCP makes her 31.4 trillion digits available via cloud images. Neat :-)
For those of us with less insane needs, they also exposed an API to grab the specific digits of interest - https://pi.delivery/
It's not irresponsible if the dude needed heat in his home. Far and few people realize that all electronics are heaters that happen to do computation in the process of conversion of electricity to heat. Sure, it's not convective if there aren't fans to disperse the heat.. but it's a beefed up computer so I am sure he had good PC fans.
In any case, heat that is created without convection fans to spread it quickly, will still diffuse into the home albeit at a slower pace. It has to go somewhere..
That's easy to say but pretty much everyone I've talked to, albeit the few oddities, thought that dedicated space heaters were way different than X-Box getting hot, or computer getting hot. Think of the cognitive dissonance to switch to all LED bulbs in a cold climate, because you're told it's more efficient, meanwhile using electric space heaters at the same time... It happens more often than you think, given the politics.
Of course, there are some heating methods that are cheaper than electricity - like natural gas. So one would have to factor that in, along with power plant emissions vs home emissions.
It's a complex entrenchment. It'd be interesting to produce a video of street interviews just to see what the average person thinks about electronics getting hot. It would make for a good case study of the intersection between common sense/basic physics knowledge/and energy/eco politics.
Every time we clean something, we simply dirty something else. Ala Einstein, energy is neither created nor destroyed. Chaos is neither created nor destroyed, simply moved. And to move chaos, requires isotropic heat emission, which releases chaos in itself - hence entropy increasing. Thus, moving chaos, increases the total chaos a little bit. Until there's chaos everywhere and nothing can be moved, ie. heat death of the universe. Prepare for the big crunch, the big pancake deep freeze, the cyclic big bang, conformal cyclic rebirth, or whatever your cosmological spirit holds to be an inkling of inclination. :-)
If anything it got me a lot more curious about this y-cruncher program and all the fun optimizations it must implement.