Putting ethics and legality aside (if you can, and you shouldn't), it's a marvellous technical achievement to be able to take a beam splitter and analyse all the traffic. I assume they must do a lot of early filtering in hardware to select traffic of interest based on IP headers and then some deeper inspection, and then gradually move up to software on the filtered traffic. Do we think they start with dedicated ASICs, perhaps add FPGAs for second line filtering which would give them a degree of adaptability, then software for the rest? What were internet backbone speeds like in the early 2000s?
I work in an environment where we do super high speed deep packet inspection for commercial customers. We use FPGAs. If all you want to do is split and exfiltrate (send to other servers for analysis) the packets as they come over the wire and maybe send them into different buckets then an FPGA is more than enough. You only need to be as fast as the slower of the two devices you're between.
Oddly enough, the "fiber optic tap" shown in the article is an extremely mundane component that you can buy out of a catalog. There are two of them in the box, called fiber directional couplers. I used one when I just wanted to combine two colors of lasers for an R&D project.
Needless to say, the stuff that those fibers would go into and out of in that particular application, is way beyond my comprehension.
Not trying to argue for the sake of it here, but _is_ that a huge achievement? I assumed this was a simple device that splits the fiber beam, sending one part on it's way to AT&T and the other to some NSA database for further processing.
Well at the speed the data comes in, you don't have time/space to store it all on disk (and just filling up also doesn't seem to beneficial for them) and also can't just double all traffic for sending it to the NSA
So they have to filter a lot of data for interesting bits really fast before storing / sending it anywhere else
