For gaming, not even close. I've often had a GPU monitor running, charted bus utilization...and it's barely a few percent no matter what game I'm running, except for a very brief spike during texture and model load-in at the start of a round/level.
Specifically for VM's, something like Looking Glass (shared framebuffer so you don't need a dedicated monitor) needs the extra bandwidth to get the video signal through without tanking performance, especially at higher resolutions and refresh rates.
Aside from that, I agree it's not too big an issue for 95% of computer users, but it's something to keep in mind.
To quantify this: a 4k 60Hz uncompressed video stream needs approximately one lane of PCIe gen4 (though only in one direction, while PCIe has the bandwidth in both directions). So any realistically obtainable resolution and refresh rate will not be a problem for sharing the framebuffer on a GPU with PCIe gen4 x16, but lower-end GPUs often have just x8 and there are even some mobile GPUs with x4 being packaged onto desktop graphics cards now.
If I remember correctly, running a 3080 10GB in PCIe 4.0 yields >5% performance average in gaming benchmarks compared to running it in PCIe 3.0.
You could probably argue that you should just settle and not take the 5%, but when you are buying in at such a price point, you are likely concerned with 5% differences. There are also higher end SKUs such as the 3090Ti where that performance gap is likely larger.
The only thing I'm aware of that will actually saturate a PCIe connection is NVME storage.