Interesting. I think that's the first time I've heard of that being a problem but fingers-crossed you've found the culprit.
Thanks for letting us know, it's good to close the loop on these things.
Good luck.
It is certainly possible that it is graphics related. Graphics cards take quite a bit of power and any resistance on the ground connection between the graphics card and the power supply could cause problems if the USB ground uses the same track on the mainboard as the graphics card.
It may be worth checking that all the connectors in your computer are clean and tight. It may also be worth seeing if there are any USB connectors (like front panel connectors) that come from a different place on the mainboard.