Ethernet Cards
Under Mac OS X, my G3 is limited to it’s on-board 10 Base-T controller. This is all right for serving things to the outside world because my cable modem’s throughput is much lower than that of a 10 Base-T link. However, I also use my server as my chief BitTorrent client. Thus, I need to transfer largeish files from my server to my laptop with some regularity. By replacing the 10 Base-T connection with a 100 Base-T would be a significant improvement.
The problem is that Mac OS X drivers for Ethernet PCI cards are rather thin on the ground, and there is no happy intersection of that set with the single 10/100 Base-T card I own. Linux, on the other hand fully supports almost every piece of ethernet hardware I own (the sole exception is a SCSI-based 10 Base-T adapter, yes, SCSI-based). SO, by switching to Linux I am also making it considerably easier to move data back and forth between my server and my laptop.
Final Result: Linux offers faster Networking for the server as a side benefit.

Hint: The Farallon Fast Ether TX 10/100 PCI card is recognized in OS X without installing drivers. It uses the common DEC 214xx chip.