ringone198,
Checking Passmark 3D baselines, the eleven highest performing GPU's in z800's are: Radeon ProWX 9100 (230W), GTX 980 Ti, GTX 1070, GTX 1080, GTX 980 Ti, GTX 980 Ti, GTX 1080 Ti, GTX 1070, GTX 980 Ti, GTX Titan Black. Ten of the systems had 3X CPU's, 6 are 56XX series and 5 are 55XX series. The W5590 and X5690 have the majority of the high 3D marks as they have higher clock speeds.
The highest 3D mark using the Xeon X5650 has a GTX 1070, scoring 3D of 8305, the second mark is 8178 from a Titan X, and third is 7963 using a GTX 980 Ti. Those are all dual X5650. The fastest single X5650 system runs a GTX 570 (specs for 550W PSU) for a 3D of 4446. I would suggest staying with GTX 7XX or newer series as they use less power, have more CUDa cores, higher clock speeds, and more modern Direct X and OpenGL.
I don't know the specifcation of the z800 power supply but the PCIe slot provides 75W and the GPU power supply rail is likely to run at least at 11A, as it does in the 825W z620. As 11A x 12V = 132W and 132W + 75W = 207W explains how the z800's above can run these high performance cards. However, if running dual CPU's, it's definitely advisable to change the PSU to the 1125W.
Our forum friend Brian1965 has a z620 overclocked to 4.8GHz and on the 825W power supply, runs a GTX 1080 Ti and a Quadro P2000. The Quadro P2000 is 75W and runs on the PCIe slot power only. I have a z620 overclocked to 4.3GHz and was running a Quadro P2000 + GTX 1070, but at the moment am running only a GTX 1070 Ti (3D = 12132).
As the CPU clock speed will impose some limits on 3D performance, In my view the best cost/performance choice for the z800 would be the GTX 980 Ti, but the GTX 970 is very good as well.
If you need strong GPU performance in 3D, consider upgrading the CPU to higher clock speed. I had very good results in Dell Precisions T5500 (similar to z600 and T3500 (similar to z400) using the Xeon X5680 6-core @ 3.33/ 3.6GHz and the 4-core X5677 @ 3.47 /3.73GHz.
BambiBoomZ