MikeArkus,
I'm wondering though if the system is shutting down based on thermal load. Have you run HWMonitor or similar to see what the CPU(s) temperatures are doing?
Another possiblity is driver conflict and running a driver cleaner and reinstalling the driver might be a first action.
In the list of GPU details:
https://www.studio1productions.com/Articles/NVidia-GPU-Chart.htm
The GTX 285 needs a 550W PSU and the GTX 1080Ti needs 600W. As GPU's needed more power when the z800 was designed than is usual today, I would be surprised if it were a power supply problem in terms of total capacity. However, if you determine the PCIe power is not uffiecint, and the GTX 1080 Ti uses the typical 2X 8-pin, consider trying a 2 into 1 power adapter; the two standard PCIe power connectors into one 8-pin, and run the other 8-pin off the Molex 4-pin. I'm currently test driving a GTX 1070 Ti in a z620 using a 2X 6-pin to 1X 8-pin connector rather than try a 1X 6-pin to 1X 8-pin.
I would note that the 3D performance of the GTX 1080 Ti will be reduced with a 3GHz CPU, but compute power should be strong. In Passmark benchmarks, there are 865 z800's tested and the highest 3D score for GTX 1080Ti is 9202, where the average for GTX 1080 Ti = 14054. The highest score for a z800 with X5675 is 8353 using a GTX 1080 (average = 12309). Even my overclocked Xeon E5-1680 v2 (8C@ 4.3GHz) with the GTX 1070Ti makes a slightly below average 3D mark of 12132 (average =12168) whereas the i7-8700 5GHz hot rods can make 15000+. In keeping the GTX 1080 Ti, consider running one or two X5687 4C@3.6 /3.86GHz (under $50 these days) to improve the single-thread performance. If the current CPU is an X5675, the single thread = 1402, whereas the X5687 is 1570. If more cores are needed, the X5690 6C@ 3.47/3.73GHz (under $100) is a good one as well.
BambiBoomZ