Multiple "PCIe Express Error Counters" for the RTX 5070 Ti

[Tatsuna]

Hello,
First of all I want to thank you for developing the tools, I've been using it for months and it's great. Today I updated to the latest beta branch and I noticed the new "PCIe Express Error Counters" that showed multiple errors. They were usually marked in red but by googling and searching for information on the forum, I've not really gotten a good answer at what exactly they mean and what causes them.
My current configuration, as the debug probably will already notify, is as follows:
MOBO: Asrock B650 Pro RS
CPU: Ryzen 7800X3D
GPU: RTX 5070 Ti
RAM: XPOWER Zenith Gaming 64 GB (2 x 32 GB) DDR5-6000 CL30
Storage: WD_BLACK 2TB SN770 NVME
PSU: NZXT C850

I'll attach the debug file generated after a 20+ minutes session while heavy gaming, in which I have not had any serious issue but I mean, I guess I shouldn't take lightly such serious issues when they appear.

I should also clarify that my 5070 Ti is negotiating PCIe Gen 5 x16 speed with 7800X3D own controller, even though the board does not officially support it. I might guess this is the issue, but I have really tested the PCIe speed and stability thoroughly and from benchmarking and gaming I've only attained benefits from using PCIe Gen 5 x16 speed, even if minor ones. Before trying to test if the errors arise even when using PCIe Gen 4 x16 speed, I'd like to be sure that the debug points to the GPU as the culprit for all the errors.

 

[Martin]

This is the best explanation we have so far:

S

Post in thread 'NVIDIA PCI Express Error Counters'

Sunday at 3:17 AM

Doing some testing with my 4090, it seems that Recovery Count is tied to power management as it increases every time power states increase or decrease, probably due to the PCI-E link needing to be reset as it changes link speeds. It also seems that Bad TLP Count and NAKs Sent are tied together as they always match, but I haven't been able to see any events that match. Hopefully more documentations will be forth coming.

• sicklown

• 

 

[Tatsuna]

On my end though the issue seems exacerbated by PCIe Gen 5 x16.
Came back from work and tested again. These are the results with PCIe Gen 4 x16:



Results with PCIe Gen 5 x16:



Again, this might be my fault as I'm using the CPU's own controller to negotiate PCIe Gen 5 speed, as the motherboard doesn't officially supprot it (though it lets me select it in the BIOS...), but (and now we'll talk about the methodology I used) I can assure that I get better results with PCIe Gen 5 x16 than I do with PCIe Gen 4 x16: I ran two identical benchmarks, heavy on the memory, and with PCIe Gen 4 speed I get worse 1% lows and some. The benchmark is a scene with lots of traversal and texture streaming and it lasts approximately 20 minutes.
Another weird thing I've noticed: HWINFO seems to catch up on PCIe Gen 5 issues from the get go, as a couple of seconds after rebooting and opening up the tool, it already showed a red mark, even though there was no error whatsoever as I had just rebooted my PC. So I don't know what to think. It may be a PCIe Gen 5 quirk when it comes to sensors, it may be that my motherboard actually gets in the way of negotiations between CPU and GPU, it may be that I get better performance because as it stands, PCIe Gen 5 is fast enough to perform better AND to repair its own errors at the same time. I don't know. If you have any technical insight, I'll be glad to hear you. Thank you again.

 

[Tatsuna]

Ok, this thing has my undivided attention now.
I've tried running the NVIDIA-smi built-in PCIe error checker, and that came out perfectly:



That's what HWINFO had to say though:


Now, it might be that NVIDIA-smi test provides results AFTER the corruption is solved? Or it may well be a mismatch between what HWINFO and NVIDIA-smi see?
I'm also performing (once again) an OCCT memory and GPU stress test just to check if it catches something about it, but I don't think so. I remember when I applied Gen 5 x16 speed from the BIOS I literally stress tested the GPU for days, and not even once an error occurred. Not one that OCCT could catch, at least.
Don't know if any of this could help out, sorry for the double posting.

 

[Martin]

NVIDIA-smi doesn't seem to list detailed PEX errors, only a single "pci errs" value.
HWiNFO and NVIDIA-smi should be using the same NVIDIA-provided interface to report errors - NVML. So maybe NVIDIA-smi has a switch to report more detailed values?