I would make sure and have the case itself along with the overdrive case for cracks. My 3rd NV4500 was cracked both front and back but still held together until the 3rd gear let go for the second time. Also check your transfer case and all your u-joints. Could be possible that your driveshaft is out of round as well. Most times you will feel it, but not always. This resonance is the only thing that I could say may cause this. You make more power in 5th than any other gear on the road, (generally speaking), and this all happens to be taking place in the weakest gear you have. You could possibly talk to someone about having a billet 5th gear made or possibly even have your countershaft done. I don't know who would do this, but have heard rumors of this being done.
BTW - I'm on NV4500 #5 in my truck. Broken 6 total. Best of luck with your venture. If it is not going to go back on the street and will stay on the track, I can probably hook you up with someone who will make it near bullet proof, just not streetable or cheap.