Moral of the story. No matter how sure you are that you don't have any boost leaks, check again!
My truck had been getting worse and worse as I drive it daily, we are in the process of working on a house we are buying so the truck has been neglected. I started getting a bunch of oil coming out around my valve covers. I finally had time to work on the truck and pulled the intercooler to replace the valve cover gaskets and low and behold, this is what I found.
The oil wasn't from the valvecovers, but from the turbos overspeeding and spitting oil! Welded it back up and rebuilt both turbos.
Boost is back up to 85lbs and it feels great! Feels better than it ever has before, or else I've just sincerely forgotten how powerful the truck used to feel. Ill know when I get to take it to the track.
So this tells me a couple things that makes so much sense now. I think I have had some hairline cracks in these runners for a while now, like ever since I put the VGT on. I remember back when I was dialing it in, I was getting max about 75lbs of boost and all of sudden all boost would drop off and I would get a big WOOSH like the turbo surged. I think that I didn't notice the crack and it was getting pushed open at higher boost levels and gradually got worse and worse.