This is looking like a good place to use a real pinion depth setting tool. It mounts into the carrier bearing caps and measures the distance to the face of the pinion gear. I have done quite a few and always found that once measured the needed pinion shim was within one or two thousandths of what the factory had for the used pinion gear I was installing. I quoted this from an email Bill sent to me- " I am using all new bearings, a used set of gears, (3:73) and a used but checked and verified Auburn 28 spline posi. I used the pinion shim from the gear set that I got, (OE pinion shim from the rear differential was .038 and the one with the gears was .034, hence the first question) and set pinion depth and pinion bearing preload to 12-15 in lbs. " Since neither shim is getting the tooth pattern I would use a depth setting tool to see what is going on.