Best of both worlds is to build a 455 and rev it like a small block.
My brother has a 403 in his '81. It was the second build that I had done, my 455 being the first so naturally, his was always faster than my 455.
However the 455 ALWAYS had more low end than his 403. His 403 with the RPM intake doesn't think twice about revving to 6000+.
Now my latest 455 is a culmination of the two. The cubes of the 455 but the rev happiness of a much smaller engine. Lightweight pistons, aluminum rods, big cam, etc.
But if you want a REALLY fast Olds, you need to look at the 350DX diesel block. The amount of money you would have to spend on the 455 or even more so the 403 block to get it to hold any kind of power north of 600HP would be easier to just buy a DX and stroke it, spray it, turbo/supercharge it, etc.
As far as fitment, yes the 403 is pretty much a direct bolt in. Anything that bolts to a 307 will bolt to it.
The 455 can be a little bit tougher but it's real issue is accessories and possibly valve cover clearance issues with the A/C evaporator box depending on how shot your body bushings are.
No need to buy expensive headers though for a mild mid-high 12 second 455. Stock W/Z exhaust manifolds work perfectly and are easier to install and live with.