Which Vin 9 cam are you using? Which Vin Y 307 do you have? If it's the original 307 for the '86, then you have the roller cam in there. You'll need to replace it with the Vin 9 roller cam and not the earlier flat tappet cam.
You also can't swap older cam/lifters into the later block due to the fact that the later roller engines had a larger diameter lifter.
To be honest, your not going to see a huge improvement in performance doing the cam swap. Maybe 30HP if your lucky. The stock roller 307 Vin Y engine was rated at 140hp, the Vin 9 roller was rated at 170HP.
It'd be smarter to find a 350 Olds and drop it in place of the 307.
And yes, you'll have to pull the lifters. That's the point in removing the intake, so that you can pull the pushrods and lifters out of the engine.