And that my friend is the main reason I haven't wired it inline with the brake switch! I actually have that fear in the event of a panic stop it not building vacuum quick enough.
However, i am contemplating on putting a resevoir can somewhere so it can store the vacuum. I had one years back thinking it would fix the issue the OP is having, but it didn't do squat! I've gotta buddy with the older style electric pump and he has a can inline for extra measure that way there is a little amount stored before te pump activates.
Good point brought up...hope the OP will figure out a solution because i recall the low vacuum crappy brakes via the lopey cam and it's scary sh*t to hit the brakes and nothing happen!