Can you hear any hissing by the booster?. If you can then the diaphragm is probably torn. Not a big tear or your engine probably wouldn't idle at all.
In both cases the engine isn't making enough vacuum to operate the booster properly, not because the engine is out of tune but the booster is bad most likely.
If you have a radical cam installed that could also give you the symptoms described, or possibly an idle speed too slow.
Roger.
