Nope, shouldn't have to. I've had my factory piece off recently to replace the seal and I only removed the clip, then the shifter rod. After that I took off the nut holding the piece the rod fit inside. They sometimes stick and need a little persuasion due to being by exhaust and collecting dirt, road grime, etc.. I would try gently wiggling/ prying around the perimeter to work it loose.