This is something I've been thinking about more and more as self driving technology is becoming more and more mainstream. The things that I find most troubling are the following.
What will happen to the auto body repair industry? If self driving cars don't wreck, the collision industry is doomed. This is a multi-billion dollar industry that employs a lot of people. Also what will happen to truck driving, if trucks can drive themselves?- A lot more people without jobs. While it may be cool to have the technology, I think it may be economically irresponsible to implement it.
I dont like the idea of self driving cars, not just because I'm a car guy, but I don't like someone/something else controlling where I'm going. This goes back to you mentioning a nanny state. The government/vehicle manufacturers, etc. will be able to keep you from going to certain places(remote areas, restricted areas, etc.) if the cars are the ones doing the driving. That's not to say that they actually will keep you from going to such places, but they will have that capability.
I am also a big opponent to Tesla. I believe that Tesla has way to much control over the people who own Tesla cars. If you own a Tesla, they can remotely activate and deactivate your car. I heard a story about a guy who bought a wrecked Tesla, repaired it, and then Tesla refused to reactivate the car after it had been wrecked, so it was essentially a paperweight. I find that disturbing.
All this said, the auto enthusiast community is very strong and there are always going to be people who want to drive their own cars, myself being one of them. I think any movement to try and outlaw human-operated vehicles would be met with great resistance from a percentage of people and would be unlikely to pass. They may greatly increase the cost of conventional vehicle ownership, but I don't think it will be possible to outlaw it.
As long as I'm here they aren't getting my cars, my guns or my Diet Coke.