I am not a vicar
Sometimes people call me a vicar, usually non-churched friends while we're sitting in a pub. I'm not exactly offended by this, the worst thing to do is judge non-churched folks at using the wrong terms, giving the impression that religious people are hyper-sensitive and easily offended so you need to walk on eggshells around them. But nevertheless I would say, no, I'm not a vicar.
The basic way of explaining this is to say vicar comes from "vicarious" - doing something for someone else, and I don't do anything for anyone else. I don't do your religion for you, you have to do it for yourself.
But I suppose the basic reason why I don't wear a clerical collar is because I'm not a vicar, this is also the reason I do not use the title "reverend." My function may seem similar, but I think if we're consistent with our theology we have to admit that something quite different is going on in Unitarianism.
We have more in common with the Quakers (and even in some senses the Baptists) than the Church of England. We do not have an ordained order of clergy with symbolic function. We may in fact have more in common with certain types of American Quakers who have ministers and with Reform Rabbis (although I don't really have enough knowledge of either movement to back that up).
As much as I may have very friendly relations with my neighbouring Anglican church and clergy (and I do) I do have to affirm that we are not the same, and that I am committed to doing religion in a quite different way. Unitarianism belongs not to Protestantism but to the radical reformation and part of what that means is a strong commitment to equality, to the priesthood of all.
The idea of a "holy person" who does religion on your behalf, who effectively is holy so you don't have to be is a bad idea. The practice of a lot of Christian denominations, and the general approach to religion in this culture around "vicars" encourages this thinking. So "vicars" are praying, not swearing, not drinking, not having sex on our behalf.
Yet this way of thinking is condemned by all the great spiritual teachers: Jesus, Buddha, Mohammad constantly tried to turn attention away from themselves towards their teaching (why do you think there's a taboo in Islam against depicting the prophet Mohammad? Because Mohammad didn't want people to start worshipping him, turning their attention away from their own relationship with God).
This approach disempowers the laity. It cam reduce your membership of a spiritual community to passive pew-sitting.
A Unitarian minister is, theologically, more of a teacher than a priest. A Unitarian minister's job is to provide guidance in helping people go deeper in their faith journey and their own ministry. Because even though we each do have our own direct relationship with the Holy and our own ministry to give, we still need guidance, resources, knowledge to help us, and the minister provides this. A Unitarian minister is different, only in the sense of their education, training and knowledge of spiritual resources, which enables them to empower others.
In short I believe giving the impression that Unitarian ministers are "basically vicars" gives the wrong impression of the type of religion Unitarianism is. Unitarian ministers are not vicars.