Libertarian and Poly?

I know I know the title seems like an oxymoron but think about it a second.
The key points of libertarianism is less government involvement in private lives, liberty, freedom. I'm talking real libertarianism, not an establishment political party.
Real libertarians do not want to dictate the actions of others, they do not want to push their beliefs on other people, they do not really care, nor want to know, what goes on in the private bedrooms of people, and surely would not legislate whom you can legally marry, or how many consenting adults could be married. (I am talking the legal form of marriage, not the faith based forms of Matrimony, since a religious institution has different rules)
Many people hear libertarian and think of the establishment conservative beliefs, the ones that are speaking out against same sex marriage and plural relationships. While, yes, we are most closely related to conservatives on the political scale, we take the distance between policy and private lives further and want less government involvement in daily lives.
As a libertarian, I feel the government has no place in telling you who you can and can not love, male, female, or other; who you can and can not marry, again, male, female or other; nor how many people you can commit your life to at one time. If all members of a relationship are consenting adults, why should it matter to whom they are choosing to commit themselves or how many people at once?
Why should the government dictate through policy if the three members of my Triad can commit to each other for life? Why should they say that I wouldn't be able to commit myself to a woman for life?
Most people think that, as a polyamorous person, or a person who leans more towards one side of human sexuality or another, or a person with a proclivity towards kink, they should be a liberal, because it just sounds like a policy structure that would welcome their own sexual preferences. But, going by the definition of liberal, the political side would want to legislate your most private acts. Libertarians, on the other hand, don't really care, and want to keep government out of private lives and bedrooms.
Just some food for thought this rainy Friday night.