I value many contributions libertarianism makes to challenging power. But here's why I no longer associate with it
Honestly, the biggest problem with any ideology is that it only creates a haze over reality that is tinted towards your preconceived notions and desires. This haze will not guide you accurately throughout the ocean of life, the universe, and everything. In fact, it may run you into shoals and rocks that you could have avoided, had you been trying to look at reality for how it works and what is present in it.
A society is a complex adaptive system, just like an economy, environment, and a government. The goal of a government should be to examine the interrelated nature of these facets of the world, in order to understand how to make policies that produce optimal well being and the greatest chances for sustainable survival within a society. Otherwise, it's just a load of philosophical crud that comes out of the mouth of people who have no idea and no inclination to have an idea about how the world actually is and works. This then makes them a danger to themselves and a danger to others, because they are not only not seeing, but refusing to see what is present and how things connect to one another, thus, increasing the odds that they'll cause harm in a society to either individual members and/or to the collective whole. Furthermore, due to the persistence of the ideological haze, it is more likely that they'll persist in their incorrect assumptions, beliefs, and poorly prioritized desires, thus increasing the odds of entrenched dysfunction both within the government and within the whole of the society.
We may not have all of the facts and relations in yet. But we should be able to tell now, that philosophical arguments are a poor substitute for scientific research and empirical study.
Think about it.