James Madison said this quote

Liberty may be endangered by the abuses of liberty as well as by the abuses of power.