James Whistler said this quote

The rare few, who, early in life have rid themselves of the friendship of the many.