As the main architect of the Constitution and a slave master of great wealth himself, Madison thought long and hard about how to protect minority rights in a government based on sovereignty of the people, a people then understood to be white men of property.