James Madison and Slavery

Born in 1751 as the first child of James Madison Sr. and Nelly Conway Madison, owners of the largest number of enslaved laborers in Orange County, Madison grew up around enslaved workers on his family's Montpelier plantation. Reservations about human bondage, consistent with the liberal opinion of his day, appear in his earliest correspondence. On the eve of the American Revolution (1775–1783), Madison expressed his fear to a former college classmate that Virginia's royal governor, John Murray, fourth earl of Dunmore, would attempt to mobilize the enslaved population against the Patriots. It was, he told William Bradford in a letter dated June 19, 1775, "the only part in which this colony is vulnerable." Later in the war, Madison, as a member of the Continental Congress (1780–1783 and 1786–1788), complained privately when the Virginia General Assembly considered offering each white recruit to state regiments in the Continental Army an enslaved laborer as an enlistment bounty. Madison believed republican ideology would be better served by freeing enslaved persons who enlisted in the Patriot cause.

Scattered criticisms of the South's "peculiar institution" continued to appear in Madison's postwar letters. The marquis de Lafayette, the French aristocrat who had fought on the American side in the Revolution, strongly supported a general emancipation. Madison told Thomas Jefferson, in a letter dated October 17, 1784, that Lafayette's antislavery stance did him "real honor, as it is proof of his humanity." When the lawyer Caleb Wallace solicited Madison's advice on a constitution for the new state of Kentucky, Madison recommended, in a letter to Wallace dated August 23, 1785, that it bar the legislature "from licensing the importation of slaves." In 1785, Madison, as a member of the House of Delegates, helped defeat a bill to repeal a Revolutionary-era law permitting private manumissions. He also received a sobering lesson in the politics of slavery. Methodists and other opponents of human bondage petitioned the assembly to pass a general emancipation bill. None of the lawmakers, including Madison, was prepared to support the proposal, but, in a letter to Jefferson on January 22, 1786, Madison seemed troubled that, while the petitions "were not thrown under the table, they were treated with all the indignity short of it."

Madison had earlier, in a letter dated July 26, 1785, confided to Edmund Randolph, a friend and prominent attorney, his wish "to depend as little as possible on the labour of slaves." He had shown an interest in religion after graduating from the College of New Jersey, now Princeton University, in 1771, and in the mid-1780s he read law with Randolph. Yet he never showed much real interest in either the ministry or the law as a career. He and James Monroe, another rising Virginia politician, combined their resources to buy land in New York's Mohawk River Valley, but neither man could afford to invest enough to produce a substantial return. As a result, he remained dependent upon the enslaved laborers of Montpelier for the majority of his income. And by 1787, if not sooner, Madison's desire to replace the Articles of Confederation, Congress's governing charter, with a new and stronger federal constitution would overshadow his discomfort with slavery.

Madison's service in the Constitutional Convention from May to September 1787 forced him to confront the sectional divisions created by slavery. On May 29, Governor Edmund Randolph, representing the Virginia delegation, presented an outline for a new constitution; Madison had written most of what came to be known as the Virginia Plan. It included a proposal for a bicameral national legislature in which representation was based on each state’s "Quotas of contribution" to the federal treasury, or their "number of free inhabitants, as the one or the other rule may seem best in different cases." The proposed "Quotas of contribution" would depend on a state's wealth, which would encompass its enslaved population, and, as a basis of representation, was not well received by northern states that had either abolished slavery or were moving to do so. Meanwhile, the slave states of South Carolina and Georgia opposed counting only free people.

Madison soon abandoned "Quotas of contribution" and, with the support of Virginia and the other large states, embraced representation based on population alone. The small states objected bitterly, demanding that each state have an equal vote in a unicameral Congress, as was the case under the Articles of Confederation. Madison, in response, argued the small states had nothing to fear from a cabal of large states; he repeatedly asserted that the real division within the nation pitted slave states against free states. The small states were not persuaded, and on July 16, the delegates, to Madison's dismay, took a different approach, approving the Great Compromise, or Connecticut Compromise, in which seats in the House of Representatives would be allocated based on population while granting each state an equal vote in the Senate.

Proportional representation made the question of counting the enslaved population unavoidable. A similar issue had arisen in 1783. Under the Articles of Confederation, Congress requisitioned money from the states based on land values, which proved difficult to determine. Madison had helped negotiate a new revenue plan in which requisitions were based on population, with an enslaved person being counted as three-fifths of a free person, and it was this formula that was adopted in the Constitution of the United States.

Slavery presented still other constitutional problems. Madison had initially believed that Congress should select the nation's chief executive, but in the course of the debates, he came to see the merits of a popular vote, in part because he wanted the president to be independent of Congress. On the other hand, he worried that a popular election would disadvantage the South "on the score of the Negroes," who could not vote. An electoral college, he ultimately concluded, partially answered that complaint and "seemed on the whole to be liable to the fewest objections. "Most of the other delegates agreed.

One issue remained to be decided: the future of the African slave trade, which was viewed as the most brutal aspect of the institution. Eighteenth-century abolitionists widely assumed the abolition of the international slave trade would mean the gradual abolition of slavery itself. Virginia, with a surplus of enslaved workers, outlawed the trade in 1778, and most of the Philadelphia delegates favored an immediate federal ban. South Carolina and Georgia, however, were still expanding and wanted to ensure their planters had access to a continued supply of enslaved laborers. They were, Madison wrote Jefferson shortly after the convention, "inflexible on the point of slaves," even threatening to leave the Union should the slave trade be stopped.

South Carolina and Georgia initially pressured the convention to allow the slave trade to continue until 1800 and then won a reprieve until 1808. Madison thought "so long a term will be more dishonorable to the national character than to say nothing about it in the Constitution." Indeed, Madison and the other delegates consistently avoided making any explicit reference to slavery in the text of the Constitution. Many of the delegates supported a tax on imported enslaved labor to discourage the trade, but Madison expressed qualms about appearing to treat enslaved persons as mere property. The convention, accordingly, included the ten-dollar tax but referred obliquely to the imported enslaved laborers as "such Persons as any of the States now existing shall think proper to admit."

When the Constitution went to the states for ratification, its Federalist supporters and its Anti-Federalist opponents attempted to exploit its ambiguous treatment of slavery. Northern Anti-Federalists criticized the three-fifths compromise and the temporary continuation of the slave trade. Southern Anti-Federalists like Patrick Henry warned their constituents that a more potent national government with a northern majority might abolish slavery altogether. In his Federalist No. 42 essay written to promote ratification of the Constitution, Madison claimed the slave trade compromise "ought to be considered a great point gained in favor of humanity," since the slave trade could have continued indefinitely under the Articles of Confederation. And if Congress could not ban the trade until 1808, he noted that it could immediately impose a tax to discourage the importation of enslaved people.

In Federalist No. 54, Madison defended the three-fifths compromise as best representing enslaved persons’ mixed status as property and persons. White southerners, he argued, could not have been expected to agree to count enslaved persons equally with free persons for the purpose of calculating a state's tax burden, as northerners wanted, and not count them for the purpose of representation. Slave states did not allow enslaved people to vote, he conceded, but laws regulating suffrage varied from state to state; the South could argue that if other nonvoters were counted for purposes of determining representation in the free states, why should enslaved individuals not be counted? Madison also argued that the three-fifths clause offered protection for property rights, somewhat comparable to New York's property requirement for voting. "Government is instituted no less for protection of the property of individuals, than of the persons of individuals," he wrote.

Madison faced a different challenge as a Federalist delegate to the convention called to consider ratification of the Constitution in Virginia in June 1788. George Mason, who led Anti-Federalist opposition to the Constitution, complained that it neither abolished the detested African slave trade nor secured the enslaved laborers that Virginians already owned. Madison reassured the convention that the state's ban on the slave trade would remain in effect. More importantly, South Carolina and Georgia would not have accepted the Constitution without the compromise. "Great as the evil is, a dismemberment of the Union would be worse," he said. At the same time, Madison pointed worried enslavers to the Constitution's vague fugitive slave clause, which provided that a "person held to Service or Labour in one State" who fled to another state "shall be delivered up on Claim of the Party to whom such Service or Labour may be due." He called it "better security than any that now exists" because, he said, "At present, if any slave elopes to any of those states where slaves are free, he becomes emancipated by their law." Madison also said he believed that Congress would never be able to muster a majority to abolish slavery in the states where it already existed. After Federalists pledged to support amendments demanded by the Anti-Federalists, the convention on June 25, 1788, voted 89 to 79 to ratify the Constitution.

In the autumn of 1789, Madison met William Thornton, a British doctor and abolitionist who supported a plan to colonize Sierra Leone as a refuge for freed British enslaved people. The encounter apparently motivated Madison to develop his own colonization plan, which he circulated privately. Colonization, he wrote, could encourage voluntary manumissions and offered "the best hope yet presented of putting an end to the slavery in which not less than 600,000 unhappy negroes are now involved." He said that many enslavers hesitated to liberate their enslaved workers because of "the ill effects suffered from freedmen who retain the vices and habits of slaves," and that enslaved persons who were freed could never be incorporated into society because of "prejudices of the Whites." Madison believed that "some external receptacle" was needed if enslaved Blacks were to be freed. He eliminated one obvious option: the American West. If formerly enslaved people were settled along the frontier, he said, they would soon clash with white pioneers. If relocated farther west, they would be massacred, he assumed, by Native Americans. Africa, he concluded, afforded the most promising site, and a successful colony there might stimulate manumissions and eventually lead slave-state governments "to forward the abolition of slavery in America." Freeing and transporting hundreds of thousands of formerly enslaved people across the Atlantic Ocean presented innumerable practical, legal, and moral problems and never won widespread public support, but Madison and other moderate critics of slavery clung to the chimera of colonization for decades.

In February 1789, Virginia voters elected Madison to the first House of Representatives to meet under the new Constitution. Hoping to exploit the uncertainty about the constitutional status of slavery, Quakers and other critics of slavery petitioned Congress to pass whatever antislavery measures it could lawfully adopt. Representatives from South Carolina and Georgia were outraged when the petitions were taken up in February 1790. Madison considered the protests counterproductive and persuaded the House to refer the petitions to a committee, expecting it to conclude that Congress could not prohibit the slave trade before 1808 or tamper with slavery in an existing state. He remained open to congressional regulation of the Middle Passage and to restrictions on slavery in the West. In 1789, Congress had reaffirmed the Northwest Ordinance of 1787, which banned slavery in territory north of the Ohio River.

The committee's report, submitted in March 1790, generally conformed to Madison's expectations, but it did criticize slavery, and one oddly written provision seemed to imply that Congress could free enslaved individuals born after 1808, which even the states of the Upper South would violently oppose. Madison brokered a compromise that deleted the controversial language and denied Congress's jurisdiction over slavery within a state, while recognizing congressional authority to bar Americans from selling enslaved persons to foreigners and to promote humane treatment of enslaved persons in transit across the Atlantic. The compromise favored slavery; for the time being, Congress would read the Constitution literally and avoid the issue. The amended report did not expressly acknowledge federal authority over slavery in the western territories, and Madison would never again support restrictions on its expansion internally. He also refused a later request to introduce an antislavery petition to Congress or to support a gradual emancipation bill in the Virginia General Assembly.

Slavery was not a major issue for Madison during his tenure as secretary of state from 1801 until 1809 or during his two terms as president from 1809 to 1817, although Congress did, as expected, ban the African slave trade in 1808. In his last annual message to Congress, Madison called for stricter laws against smuggling enslaved persons into the United States.

Madison addressed slavery more often in his retirement but usually in private correspondence. He told Robert Walsh in a letter dated March 2, 1819, that the treatment of those who were enslaved had improved "in every respect" since the American Revolution, which had unleased a new "sensibility to human rights" and broken up large estates through the repeal of primogeniture and entail. Madison also elaborated on his colonization plan, adding the condition that enslavers should be compensated for emancipating their enslaved workers. Madison recognized that private charity would be inadequate to the task and proposed that public land be sold to finance the purchase of enslaved persons and their resettlement. He thought a constitutional amendment would be needed to authorize such an unprecedented undertaking and seemed blithely optimistic about its prospects. Southerners, he assumed, wanted to be free of their enslaved laborers, and northerners would surely recognize that enslaved property "could not constitutionally be taken away without just compensation," he wrote to Robert J. Evans on June 15, 1819.

Missouri's application for admission into the Union as a slave state in 1819 sparked the most bitter debate over slavery of Madison's post-presidential years. Madison took what was for all practical purposes a pro-slavery position. He denied that the Constitution's slave trade clause had ever been intended to empower Congress to regulate slavery internally or that the framers had intended to allow discrimination against new states in the matter of slavery. He denied, less plausibly, that Congress's power to govern federal territories included the ability to ban slavery in them. He argued that the Confederation Congress had no authority to outlaw slavery in the Northwest Territory and had done so only because it had no other way to discourage the African slave trade. Madison now embraced the theory of diffusion: that spreading slavery over a larger area would reduce the concentration of enslaved labor in any one state and expedite a gradual emancipation.

Under the Missouri Compromise, Congress admitted Missouri as a slave state in tandem with Maine, a new free state. Madison told James Monroe in a letter dated February 10, 1820, that he considered the deal to be "a very doubtful policy" but grudgingly recognized the need for a peaceful settlement. Privately he accused the declining Federalist Party of exploiting antislavery sentiment in an attempt to create a new party system organized along sectional lines, a scheme he feared could split the Union.

Madison continued to criticize slavery but refused to act decisively against it. He treated successful free Blacks respectfully, and he never suggested Blacks were innately inferior to whites, but he cited the woes of less prosperous formerly enslaved persons as an argument against immediate emancipation. He blamed their predicament on discrimination and on the failure of slavery to prepare them for freedom. Gentle by temperament, Madison acquired a reputation as a relatively benevolent enslaver, and he generally avoided selling the people he owned. However, faced with mounting financial pressure as farming income from Montpelier declined, he did sell sixteen enslaved persons to William Taylor, a cousin in Louisiana who owned a sugar and a cotton plantation.

Madison freed none of his approximately 100 enslaved workers in his will, but he requested that Dolley Madison, his widow, not sell enslaved people, except for misbehavior, without their consent. Some evidence suggests that Madison expected his wife to free Montpelier's enslaved population at her death. They supposedly kept their understanding secret to avoid giving any person who was enslaved an incentive to kill her. She may have had fewer reservations about slavery than did Madison, however, and under increasing financial pressure, she sold some enslaved persons and kept the rest in bondage. Like many of his fellow founders, Madison had qualms about slavery, but he never found a way to bring about the painless and peaceful emancipation he sought.