Robert E. Lee and Slavery

Lee was born in 1807, into two of Virginia's most prominent families. His father, Henry "Light-Horse Harry" Lee, served as a cavalry officer in the American Revolution (1775–1783), a governor of Virginia (1791–1794), and a member of the House of Representatives (1799–1801), while his mother, Ann Hill Carter Lee, was the great-granddaughter of colonial-era Virginia's most prominent slaveholder, Robert "King" Carter. Lee spent his early childhood at Stratford Hall, the family plantation on the Northern Neck, surrounded by more than thirty enslaved African Americans. Even after various financial setbacks and a move to diminished quarters in Alexandria, the family still retained slaves, including at least six at the time of Ann Carter Lee's death in 1829. There is no record of Robert E. Lee owning slaves prior to that year, which coincided with his graduation from the U.S. Military Academy at West Point, New York. Ann Carter Lee itemized the slaves she bequeathed to her daughter, Ann Kinloch Lee, but the only designation of property to her youngest son was a vague division of "the remainder of my estate" among Robert and his two older brothers, Charles Carter Lee and Sidney Smith Lee.

This "remainder," however, may have included other slaves, since a letter written by Lee to his brother Charles Carter Lee, on February 24, 1835, mentions "Mrs. Sally Diggs" and "Mrs Nancy Ruffin & her three illegitimate pledges," who are "are all of the race in my poss[ession]." Lee may well have owned one other slave as a result of his mother's estate, a man known only as Nat (or "Nate"), who accompanied Lee to his first posting as an engineering officer on Cockspur Island, Georgia, at the mouth of the Savannah River. But Nat was evidently both elderly and ill—"very weak & his cough is still bad," Lee wrote to his brother Charles on January 4, 1831—and died soon after.

But even if Lee owned few slaves in his own name, he became part of a large slaveholding household when he married Mary Anna Randolph Custis in 1831. Mary Custis Lee was the only surviving child of George Washington Parke Custis, the grandson of Martha Washington and the owner of three major plantations—Arlington, overlooking the Potomac River in the District of Columbia, and Romancocke on the Pamunkey River and White House on the York River—and 198 slaves by the 1850s. Thereafter, Lee had little need to acquire slaves himself, and the family's summer-vacation peregrinations to visit various relatives included, according to one 1839 account, "a squad of children, Negroes, horses, and dogs." On the other hand, the various postings to which Lee was ordered after his marriage did not accommodate large numbers of slaves, especially in the North. In 1840 Lee was stationed in Saint Louis, where he was in charge of a multiyear project to dredge the Mississippi River ship channel. That year the federal census lists him as owning twenty-two slaves, but it is more likely that these were enslaved laborers hired by the U.S. Army Corps of Engineers for the dredging project. In a letter to his wife, dated April 18, 1841, Lee noted that his quarters at Fort Hamilton in New York harbor offered "poor encouragement about Servants," since "every one seems to attend to their own matters." In 1850, when Lee was assigned to Baltimore to begin the construction of Fort Carroll, a census taker only itemized three "mulattoes" as household slaves, all of them drawn from the Arlington slave population.

Despite the family's extensive investment in slaves, the Custises were not entirely comfortable with owning slaves. George Washington Parke Custis's step-grandfather, George Washington, had manumitted, or freed, through his will the slaves owned in his own name, and like many elite Virginia slaveholders, the Custises took the opportunity of slavery's steady drain toward the southwestern states to profess their distaste for the "peculiar institution," although without necessarily doing anything about it. The elder Custis described slavery as a "vulture" that gnawed at the "vitals" of southern society; Custis's brother-in-law, William Henry Fitzhugh, had scheduled the postmortem manumission of his own slaves "after the year 1850." Mary Custis Lee herself operated a small Sunday school for enslaved children at Arlington. While it was legal under the laws of the District of Columbia, the classes became suspect under Virginia law once Arlington was retroceded to Virginia in 1847. And, like his Custis in-laws, Lee supported the American Society for Colonizing the Free People of Color of the United States, founded late in 1816 and popularly known as the American Colonization Society. In 1855 he purchased a life membership for his wife.

With his connection to slave owning being mostly tangential rather than direct, Lee preferred to treat the enslaved people around him as invisible. In an 1832 letter to his brother Charles, written during his first posting in Georgia, he noticed only that "Blacks in this part of the Country are mostly Labourers, & Mechanics white." Twenty-five years later he was posted in Texas as lieutenant colonel of the 2nd Cavalry. In a letter to his wife, dated March 28, 1857, he remarked obliquely on "our troubles" in maintaining "servants" at Camp Cooper. Before the 1850s, his only comments on the larger controversies roiling the nation about slavery were brief and disapproving.

It was not until the mid-1850s that Lee offered his first extended commentary on slavery. After the Mexican War (1846–1848) brought to the United States a substantial amount of new territory known as the Mexican Cession, Congress began debating the possibility of legalizing slavery there. The controversy was abated by the Compromise of 1850, which permitted slavery in the Cession under the rule of "popular sovereignty," which allowed the settlers of a given territory to decide for themselves whether to allow slavery. But peace lasted only until 1854, when the Kansas-Nebraska Act attempted to apply the "popular sovereignty" rule to the other western territories acquired earlier by the Louisiana Purchase. The Whig Party dissolved under the weight of the controversy. A new political party, the Republican, emerged in adamant opposition to any expansion of slavery into the territories and in 1856 nominated its first presidential candidate, John C. Frémont, of California.

From his post in Texas, Lee wrote his wife on November 19, 1856, that he was "much in the dark as to what is going on in the outer world." But he was apprehensive that there was "no hope" of electing the last of the Whig candidates, Millard Fillmore, in order to promote national compromise. He was certain that Frémont could not be elected, and predicted that a victory by the Democrat, James Buchanan, would best guarantee "the Union and Constitution." His guesses were confirmed the next month. He wrote his wife on December 27 that "full files of papers" had arrived "from New Orleans," along with news of outgoing President Franklin Pierce's final State of the Union Address. In the same letter he wrote at length to condemn "the systematic & progressive efforts of certain people of the North, to interfere with & change the domestic institutions of the South." The driving force behind the movement to restrict the legalization of slavery in the territories was a movement to abolish slavery entirely—not just in the territories, but in the southern states as well. This, Lee believed "can only be accomplished by them through the agency of a civil and servile war."

It was, however, the violence of the abolitionist solution that alarmed Lee, not a love for slavery itself. "In this enlightened age, there are few, I believe, but what will acknowledge, that slavery as an institution, is a moral & political evil in any country," he wrote his wife. The burden of the evil actually fell more to "the white than to the black race," since "the blacks are immeasurably better off here then in Africa, morally, socially & physically," and their "painful discipline" as slaves will "prepare & lead them to better things." What he dreaded was the disruption caused by a sudden and total emancipation of the slaves. It will accomplish nothing, Lee argued, for the abolitionist to "create angry feelings in the master […] if he means well to the slave," since the abolitionist has no more ground for interfering in the legal relations of masters and slaves than in any other "kind of interference with our neighbors when we disapprove their conduct." The "final abolition of human slavery is onward, & we give it the aid of our prayers & all justifiable means in our power," he wrote, but it is a process that cannot be rushed, and "will sooner result from the mild & melting influence of Christianity, than the storms & tempests of fiery controversy."

Events did not permit Lee the luxury of dealing with slavery in such detached terms. His father-in-law, George Washington Parke Custis, died in October 1857, having designated Lee as the executor of his estate. Taking a leave of absence from his regiment, Lee found both the will and the estate a tangled mess, not the least because Custis had stipulated that his slaves were "to be emancipated by my executors in such manner as to my executors may seem most expedient and proper." He further stipulated for "the said emancipation to be accomplished in not exceeding five years from the time of my decease." To accomplish that, Lee had to satisfy Custis's substantial debts and fund several benefactions to Lee's own children, which could only be created by overhauling the management of the Custis estates and, according to an anonymous letter critical of Lee and published in the New-York Tribune on June 24, 1859, keeping the slaves "harder at work than ever," with "no time given them" for "making a little now and then for themselves, as they were allowed to do during Mr. Custis's life." The slave population, especially at Arlington, balked at Lee's new discipline, especially since a number were convinced that Custis had provided in his will for their immediate freedom. In a letter to A. E. S. Keese, dated April 28, 1858, Lee described how three of the Arlington slaves had "refused to obey my orders, & said that they were as free as I was," and attempted to run away. They were arrested in Maryland, he later told his son George Washington Custis Lee, "making their way to Pennsylvania," and Lee, at least according to a newspaper report, ordered them whipped and hired out. These difficulties required Lee to apply to the Alexandria County circuit court for an order confirming his authority to maintain the Custis slaves in bondage through the five-year term, until payment of the legacies and debts had been finished.

Emancipation in another form struck close to Lee in October 1859, with the abolitionist John Brown's raid on the federal arsenal at Harpers Ferry. Although Lee was still on leave, he was immediately ordered by the secretary of war, John B. Floyd, to take command of a detachment of ninety U.S. Marines and suppress Brown's attempt to provoke a slave uprising. Lee was able to capture Brown with surprisingly little resistance, and, in his report, dismissed the raid as "the plan […] of a fanatic or a madman that could only end in failure." Lee remained on duty at Harpers Ferry until December, but his comments on Brown's trial and subsequent hanging were derisive, writing to his nephew Henry Lee: "'Poor fly he done buzz' as the crazy man said." Nevertheless, the Brown raid terrified the white population of the slave states, and Lee was invited by Virginia state authorities to consult with the General Assembly's military committee for the organization and arming of the state militia—an invitation he declined.

Lee returned to duty in Texas in February 1860, just as the furor over the 1860 presidential election was mounting. The Republican Party nominated an avowed antislavery candidate, Abraham Lincoln, of Illinois; the Democratic Party split into northern and southern factions, nominating Stephen A. Douglas, of Illinois, and John C. Breckinridge, of Kentucky, while a rump of the Whig Party, calling itself the Constitutional Union Party, nominated John Bell, of Tennessee. In a letter to his fellow officer Earl Van Dorn, dated July 3, 1860, Lee said that he hoped that "Judge Douglas would now withdraw & join himself & party to aid in the election of Breckinridge," but no such withdrawal took place, and Lincoln was elected in November. The slave states began seceding from the United States and in February 1861 formed a breakaway republic, the Confederate States of America. Lee was recalled from Texas by the U.S. Army's general-in-chief, Winfield Scott, but after Virginia joined the secessionists on April 17, 1861, Lee resigned his commission and instead took command of Virginia's state forces.

Lee did so with serious reservations about the constitutionality of secession. In a letter to his son, George Washington Custis Lee, dated December 14, 1860, he blamed in equal parts "the aggressions of the North" and the "selfish and dictatorial bearing […] of the cotton states." According to the postwar recollection of Lee's friend William Allan, choosing to serve Virginia and the Confederacy "was a hard thing for him […] thinking as he did that Secession was foolish." He was even more alarmed at how "unprepared" the South was to wage a war, and that unpreparedness and the unpopularity of defending a regime built on chattel slavery, in Lee's eyes, spelled doom from the start.

President Lincoln sent his advisor Francis Preston Blair to offer Lee command of the U.S. Army. Blair later recalled that Lee was insistent that his own decision to ally himself with the Confederacy had nothing to with defending slavery, claiming that if "he owned all the negroes in the South, he would be willing to give them up […] to save the Union." Nevertheless, in a letter to his brother Charles Carter Lee, dated March 14, 1862, he praised the Confederacy as "the noble cause we are engaged in," and kept two of the Arlington slaves, whose manumission he was otherwise working through the courts, as servants on his first field campaign in western Virginia. In a letter to the governor of South Carolina, F. W. Pickens, dated January 2, 1862, he also urged on Southern governors "the employment of slaves on works for military defense," and during both of the campaigns he conducted north of the Potomac River, in 1862 and 1863, officers of his army rounded up free blacks in their path and sold them into slavery.

And yet Lee continued to push forward the manumission of his father-in-law's slaves, relying on the location of George Washington Custis Lee, in Richmond, as he wrote his son on November 28, 1862, to "arrange for the people whom I wish to liberate," including "the whole list at Arlington, White House, &c." by the close of December 1862. The incursions of the Union army into Virginia overran not only Arlington, but also White House and Romancocke, leaving Lee to wonder, in a letter to William Henry Fitzhugh Lee on February 16, 1862, whether the Custis slaves "might continue as they are until circumstances permit me to emancipate them." But Lee recorded the deed of manumission on January 2, 1863, in Richmond, "embracing all the names" of the Arlington and Romancocke slaves, and suggesting a "supplementary deed embracing those who have been omitted." Curiously, he included among the manumissions the one enslaved family he still owned in his name, that of Nancy Ruffin. Even so, as he wrote his wife, many of the Custis slaves were not actually released from bondage "at the White House & Romancocke" until late in 1863. Lee was aware, as he wrote a couple months later, that "there is some desire on the part of the community to continue them in slavery," but this, he added, "I must resist."

The Emancipation Proclamation, issued by President Lincoln on January 1, 1863, committed the United States to recruiting African Americans for its armies. The Confederacy, however, refused to recognize any fugitive slaves as soldiers, especially for the purposes of prisoner-of-war exchange, a policy which Lee enforced. According to William Allan, though, Lee claimed to have urged the Confederate president Jefferson Davis "often and early in the war […] that the slaves should be emancipated, that it was the only way to remove a weakness at home and to get sympathy abroad, and to divide our enemies, but Davis would not hear of it." Lee's son, William Henry Fitzhugh Lee, also insisted that his father "was always for gradual emancipation." And Lee told William Preston Johnston in 1868 that "he knew the strength of the United States Government; and saw the necessity at first of […] a proclamation of gradual emancipation and the use of negroes as soldiers."

It was not, however, until September 2, 1864, that Lee addressed himself officially on the subject to Davis, and then only with the intent of "relieving all able bodied white men employed as teamsters, cooks, mechanics and laborers, and supplying their places with negroes." Lee continued to recommend the impressment of slave workers "to labor on the fortifications" of Richmond. But in October, Lee took the further step of writing to William Porcher Miles, the chair of the Confederate House of Representatives' military affairs committee, to urge a program of African American enlistment, accompanied by emancipation.

Rumor of Lee's endorsement of slave enlistment was met with incredulity by Confederate hardliners. Howell Cobb, a general and politician from Georgia, denounced the notion of enlisting and emancipating slaves as "the most pernicious idea that has been suggested since the war began." Cobb found it "a source of deep mortification and regret to see the name of that good and great man and soldier, General R.E. Lee, given as authority for such a policy." Still, on January 7, 1865, the Virginia state senator Andrew Hunter wrote directly to Lee to ask his views on "the expediency and propriety of bringing to bear against our relentless enemy […] the element of military strength supposed to be found in our negro population."

Lee replied swiftly, writing that "the relation of master and slave, controlled by humane laws and influenced by Christianity" was "the best that can exist between the white and black races." But the Union armies will "in course of time penetrate our country and get access to a large part of our negro population" and recruit them into the Union forces. It was time for Southerners to decide whether "slavery shall be extinguished by our enemies and the slaves used against us, or use them ourselves at the risk of the effects which may be produced upon our social institution." That would, however, require "giving immediate freedom to all who enlist, and freedom at the end of the war to the families of those who discharge their duties faithfully." Nor could it stop merely with those who served in the Confederate forces. Once begun, military emancipation would have to be accompanied by "a well-digested plan of gradual and general emancipation."

Within days, Lee's letter "on the subject of negro enlistment" was being copied, circulated, and discussed. Robert Garlick Hill Kean, a Lynchburg native working in the Confederate war department, confessed to his diary on January 24, 1865, that he was "astonished" that Lee "favors emancipation per se," as well as advocating "large enlistments accompanied by the promise of prospective emancipation of the families of the negro soldier." But on February 11, 1865, the Confederate secretary of state Judah P. Benjamin sought from Lee and his army "an expression of its desire to be reinforced by such negroes as for the boon of freedom will volunteer to go to the front." One week later, Lee wrote to the Mississippi congressman Ethelbert Barksdale, who had introduced legislation to recruit slave soldiers without requiring their emancipation, to urge "the employment of negroes as soldiers" as "not only expedient, but necessary." He insisted, however, that "placing them on the footing of soldiers" should also come "with their freedom secured." Finally, Lee asked Jefferson Davis for direct authorization to "carry" black enlistment "into effect as soon as practicable," and on March 27 he anxiously requested confirmation from the Confederate secretary of war John C. Breckinridge "for raising and organizing the colored troops." But it was too little, too late. Lee evacuated Richmond on April 2 and surrendered the Army of Northern Virginia a week later at Appomattox Court House.

After the war, Lee remained adamant that the war had been fought by the Confederates not for slavery but "for the Constitution and the Union established by our forefathers." When, in the autumn of 1865, he took up the presidency of the struggling Washington College, he was careful to restrain rambunctious students (a number of whom were Confederate army veterans) from harassing black schools and churches and personally expelled a student involved in a harassment incident. When called to testify before the congressional Joint Committee on Reconstruction in 1866, Lee averred that "every one with whom I associate expresses kind feelings towards the freedmen. They wish to see them get on in the world, and particularly to take up some occupation for a living and to turn their hands to some work." The freed slaves, he added, were "as capable of acquiring knowledge as the white man is."

On the other hand, Lee told Congress that he had no desire to see Washington College become an instrument of free blacks "acquiring knowledge" by becoming racially integrated, and he was adamant in his personal opposition to proposals for equal civil rights for the freedpeople in the new Virginia state constitution. "The idea that the Southern people are hostile to the negroes, and would oppress them if it were in their power to do so, is entirely unfounded," Lee protested, but he opposed "any system of laws which would place the political power of the country in the hands of the negro race" because "the negroes have neither the intelligence nor the qualifications which are necessary to make them safe depositories of political power." In a letter to his nephew Edward Lee Childe, he wrote that he dreaded the prospect of "the South" being "placed under the dominion of the negroes," and, in a letter to a cousin on February 22, 1867, that he was so contemptuous of the "farce" of Reconstruction that he expected that "all decent white people would be forced to retire" from Washington.

Lee was never an enthusiast for chattel slavery. He had only a small legal involvement of his own in the institution and generally tolerated it as part of the give and take of the prewar southern landscape. When it became apparent that Confederate survival depended on jettisoning slavery, he was willing to do so. And although he was never reconciled entirely to the war's outcome and never promoted any form of racial egalitarianism, his attitude toward the freed slaves did not embrace the manic and violent hostility manifested by the Ku Klux Klan. Lee preferred to think of the postwar racial geography in terms of separate spheres, in which black and white people went their separate and politically unequal ways. "You will never prosper with the blacks," he warned his youngest son in 1868. "I wish them no evil in the world—on the contrary, will do them every good in my power." But it remained "abhorrent to a reflecting mind to be supporting and cherishing those" whom Lee would always suspect of "plotting and working for your injury, and all of whose sympathies and associations are antagonistic to yours."