I was just working on the records of this family recently and just came across your question. I hope this information might be helpful.
Judge William Bolling was only 14 at the time of the 1850 census. In 1860, he was a 25 year-old lawyer, living with his aunt and uncle (his uncle was also a lawyer and they were practicing law together). His uncle, William Terry, held 5 enslaved people--two adults and three children. William Bolling is not shown on the 1860 slave schedule.
However, William's father, Dr. Archibald Bolling, is shown on the 1860 schedule holding 8 enslaved people. Dr. Bolling lived in the household of his father-in-law, Benjamin Wigginton, in Bedford County, Virginia; Benjamin Wigginton held 38 slaves as shown on the 1860 slave schedule. When Benjamin died in 1863, his will divided his slaves between his two daughters, Anne (Wigginton) Bolling, the wife of Dr. Archibald Bolling and mother of Judge William Bolling, and her sister, Emma (Wigginton) Terry, the wife of William Terry, with whom William Bolling was living in 1860.
William Bolling would have been in close proximity to half of the Wigginton enslaved people as well as the Terry enslaved people, and Archibald Bolling would have had access to his own enslaved people as well as been in close proximity to the other half of the Wigginton enslaved people (his wife, Anne, inherited them free and clear of his control, contract, or debt, so they were hers, not his).
Dr. Archibald Bolling also had several brothers who lived into the 1830s. I haven't looked at their records, but chances are good that they were also slave holders.