The images for this soldier's Compiled Service Record are all available at FamilySearch:
In some documents, he is very clearly identified as a "slave" (ex. image 1151) while in others he is shown as being a Farmer born in Howard Co, Maryland (image 1141). One does not preclude the other.
All of the data in the documents combined leads me to wonder if there were two Oliver Scotts. While everything is for a man named Oliver Scott from Howard County, Maryland, with gunshot wounds, one says hip and the other says back (image 1156). There are two casualty reports and two discharge documents, only one of which includes a photograph of the soldier (image 1168). Image 1156 actually has two documents; one from 1864 showing his "wife or nearest relative" as Mary Scott of Clarksville, Howard County, Maryland; the other from 1865 has his "wife or nearest relative" as Ellen Miles of Clarksville, Maryland.
There are two 1870 Census records for Oliver Scott in Howard County, one with Ella as the adult female, one with M J as the adult female.
- "United States Census, 1870", database with images, FamilySearch (https://www.familysearch.org/ark:/61903/1:1:MN3N-CKF : 29 May 2021), Oliver Scott and Ella, 1870.
- "United States Census, 1870", database with images, FamilySearch (https://www.familysearch.org/ark:/61903/1:1:MN3F-Y2W : 29 May 2021), Oliver Scott and M J, 1870.
When writing his story, you might find context for his Civil War service in these books: