Hi
In 1850 only ONE James A Plummer shows up. He's living with Margaret and Allen Plummer in Maryland, and several other people who most likely are his brothers and sisters.
Consider this a clue: this may or may not be the family you are looking for. I hope it's a stepping stone for you!
Sylvanus Plummer, Plummer-409, has been well sourced and might provide more clues to your family line.
Source Citation