Tammy, our grandmother Adams told us the same thing. She was right and wrong. We ARE descended from John Quincy Adams, but the one born 1843 in New York state. (In fact there were many children named that around that time.)
And, she was wrong because when that name is used the assumption is that PRESIDENT John Quincy Adams is meant.
You will need to do a lot of census searching to prove or disprove your situation. I would start with the Adams spouse of Leovicy Milam Adams.