I suspect those different census records are for different Charles Browns. It is a very common name, and it's unlikely that someone will tell consecutive census-takers that their birthplace was a different state.
You might want to match his profile to the familysearch record for him: 293H-2ND
The 1860 and 1880 Census records both say his birthplace is Pennsylvania and his parents were born in New Jersey.
I added his marriage and probate records to his profile.