Checking Ancestry: The 1860 Census of Olean, NY, lists a Joseph Simpier, 34 or 39, livery keeper, b. Canada married to Harriet, 32. They have a son, Henry, 14. b. Pennsylvania.
Does that sound like the same person to you? Sounds likely to me. Likely enough to follow the trail.
In the 1855 New York State Census: Joseph Simpier, 36, b. France (I'm wondering if French Canada), liveryman; Harriet, 25, b, Cattaraugus; Henry, 10, b. Pennsylvania; Joseph, 6 mo.