I would consider looking for his family in Connecticut, particularly around New London County, before a move to Nova Scotia, because:
A) there was a family of Ledyards there, and
B) A lot of New London County folks were early settlers of Vermont.
Granted, you have Nova Scotia in the mix, but it could still be a case of going where the cousins were when relocating to Vermont.