Ooooh, you picked someone born in what I call the sweet spot. It is not a sweet spot at all. Since he was about 40 at the 1850 Census, there is every likelihood that his parents had already passed. It is awesome when they are living with their children in their old age. Some questions to ask yourself are:
When was Rusk, Texas settled?
Is there a history?
Could Sylvanus' parents have been settlers in Rusk?
If the parents went to Rusk, you could possibly guess who the father was from 1810 to 1840 census records. Hopefully there were not too many Walkers of a corresponding age. Georgia would be a big state to look for Walkers. If you could narrow down the town in Georgia where Sylvanus was found that might help.