Well, his first name in Sweden would most likely been Svante which became Swonte.
However, Swedenburg might be a "taken name", many changed their names when settling in US. There needs to be more research done on "your end" first I'm afraid. According to the database Emiweb there are more than 200 persons named Svante born between 1840-1850 who emigrated before 1891. There are 17 persons emigrating with last name Svedenborg but none of them is named Svante.
Add all different birth dates and emigration years for Svante mentioned in different Census to the profile, if any of them matches an emigrant exactly it might be a lead. Also check the childrens birth notices if there can be any information on them about another last name.
Does the wedding notice mention his parents? SInce it is on Ancestry I can't see it, perhaps its available on FamilySearch as well? We simply need more to work on if we should be of any help with the Swedish part.
Edited to add...
There is a place in Örebro called Svedenborg, perhaps that is where he took his name from? It just might be a clue, unfortunately I have be off, have an appointment time to keep.