I can certainly understand the Southern viewpoint. Being labeled a "Carpetbagger" definitely was not a term of endearment. I am sure they resented having folks come in and buy up the local assets. (It is probably similar to the feelings I have when a large chain store moves in and kills off the little local businesses.)
Looking back now, I am not sure we could establish a workable set of guides for when to call someone a "carpetbagger." Over history, we have had folks with money who did buy up assets and that action was resented by poorer locals seeing things bought up my wealthier outsiders.
The unique conditions of the Civil War coupled with the reconstruction era policies (that had in some cases a sense of punishment) that yielded so many fertile opportunities. All this led to the phrase "carpetbagger," It became to widely used whenever someone who was not local acquired a local asset that I don't know how we now could set guidelines to limit the term's use.