Line an 8x8 baking dish with foil or waxed paper leaving edges to hang over the side that will help in getting the fudge out of the pan once it's hardened.
Heat a skillet over medium high heat. Add pecans and butter. Stir constantly until butter is melted and pecans start to toast slightly, about 3 to 4 minutes. Sprinkle with salt. Set aside.
Add Eagle Brand Sweetened Condensed Milk, cinnamon, ginger, nutmeg, and sweet potato to a medium saucepan over medium heat. Stir constantly until mixture begins to bubble. Remove from heat, add vanilla and stir.
Add baking chips and stir until melted. Immediately stir in about 3/4 of the buttered, salted pecans reserving remaining 1/4 for the top of the fudge.
Pour mixture into prepared pan. Sprinkle with remaining pecans.
Cover and place in refrigerator 2 hours or until firm.