Preheat oven to 375°F. Spread chopped pecans on a baking sheet and bake for 7 minutes until toasted. Set the pecans aside.
Lower oven temperature to 350°F. Spray a 9 ½-inch loaf pan with nonstick cooking spray and line with parchment paper, leaving an overhang on two sides to help remove the loaf after baking. Set the pan aside.
In a large mixing bowl, whisk sweet potato, brown sugar, vegetable oil, sour cream, eggs, and vanilla until combined.
Fold in the flour, baking soda, cinnamon, ginger, nutmeg, allspice, cloves, cardamom, and salt until no more dry flour is visible. Be careful not to overmix. Gently fold in 1 cup of the chopped and toasted pecans.
Pour the batter into the prepared loaf pan and bake for 50 to 60 minutes until golden brown and a toothpick inserted into the center comes out with no wet batter. Allow the loaf to cool completely in the pan.
Meanwhile, whisk the powdered sugar, butter, milk, and vanilla until smooth, thick, and pourable. Drizzle the glaze over the completely cooled loaf then top with the remaining chopped pecans.