Preheat the oven to 350 F, and position a rack in the center of the oven. Lightly spray a 9-inch square baking pan with nonstick baking spray.
In a saucepan, melt the butter over medium heat. Continue to cook the butter for about 5 minutes, swirling the pan occasionally, until nutty golden brown solids form at the bottom of the pan. Throughout the process, the water will evaporate, hiss and splatter a little. When a layer of foam starts to form on top of the butter, and it quiets down, it's just about ready.
Use a spatula to scrape the browned bits off the bottom of the pan, and pour the butter and browned bits into a mixing bowl.
Add the brown sugar to the hot browned butter and whisk until combined, then whisk in the eggs and vanilla until smooth. Next, whisk in the milk and sour cream.
In a separate bowl, sift together the flour, salt, baking powder, cinnamon and nutmeg. Add the dry ingredients to the wet, and whisk just until combined and mostly smooth.
Pour the batter into the prepared pan. Bake for about 28-30 minutes, until a cake tester inserted in the center comes out clean, and the sides of the cake are pulling away from the edges slightly.
Set the pan on a cooling rack, and let the cake cool for at least 45 minutes before serving the cake warm, or let it cool to room temperature. Cut into squares and serve with the toffee sauce and a scoop of vanilla ice cream.
While the cake is cooling, get the toffee sauce ready.