Pour the milk into a small saucepan, and bring just to a simmer over medium heat. Remove from the heat, stir in the lavender, and let steep and cool for about an hour at room temperature. Strain out the lavender bits and discard, so that you just use the lavender-infused milk.Whisk the lemon juice into the milk, and let sit for several minutes to curdle.
Preheat the oven to 350 F. Line three 8-inch cake pans with parchment paper in the bottoms of the pans, and spray the paper with non-stick spray. The baked cakes are sticky, and you should not skip this step of lining your pans with parchment paper.
Measure the sugar into a large bowl, and zest the lemons over the sugar. Use your fingers to rub the lemon zest into the sugar to really infuse the lemon flavor.
Set a fine mesh strainer over the sugar, and sift in the flour, baking soda, baking powder and salt, then whisk together the dry ingredients with the lemon sugar.
In a separate bowl, whisk together the lavender-lemon milk, eggs, egg whites, sour cream, oil and vanilla until smooth. Add the wet ingredients to the dry and whisk by hand until smooth, about 30 seconds.
Divide the batter between the pans. Bake for 25-27 minutes, until a cake tester in the center of the cakes comes out clean and the tops spring back when lightly touched. Set the pans on a wire rack, cover with a clean kitchen towel, and cool completely.