Preheat oven to 350 degrees. Spray a 9 by 9-inch square dish well with non-stick cooking spray.
For the topping: In a medium mixing bowl stir together brown sugar, melted butter, cinnamon nutmeg and salt. Spread evenly into greased baking dish.
Toss apples with lemon juice (I just do this on the cutting board I cut them on). Align apples in rows over brown sugar layer. Set aside.
In a medium mixing bowl whisk together flour, baking powder and salt, set aside.
In a mixing bowl using an electric hand mixer (or in a stand mixer fitted with a paddle attachment) cream together butter and sugar until pale and fluffy. Mix in eggs one at a time, scrape down bowl, then blend in vanilla.
Add 1/3 of the flour mixture to the butter sugar mixture and mix until combined. Add in 1/2 of the buttermilk and mix. Mix in another 1/3 flour mixture, then remaining 1/2 buttermilk. Finish by mixing in last 1/3 of the flour mixture. Scape down bowl and fold batter to ensure ingredients are evenly incorporated.
Spread batter over apple layer in baking dish. Bake in preheated oven until toothpick inserted into center comes out clean, about 35 - 45 minutes. Let cool 15 minutes on a wire rack.
Run knife around edges of cake to ensure it's loosened then invert onto a platter. Cut into slices and serve warm with vanilla ice cream if desired.