Whisk eggs, half-and-half, light brown sugar, vanilla extract, ground cinnamon, kosher salt, and ground nutmeg together in a large bowl until smooth and combined.
Prepare the topping by tossing chopped nuts, flour, light brown sugar, ground nutmeg, and kosher salt together in a large bowl. Add the softened butter and incorporate into the dry mixture. Refrigerate until ready to bake.
Heat the oven to 350°F. Let the casserole sit at room temperature for 30 minutes while the oven heats. Sprinkle the topping evenly over the casserole and bake for about 45 minutes.