Preheat oven to 375 degrees, with rack in lower third. Line a standard muffin pan with paper baking cups.
In a large bowl, whisk together flour, graham cracker crumbs, brown sugar, baking powder, ground ginger, cinnamon, and salt. Make a well in the center of dry ingredients.
In a small bowl, whisk together buttermilk, eggs, vegetable oil, vanilla, and lemon zest. Add egg mixture to dry ingredients. With a rubber spatula or wooden spoon, gently stir ingredients until batter is just moistened (do not over mix; a few lumps will remain).
Spoon batter into muffin cups, filling each about two-thirds full. Bake until a toothpick inserted in center of a muffin comes out clean, 15 to 20 minutes. Allow muffins to cool in pan for 2 to 3 minutes then transfer muffins to a wire cooling rack to cool completely. Serve slightly warm or at room temperature according to preference.