In a medium bowl, whisk together the brown rice flour, almond flour, cornstarch, baking powder, baking soda, salt, cinnamon, nutmeg, and ginger. Set aside.
In a large bowl, beat the oil and brown sugar together. Add the sweet potato puree and mix until combined. Add the eggs one at a time, beating well after each addition, then mix in the vanilla extract.
Sift the dry ingredients into the wet mixture, add the milk, and mix until just combined.
Pour the batter into the prepared cake pan and spread it evenly.
Bake for 50 minutes. To check for doneness, insert a toothpick into the center of the cake; it should come out clean, without streaks of wet batter.
Let the cake cool before removing it from the pan. Allow it to cool completely before frosting.
Spread the cream cheese frosting over the cooled cake and finish with chopped toasted walnuts, if desired.