Preheat the oven to 350°F.
In a medium bowl, mix together graham crumbs, brown sugar, and melted butter. Press the crust mixture into a 9-inch pie pan.
Bake the graham crumb crust for 10 minutes.
While that is baking, prepare the key lime filling. Using an electric hand mixer, combine the key lime juice, zest, sweetened condensed milk, sour cream, 2 egg yolks, and powdered sugar.
Pour the filling into the baked crust.
Bake the pie for 15 minutes. It should be just slightly jiggly in the center. If the pie is very jiggly, leave it in and check it every 5 minutes.
Remove the pie from the oven and allow it to cool to room temperature. Then refrigerate the pie for at least another 2½ to 3 hours.
Serve with whipped cream and lime zest or lime slices if desired.