Bring 4 cups of water to a boil. While the water is heating, add the tea bags and baking soda to a heat-proof glass pitcher. Once the water has come to a boil, pour it into the pitcher and let the tea steep for 15 minutes.
Remove the tea bags from the pitcher and discard. Stir in the sugar until dissolved. Stir in the remaining 4 cups of cold water and the vodka (if including) and transfer to the fridge to chill. Serve with lemons slices and ice.