Removing lower growth will not stunt the plant but make the tops grow faster. To slow overall growth of a plant is to allow all the undergrowth to exist until you want the plant to take off by removing it all.

I would remove any growth about 2 feet above the soil so you have some airflow at the bottom. Depending on how wide the side branching is, you may want to remove some of the inside growth on the lower branches not getting good light. Remove anything not getting good light.