Speech signals and glottal signals convey speakers’ emotional state along with linguistic information. To recognize speakers’ emotions and respond to it expressively is very much important for human-machine interaction. To develop a subject independent speech emotion/stress recognition system, by identifying speaker's emotion from their voices, features from OpenSmile toolbox, higher order spectral features and feature selection algorithm, is proposed in this work. Feature selection plays an important role in overcoming the challenge of dimensionality in several applications. This paper proposes a new particle swarm optimization assisted Biogeography-based algorithm for feature selection. The simulations were conducted using Berlin Emotional Speech Database (BES), Surrey Audio-Visual Expressed Emotion Database (SAVEE), Speech under Simulated and Actual Stress (SUSAS) and also validated using eight benchmark datasets. These datasets are of different dimensions and classes. Totally eight different experiments were conducted and obtained the recognition rates in range of 90.31%–99.47% (BES database), 62.50%–78.44% (SAVEE database) and 85.83%–98.70% (SUSAS database). The obtained results convincingly prove the effectiveness of the proposed feature selection algorithm when compared to the previous works and other metaheuristic algorithms (BBO and PSO).