Cyber-security systems collect information from multiple security sensors to detect network intrusions and their models. As attacks become more complex and security systems diversify, the data used by intrusion-detection systems becomes more dimensional and large-scale. Intrusion detection based on intelligent anomaly detection detects attacks based on machine-learning classification models, soft computing, and rule sets. Feature-selection methods are used for efficient intrusion detection and solving high-dimensional problems. Optimized feature selection can maximize the detection model performance; thus, a fitness function design is required. We proposed an optimization algorithm-based feature-selection algorithm to improve anomaly-detection performance. We used a genetic algorithm and proposed an advanced fitness function that finds the most relevant feature set, increasing the detection rate, reducing the error rate, and enhancing analysis speed. An improved fitness function for the selection of optimized features is proposed; this function can address overfitting by solving the problem of anomaly-detection performance from imbalanced security datasets. The proposed algorithm outperformed other feature-selection algorithms. It outperformed the PCA and wrapper-DR methods, with 0.99564 at 10%, 0.996455 at 15%, and 0.996679 at 20%. It performed higher than wrapper-DR by 0.95% and PCA by 3.76%, showing higher differences in performance than in detection rates.