Baited underwater video sampling is a common method to monitor fish populations, yet the data requirements associated with imagery leads to bottlenecks in productivity. Image analysis that incorporates automated methods through deep-learning models could provide solutions. These models have the potential to improve efficiency, and decrease the cost of producing information on fish populations and habitats. In order to reduce human intervention, these models must produce precise, accurate results. While methods for gauging model performance through metrics such as mean-average-precision are helpful during the model training process, evaluating the performance on years of survey data requires a different approach. An otolith age-reader comparison method has been adapted to compare automated counts to true counts. The metrics produced in this analysis are then compared across a span of the model confidence levels in order to find the optimal settings per species to filter output and improve processing speed. For most species, increasing annotations for model training results in better performance, however issues persist with occlusion, turbidity, schooling species, and cryptic/conspecific appearances. With focus on Red Snapper (Lutjanus campechanus), this process of evaluation was carried out with multiple years of video data to test for fidelity based on location, time, and environmental conditions. Identifying common failures and adapting active learning algorithms can lead to targeted training for more efficient models in the future. These quality assessment and quality control methods of evaluation provide a framework for tracking performance drift and integrating automated methods properly with existing surveys and manual video count protocols.