In this paper, an agricultural robot vision system is proposed for two typical environments—farmland and orchard—combined with weeding between crops. The system includes orchard production monitoring and prediction tasks, the target information recognition approach, and visual servo decision making. The results obtained from the proposed system show that using the region combination features of image 2D histogram as the decision-making basis, the accurate and rapid indirect identification and positioning of crop seedlings can be accomplished while skipping the complex process of accurately identifying crops and weeds. The algorithm performs reasonably good as the time of target recognition in the prototype system is found to be less than 16 ms, and the average accurate recognition rate of 97.43% is achieved. The benefits of the proposed system are the continuous improvement of the quality of agricultural products, the rise of production efficiency, and the increase of economic benefits.