In a dynamic construction site, sound generated by work activities and equipment operations is one of vital field data indicating construction progress, work performance, and safety issues. However, because of an enormous number of construction work types, accurate sound classification is currently limited. To address this challenge, this study proposes a schedulebased sound classification for establishing dynamic sound training data. With the schedule-based dynamic training method, this system retrieves the types of sounds of daily planned construction activities for flexibly restricting training data types of working sounds and ultimately improving the accuracy of sound classification. To reveal the implications of audio-based construction activity detection and site monitoring frameworks, this study also involves the development of a construction sound library, a hardware system, a sound classification framework, and a web-based visualization method. This proposed method is expected to play a critical role in managing a construction project by supporting site monitoring and progress analysis, and safety surveillance.