Safety is a central challenge in human-robot collaboration. Particularly in higher collaboration levels, separating safety devices, such as fences, are no longer needed and must be replaced by intelligent sensor-based systems. Of particular interest is the adaptive speed control of the robot. This work presents a methodology to adaptively control the end-effector velocity of the robot based on the distances to dynamic environmental objects. The method combines distance measurement and environmental subtraction with conservative velocity estimation using robot-specific stopping distances and is available in real-time. Data acquisition is performed using a co-moving 3D camera sensor attached to the robot structure.