In this paper, we introduce an adaptive robotic manipulation framework to respond to the flexibility needs of common industrial tasks such as box-filling and item sorting. The proposed framework consists of a vision module and a robot control module. The vision module is responsible for the detection and tracking of the environment (e.g., box and the items), which is also capable of creating an occupancy grid in real-time, to continuously update the robot trajectory planner with the occupied portions of the detected box and their coordinates. The robot control module includes a trajectory planner and a self-tuning Cartesian impedance controller, to implement an adaptive strategy for the picking, placement, and sorting of the items in the box. The item-sorting strategy is based on our preliminary observations on human motor behavior, implementing a trade-off between the task execution accuracy and environmental perception uncertainty. The efficacy of the framework in performing a flexible box-filling task using a robot, autonomously or in collaboration with a human, is evaluated through several experiments.