Over the past decade, there has been an increasing demand for mobile devices to perform computationally intensive tasks. However, the computational capability of these devices is limited due to memory, power and portability constraints. One of the feasible and attractive ways to enhance the performance of the resource-limited mobile devices is to offload their computationally intensive tasks on to the cloud servers when internet connectivity is available. However, when cloud servers are involved in processing, the latency and cost of computation increases. To mitigate these problems, devices with high computational resources, called cloudlets, can be deployed in the locations close to the mobile users/devices. The mobile devices can then offload their computationally intensive tasks on to them. Due to easier access and nearness of the cloudlets, the cost and latency in processing the tasks decreases. In this work, we focus on task assignment problem in a multi-cloudlet network connected via a wireless SDN network, which services the task offload requests from mobile devices in a given locality. The aim of the proposed solution is to minimize latency and thus enhance the quality of service for mobile devices. We prove the optimality of the proposed solution mathematically and employ an admission control policy to maintain this optimality even in heavily loaded networks. We also perform numerical simulations for two scenarios of small and large networks and evaluate the performance for varying traffic and network parameters. The results demonstrate that the proposed task assignment method offers reduced latency compared to state-of-the-art task assignment approaches and hence improves the quality of service offered to mobile devices.