Recently, visual object detection and tracking has become a vital role in many different applications. It spans various applications like robotics, video surveillance, and intelligent vehicle navigation. Especially, in current situation where the use of UAVs is expanding widely, detection and tracking to soot down illegal UAVs flying over the sky at airports, nuclear power plants and core facilities is becoming a very important task. The remarkable method in object tracking is correlation filter based tracker like KCF (Kernelized Correlation Filter). But it has problems related to target drift in tracking process for long-term tracking. To mitigate the target drift problem in video surveillance application, we propose a tracking method which uses KCF, adaptive thresholding and Kalman filter. In the experiment, the proposed method was verified by using monochrome video sequences which were obtained in the operational environment of UAV.