Railway systems are one of the most preferred transport means worldwide. Some faults may occur on railway tracks due to several reasons and such faults may cause accidents. For this reason, railway tracks should be periodically inspected. In this study, a computer vision based approach was proposed for inspecting the faults in railway tracks. It was aimed to inspect the faults which may occur on rail surfaces such as scouring, breaking, and deficient fasteners such as bolts and sleepers with the experimental study presented. In this study, feature extraction was performed on a video image containing especially a healthy railway track. Then, feature extraction was again performed on the image containing healthy railway track by generating virtual faults, and these two data sets were labelled as faulty and healthy, and trained. Algorithm was applied on a video image including faulty and healthy frames, operating time and accuracy performance was measured and a decision making mechanism was established during test phase.