Lane lines play a key role in indicating traffic flow and directing vehicles; lane detection serves as a core component in most of the modern-day advanced driver assistance systems (ADASs). Computer vision-based lane detection is an essential technology for self-driving cars. This paper proposes a lane detection system to detect lane lines in urban streets and highway roads under complex background. In order to nullify the distortions caused by the camera lenses, we generate a distortion model by calibrating images against a known object, and apply a generalized filtering approach using Sobel operator (Canny edge detection) in HLS color space. A bird eye view of image is generated using perspective transformation. A special search strategy using sliding window algorithm is used to detect lane lines, and later, curve fitting is done using polynomial regression. Thus, the obtained lane detector is overlaid upon a video to fill the detected portion of the lane. Then, it is applied to the video to detect lane lines. The image processing pipeline is written in Python using OpenCV libraries, and video processing is done using MoviePy. In this paper, the system developed is tested by applying it on a video taken from a camera mounted over the car. The environment used to implement the system is Anaconda. The results obtained show that the proposed system for lane detection, self-calibration and vehicle offset estimation is effective, accurate for both straight and curved lanes and robust to challenging environments. © 2021, Springer Nature Singapore Pte Ltd.