Web page classification has become a challenging task due to the exponential growth of the World Wide Web. Uniform Resource Locator (URL)-based web page classification systems play an important role, but high accuracy may not be achievable as URL contains minimal information. Nevertheless, URL-based classifiers along with rejection framework can be used as a first-level filter in a multistage classifier, and a costlier feature extraction from contents may be done in later stages. However, noisy and irrelevant features present in URL demand feature selection methods for URL classification. Therefore, we propose a supervised feature selection method by which relevant URL features are identified using statistical methods. We propose a new feature weighting method for a Naive Bayes classifier by embedding the term goodness obtained from the feature selection method. We also propose a rejection framework to the Naive Bayes classifier by using posterior probability for determining the confidence score. The proposed method is evaluated on the Open Directory Project and WebKB data sets. Experimental results show that our method can be an effective first-level filter. McNemar tests confirm that our approach significantly improves the performance. © 2018 Wiley Periodicals, Inc.