Detection of Web subsites: Concepts, algorithms, and evaluation issues

Web sites are often organized into several regions, each dedicated to a specific topic or serving a particular function. From a user's perspective, these regions typically form coherent sets of pages characterized by a distinct navigation structure and page layout - we refer to them as subsites. In this paper we propose to characterize Web site structure as a collection of subsites and devise a method for detecting subsites and entry points for subsite navigation. In our approach we use a new model for representing Web site structure called Link Structure Graph (LSG). The LSG captures a complete hyperlink structure of a Web site and models link associations reflected in the page layout. We analyze a sample of Web sites and compare the LSG based approach to commonly used statistics for Web graph analysis. We demonstrate that LSG approach reveals site properties that are beyond the reach of standard site models. Furthermore, we devise a method for evaluating the performance of subsite detection algorithms and provide evaluation guidelines.