A website crawler is a crucial tool for navigating the vast landscape of the internet. It systematically scans websites