crawl deep and unique links csefo crawling and spidering Deep crawling – now the crawler will follow all internal links recursively and deeply. pip install requests beautifulsoup4 pyfiglet threading to run >> python3 csefo.py targets.txt