Project_CSE305

This a git repository for the CSE305 Project: Parallel Web Crawler

How to execute:

Make sure libcurl is properly installed on your comp.
Edit the makefile: replace this directory with where you have libcurl installedLIBCURL_DIR = ----/curl
Execute on wikipedia with: make clean, then make, then ./crawler, if you wanna change the link it can be done in the main file.
Please make sure you have set up your tests properly when executing crawler.

Name		Name	Last commit message	Last commit date
Latest commit History 36 Commits
.vscode		.vscode
.DS_Store		.DS_Store
HashTable.cpp		HashTable.cpp
Integer.h		Integer.h
Makefile		Makefile
README.md		README.md
Website.h		Website.h
crawler		crawler
crawler.h		crawler.h
default.profraw		default.profraw
downloader.h		downloader.h
hashtable 10000.txt		hashtable 10000.txt
link_extractor.h		link_extractor.h
link_extractor.h.gch		link_extractor.h.gch
main.cpp		main.cpp
main.o		main.o
output.txt		output.txt
prof.out		prof.out
test.h		test.h
test_output.txt		test_output.txt
test_output_16.txt		test_output_16.txt
test_output_I_ url_2020.txt		test_output_I_ url_2020.txt
test_output_html_2020.txt		test_output_html_2020.txt
test_output_html_2020_2.txt		test_output_html_2020_2.txt
test_output_html_2020_3.txt		test_output_html_2020_3.txt
test_output_html_2020_4.txt		test_output_html_2020_4.txt
test_output_html_2022.txt		test_output_html_2022.txt
test_output_html_2022_2.txt		test_output_html_2022_2.txt
test_output_url_2020_2.txt		test_output_url_2020_2.txt
test_output_url_500000_2020.txt		test_output_url_500000_2020.txt
thread_pool.h		thread_pool.h
url_parser.h		url_parser.h

Provide feedback