Skip to content

README.md

Latest commit

 

History

History
48 lines (33 loc) · 1.04 KB

README.md

File metadata and controls

48 lines (33 loc) · 1.04 KB

darkweb scraper

A Typescript generic scraping platform that connects to Tor browser through a docker container. The scraper runs through config files (. yaml), Analyse the results in gcloud (GCP), And saves the results in postgreSql database. An Express server then exposes the server to A React client side which uses Redux to handle the data.

Front-end

Dashboard: home Custom Scraping: search Live Data: results

Architecture:

defenition

Installation

  1. put your configuration in ./configs/sites

  2. add .env file with:

  • postgresql link as DATABASE_URL2="your link here"
  • google as env:GOOGLE_APPLICATION_CREDENTIALAS =C:\dev...your path to env file
docker compose up

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

MIT