Improved script for downloading all games data without incurring in errors

.gitignore

@@ -1,3 +1,8 @@
+.idea
+.vcs
+
+*.csv
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

README.md

@@ -1,13 +1,66 @@
-vgchartzfull is a python script based on BeautifulSoup.
-It creates a dataset based on data from 
-http://www.vgchartz.com/gamedb/
+# vgchartzfull - A crawler to download data from Global Videogame Sales
 
-The dataset is saved as vgsales.csv.
+vgchartz-full-crawler.py is a python@3 crawler script based on BeautifulSoup.
+It creates a csv dataset with data from more than 57,000 games. based on data from [VGChartz Site](http://www.vgchartz.com/gamedb/).  
+
+## Output
+
+The dataset is saved in the file specified at cfg/resources.json, by default "dataset/vgsales.csv".
+
+## Install & execution
+
+You will need to have some depencies compiled at **requirements.txt**.
 
-You will need to have BeautifulSoup added.
 It can be installed by pip.
 
-sudo pip install BeautifulSoup
+```bash
+
+  # Install dependencies
+  $> pip install -r requirements.txt
+
+  # Run
+  $> python vgchartzfull.py
+
+
+```
+
+## Dictionary
+
+The dataset it's composed by this fields, and the data is collected with this [methodology](https://www.vgchartz.com/methodology.php).
+
+| Field | Description              |
+|-------|--------------------------|
+| Rank  | Ranking of overall sales |
+| Name | The games name |
+| Genre | Genre of the game |
+| Platform | Platform of the games release (i.e. PC,PS4, etc.) |
+| Developer | Developer of the game | 
+| Publisher | Publisher of the game |
+| Vgchartz_Score | Score at VGcharz site | 
+| Critic_Score | Score at Critic | 
+| User_Score | Score by VGcharts users' site | 
+| Total_Shipped | Total worldwide shipments (in millions) | 
+| Total_Sales | Total worldwide sales (in millions) |
+| NA_Sales | Sales in North America (in millions) |
+| EU_Sales | Sales in Europe (in millions) |
+| JP_Sales | Sales in Japan (in millions) |
+| Other_Sales | Sales in the rest of the world (in millions) |
+| Release_Date | Year of the game's release |
+| Last_Update | Last update of this register |
+
+## TODO
+
+- [ ] Remap the columns according the selected values at resources.json
+- [ ] Add some unit testing
+- [ ] Dockerize (w/ alpine-python) to ease use and avoid intallations
+- [ ] Publish at Docker hub
+
+## Links
+
+* [vgchartz.com](https://www.vgchartz.com)
+* [Original Crawler](https://github.com/GregorUT/vgchartzScrape)
+* [Kaggle Dataset](https://www.kaggle.com/gregorut/videogamesales)
+
+## Greetings
 
-Thanks to Chris Albon.
-http://chrisalbon.com/python/beautiful_soup_scrape_table.html
+Thanks to [Chris Albon](http://chrisalbon.com/python/beautiful_soup_scrape_table.html) 

cfg/resources.json

@@ -0,0 +1,44 @@
+{
+  "application_log_filename": "log/app.log",
+  "output_filename": "dataset/vgsales.csv",
+  "separator": ",",
+  "encoding": "utf-8",
+  "start_page": 1,
+  "end_page": 2,
+  "include_genre": false,
+  "base_page_url": "https://www.vgchartz.com/gamedb/?page=",
+  "query_parameters": {
+    "results": 100,
+    "region": "All",
+    "boxart": "Both",
+    "banner": "Both",
+    "ownership": "Both",
+    "showmultiplat": "No",
+    "order": "Sales",
+    "showtotalsales": 1,
+    "showpublisher": 1,
+    "showvgchartzscore": 1,
+    "shownasales": 1,
+    "showdeveloper": 1,
+    "showcriticscore": 1,
+    "showpalsales": 1,
+    "showreleasedate": 1,
+    "showuserscore": 1,
+    "showjapansales": 1,
+    "showlastupdate": 1,
+    "showothersales": 1,
+    "showshipped": 1,
+    "keyword": null,
+    "console": null,
+    "developer": null,
+    "publisher": null,
+    "goty_year": null,
+    "genre": null
+  },
+  "minimum_sleep_time": 6,
+  "maximum_sleep_time": 15,
+  "minimum_major_version": 1,
+  "maximum_major_version": 56,
+  "minimum_minor_version": 1,
+  "maximum_minor_version": 10
+}

dataset/.gitkeep

@@ -0,0 +1 @@
+Git doesn't like empty folders

log/.gitkeep

@@ -0,0 +1 @@
+Git doesn't like empty folders

requirements.txt

@@ -0,0 +1,8 @@
+beautifulsoup4==4.8.2
+bs4==0.0.1
+numpy==1.18.2
+pandas==1.0.3
+python-dateutil==2.8.1
+pytz==2019.3
+six==1.14.0
+soupsieve==2.0

run.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+python --version >/dev/null 2>&1 || { echo >&2 "I require python@3 utility but it's not installed. ¯\_(ツ)_/¯ Aborting."; exit 1; }
+pip --version >/dev/null 2>&1 || { echo >&2 "I require pip utility but it's not installed. ¯\_(ツ)_/¯ Aborting."; exit 1; }
+
+clear
+
+echo "\nInstalling deps... "
+pip install -r requirements.txt
+
+echo "\nStart crawling... (remember a crawler is the friend nobody likes)"
+python vgchartz-full-crawler.py
+