Remove hadoop, keep spark

Hadoop is outdated. It's better not to use it now-days... Let's migrate all to spark.

- [ ] Review the code
- [ ] Identify pieces to keep
- [ ] remove hadoop
- [ ] update test
- [ ] update doc 