Seperate profile parsing from scrapers/profiles.go

[SoggyRhino]

Currently scrapers/profiles.go also does the parsing which does not match our design.

Here is what I am proposing

• Update scrapers/profiles.go to save .html files similar to scrapers/coursebook.go
  • add /professors to outDir
  • save profiles as {fist}-{last}.html
• Create a parser/profiles.go
  • Copy all of the parsing logic into here, modified to use goquery instead of chromedp
• Update flags in main.go
• Bonus
  • Add resume support to scraper
  • Add a unit test for the parser
• Side effects
  • parser.go uses utils.GetAllFilesWithExtension which would create an issue if the proposed /poffessors is added so we might consider scraping coursebook into outDir/coursebook/... instead.

Sample dir structure: 

 outDir (ie data)
    ├───coursebook
    │   ├───24f
    │   │   └───cp_acct
    │   │           acct2301.001.24f.html
    │   │           acct2301.002.24f.html
    │   │           ...
    │   │    ...
    └───professors
            first-last.html
            ...

I haven't worked with the profiles scraper very much but there does not seem to be any technical reason why this should not be possible.

If this is added as a task I don't mind working on it but if someone is interested feel free.

[mikehquan19]

I’d say the idea seems great, but it’s a bit of an overkill, for 2 reasons:

• The profile page in which to scrape data from is not complicated and intertwined like the coursebook (where you have to scrape and later parse both courses and sections simultaneously).

• There are way less professors than courses and sections (2-76 ratio).

So try matching the design of coursebook scraping just accidentally makes it more over-engineered than it’s supposed to be.

However, I like your idea since it will help with the -resume flag for profiles scraping in the future. So how about this?

Try figuring out if there’s anyway you can implement the -resume flag in the current profile scraper. If there are, then we can keep the current one. Otherwise, we can start migrating to HTML-saving approach, where I will break it down to smaller subtasks, for people to share their work.

[SoggyRhino]

The issue with the resume flag is that we need to save something every time scrape a profile. My thought process is might as well just save the html.

Although a bit over engineered I think separation is worth it. It would make the project more consistent and would make adding unit tests easier since most of the boiler plate is already implemented in the parser.

Also then then the -resume could match what I did for coursebook but I'm kind of biased.

This is the cleanest way I can think of without separation:

  load profiles.json 
  scrape profile urls 

  for each profile not loaded
       scrape profile 
       update profiles.json 

[mikehquan19]

I think you have a point. Alright, I will convert this to a task. But I will break it down to small tasks. One will be responsible for scraping and saving them in the HTML. The other will be responsible for parsing using GoQuery.

If you want to work on first task, feel free :))

[AnshSharma100]

I would like to work on this.

[SoggyRhino]

I've been really busy recently so haven't gotten a chance to work on this but should be able to finish the first task in the next couple days. Or I can do the other part if @AnshSharma100 wants part 1.

[AnshSharma100]

Hey, @SoggyRhino by "part 1" do you mean scraping & saving the raw profile pages as HTML or should I go a little deeper?
Also if you are busy I do not mind tackling issue alone.

[mikehquan19]

@AnshSharma100 , yes part 1 means you rewrite scraper to save html. Part 2 means you will rewrite the parser to parse the html to object.

I’m really busy lately so haven’t got the chance to do it but I will break this down to 2 smaller issues and then assign each to two of you.

[AnshSharma100]

Alright!

[mikehquan19]

@AnshSharma100 , I think you can start working on the parsing of the professor right now. The new changes are from parse-professor branches.