Rebuild the Academic API

[joengelm]

The registrar's recent switch from Banner to Courses@Brown has killed the Academic API's scraper. Until this is fixed, there will be no Academic API.

[hpincket]

Scrapy (http://scrapy.org/) now supports Python 3.

[jbrower95]

The issue is that our point of authentication is now gone. We used to use this sketchy banner URL which didn't require 2-fac to authenticate, but CIS seems to have removed this when deprecating banner.

If we want to have a courses API, we unfortunately need to get into a meeting with CIS and have credentials drawn up for shibboleth, which won't be fun. Most likely, this project is dead. RIP.

[jbrower95]

Perhaps this is a bug, but Courses@Brown currently uses a JSON-based Api for accessing course data which, for some queries, does not require authentication. We could look into this, but most likely we'll hit a dead end (as shibboleth will block us from getting any automated work done). If we just want to scrape though, we could authenticate a session over shibboleth and then make API calls ourselves, which would be pretty chill.

[hpincket]

Good point. I think the immediate solution is to have the script pause for 2-fac approval. We'd have to handle the automatic forwarding somehow... but I'm sure there's a way.

[macmccann]

Hi, I know this is a really old issue, but I'm a sophomore at Brown (class of '20) looking to enhance course planning with a webapp and I need a source of truth for the courses and course requirements. I'd be happy to look more into whether there's a way to scrape Courses@Brown, but I'd love any help I can get.