Blackboard Scraper

BLACKBOARD SCRAPER HAS BEEN UPDATED TO WORK WITH THE LATEST BLACKBOARD

Blackboard Scraper: A tool for scraping unit material off Curtin University’s Blackboard written in Python. This was put together in a couple of days so it may/will have bugs. Current functionality allows scraping of unit materials and scraping of iLectures if you visit the iLecture page and paste in the RSS feed link. I should probably mention that this only works at for curtin.edu.au blackboard accounts and it was written in a couple of days so it is pretty hacky.

There are three options for using the scraper, if you are on windows, you can download the frozen executable. If you are using Mac you can download the frozen app or if you are using anything else (ie Linux) you can download the source from GitHub and run it in Python. If you are running from Github, you will need to download the dependencies listed in the requirements.txt.

Windows Application
Mac OSX Application
GitHub

Using the program:

1) Log in by typing in your student ID and password. After loading, the fields should be populated with units and iLectures available to you on blackboard.

2) Choose the directory to download things to up the top.

3) To download unit materials select one or more item from the Blackboard Materials list and click Scrape. You can monitor the progress in the command window (it just lists the current link being downloaded). When a unit is done, it will print done, so if the last thing in the command window is done, the scrape is finished.

4) To download ILectures, visit the iLecture site by clicking one and pressing Visit URL (You may need to be logged into blackboard in your browser for this). Go to the bottom left hand corner of the Echo360 screen where there is the RSS feed symbol, hover over it and press vodcast. Now copy the URL from the browser to the iLecture RSS field and press scrape. Progress is shown in the command window. This is an experimental feature and I’m trying to find other ways to automate this process.

You end up with all the files neatly ordered by the page they were on like this:

Capture

If you are wondering where the iLecture RSS link is, I have circled it in the picture below.

RSS Link

5 thoughts on “Blackboard Scraper

  1. Pingback: Linux.conf.au: Day 3 | JasonGi

  2. Pingback: Blackboard Scraper now has stand-alone download option. | JasonGi

  3. Pingback: Updated Blackboard Scraper | JasonGi

  4. E

    Hi there,

    I’m currently looking to do a similar Blackboard scraper. Mine however is intending to scrape parts of blackboard (and other university services) into a mobile app.

    My question to you is how did you find out the required authentication and certificates you required to log into blackboard through your application? Also how did you deal with cross site request forgery?

    Any help you could provide would be great 🙂

    Thanks,
    E

    Reply
    1. jasongi Post author

      Hey Mate

      The requests library deals with the SSL authentication. As for figuring out what url/payload to log in – I just used Chrome developer tools to watch what happens when you log in.

      Not sure what you mean by how did I deal with cross site request forgery? This is a python script, not a website.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *