Author: jasongi

Hottest 100 Predictions – A Comparison

This Hottest 100 I made a program to scrape Instagram for hottest 100 votes. I then collated the predictions from other programs (100 Warm Tunas and ZestfullyGreen’s Twitter scraper) and scored them based on performance, you can see the results here (I also opened this up to manual entries, one which outscored all the predictors).

I also decided to combine the results of the twitter scraper and my Instagram scraper, which turned out the be a better predictor than any of them. Next year I will have to incorporate a twitter scraper into my predictor.

Below is a summary of some interesting stats about the three automated prediction methods, plus the combination of 100 Toasty Tofu(s) and ZestfullyGreen’s Twitter scraper. I decided to take the results from ZestfullyGreen’s twitter scrape and add them to my results to see if this would be any better. I had a look at my predictions that included duplicate votes, however these performed worse than everything except the twitter prediction, so I have excluded them. This means my hypothesis on excluding duplicate votes (that they make the prediction less accurate) seems confirmed.

The final question that remains is, who truly is the internet’s most accurate Hottest 100 predictor? As you can see below, there isn’t really an answer for this. By my (somewhat arbitary) scoring system, 100 Warm Tunas and myself have a very similar accuracy. I think we will have to wait until next year to really test them.

JG 100 Tunas ZG JG + ZG
Points 7289/10000 7288/10000 5679/10000 7297/10000
Number of Songs in Correct Position 7/100 4/100 1/100 3/100
Number of Correct Songs in any Position 83/100 83/100 70/100 83/100
Number of Correct Top 5 Songs in Correct Position 2/5 2/5 1/5 2/5
Number of Correct Top 5 Songs in any Top 5 Position 4/5 4/5 4/5 4/5
Number of Correct Top 10 Songs in Correct Position 2/10 2/10 1/10 2/10
Number of Correct Top 10 Songs in any Top 10 Position 8/10 8/10 5/10 8/10
Number of Correct Top 20 Songs in Correct Position 2/20 3/20 1/20 2/20
Number of best predictions (see below) 45 50 34 45
Number of worst predictions (see below) 16 20 65 15
Number of Correct Top 20 Songs in any Top 20 Position 16/20 16/20 11/20 16/20
Guessed #1? Yes Yes No Yes

Song-by-song comparison of predictors

# JG 100 Tunas ZG JG + ZG Title Artist
1 1 1 2 1 HUMBLE. Kendrick Lamar
2 2 3 4 2 Let Me Down Easy Gang Of Youths
3 6 6 25 6 Chateau Angus & Julia Stone
4 3 4 3 3 Ubu Methyl Ethel
5 4 2 5 4 The Deepest Sighs, The Frankest Shadows Gang Of Youths
6 10 8 1 10 Green Light Lorde
7 5 5 13 5 Go Bang PNAU
8 11 10 43 11 Sally {Ft. Mataya} Thundamentals
9 16 15 33 16 Lay It On Me Vance Joy
10 9 13 14 9 What Can I Do If The Fire Goes Out? Gang Of Youths
11 7 7 29 7 SWEET BROCKHAMPTON
12 15 16 39 15 Fake Magic Peking Duk & AlunaGeorge
13 23 24 30 23 Young Dumb & Broke Khalid
14 29 30 6 29 Homemade Dynamite Lorde
15 12 11 24 12 Regular Touch Vera Blue
16 30 32 36 30 Feel The Way I Do Jungle Giants, The
17 13 12 20 13 Marryuna {Ft. Yirrmal} Baker Boy
18 14 14 9 14 Exactly How You Are Ball Park Music
19 17 19 15 17 The Man Killers, The
20 35 38 59 35 Let You Down {Ft. Icona Pop} Peking Duk
21 8 9 22 8 Birthdays Smith Street Band, The
22 26 26 27 26 Lemon To A Knife Fight Wombats, The
23 19 18 10 19 Not Worth Hiding Alex The Astronaut
24 78 86 N/A 77 rockstar {Ft. 21 Savage} Post Malone
25 34 31 18 33 Weekends Amy Shark
26 39 39 23 39 Feel It Still Portugal. The Man
27 43 41 N/A 43 Be About You Winston Surfshirt
28 47 51 76 47 Mystik Tash Sultana
29 28 27 37 28 Mended Vera Blue
30 36 35 26 36 Low Blows Meg Mac
31 25 25 48 25 Lay Down Touch Sensitive
32 27 28 91 27 NUMB {Ft. GRAACE} Hayden James
33 22 23 58 22 Slow Mover Angie McMahon
34 37 37 19 37 DNA. Kendrick Lamar
35 51 46 31 51 Passionfruit Drake
36 18 17 12 18 I Haven’t Been Taking Care Of Myself Alex Lahey
37 63 70 52 62 Slide {Ft. Frank Ocean/Migos} Calvin Harris
38 46 48 34 46 Bellyache Billie Eilish
39 53 49 N/A 52 Got On My Skateboard Skegss
40 24 21 44 24 True Lovers Holy Holy
41 41 40 35 41 Blood {triple j Like A Version 2017} Gang Of Youths
42 59 56 N/A 59 Cola CamelPhat & Elderbrook
43 91 74 74 91 Murder To The Mind Tash Sultana
44 49 50 42 49 In Motion {Ft. Japanese Wallpaper} Allday
45 21 20 7 21 Every Day’s The Weekend Alex Lahey
46 57 54 17 57 Better Mallrat
47 45 52 16 45 Want You Back HAIM
48 54 47 N/A 53 The Comedown Ocean Alley
49 33 34 82 34 Passiona Smith Street Band, The
50 77 84 84 74 On Your Way Down Jungle Giants, The
51 N/A N/A 56 N/A Man’s Not Hot Big Shaq
52 N/A N/A N/A N/A Glorious {Ft. Skylar Grey} Macklemore
53 62 68 87 63 Moments {Ft. Gavin James} Bliss N Eso
54 50 57 N/A 50 Homely Feeling Hockey Dad
55 42 44 N/A 42 6 Pack Dune Rats
56 32 29 72 32 Watch Me Read You Odette
57 67 67 N/A 67 Bad Dream Jungle Giants, The
58 20 22 11 20 The Opener Camp Cope
59 80 79 N/A 80 Used To Be In Love Jungle Giants, The
60 69 66 8 69 Boys Charli XCX
61 73 77 N/A 73 21 Grams {Ft. Hilltop Hoods} Thundamentals
62 92 89 N/A 92 Saved Khalid
63 40 43 28 40 Life Goes On E^ST
64 60 58 45 60 Fool’s Gold Jack River
65 65 62 38 64 Everything Now Arcade Fire
66 66 65 93 65 Lemon N.E.R.D. & Rihanna
67 38 36 N/A 38 Shred For Summer DZ Deathrays
68 48 45 80 48 Golden Kingswood
69 44 42 96 44 I Love You, Will You Marry Me Yungblud
70 31 33 54 31 Amsterdam Nothing But Thieves
71 N/A N/A 21 N/A Perfect Places Lorde
72 88 85 71 88 In Cold Blood alt-J
73 83 64 N/A 82 Nuclear Fusion King Gizzard & The Lizard Wizard
74 N/A N/A 98 N/A XO TOUR Llif3 Lil Uzi Vert
75 61 60 N/A 61 Braindead Dune Rats
76 76 76 N/A 75 Cloud 9 {Ft. Kian} Baker Boy
77 N/A 100 66 N/A Million Man Rubens, The
78 N/A N/A N/A N/A Electric Feel {triple j Like A Version 2017} Tash Sultana
79 N/A N/A 69 N/A Hey, Did I Do You Wrong? San Cisco
80 90 90 61 90 Say Something Loving xx, The
81 N/A N/A 32 N/A Liability Lorde
82 N/A N/A 46 N/A 1-800-273-8255 {Ft. Alessia Cara/Khalid} Logic
83 74 72 60 76 Blood Brothers Amy Shark
84 84 73 N/A 85 Oceans Vallis Alps
85 58 59 N/A 58 Does This Last Boo Seeka
86 94 91 95 94 Maybe It’s My First Time Meg Mac
87 72 63 78 71 The Way You Used To Do Queens Of The Stone Age
88 56 61 N/A 56 Edge Of Town {triple j Like A Version 2017} Paul Dempsey
89 N/A N/A N/A N/A Dawning DMA’s
90 N/A N/A N/A N/A Hyperreal {Ft. Kučka} Flume
91 N/A N/A N/A N/A Big For Your Boots Stormzy
92 N/A N/A N/A N/A LOVE. {Ft. ZACARI} Kendrick Lamar
93 95 95 85 96 Do What You Want Presets, The
94 99 93 N/A 98 Second Hand Car Kim Churchill
95 N/A N/A N/A N/A Mask Off Future
96 100 97 55 100 Chasin’ Cub Sport
97 N/A N/A N/A N/A LOYALTY. {Ft. RIHANNA} Kendrick Lamar
98 N/A N/A N/A N/A Snow Angus & Julia Stone
99 64 N/A N/A 66 Arty Boy {Ft. Emma Louise} Flight Facilities
100 N/A N/A N/A N/A Don’t Leave Snakehips & MØ

100 Toasty Tofu(s) – Submit your prediction

Think you can predict the hottest 100 better than me? I have made a form for submitting your own predictions, and a leader-board will be shown song-by-song on Triple J day. Check it out: Triple J Hottest 100 Prediction tracker submission.

The scoring will be as follows:

  • 100 points for each correct guess (song and place).
  • If you pick a song that gets in the top 100, but not the right place, you lose 1 point per place you were off. For example if you pick Never Gonna Give You up for number 90 but it gets 75, you get 85 points.
  • 0 points for a song that isn’t in the top 100 at all.
  • Therefore, a perfect score will be 10000 points.

100 Toasty Tofu(s) – Another Triple J Hottest 100 Predictor

Update: Think you can do better than my prediction? Prove it by filling out your prediction here: Triple J Hottest 100 Prediction tracker submission. Also, you can look at the leaderboard of predictions over here.

100 Toasty Tofu(s) is another Triple J Hottest 100 Predictor, made for your entertainment with no guarantees what-so-ever.

Since 2012, various people have been predicting the Hottest 100 using social media scrapes and OCR. This started with The Warmest 100 and was continued by 100 Warm Tunas. I’ve long thought it’s an awesome experiment because the conditions are good for using social media as a predictor. Two factors make this a good experiment – the average person is willing to share their hottest 100 votes and the stakes are so low, unlike political elections, that there aren’t hoards of true believers/trolls/Russian government agents trying to manipulate public sentiment.

I use instagram-scraper to scrape the hashtags (the same as 100 Warm Tunas) and then a python script that uses Tesseract OCR to convert them to text. They are then matched with the Triple J song list (PDF) and saved. I removed any duplicate votes I found, that is people who voted for the same songs in the same order when there are greater than 3 songs in the image (a very unlikely occurrence). I figure these are probably the same person uploading the same image twice.

This is an initial cut, there’s still some extra work to do including:

  • Manually add songs that would be in the hottest 100 to the song list
  • Tune the OCR, including doing some pre-processing to images if needed
  • Tune the matching algorithm – currently using Levenshtein distance
  • Do more analysis on voting combinations (e.g are there factions who vote for particular songs together and what can we learn from this).
  • Make the table pretty like the other ones.
  • Make a form for people to upload their own predictions and show a leaderboard as they come in on the 27th.

The results are quite different to 100 Warm Tunas – I seem to be picking up more votes. I’m not sure if this is due to some sort of filtering I’m not doing or just algorithm differences, but we will see if 100 Warm Tunas still is the internet’s most accurate prediction of Triple J’s Hottest 100 for 2017 on January 27!

This table is updated automatically every few hours.
Total number of images: loading…
Total number of duplicates: loading…
Total number of votes: loading…

# Title Artist Votes % Votes Inc dupes %
Loading… Loading… Loading… Loading… Loading… Loading… Loading…

Routing certain IPs over VPN with DD-WRT without IPTables

I decided I wanted to be able to route certain devices on my network over a VPN connection for reasons that I am sure you can use your imagination (geo-restrictions etc). I didn’t want everything to go through the VPN because that would slow down my connection for things I didn’t need it for.

It’s worth noting that before this year you could just use some fancy DNS tricks to route only traffics from a certain domain over your VPN, but I found this failed on devices with hard-coded DNS (like the chromecast or the Android Netflix app).

Media devices like Smart TVs and Chromecasts can’t run OpenVPN so it has to be done on the router. If you want to do this, make sure your router is up to scratch. Encryption uses processing power which most routers lack. You want to be getting minimum 5Mbps with a recommended 10 for this to be usable. I forked out for an R7000 which is probably overkill. Another option is choosing a VPN provider (or setting up your own) that enables you to use weaker encryption – the idea being that it doesn’t really matter that the NSA can snoop on your netflix, it’s up to you.

My first idea was to have a separate WLAN (Wireless LAN) with it’s own subnet and DHCP and route all connections through the VPN. That way you could choose to go over the VPN just by switching networks. I’m sure there is a way to do this, but I couldn’t get it working with my limited knowledge of dd-wrt and iptables and the like. Issues I ran into went from not being able to access the other local subnets (which I wanted to for things like Plex) and just generally getting it to play nice.

So I scrapped that idea and moved onto the next. Give every device you want to route over the VPN a static DHCP lease (i.e their IP doesn’t change) and then use the Policy Based Routing field to tell the router to route internet traffic over OpenVPN. This worked perfectly. The only catch is with Chromecast your mobile device also has to be over the VPN or you won’t be able to see the geo-restricted content. If you don’t always want your phone to go over the VPN for wifi then you could use a cheap tablet as a Chromecast remote OR install OpenVPN on your phone and only connect when you want to access geo-restricted content.

OK so here is how you do it. These instructions assume that you have set up your router to the point of having an internet connection and a single subnet with DHCP turned on.

  1. Put your devices on a Static Lease
    Go to Services > DHCP Server > Static Leases
    Add each device one at a time, pressing save and apply after each time. Note that the hostname doesn’t really matter here, MAC Addresses do and I found some of the hostnames made nothing resolve so if there are any special characters in your hostname just name it something else.
  2. Set up OpenVPN
    Instructions will be different for each provider. OpenVPN is under Services > VPN > OpenVPN Client.
    The only deviation will be that you don’t want to redirect your gateway so remove redirect-gateway from the additional commands
  3. Add you IPs to OpenVPN Client config
    Under Services > VPN > OpenVPN Client > Policy based Routing add each IP in the form of X.X.X.X/32 with one per line. I put both my Chromecasts and my TV on it as well as my cheapo tablet that I use solely for Plex/Netflix.
  4. Bingo. You’re done. No telneting, no iptables no messing around.

I wish I had found this earlier and maybe I would have saved myself some messing around.

University Portfolio

I think one of the most important parts about studying computer science or software engineering at university is that it gives you the ability to slowly build a portfolio of small pieces of code which can demonstrate what you are capable of. I have embarked on a project over the last month to collate all of my significant university programming assignments. This is a general snapshot of what you learn in a CS degree these. If you are a student – don’t plagiarise my code for your assignments, you’ll get caught and lose your marks. It also violates the license on the code (GPL v2.0) where you must reference the author. If you’re a lecturer – I hope this doesn’t bother you, you should really be changing the assignments every semester anyway to allow for things like this :).

The projects are listed below. I haven’t included all projects as many were fairly trivial and not all computing units assess with programming assignments (much to my annoyance). The code has not been updated since it was first written, so please take note that generally my style of coding has evolved since first learning 4 years ago:

First Year

Second Year

Third Year

 

 

Youtube Song Downloader

A little project I worked on for the last day was getting a program to make downloading a list of songs off youtube easier. Initially it was just going to be command line, import from a CSV file. But this only works when you know the first hit will be the correct song. So I decided to flesh it out into a GUI.

This was inspired by looking at the sexy lists on the Triple J Hottest 100 Wikipedia page and deciding there should be a way to grab all those songs easily.

Unfortunately, the Google API restricts this kind of thing. But I’m sick of this project now, so here it is. You’re limited to downloading about ten songs at a time, otherwise you get service abuse messages. Hopefully in the future I can find a better method of searching for songs.

Youtube Downloader

Blackboard Scraper now has stand-alone download option.

I have finally gotten around to making the Blackboard Scraper stand alone so you no longer need to install lots of different things to get it working.

Hopefully this makes it easier for non-computing students to access and use. It still only works on Curtin’s blackboard system, however a UWA one is in the works.

Head to the Blackboard Scraper page to give it a whirl.