Re-Scraping Instagram

Back when Instagram’s API rules didn’t completely suck, I wrote a few posts on scraping it so that some of our faculty could use those data in their research. Then all their rules changed and everything broke. That’s their prerogative but it’s also my option to complain about it. But because I posted about it, I got a comment from raiym1 who let me know he wrote a PHP scraper that avoided the API limitations. I’ve now got that up and running and set up a simple GET so that the URL determines the tagged content that is returned. The PHP for that page is below and allows you to replace the API URL in the old Google Scripts with a new url like http://bionicteaching.com/creations/ig/scrape.php?tag=fish You can then make your own custom displays based on that. I made a quick custom page template for the artfulness WP theme (currently showing filler data from the exciting ‘fish’ tag). This example has the tag hardcoded in but could easily use a custom field to pass the value. 1 On this post. And apparently this theme doesn’t support direct links to comments. About time I wrote my own theme . . .

Grabbing JSON

flickr photo shared by Library Company of Philadelphia with no copyright restriction (Flickr Commons) What I wanted to do was grab data from the WordPress API and use that to provision chunks of my new portfolio site. The portfolio is hosted on GitHub and GitHub is HTTPS. At the moment my bionicteaching site is not HTTPS.1 That causes problems as secure and insecure are not friends. I wanted a quick and easy solution so I could continue until I do the HTTPS switch. The following is how I wandered towards a solution. A number of the things worked but don’t quite work for what I wanted. So they’re worth remembering/documenting for later and it’s kind of fun to see a mix of javascript, php, url manipulation, Google API, and the WordPress V2 API all in one little bit of wandering. My first thought was to grab the JSON via a Google Script and store it in Google Drive. I can do that but can’t seem to make it available for use the way I want. I tried messing with various URL parameters but it wasn’t flowing and I only started there because I thought it would be easy. I did eventually get the file accessible in DropBox (the only other place I could think of immediately for https file storage) […]

Many Paths in Programming

flickr photo shared by San Diego Air & Space Museum Archives with no copyright restriction (Flickr Commons) Minor Thoughts on Computational Thinking Probably obvious stuff but I’m trying to jot things down for my own reference. The first thing one ought to know about computational thinking/programming is that there are many correct paths (although some are better1 than others). This is true for just about anything but I think people think technology will be much more . . . binary. Searching for cleaner paths can be kind of fun. Computational thinking is powered by vocabulary. Vocabulary, like in language, is closely tied to concepts (maybe analogies). Having never heard of the range function, it didn’t occur to me that it existed . . . let alone that I should use it. To make it work properly I need grammar but just knowing the word exists and means something starts to change things for me. It brings to mind setting up programming challenges much more like Dan Meyer’s 3 Act math lessons . . . with the scenario really begging for the addition of a particular concept but letting students struggle with it rather than providing it ahead of time. A Path This is a little bit of real-life progression which demonstrates how one thing can be done in a variety […]

Grabbing Flickr Photos

flickr photo shared by goosmurf under a Creative Commons ( BY ) license This one will be improving considerably in the near future but given I’ve just been talking to many interesting people about APIs, reclaiming various things, and Indie-Ed Tech1 I figured I’d get it out early and that’d force me to follow up. Nothing like ugly betas to drive development. It’s also a chance to test my new blog to Twitter system as I disentangle myself from IFTTT. Nothing kills momentum like not doing something . . . So this script currently works on public photos. You’ll need a file named imgs and a file named data.json. This thing should chew through all your photos and download the original size image to the folder. It’ll also make a giant json folder with the image title, any lat/long coordinates, tags (not machine entered though), and the photo date. I will warn you that I’ve only run it on 100 photos so far. I’ll give the full thing a shot once I get things setup to put it on S3. 1 Count down to book . . .

20

Grabbing FaceBook Comments – More Primitive Programming

Functional is a kind way to describe this . . . but it’s hard to argue with computers doing tedious work that was previously done by hand. The goal here was to automate the collection of the comments on 300 or so posts from the CDC’s Facebook page so they could be analyzed. Ebola is in the mix which leads to some comments that may very well rival YouTube comments in terms of causing me sadness. We did end up with around 34,000 comments which would have been hand copied in the past. So some karma balance was achieved. wTake all this with a grain or two of salt. It works but may very well be illegal in some states or do things in a way that would make real programmers weep. Consider my programming skills to be like a man trying to swat flies with a broomstick- lots of furious action, panting, cursing, and every so often I will kill a fly and celebrate wildly.1 The Facebook This was my first experience trying to do anything with Facebook. The hardest part here for me was just getting the stupid access token needed to get at the JSON feeds. I eventually found the secretly named Facebook Login Example. Part of my issue was that I thought I could skip the […]

Flickr Group Widget – Round 1

flickr photo shared by Little Orange Crow under a Creative Commons ( BY-NC ) license The goal here was simply to take the Flickr API knowledge I’d gained earlier and apply it within a WordPress widget. In doing so, I learned a few things. The primary one being that I often harm myself by being fairly good at making stuff work rather than understanding what I’m doing. That’s what I did here. The Flickr API thing was pretty legitimate. I knew what I was doing there but I pretty much crammed it into a widget plugin template without really understanding the whole thing. That led to more confusion around print

results of Flickr api group request

Flickr API Basics – Pulling a Group Feed

Someone at work mentioned that the Flickr WordPress widget didn’t work with group photos. We use this group as part of our (sort of just getting started) Friday photo walks. Since I’m trying to learn stuff I figured I’d play a bit with the Flickr API and see how that worked. After getting an API key, my first stop was at URLs. I thought that’d mean endpoints like on Instagram but it was the URL structures for photos. Useful and needed but not what I thought it’d be. I then saw in the right hand sidebar groups and browse. That turned out to be a list of groups. I ended up finding what I wanted at groups.pools get photos. I planned to get the information in json if I could given I’d just spent some time figuring out how that worked so I went back to the main index thinking the json response format would show me how to ask for json data given the variables I’d seen as options in groups.pools.getphotos. It didn’t. This section shows the structure of the json response. Which makes sense it just wasn’t what I was looking for. I found what I was looking for under request.formats which makes sense. This shows the URL structure. https://api.flickr.com/services/rest/?method=flickr.test.echo&name=value is the basic URL. Method is the thing […]

Scraping Instagram – Take 3

flickr photo shared by JeepersMedia under a Creative Commons ( BY ) license I started to apologize for writing three posts on this and promising not to do any more but I reconsidered. This is my site. I’ll write whatever I want. Skip it if it bores you or exile me from your feed reader.1 Alan’s comment got me thinking that using spreadsheet formulas was not necessary and it felt awkward to me anyway. So I figured out how to do it all in the php. I’ll include the relevant portion of the code below. You can get the whole thing here. The key is substr_count which will find stuff in a string and count it. The other little piece is boolval which returns true if it’s greater than 0. 1 Plus no one reads blogs any more. Shouldn’t you be on Twitter or vaping?

Electable? – Freedom Through Randomization

Try the POC here. Talking to Britt and John “Aughie” Aughenbaugh (who will be teaching an intro to government course online this summer), Aughie mentioned a conversation he had with some students about whether Chris Christie could be a legitimate contender for president after all the recent bridge drama. That led to some talk of other historical political drama and how things were slightly different back then. It led to some wondering about how previous presidential candidates/winners might fare in today’s 24 hour news cycle (AKA the CNN Effect). The rough idea was to build a randomizer that would present three randomized US presidents with three randomized traits/characteristics. Students would be able to argue whether a person with those traits would be electable. Clearly, I like randomization but it’s more than just the chaotic serendipitous aspect. I think it adds an element that makes it “safer” for students. It is random after all, there can’t be a specific answer the instructor wants because the question can’t be totally predicted. Plus, it takes work off the instructor and adds variety to the responses since everyone gets different prompts. I had this meeting Wednesday and wanted to get a semi-decent looking proof of concept built fairly quickly. I knew I wanted images of the presidents and figured I could re-use some of […]

22

Revision to Twitterbot Monster

So tying the trigger to the Calendar works . . . but only if the computer is not asleep. I had considered this initially but may have over estimated how often my computer was running. It was done more for amusement anyway so here’s a way to move the whole email/trigger piece to the server using PHP. It replaces the whole AppleScript portion of the previous iteration. I had no idea how to do this prior to Google leading me to the examples that are linked below. The first chunk runs the cURL portion and gets the Markov response from the other page and passes it to a variable- creatively named $DICKINSON in this case. I left the echo line in there as I used it to make sure things were working. The second chunk sends the email. You can see the $DICKINSON variable being passed in as both the message body and the subject. The final piece is triggering this page every so often using a cron task. I am not yet positive I’ve done that right. When I’m more sure, I’ll come back and update this.