DoOO Data Considerations

This is for the Domain of One’s Own Roadshow workshop/conference thing. Reclaim stuff is always fun and I should have gotten this down earlier but I didn’t so I’m just trying to get down some ideas on assessment and data.

I read it now and I’m kind of bored myself but overthinking the value of posts is why so many people don’t blog. With that . . .

Dearest Web Log,

My normal warning about measures.

When a measure becomes a target, it ceases to be a good measure.

Marilyn Strathern (making Goodhart understandable)

Claims

What are you saying that your DoOO will do? Why did people agree to spend money to do this? Who do you need to convince ?

If you look across the DoOO homepages, you will see lots of statements about identity, digital fluency, ownership, experimentation etc. Some of these are easily defined. Others require more digging and definition. What is digital fluency at your institution? How do you break it down? You have to get specific and it’ll lead to annoying conversations but useful ones. It often exposes different beliefs in your group. Specificity will lead to statements that have some hope of being measured.

I took a bunch of the main statements from DoOO sites and put them in a spreadsheet if you want to wander around looking at other people’s statements. It’s also in Voyant.

How do you know that’s happening?

You said the DoOO would do x, y, and probably z . . . how do you know that or are you just assuming?

This is usually the trickiest part and it’s heavily influenced by personal and institutional beliefs around proof and data. I think it’s also important to keep in mind that the data you use in conversations with supervisors or other outside groups can end up being the tail that wags the dog. So I suggest thinking hard about internal data (where you can look at more and add/remove as it makes sense) and thinking extra hard about external data which will be harder to change and plays a major role in shaping how people think about and understand your initiative.

Another complication is that you don’t want to become like the data-hoarding-social-media sites that you’re often encouraging students to escape. What’s the least data you can collect? How do you keep it? How long do you keep it? How do you anonymize it? What kinds of usage are ethical? Having some policies and structures there will help you avoid drama and maintain sanity down the road.

Quantitative/Qualitative

Most places talk about “data-driven decision making” but are swayed far more by anecdotal narratives. Same with most people. There’s power in the stories but balance is useful. If I’ve got three people doing amazing work and 3,000 abandoned sites then I need to know that. It also brings up the complexity of scale and determining what “abandoned” even means . . . let alone how to determine it across so many sites. There are ways but they need to be built and part of any good assessment plan is looking at whether it’s sustainable for you.

Some example scenarios

If you’re trying to figure out if people are experimenting, how would you know this? You might look at Installatron data for applications installed and deleted. Seeing short-duration installs might be indicative of a healthy culture of experimentation. It could also be that experiments are happening directly (which is awesome) but isn’t going to leave much evidence. Maybe you look at the total installed applications.

Say you wanted your DoOO to teach digital fluency. You will probably have to extrapolate that as you’re unlikely to get pre/post skill tests. Maybe you offer workshops that teach digital literacy through the tools available in your DoOO. This is also a good time to look at partnerships with departments that will be assessing their students. Make partners that say they’re teaching digital fluency. Not only will you have the opportunity to talk about assessment with them but you’ll be tying your DoOO into the curriculum. My goal was to have as many institutional connections as possible. I wanted an army of infuriated people if you tried to take this resource away.

If you set the number of accounts as the quantitative definition of success then that can lead to 3,000 accounts with little actually happening. You can spend all your energy getting people to sign up while losing sight of the actual goal. That’s easy to do when so many organizations are focused on charts that always go up¹. It may be that the rapid growth makes a nice chart but exceeds your support structure. That leads to bad experiences and abandonment of the sites. Students end up with a bad taste in their mouth and only use Instagram for the rest of their lives. You don’t want that.

Money is almost always part of the conversation. So showing how the price of the DoOO compares to an internally hosted project of similar complexity is one metric that’s useful. IT groups are usually happy to have this offloaded so asking them for a budget estimate can usually result in a pretty high number. I would also compare the DoOO to the ridiculously priced student portfolio products out there.

Showing

We are usually serving a couple of audiences with different goals so we’ll want different messages and different evidence.

Think hard about how you want to capture and show the powerful stories. Aligning them to your goals (and then upwards to institutional priorities) is very useful. Consider what a faculty member looking for innovative course designs would want vs a student trying to figure out what a good portfolio looks like. These narrative collections also give you a chance to reinforce the language of your goals while providing understanding, proof, and inspiration. That combination is worth investing time in. There’s a chunk of this that just isn’t going to be automated so deciding where you want to spend time like this is important.

Giving people a path to submitting their own examples like Berg Builds does is useful for a couple of reasons. It’s opt in so it save you a step. You are also sharing out the work. One person adding 20 examples is a big pain. 20 people adding their own examples (even if it’s not perfect) is far less work and you can build on top of that effort, course correct things etc. Think about categorization/tags so you can show found sets in ways that make sense down the line. If portfolios are a major focus, you can have a page that presents the idea for faculty and includes examples associated with disciplines while using those same examples with different context for students or for some select examples for admin reports. Keep your examples portable and you’ll save huge amounts of time and effort.

Now administrative evidence will overlap with some of these examples but will contain different arguments. In both cases, I assume the person is busy and is skimming. So thinking through how you present the data for scanning is important. Often libraries do a really good job of creating reports like this for their work. It’s worth seeing if your library does and then imitating them. If not, take a look around the Internet.

If you’re still with me, I’ll walk you through a rough draft we made for administrative report below. You can see we are claiming that our WordPress Multisite is doing a few things. We lead with scale. I wanted people very nervous about upsetting (at that time) 26,000 people.² Student portfolios were a lead argument because it tied us into 12 or so departments. That’s a lot of angry people if this resource went a way and lots of time would be required to make the curricular changes to move to another platform. It also let me compare prices with outside portfolio vendors. Since Ram Pages gave students 2GB of storage, didn’t delete accounts on graduations, included paid plugins . . . the comparisons were pricey! Anyway, you get the idea. We also had plans for a way to dig deeper into money related to OER. Money talks. Students savings wer a strategic priority. Line up those connections. Speak the language the admin understands.

See the Pen
ALT Lab R&D by Tom (@twwoodward)
on CodePen.

It’s worth noting that this took a good bit of effort . . . and got zero interest from at least one supervisor. It was a preemptive defensive action on my part in case someone wanted to get rid of Ram Pages. So maybe I have no idea what I’m doing. In our scenario, it turns out no one cared about the relatively trivial amount of money Ram Pages required. They just wanted charts about online programs and courses. Cool. Cool. But if they did care, we’d have built workflows that constructed those reports incrementally rather than through one-time-herculean efforts. That pattern usually happens for me. The first time is custom and lots of hassle but with an eye on automating for the next round. Round two, we have a frame that’s being filled in by other actions and from there we can keep refining and add/remove arguments.

Events like OU’s Creaties are worth imitating for lots of reasons. It gets cool examples given to you rather than having to look for it. It reinforces community. It can give a place for admins to see things and talk to people in a way that is incredibly influential. We had something similar (H21) when I worked in Henrico County for 21st century lesson plans and it’s probably the only thing I had anything to do with that they’re still doing.³

Technical Stuff

Now that you’ve sketched out some goals and thought about proof and audience . . . let’s take a look at where existing data sources live. If you haven’t looked at the DoOO documentation around analytics here, then go do that. I started there and went a bit deeper on a few things that were mentioned.

The long term goal here is to build a dashboard that looks across a couple data sources. Likely it’ll mix some pragmatic data (is someone eating all our storage/processing/bandwidth?) with more aspirational/strategic data (what’s our app installation breakdown? anyone getting massive external views?). We also have the ability to take data that’s hard to digest in the native views in WHM and make it into dataviz. How much better would disk usage be to understand in a treemap?

installatron database

It lives at /var/installatron/data.db and is SQLite. You can see it mentioned in item 5 on this page.

It has a four or five tables and I started to write out the tables and the columns but it was really boring and I already have a PDF with the schema. Anyway, take a look at what’s in there. I think most of what you want is going to be in i_installs. I exported those to CSV using SQLite Browser and then started messing with it in Google Sheets. You’ll see there are backups and updates and some other things in there that you’ll want to sort out. Once you have done that, it’s a decent way to look at installs over time. I added a little custom function to my Google Sheet to try to filter out dead domains and uninstalls. It’s not overly robust but it points towards making this automated down the road.

function checkDomain(input) {
  var response =  UrlFetchApp.fetch(input,{muteHttpExceptions: true, followRedirects : false});
  if (response.getResponseCode() == 404 ||  defaultPage(response)) {
    return 'MIA'
  } else {
    return 'OK!'
  }
}


function defaultPage(url){
  var response = String(url)
  if(response.includes("/cgi-sys/defaultwebpage.cgi")){
    return true;
  } else {
    return false;
  }
}

yourDoOOName_last_logins.csv

This is created by a python script located at /root/last_logins_py.py

It saves the data to the root directory and contains the following information. It’d be nice if you want people in cPanel (and are doing things to encourage that) but it likely doesn’t give you much of a view into usage given most people will install apps and enter through that path rather than using cPanel.

• Data
• Date of last cPanel login
• cPanel Username
• Domain Name
• Email Address
• Disk Usage for Account
• Date of Signup

It does point to ways you can script data dumps using Python and run them via chron tasks. In this case it’s CSV and runs nightly but it could be JSON or you could get fancy and create a central DB that writes data from multiple sources.

WHM/WHMCS APIs

You also have the option to use APIs with WHM and WHMCS. This’ll be a better documented path to get at information in these systems.

Custom Stuff

After getting the right mix of data that’s being recorded already, we start to think about what might need to be customized. Like for a WP site, we can look at last_modified from the posts and pages API endpoints and know if the site is actively being authored. It’s important to realize the limits here though. Maybe it’s done and it’s being used but not authored. Maybe new posts are being created but it’s done through syndication and the person has nothing to do with it.

Within major application categories, we can probably do stuff like that but each check for activity will be custom. So you’d set some sort of checker based on centralized data (if wp then use this function). For the more complex things, like Node Apps or plain HTML, we have to figure out other indicators of activity. Likely, I’ll dedicate focus on our installs based on percentages and work my way to outliers.

My focus is on improving data views and realizing they are flawed and will always be flawed. We can keep improving them but it’s not like this is medical equipment. We don’t have to be perfect. We’re just trying to provide ways to think better about what we’re doing.