ESRI Scraping

ESRI is sunsetting the Classic StoryMaps. Apparently that was announced in 2019 so they’ve been pretty generous. They provided a link to the content that is going to be retired. I couldn’t figure a decent way to export that content and copy/pasting was a mess.¹

It is HTML though and that means I can take some javascript and run it in the console to grab that information. This gets a bit trickier because of shadowRoots.²

const items = document.querySelectorAll('arcgis-item-browser-table-row');

const data = Array.from(items).map(item => {
  const shadow = item.shadowRoot;
  const title = shadow.querySelector('.checkbox__title .cell--clamp')?.textContent.trim();
  const link = shadow.querySelector('.cell--link__container a')?.href;
  
  const userPopup = shadow.querySelector('arcgis-user-popup');
  const userAvatar = userPopup?.shadowRoot?.querySelector('arcgis-user-avatar');
  const owner = userAvatar?.shadowRoot?.querySelector('.title')?.textContent.trim();
  
  return { title, link, owner };
});

copy(JSON.stringify(data, null, 2));//copies the JSON to my clipboard

I have to do it for each page and do some minor combining, but once I have the content, I can do with it what I will. In this case, I took the combined JSON and then ran it back out to make the HTML I wanted to paste into an email.

const destination = document.querySelector('#list')

data.forEach(item => {
  const title = item.title
  const link = item.link
  const owner = item.owner
  const html = `<li><a href='${link}'>${title}</a> - ${owner}</li>`;
  destination.insertAdjacentHTML('beforeend', html)
})

The HTML is very exciting.

<ol id="list"></ol>