This is an interesting example to me for a variety for a variety of reasons. One of them being that I don’t really know enough about graph theory to evaluate the responses. That’s kind of fundamental, but I made the effort anyway. I did learn some as I went on and it does also reinforce the idea that the AI tools, models, and interaction patterns for different disciplines/courses can, maybe should, be really different.
The process moves upwards from some prompting strategies in basic general LLM upwards to more complex options. My goal is to move them between different LLMs so they can see how different tools have different options. It’s tough to speak in generalities here as you have a mix of model behavior/capacity mixing with UI options. Claude runs its artifacts in a publishable iframe sandbox for html/js/css while ChatGPT now runs a Python sandbox (that I don’t think is shareable). Even simple things like how LaTeX gets processed (or doesn’t) is different between systems. That all makes for very different experiences and different choices based on what you want.
Claude
The ability to make small, interactive tools to understand different elements (in Claude in particular) is pretty interesting to me. Would it be better if you could program to do it? Probably. Maybe? I don’t know. Can you make a tool that helps you understand a thing you don’t understand and know whether it’s helping you or not? I would really like to be able to embed those artifacts in this page but they’re all linked in the presentation. I’ve always like explorable explanations. The idea of a low code/no code path to that is attractive to me.
chatGPT
This section is an attempt to hit the mode choices (o1-preview), GPTs (Wolfram), and a custom GPT (RAG, a few prompt tweaks)- all elements that might impact math usage and effectiveness. That level of complexity in a single choice is a headache to explain. For the custom GPT, I put in some things about questioning patterns that I think might help in addition to a bunch of OER graph theory textbooks.
Notebook LM
This is basically just another version of a custom GPT but with a few different UI options and a audio summary creator. I threw a variety of file types in there just because I could. I shared editor level access with the professor and set the students as viewers.
Lean and focused LLMs
It’s really difficult to provide guidance when the AI starts blending with other tools that I’m not familiar with. There’s just so much stuff out there. I think a decent path would be to delve into Lean and how theorems are proved in combination with Lean. There’s a decent Scientific American article on AI as mathematics assistants. Deep Seek seems to have a math focus and provide some interesting responses. I do think the benefit is likely to be found in more fringe technologies. It makes support and guidance more difficult from outside that disciplinary perspective. Can outside people guide the better uses of any technology in discipline-specific contexts? They certainly can’t do it as well as someone with that deep disciplinary knowledge.
What of it?
I really don’t know what this looks like in the long run. We’re going to talk about AI again in our department as part of the Digital Detox we do each year. I’ve been talking to my kids about aspects of this. I’ve been talking to k12 educators. It’s a really messy space in so many ways. The energy usage/water usage is far blurrier than I think most people appreciate. It brings up lots of other questions about how and when we measure things like this. Do you compare AI interactions to Google queries? You might if that’s what you’re using AI for. If you’re making an image, what do you compare? If you’re putting AI as the thing that pushes us over the edge of ecological disaster, do you compare AI vs old data centers vs electric vehicles or vs the massively larger electricity usage that’s outside of data centers? I’d link to my energy/water research links but that’s not working currently.1
I am not happy that companies are making a profit on the work I’ve done on the Internet but that’s also the scenario we’ve setup in all the other ways so it seems crazy to think this would be different or to get extra upset about it here but not on whatever social media service.2 Replace public internet content with publicly-funded research, publicly-funded infrastructure, tax breaks for whatever industry, bailouts for whatever company etc. etc. Be mad, just be mad at all the other stuff. Be madder. If you make an LLM out of public domain data, does that alleviate the problem?
I also feel like the ownership of ideas in the particular way we seem to view them these days is not great. Citations are weird. We ought to question our goals there. I get citing people to give them credit. I get citing things so they can be verified. If you don’t cite things that are common knowledge, is anything/everything in this giant repository underlying the LLM common knowledge? I’m not sure what you’re doing when you cite a non-repeatable element that was created by your input into a massive math equation designed to not repeat the same outputs. Is this a monkey selfie? Is it more of a camera trap? Where is the human action and intent vs the outcome?