hidden_variable

This post was partially inspired by cahn’s review of Emily Tesh’s Some Desperate Glory, and some of the resulting discussion about AI. As I mentioned in my comments there, my experiences with AI over the past few months have mostly been very irritating, thanks to people trying to use ChatGPT as a cheating tool in my classes. I wrote some exam problems in which I asked the students to evaluate ChatGPT’s responses to physics questions, and found that the chatbot itself seems incapable of evaluating its own output in this way—it either claims the solution is fabulous and perfect, or just says, “As an AI language model, I am unable…” My assessment of ChatGPT’s AI capabilities aligns pretty closely with this blog post, which I’ve already seen shared in a lot of places, but is worth a read if you haven’t seen it. The author is a history professor, so the examples are in a very different context than what I’ve seen, but many of the general observations and conclusions still apply. Here’s some more detail on what I’ve seen in a physics context.

ChatGPT physics is highly recognizable, and bizarre. First of all, it has a distinctive writing style I’d describe as “bad undergrad paper trying to pad its word count.” It’s very fond of meaningless filler sentences that begin with “Overall…,” or “In summary…,” which is very weird to see on a timed physics test. As a cheating aid, it’s not particularly impressive—not nearly as reliable as something like Chegg, where humans are illicitly doing the problems. (As a side note, it’s kind of darkly amusing to watch the interplay between Chegg and ChatGPT. Apparently competition from ChatGPT has caused a 40% drop in Chegg’s stock, while a large percentage of Chegg postings now open with statements like, “Work must be correct! Do NOT use ChatGPT!” It’s not quite an “enemy of my enemy is my friend” situation, more like, “I hope my enemies both manage to inflict serious damage on each other before one of them ultimately perishes.”)

If I were to score its exam work in the way I normally grade, I think ChatGPT would typically earn itself something around a C minus. Its strong point is that it’s fairly consistently good at solving problems requiring only one essential step to get to the answer—even if that step involves doing some math. Example: given an electric potential as a function of position, it can calculate the electric field by taking the gradient. I had a problem like this on one of my exams last semester, and ChatGPT smoothly handled every variation I handed it. The original problem had just a simple polynomial function; then I decided to try giving it functions whose derivatives are more annoying, like inverse trig functions, which still didn’t seem to give it any trouble. At this point, I started to wonder whether it actually “knows” how to take derivatives. This seemed like not an unreasonable possibility: software that can do symbolic manipulation, such as Mathematica and Matlab, has already existed for decades, so why couldn’t ChatGPT take advantage of that capability?

So I flat-out asked it: “Do you know how to differentiate functions?” Its response included an unforced error—what one of my high school math teachers used to call “Unnecessary Display of Ignorance”:

( content warning: bad math )

If it’s not really doing calculus, what about arithmetic?

( way too much detail on ChatGPT arithmetic )

Going beyond straightforward, one-step problems, ChatGPT is remarkably bad at doing any sort of problem that involves multiple steps, either conceptual or mathematical. It’s very good at reproducing the surface appearance of physics. It does great solution-manual-esque prose (“Substituting the result into Bernoulli’s equation, we find…”). It meticulously defines every symbol used in every equation (even if it already defined the same symbol two sentences ago). It writes out vector components of Newton’s second law in a systematic way. It even occasionally draws ( weird free body diagrams using keyboard symbols )

But the actual content of what it’s saying is wildly inconsistent, with the physical world and even with itself. I’ve repeatedly seen it state, in both words and equations, that gravity pulls sideways. It will say things like, “The tension in the string cancels the horizontal component of the weight.” And my favorite example was its response to a mechanics problem that required conserving both energy and momentum. The whole thing was mildly hilarious in terms of wrong physics, but my favorite part was ( the conclusion: )

Also, it’s terrible at circuits—not surprising, because it cannot parse diagrams, so it doesn’t get any information about how things are connected in the circuit. To make the test fairer, I tried replacing the circuit diagrams with verbal descriptions of the circuit—imagining I was writing alt text for someone who couldn’t see the diagrams, and trying to make it as clear as possible. Which… didn’t make any difference at all to the quality of the solutions. This really brought home to me that ChatGPT is not using or retaining information in any way that would conceivably define it as “intelligent.” Circuit problems are nearly always presented with diagrams, so its training material just didn’t have any examples like my verbal circuit descriptions for it to draw from.

Finally, just for entertainment’s sake, I asked it if it knew any good math jokes. Two iterations of this question produced a total of ( 13 jokes: )

In conclusion: I think we can all safely cross off “super-intelligent AI destroys humanity and takes over the universe” from the immediate-worry list.