Identify, solve, verify

My job is to identify problems that can be solved with code, then solve them, then verify that the solution works and has actually addressed the problem.

A more advanced LLM may eventually be able to completely handle the middle piece. It can help with the first and last pieces, but only when operated by someone who understands both the problems to be solved and how to interact with the LLM to help solve them.

No matter how good these things get, they will still need someone to find problems for them to solve, define those problems and confirm that they are solved. That’s a job – one that other humans will be happy to outsource to an expert practitioner.

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity – Apple Machine Learning Research

Through extensive experimentation across diverse puzzles, we show that frontier LRMs face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counter-intuitive scaling limit: their reasoning effort increases with problem complexity up to a point, then declines despite having an adequate token budget.

Scrappy: make little apps for you and your friends

Scrappy is an infinite canvas of interactive objects. The workflow is similar to an app such as Figma, Miro, or Google Slides — except you can attach behaviors to the objects.

You drag objects out on the canvas — a button, a textfield, a few labels. Select an object, and you can modify its attribute in an inspector panel. Certain objects, like buttons, has attributes like “when clicked” that contain javascript code. When the button is clicked, that code is run — maybe it records the contents of the textfield to a label that acts as a log. You build your app step by step: tweaking and rearranging the objects, and attaching a little bit of code to them.

—in the realm of explorable explanations and other good things