AI ROI

The backlash to “tokenmaxxing” has begun. In the earliest wave of AI adoption, we needed leaderboards to incentivize employees to spend the time to bring AI into their workflow. As internal token bills have risen to 6 or 7 figures for most organizations, many are looking for ways to separate AI value from AI spend. Yet there are immense challenges for most firms to do so!

• Coding is the domain where ROI is most easily measured, for the same reason it’s the easiest area for models to improve. Correctness is directly measurable through tests, compilation, and running the code, resulting in tight feedback loops and iterative AI loops until the task is complete. Most business tasks do not share this “verifiable reward” structure naturally.
• Switching to lower cost models does generally incur a reduction in model performance. There are few ways to determine what level of intelligence is needed for a given task.
• Employees are the AI adopters, and their willingness and capability to use it is often beyond management’s vision or control. Some are eager and ask frontier models to craft every response in Slack. Others are reluctant adopters and haven’t spent the upfront cost to learn how to take a blank chat window and get value out of it.
• Each platform offers a spend dashboard but little beyond that to understand how or why that spend occurred.
• As the number of products has proliferated from chat to code and cowork, users are also blind to how different modes of interaction affect spend.

On top of this, we see a widening gap in firms that build internal tools that create optimization loops that recursively compound value. Ramp announced first a background coding agent then a fully enabled AI coworker to great fanfare and jealousy for many organizations. For those working in Chat and Cowork primarily, these compounding loops are hard to replicate, resulting in a yawning gap in competitiveness.

Building your flywheel from chat

Chat is more unstructured, iterative, and individualized. But every employee is still doing hundreds of tasks in chat every day, most of which define the core of what your business actually does. There is real value in those logs worth unlocking. I’ve built an AI tool to identify all of the tasks performed within a firm and optimize the highest value tasks to produce reliable output while eliminating wasted tokens. The core idea is to build a reinforcement learning loop for each firm, automatically:

• Create a hierarchical task map across conversations to see how AI spend goes into actual work
• Identify positive and negative examples of AI’s implementation of that workflow, constructing both rubrics and synthetic examples.
• Train a discriminating judge to learn those preferences across all firm tasks.
• Optimize performance on each task, producing a guiding Skill file that gives consistent guidance available to AIs used by any employee in the firm.

It all drives higher transparency in how AI spend drives firm work and the ability to own the core knowledge of the firm in a form that makes it possible for any employee to perform tasks at the upper tier of performance.