Which Industries Survive AI, The New AI Benchmarks, and the 2026 Recursive Learning Timeline | #218

Matt details that the fundamental challenge is not the technology but the organizational and data architecture. Most companies' tech stacks are decades old and their data fragmented. He urges starting with a value-first mentality: choose the 2-3 things that materially move the needle — customer service, FP&A forecasting, inventory management, digital marketing. Then, rather than writing a strategy, get a working prototype in weeks. He stresses that the machine learning paradigm of building for months then underwriting statistically is inverted here: you can prototype fast but must test relentlessly. The failure mode is when the initiative sits in the tech org without an operational KPI; the fix is to put your best ops person in charge. Finally, he recommends an RFP to a third-party vendor compensated on outcomes (like Invisible) because it limits risk and holds someone accountable for results. This roadmap is his direct advice to any CEO.

Matt explains that most enterprises lack a way to say 'it works' for generative AI because they don't have a baseline. For contact centers, you can measure time per call, CSAT, cost per call; for mortgage underwriting, back-tested credit decisions. But for tasks like generating an investment memo, formats vary wildly. The custom eval process involves collecting human-generated outputs for the exact task, then measuring the AI's output against that. This becomes the foundation for scaling. He suggests that whoever builds the benchmark for an industry vertical becomes the instant authority, creating a massive entrepreneurial opportunity.

Matt brings up the Jane Street example: a trading firm will never give its proprietary data to OpenAI because that data is the entire source of alpha. But their back-office FP&A data might be less sensitive. He cites banks with 300 customer databases where internal silos exist; segmenting which parts need air-gapped AI treatment and which can use cloud models is the practical way forward. He also notes that small language models are an option for on-premise sensitive use cases. This segmentation avoids the all-or-nothing trap.

Matt discusses the Klarna case where they announced fully agentic contact center, touted huge cost savings, then silently rolled it back. He speculates that the hardest part wasn't the tech but the non-first-line resolution tasks — things like processing a refund that require writing back to multiple systems and judgment calls. He argues you should always start with a human-AI mix and evolve the ratio. The architecture he suggests: an orchestrator that classifies call types, routes level-1 to AI, and escalates complex or emotion-laden calls to humans. This avoids the PR backlash of seemingly removing human empathy.

Matt uses the Lifespan MD example, a concierge medicine network. The first step was using Invisible's Neuron platform to bring together all patient and provider data into a HIPAA-compliant instance that gave a 360° view. They could then ask questions like 'What longevity tests do male patients 35-50 use most?' and get practice performance metrics. The generative AI layer acts as a chat agent and knowledge management system over that data. He emphasizes that AI should not replace physician decision-making but should handle admin, scheduling, and information retrieval — the painful overhead that consumes 30-40% of healthcare spending. This approach keeps the human expert empowered while removing drudgery.

Matt explains that for a contact center, you'd need to compare AI agents against expert human agents on metrics like first-call resolution, whereas for insurance claims you'd need human-equivalent outputs. The difficulty of building those baselines has slowed adoption in many sectors. Unlike the SaaS paradigm where off-the-shelf tools just work, AI requires fine-tuning on proprietary data and then building a custom eval to validate performance. This is hard, but it's where the real value lies. He believes the benchmark itself becomes the IP, and whoever builds it first in a sector becomes the instant star because nobody else has claimed topic ownership. This creates a new calling for domain experts to define the standard of 'good enough' for AI in their field.

Matt acknowledges that the nature of RLHF is changing — moving from commodity cat-dog labeling toward expert work by PhDs and masters, plus RL gyms and simulated environments. However, he insists that for any highly specific task (legal services, insurance claims, non-standard contact center interactions), you need human equivalence testing. The data for these tasks isn't in any training corpus; it sits in people's heads. As models venture into niche domains, the need for human feedback actually grows. He cites the Klarna contact center example: moving to 100% agentic without a human escalation path led to a rollback, illustrating why a human-AI mix is the winning architecture. He sees human-in-the-loop as a feature that will persist for many years.

Matt explains that rather than one decisioning agent, companies will train narrow agents for subtasks — e.g., one for balance checks, another for refunds — each with its own accuracy benchmarks, and have an orchestrator LLM coordinate them. This modular approach lets you validate each piece independently and pinpoint failures, making the whole system trustworthy. He thinks contact centers are an early example and that more success stories will emerge in 2026. This architecture also simplifies data requirements because you only need clean data for each specific task, not an entire corporate data lake.

Matt describes these as controlled environments where you can simulate a series of function calls, tasks, or interactions — for example, manufacturing operations or multi-step customer service calls. By running the AI through these simulations, you can see how it performs on edge cases, measure coherence in complex agent chains, and refine prompts or fine-tuning before touching live systems. This concept ties into the enterprise need for guaranteed outcomes; digital twins allow you to answer the question 'would you bet your annual bonus on this deployment?' before going live.

Matt notes that for many enterprises, the most impactful use cases — sales, underwriting, service — rely on data about how they do things, not just what databases store. That data is often unstructured and resides in documents, emails, and human expertise. Companies must balance the convenience of using frontier APIs with the risk of leaking their data. He suggests segmenting data: truly sensitive information stays on-premise or uses small language models; less sensitive data can leverage public APIs. This segmentation will define the competitive landscape.