Frontwave Credit Union

Shane Trefftzs, Director of Business Intelligence

The AI That Says I Don't Know Business intelligence that knows what it doesn’t know

Shane Trefftzs

A few months ago, a lending stakeholder asked me why pull-through rates showed different numbers across two dashboards. He needed to know which number to trust.

I did not have an answer, but I knew the problem. Two analysts had written two queries using different definitions of pull-through. Both were defensible. Neither was wrong. They were simply not working from the same playbook.

Every organization has a version of this problem. I saw it across a decade of building data platforms at Fortune 500 firms. Business logic lives in people’s heads. Definitions get written into queries by individual contributors. Nobody is working from a single governed source of truth.

When I took the BI Director role at Frontwave Credit Union, a 300-person credit union serving the military community in North San Diego County, that conversation made something click. We did not need more dashboards. We needed a formal process for governing business logic and a platform that could enforce it.

The semantic layer industry is solving the wrong problem

DBT’s Semantic Layer, Cube, AtScale, LookML, Atlan. Every one of these tools ships with a clean interface for shared business definitions and assumes the rest will follow. The interfaces are beautiful. The governance question is mostly absent. Who reviews a new definition? Who approves it? What happens when a bad one slips in? By month twelve, analysts are writing their own definitions again because nobody trusts the layer.

Pointing an LLM at your warehouse has the same flaw. The SQL it writes will sometimes be wrong. Confidently wrong. Polished enough that nobody questions it until the wrong number lands in front of the board. That is not intelligence. That is automation without accountability.

We built ARC-ai

ARC is Frontwave’s internal business intelligence platform within Databricks. We built it to turn the mountain of data a financial institution like ours sits on into a single governed environment the entire organization can use, from executive reporting and machine learning to regulatory analytics. ARCai sits on top, grounded in hundreds of tables and thousands of columns from core banking alone, plus every approved business definition and policy document.

ARC-ai does not answer from its training data. It answers only from approved internal documentation, and only after verification. Every question is broken into eight components, each requiring evidence in the governed corpus: business terms, source tables, columns, joins, time windows, filters, grain, and calculation logic. If everything verifies, ARC-ai writes a candidate query for engineer review. If anything fails, the system blocks the request and reports which evidence is missing.

Most AI systems are designed to always produce an answer. ARC-ai was designed to produce one only when it should.

The blocked questions are the product

This is the part many companies miss. Every blocked question is logged with the precise documentation gap that caused it. Leadership pulls a report showing what the organization is asking that cannot yet be answered, along with what it would take to close each gap. That report becomes the agenda for our Feature Review Committee, which meets monthly, approves new definitions one at a time with a full audit trail, and promotes them to gold tables.

“The organizations that lead in BI over the next decade will not be the ones with the most sophisticated AI. They will be the ones whose AI is honest about what it does not know.”

This is how the semantic layer grows. Not through a onetime modeling exercise. Not through a vendor implementation. Through a living feedback loop where the questions the organization is asking drive the definitions it produces next. The demand signal is built into the architecture.

If that lending stakeholder asked the pull-through question through ARC-ai today, the system would block the request, log the gap, and route it to the committee. Once approved, every report, dashboard, and model uses the same number.

The structural advantage of the mid-market

A Fortune 500 company has the headcount to absorb the cost of inconsistency. A 300-person credit union does not. That constraint pushed us toward something simpler and more disciplined than I would have built with unlimited resources, and it made the platform better.

The other piece was our CIO, Ryan Huff. He invested in a modern stack and then did the harder thing: trusted his BI team to build on it. That combination is why a community credit union is shipping governance infrastructure that enterprise data teams are still debating.

The real question for leadership

Your dashboards look fine right now. Somewhere, two analysts are defining the same metric differently, and that discrepancy will surface at the wrong time in front of the wrong audience. Buying a semantic layer product will not fix it. Neither will pointing AI at your data.

The bottleneck is treating business definitions as assets worth governing and building a system where governance is self-sustaining.

The organizations that lead in BI over the next decade will not be the ones with the most sophisticated AI. They will be the ones whose AI is honest about what it does not know.