Institutional Intelligence: The Hidden Cost of Tribal Knowledge in Data-Intensive Firms
7 min read
Most firms in capital markets can account precisely for what they spend on data. Far fewer can say what that data returned, because valuing data remains a genuinely hard measurement problem and one term in the equation is almost never recorded: which data products held up, where they failed, and what they were trusted for.
That evidence does exist. It sits with the people who built the pipelines, in message threads, in analysts’ notebooks, and in the reasoning behind decisions nobody wrote down. We call it tribal knowledge because it belongs to a group rather than to the firm: available to whoever knows whom to ask, and unavailable to everyone else, including every system built on top of it and every attempt to put a number on what the data is worth.
Because tribal knowledge lives with people, firms file it as a staffing risk and manage it like one, with handover checklists when someone resigns. That puts the entire loss at the moment of departure, which is the wrong place to look for it. Drawing on knowledge that sits in somebody’s head requires two conditions: knowing that the knowledge exists, and knowing who holds it. Anyone meeting a data product for the first time has neither. There is nothing to search for and no obvious person to ask, so the answer stays one conversation away and that conversation never happens. A fully staffed firm loses much of what it would have lost had the expert resigned, without anything occurring to make it notice.
The two halves of a data product
A data product has two halves: the data, which a firm licenses or produces and accounts for exactly, and the interpretive layer that decides whether it can be used at all, recorded nowhere and therefore impossible to value.
| Category | The question it answers |
|---|---|
| Revision behavior | Is this series point-in-time, or does it restate? A backtest on the wrong assumption is wrong and hard to detect |
| Definitional drift | Has a field changed meaning without the schema changing? Vendors reclassify; history does not always follow |
| Source preference | Where two data products overlap, which is authoritative, for what purpose, and on what basis |
| Quality boundaries | Where does coverage thin out by market, asset class, or period, and what is known of past gaps |
| Permitted use | What has this been approved for, and what does the license exclude |
None of it is discoverable from a schema. It is the accumulated judgment of people who have worked with the data, and it is the difference between holding a data product and knowing what it is good for.
A firm can account precisely for what it pays for data. The knowledge that makes that data usable has never had the same accounting.
Not a retention problem, a circulation problem
Consider a firm with no attrition at all, where everyone who has ever understood a data product is still employed and still reachable. That firm is still paying for tribal knowledge every day. Two analysts on different desks independently work out the same quirk in the same feed. A team evaluates a data product the firm already licenses, because the subscription is invisible from where they sit. A model goes into production on a series whose revision behavior its author never checked, because checking meant finding the right person and the right person was busy.
None of that is a retention failure. It is a circulation failure, and it is the permanent condition rather than the exceptional event. Departures matter because they turn a circulation problem into a permanent loss, but they are the visible instance of a cost already being paid.
The distinction decides where the money goes. Treat it as retention and the response is procedural: handover documents, transfer sessions, a checklist. Treat it as circulation and the response is architectural: make knowledge move without anyone having to remember to move it.
Why tribal knowledge stays tribal
Management theory settled this argument decades ago: an organization’s advantage lies in converting what its experts know into shareable form, not in employing the experts1. Anyone who has worked closely with data also knows the harder half of the problem, that we know more than we can readily tell. The theory is agreed. The mechanism is what firms lack. Knowledge stored apart from the asset it describes has no forcing function, so nobody notices when it goes wrong. Knowledge that takes extra effort to record loses to the work in any busy quarter. Knowledge given once in conversation is unfindable within a month.
Gartner’s forecast that 80% of data and analytics governance initiatives will fail by 2027 for want of a real or manufactured crisis2 describes the same pattern: programs asking people to do something additional, for a benefit landing on someone else later, do not survive. Which makes this a design problem with design solutions. Knowledge has to be attached to the asset it describes, produced as a by-product of the work, and surfaced where decisions get made.
Four tests of whether knowledge is an asset
Most firms hold some documentation and some institutional memory, with no way to tell whether it works. These tests concern form rather than volume, because volume is not the constraint.
| Test | The question | What failure looks like |
|---|---|---|
| Attached | Does it live on the data product, or beside it | A wiki describing an estate it is not connected to, decaying invisibly |
| By-product | Is it created by doing the work, or after it | A process kept up in quiet quarters, abandoned in busy ones |
| Surfaced | Does it appear at the point of use | Accurate documentation nobody reads, because nothing puts it in their path |
| Independent | Does it survive its author | Notes that need their author present to interpret |
The tests are cumulative. Attached but not surfaced is an archive. Surfaced but not independent is a personal aide-memoire. Only knowledge passing all four compounds.
What tribal knowledge actually costs
The cost is hidden because it arrives as four separate things, none of them a line item.
Work is repeated. Every analyst who sees a data product learns its behavior the same way, by making an error or interrupting one of the few people who already know. The firm pays that tuition every time.
Data is bought twice. Overlapping subscriptions persist because no individual sees the whole estate, so nobody is positioned to notice.
Data is bought and left unused. A data product nobody trusts well enough to use becomes shelfware while the invoice keeps arriving.
Decisions are made without the caveat. The largest and least visible: a position taken, or a model shipped, on data whose known limitations were known only to somebody else.
What makes this hidden rather than fixed is the direction it travels. Tribal knowledge never gets cheaper as a firm accumulates it, because none of it accumulates anywhere shared, and it resets whenever a custodian becomes unavailable. Recorded context moves the other way: the first note on a data product serves the second person to arrive, whose correction serves the third.
One effect turns this from a productivity argument into a budget one. Failed searches are procurement intelligence: when analysts repeatedly look for something the firm does not have, that is recorded, quantified demand from the people who would use it.
Tribal knowledge is expensive, recurring, and invisible. That combination is why it stays unmanaged.
The newest consumer of institutional data knowledge
All of this predates AI. What AI changes is how many consumers now arrive without judgment. An agent querying a firm’s data has the schema and none of the interpretive layer. It has no sense that a series looks wrong and no colleague to ask. Where an experienced analyst hesitates over a field, an agent uses it and returns an answer that is fluent, well-structured, and incorrect.
The failure mode of an uninformed human is a question. The failure mode of an uninformed agent is an assertion.
This is the concrete content of the claim that data rather than models is the constraint. Institutional data knowledge and AI readiness are one program, not two. The context captured to make the next analyst faster is the context that makes an agent correct, and the Model Context Protocol has become the standard channel for delivering it. Judgment that was a private asset becomes an interface.
What to measure
Knowledge stays unmanaged because it is unmeasured. Two metrics are worth standing up first. Single-custodian exposure, the number of data products exactly one person can explain, turns an abstract concern into a list of named positions, and the answer is usually worse than expected. Duplicate coverage identified produces savings inside the current budget cycle, which is what funds everything else.
From tribal knowledge to institutional knowledge
The difference between the two is ownership. Tribal knowledge is held by a group and reaches only whoever thinks to ask. Institutional data knowledge belongs to the firm: it outlasts the reorganization, transfers to whoever needs it next, and can be inspected by anyone entitled to see it. The judgment itself is unchanged in that conversion. What changes is that the firm now holds an asset where it previously held a dependency.
Which answers the question the data budget has always begged. A firm that records which data products held up, where they failed, and what they were trusted for can say what its data returned rather than only what it cost, which turns spend from an expense to be defended into a portfolio to be managed. The same record decides whether an AI estate repeats the firm’s best thinking or its worst assumptions.
None of this asks a firm to know more than it already knows. The knowledge is present today, held by people who are not leaving, and it simply belongs to them rather than to the institution. The firms that pull ahead will not be the ones with the most data, or the longest-tenured teams. They will be the ones that stopped letting their most valuable knowledge stay tribal.
How DataHex Data Library fits
DataHex Data Library is an AI-native business data catalog for capital markets, built so the interpretive layer is part of every data product rather than an attachment to it.
Each product carries its own context: engineer’s notes and discussion on the asset page, a named owner, quality history, lineage, and a business glossary so a term means one thing across the firm. Usage and search behavior are recorded, including searches that returned nothing, so the catalog improves through use and reports what analysts could not find. That same context reaches AI agents through a governed MCP interface, so an agent inherits the judgment an experienced analyst would apply along with the entitlements that analyst would respect. It runs as a metadata layer over existing platforms, with no migration.
See it in action