# The tribal knowledge tax: what walks out when your senior engineers leave (Problem Scope 3/5) > Why a pinned BIOS setting or held-back firmware version often lives in one engineer's head, and what that costs in investigation time and deferred maintenance. - URL: https://blog.asama.ai/blog/issue-03-asama-the-tribal-knowledge-tax/ - Author: Manish Gautam - Published: 2025-11-04 - Tags: Fleet problems Tribal knowledge is the operational context that lives in a few engineers' heads instead of in any system: why a BIOS setting is pinned, why a firmware version is held back, why one rack is configured differently from the rest. When those engineers are unavailable, investigations take longer, maintenance gets deferred and the fleet drifts. Every infrastructure team has a knowledge distribution problem. It rarely appears on any risk register, never features in a capacity plan, and only becomes visible at the worst possible moment: when the engineer who knows [why that BIOS setting is pinned](https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts) is on leave, on a call with another incident, or gone entirely. The knowledge itself is not complicated. It is specific. Why this firmware version is running and not the newer one. What that kernel parameter change in 2022 was fixing and why it has never been reverted. Why the rack in row 7 is configured differently from every other rack in the data centre. Take a NIC firmware version pinned across one rack because the newer build drops links behind a particular switch model. When the engineer who found that leaves, the pin looks arbitrary. A routine update rolls the newer build out, links start flapping, and the investigation starts from zero. > This is not a people problem. It is a systems design failure, and it has a compounding cost that most organisations never calculate. ## How the tax accumulates --- > ### Incident cost > > When the engineer who understands the relevant system is unavailable, investigation time multiplies. > > What takes an hour with the right person takes a day without them. At scale, this becomes structural overhead embedded in every MTTR number across the team. --- > ### Maintenance risk > > Maintenance cycles become riskier as the institutional knowledge of why things are configured a certain way thins out. > > Teams defer changes they don't fully understand. Deferred changes accumulate. [The fleet drifts further from its intended state](https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer). --- > ### Scaling friction > > Every new team member is an apprentice until they have absorbed enough tribal knowledge to operate independently, a process that takes months, not weeks. > > Onboarding becomes knowledge transfer from people, not systems. --- ## The structural fix The fix is not documentation. [Documentation gets stale faster than infrastructure changes](https://blog.asama.ai/blog/article-01-asama-your-team-is-trapped-in-a-loop). A wiki page about a configuration decision made in 2021 is worse than no documentation at all if it describes a state that no longer exists. The fix is systems that capture operational knowledge automatically and keep it current: configuration baselines that reflect actual fleet state, change history that records not just what changed but why, maintenance records that preserve the reasoning behind decisions. > ### From Asama > > Asama is designed to keep this context as infrastructure state: configuration, change history and fleet topology, captured continuously as a by-product of normal operations rather than as a documentation exercise. The operational knowledge your team has built over years then lives in the system, not only in the heads of your two most senior engineers. ## Food for thought If your two most senior engineers left tomorrow, how long before the next incident would reveal what walked out with them? **Update, October 2026.** On the fleets Asama has measured, firefighting of this kind takes 40 to 60 percent of senior engineering time, and Asama's fleet observations put about 18 percent of manual fixes at the wrong procedure for the node's actual hardware state. Asama's fleet visibility is generally available, and detection built on it has been validated; investigation and remediation are expanding. To see how much of your fleet's state lives only in people's heads, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.