# Bring-up is where schedules go to die: what new servers need before production (Problem Specifics 3/5) > Bring-up is the work between racked and first workload accepted. What a slip costs while purchased capacity waits, and a five-step gate for every new server. - URL: https://blog.asama.ai/blog/issue-08-asama-bring-up-is-where-schedules-go-to-die/ - Author: Jain Johny - Published: 2026-02-02 - Tags: Fleet problems Bring-up is the work between hardware racked and first workload accepted: firmware and BIOS validated against a baseline, drivers aligned, links and network checked, and burn-in run to shake out early failures. It is often missing from the project plan, so its overruns land on the schedule unplanned, and the hardware waiting on it is already bought. Every infrastructure deployment timeline has a visible part and an invisible part. The visible part (procurement, shipping, rack installation, power provisioning) has lead times, purchase orders, and project plans. It is managed, tracked, and reported. The invisible part begins the moment the hardware is racked and powered on. Firmware validation. Driver alignment across the fleet. [BIOS baseline configuration](https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer). Network stack verification. [Burn-in testing to identify infant mortality failures](https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts) before they surface in production. > ### 1,200 server-weeks > **An assumed six-week slip on 200 new servers** > Capacity already bought, delivering nothing. > > ### $3 million > **Committed before bring-up starts** > 200 servers at an assumed $15,000 each, racked and waiting on validation. > > ### At purchase > **When the money is committed** > Under [IAS 16](https://www.ifrs.org/issued-standards/list-of-standards/ias-16-property-plant-and-equipment/), depreciation starts only when an asset is available for use, so the books may not show the wait. The money was committed when you bought it. > "The money is committed when the hardware is bought. The capacity arrives when the first workload is accepted. The gap between them is capital doing nothing." --- ## The permutation problem The reason bring-up takes so long is not that any individual step is difficult. It is the number of configuration permutations that must be resolved before the system is reliably operational. Bootloader settings, driver versions, kernel parameters, firmware revisions: one mismatch anywhere in the stack and the system misbehaves in ways that take time to investigate. The knowledge required to navigate these permutations efficiently [lives in the heads of the two engineers](https://blog.asama.ai/blog/issue-03-asama-the-tribal-knowledge-tax) who have done the most deployments. When they are available, bring-up goes reasonably well. When they are not, everyone else starts from the datasheet. --- ## A bring-up gate before the first workload Run the same checks on every new node before it takes work: 1. **Firmware, drivers and BIOS against the baseline for that model.** BMC, BIOS, NIC, drive and GPU firmware, the kernel and driver versions, and the BIOS settings the workload depends on, such as the power profile. 2. **PCIe and network links at the speed and width they should train at.** Under load, compare LnkSta in `lspci -vv` with LnkCap on both the device and the port above it, since many GPUs drop link speed at idle and a slot can be wired narrower than the card. Read [the correctable error counters](https://blog.asama.ai/blog/issue-07-asama-the-correctable-error-trap), and check that each NIC port negotiated the speed of the switch port it is cabled to. 3. **Burn-in under real load**, long enough for infant mortality to show up before production does. 4. **GPU diagnostics at full depth.** On NVIDIA, run [`dcgmi diag -r 4`](https://blog.asama.ai/blog/gpu-health--diagnostics----why-they-matter), DCGM's deepest level, which adds a memory test and a pulse test to level 3; on AMD, [the RVS stress modules](https://blog.asama.ai/blog/gpu-health--diagnostics-part-2-amd-gpus-with-amd-smi) play the same role. 5. **A record of what passed and what was changed**, so the next deployment starts from it rather than from memory. --- ## What faster bring-up requires | Capability | Operational impact | | :--- | :--- | | **Automated firmware validation** | Firmware validated against known-good baselines before workload allocation, not after first production failure | | **Configuration drift detection** | BIOS and system deviations identified in seconds instead of during post-incident investigation | | **Encoded bring-up knowledge** | Hardware quirks, firmware combinations, and deployment learnings captured in systems instead of tribal knowledge | --- ## Food for thought > What was the actual gap between hardware racked and first production workload accepted on your last deployment, and how much of that time was recoverable with better tooling? **Update, October 2026.** Asama's fleet visibility, down to BIOS settings and firmware levels, is generally available. Detection built on it, which compares each node against its golden config and against its peers, has been validated; investigation and remediation are expanding. To see how a new rack differs from your baseline before it takes a workload, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.