# Clustering for infrastructure observability: finding outlier servers with HDBSCAN

> How HDBSCAN finds servers that behave unlike their peers, how to judge the clusters, and how to compare each outlier's infrastructure state with its peers'.

- URL: https://blog.asama.ai/blog/clustering-for-infrastructure-observability/
- Author: Jain Johny
- Published: 2025-10-28
- Tags: Field notes

Clustering groups servers by how they behave, without labels, and the servers that fit no group are the anomalies worth a look. Density-based methods such as HDBSCAN suit fleet telemetry: they find clusters of any shape and mark the points that belong nowhere as noise. When a server lands outside every cluster, look at its infrastructure state early, since it is one of the cheapest things to check: the parts, firmware, BIOS settings, kernel parameters and drivers that make it what it is, as opposed to what it is doing.

Plot a few hundred servers of one model on two numbers, CPU frequency under load and inlet temperature, and groups appear without anyone drawing them. Nodes running the same workload line up on the frequency axis, and racks in the room's cooler spots sit lower on the temperature axis.

A few servers sit apart from every group. What sets them apart is often their state: a BIOS power profile that a firmware update reset, a firmware version nobody else runs, a kernel parameter [changed by hand](https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer). When [ten servers came back from maintenance with a BIOS setting changed](https://blog.asama.ai/blog/article-03-asama-what-happens-when-the-loop-ends), their dashboards stayed green, but on a plot like this they would have left their group together.

## What Is Clustering
Clustering is the process of automatically grouping similar data points.

In infrastructure work, that means:

- Anomaly detection: grouping normal vs abnormal signals
- Workload segmentation: clustering GPUs, jobs, or nodes by behaviour
- Log and metric deduplication: grouping repeating fault patterns
- Embedding analysis: grouping semantically similar vectors

Each of these turns a wall of telemetry into a short list of things to look at.

## How Clustering Works
At its core, every clustering method follows the same steps:

1. Measure similarity: define how close two data points are using a distance metric (e.g., Euclidean, cosine).
2. Group related points: combine points that are close together into clusters.
3. Evaluate stability: check how cohesive and distinct each cluster is, and merge or split as needed.

## Two Broad Types of Clustering
There are many algorithms, but most fall into two broad categories:

- **Flat Clustering**
Flat algorithms create a single partition of the data: every point belongs to exactly one cluster.
They’re simple and efficient but can struggle when cluster densities vary or when data has complex shapes.
Examples: k-means, k-medoids.

- **Hierarchical Clustering**
Hierarchical algorithms build a tree (dendrogram) of clusters, capturing how groups form, merge, or split at different similarity levels.
This approach helps reveal structure at multiple scales, from fine-grained subgroups to broader categories.
Examples: BIRCH, HDBSCAN.

HDBSCAN is also density-based. It builds clusters from regions where points are packed tightly and labels the points that sit in no dense region as noise, with the label -1. For finding the odd server out, that noise label is the output you want.

### Evaluating Clustering Quality
Because clustering is unsupervised, there’s no single “correct” answer.
Common ways to assess cluster quality include:

- **Silhouette score**: measures how well points fit within their cluster vs others.
- **Davies-Bouldin index**: compares internal cohesion to external separation.
- **Stability checks**: ensure clusters persist across different samples.
- **Visualisation**: project high-dimensional data into 2D (UMAP or t-SNE) to inspect structure and overlap.

Use the first two with care. Both assume compact, convex clusters, so they can misjudge good density-based clusters like HDBSCAN's ([scikit-learn](https://scikit-learn.org/stable/modules/clustering.html#clustering-performance-evaluation)). For HDBSCAN, use density-based cluster validity (DBCV); the hdbscan library's `relative_validity_` approximates it, for comparing runs with different settings.

## Finding Outlier Servers with hdbscan

The example uses the standalone hdbscan package (`pip install hdbscan`); scikit-learn's own HDBSCAN, added in version 1.3, has no outlier scores or validity measure.

```python
import numpy as np
import pandas as pd
import hdbscan
from sklearn.preprocessing import StandardScaler

# features: one row per server, one column per metric, for example mean
# CPU frequency under load, inlet temperature, fan speed, correctable
# ECC rate and PCIe replay rate
X = StandardScaler().fit_transform(features)

clusterer = hdbscan.HDBSCAN(min_cluster_size=10, gen_min_span_tree=True)
labels = clusterer.fit_predict(X)

noise = np.flatnonzero(labels == -1)            # servers that fit no cluster
scores = clusterer.outlier_scores_              # GLOSH: higher is more outlying
edge = np.flatnonzero(scores > np.quantile(scores, 0.99))
print(clusterer.relative_validity_)             # relative DBCV: compare settings

# state: one row per server, in the same order, one column per setting,
# for example BIOS power profile, BMC firmware, kernel and NIC driver
state = pd.DataFrame(state_rows)
usual = state.mode().iloc[0]                    # the most common value of each
for i in np.union1d(noise, edge):               # odd servers
    row = state.iloc[i]
    differs = row[row.ne(usual) & row.notna()]  # skip settings it couldn't report
    if len(differs):
        print(i, differs.to_dict())
for k in np.unique(labels[labels >= 0]):        # a batch can form its own cluster
    common = state[labels == k].mode().iloc[0]
    differs = common[common.ne(usual) & common.notna()]
    if len(differs):
        print('cluster', k, differs.to_dict())
```

Start with the noise points, then the top of the outlier scores: a server can sit inside a cluster and still be on its edge. If HDBSCAN labels all or nearly all of the fleet noise, as it can for a fleet running one workload, even with `allow_single_cluster=True`, go straight to the scores. Scale the features first, or the one with the biggest units decides the clusters. If many servers report identical values, cluster the unique rows and map the labels back (`np.unique(X, axis=0, return_inverse=True)` gives you both): with many duplicate rows, hdbscan can return NaN outlier scores, even for the real outliers.

Keep state out of the distance calculation. Most settings are categories, so Euclidean distance has no meaning for them, and a firmware version found on one server in 300, one-hot encoded and scaled, lands 17.3 standard deviations out and outweighs every metric. Cluster on behaviour, then compare each odd server's state with the most common value of each setting, as the last lines do. That comparison assumes one hardware model; with several, run it within each. Ten outliers that share one odd setting are one finding, not ten: one maintenance window, one rollout, or one delivery batch with a different default. A batch like that shows up as several noise points sharing a setting or, once it is tight and larger than `min_cluster_size`, as a small cluster of its own, which is why the code checks each cluster's common settings too.

## How Asama AI Uses Clustering
Asama is designed for fleets too large to eyeball, with clustering as the first pass: [grouping similar devices](https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts) by operational metrics within each hardware model, so that a deviation means something. A deviation usually signals [performance degradation](https://blog.asama.ai/blog/issue-01-asama-uptime-is-not-reliability), and the common causes are [configuration drift](https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer) and failing components.

For each outlier, and for each small cluster that appears, Asama is designed to compare its infrastructure state with its peers' and name the setting that differs.

That answer is what makes [remediation](https://blog.asama.ai/blog/article-03-asama-what-happens-when-the-loop-ends) targeted: your engineers start from the affected servers and the setting to change.

## References
- [How HDBSCAN works](https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html), the hdbscan documentation
- R. Campello, D. Moulavi and J. Sander, "Density-Based Clustering Based on Hierarchical Density Estimates", PAKDD 2013
- L. McInnes, J. Healy and S. Astels, ["hdbscan: Hierarchical density based clustering"](https://joss.theoj.org/papers/10.21105/joss.00205), Journal of Open Source Software, 2017
- [scikit-learn: clustering performance evaluation](https://scikit-learn.org/stable/modules/clustering.html#clustering-performance-evaluation)
- [StatQuest: Hierarchical Clustering](https://www.youtube.com/watch?v=7xHsRkOdVwo), StatQuest with Josh Starmer (video)
- [The DBSCAN Clustering Algorithm Explained](https://www.youtube.com/watch?v=4AW_5nYQkuc), Kilian Weinberger (video)

## Summary
Cluster servers on what they do and the ones that fit no group surface on their own. Compare their state with their peers' to find out why. HDBSCAN's noise label and outlier scores give you the list; DBCV, not the silhouette score, judges the clusters.

**Update, October 2026.** Asama's fleet visibility is generally available, and detection built on it has been validated; investigation and remediation are expanding. Asama now correlates each finding on four dimensions: time, topology, cohort (do identical peers show the same symptom?) and state and change (what moved, and when). To see which of your servers sit outside their cluster, and how their state differs from their peers', book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.