Feature selection for industrial sensor data

The problem

Industry 4.0 has made production data abundant: connected lines now log every screwdriving and punching cycle as a high-dimensional time series with hundreds of measurement points per signal, accumulating faster than anyone can analyze them. Hidden in that data are the process states a plant needs to see: worn tools, cross-threaded screws, contaminated threads. Clustering can surface these states without labeled training data, but it breaks down exactly at this scale: with rising dimensionality, distance measures lose their meaning (the curse of dimensionality), redundant and irrelevant time points blur the cluster structure, and runtime and memory grow without adding any insight. Feature selection is the lever that restores the explanatory power of these algorithms. But there are dozens of competing methods, and picking the wrong one silently produces wrong clusters.

The solution

Instead of betting on one method, the thesis builds a Python toolchain that makes the choice measurable: it runs many feature-selection methods on the actual dataset, clusters the reduced data, and scores every method on two axes, an internal cluster-validity score (silhouette, Davies-Bouldin, DBCV, noise ratio, stability) and a computational-efficiency score (runtime, CPU, peak memory). The result is a ranked, reproducible answer to “which method fits this data” instead of a guess. The toolchain runs as a web application, so it can be operated centrally and used without ML expertise.

What I did

• Designed and implemented the full pipeline as a modular web application: data import (JSON/CSV), signal preprocessing with trigger-based cycle segmentation and cropping, feature selection, Ward clustering, evaluation, and automated visualization and export.
• Implemented eleven unsupervised feature-selection methods, from variance and entropy filters to Laplacian score, SPEC, MCFS, UDFS and NDFS.
• Defined the evaluation framework: a weighted internal-validity score, a computational-efficiency score, and an external check against known process states used strictly for validation, never for tuning.
• Validated everything on two real production datasets: screwdriving cycles across 13 process states (including fault cases like cross-threaded or contaminated screws) and 92 punching measurement series covering three steel grades (DP600–DP1000) and three tool-wear states, each at three reduction levels (keeping 10/15/20% of time points).

Results

• Feature selection raised internal cluster validity on the screwdriving data from 0.20 (no selection) to 0.90 (Laplacian score), while keeping only a fifth of the data.
• The most important finding: the internally best method can still be factually wrong. Laplacian score produced near-perfect geometric clusters that missed the real process states almost entirely (external agreement 0.04), while MCFS and SPEC matched them at up to 0.91 and 0.93. No method wins universally, which is exactly why a per-dataset benchmark beats a fixed choice.
• On the punching data, Laplacian score identified all real process states perfectly (external score 1.00 across every reduction level).
• The thesis, with the toolchain as its deliverable, was graded 1.3.