High-Performance Computing for Machine Learning and Artificial Intelligence in Brain-Computer Interfaces with Big Data

Abstract

The article explores approaches to optimizing the processing of big data of EEG signals in BCI by combining dimensionality reduction methods and HPC. The relevance of the problem is due to the fact that modern BCIs generate large datasets of signals, the processing of which in real time often creates a critical load on hardware and software resources. The aim of the work is to establish an optimal balance between classification accuracy, model robustness, and data processing time using various dimensionality reduction methods – PCA, ICA, LDA – in combination with the MLP classifier and the Dask library for parallel calculations. A series of experiments was conducted by varying the number of components for each decomposition. It was found that when using PCA with n_components=0.999 or LDA with n_components=13, the accuracy and f1_weighted remain practically the same as in the model without dimensionality reduction, but the processing time is reduced by 1.5-4 times, depending on the settings. The use of fewer components allows for even higher performance, but is accompanied by a noticeable decrease in accuracy, which is critical for neuroengineering and rehabilitation tasks. The use of Dask for organizing parallel calculations made it possible to effectively scale experiments and avoid excessive load on individual system nodes. A comparative analysis of the accuracy, robustness, f1_weighted, roc_auc_ovr_weighted metrics and execution time showed that the optimal settings of matrix layouts allow preserving key information in the signal without significant loss of classification quality. The developed approach has proven its effectiveness for tasks where resource limitations are combined with requirements for stability and accuracy of the system in real-time mode. The practical value of the results lies in the possibility of adapting the proposed pipeline for a wide range of biomedical and engineering applications, where speed, reliability, and robustness of brain signal processing are critical