Dynamic data reduction for foundational AI models

Abstract

Arti cial Intelligence (AI) fundamentally relies on data to generate useful outputs. The accuracy of an AI system is not only determined by the e ectiveness of the model, but also both the quantity and quality of the data it is trained on. Therefore, a relatively simple way to maximise AI performance is to use more data. However, as we continue to push the boundaries of AI, the demand for larger and more diverse datasets has only intensi ed, highlighting the importance of high quality data acquisition and management in the development of advanced AI technologies. This causes a growing concern about the environmental impact associated with these methods. The escalating energy requirements for training and operating sophisticated AI models contribute to a larger carbon footprint, prompting discussions on sustainable practices within the eld of arti cial intelligence. Addressing these concerns necessitates a balance between pursuing technological advancements and implementing strategies to mitigate their environmental repercussions. This thesis explores dynamic data reduction methods as a means of exploring data quality and usefulness. For example, by reducing the amount of training data used, less computation is needed, reducing the environmental cost of the algorithm. Random dynamic data reduction is explored rst by reducing the amount of data used for training at di erent stages of training, for image classi cation tasks. Three dynamic data reduction methods are introduced to observe di erent strategies of data reduction: data step, data increment, and data cut. These strategies present varying amounts of training data application which means that fewer image evaluations are performed to train the model than if the model is trained on the full dataset. Meanwhile, the strategies ensure that the full dataset is used, to ensure maximum data coverage. Results show that the data step method is best performing, and for datasets MNIST and smallNORB, model accuracy may be increased by 0.04% and 0.593% respectively, while 12% and 6% fewer evaluations image are performed. Random data reduction is so far limited by uncertainty of the quality of data that is removed for a portion of the training. This may be alleviated by introducing a method to determine speci c data to remove. By rst applying dimensionality reduction algorithms, data may be ordered and then removed based on distance from each classes' centroid. This allows for more intelligent data selection. Data step reduction is performed on larger image datasets, namely Imagenette and the Micro-PCB dataset, yielding inferior accuracy for the Imagenette dataset with an accuracy decrease of 0.114%, and superior accuracy for the Micro-PCB dataset by 0.013%. The data step method has shown promising results as a dynamic data reduction strategy. An alternative method using a sliding window of data is explored, which ensures unique data each epoch throughout training, instead of a step-up in data use. This method is combined with the previously used non-random data reduction, on image data of di ering sizes. Results show that model accuracy is worse for MNIST, CIFAR-10, Imagenette and the Micro-PCB datasets by up to 0.16%, but accuracy increases for the smallNORB dataset by 0.654%. However, statistical signi cance against benchmark experimentation informs us that after reducing data, results show no statistically signi cant di erence from the benchmark, even though training data has been reduced. Still, despite the underwhelming performance, insight is given into which data is more useful at the beginning or end stages of training by comparing the direction of the sliding data window. Results show that for unstructured datasets MNIST, CIFAR-10 and Imagenette, the forward data ow direction yields superior accuracies, while for the highly structured smallNORB and Micro-PCB datasets the opposite is true. This supports the theory that some data is more useful in the ne-tuning stages of training, depending on how structured the data is. Having explored dynamic data reduction in image data, data reduction in imbalanced, timeseries occupancy datasets is investigated. Occupancy detection from non-invasive sensor readings is useful for building interior management, while maintaining personal privacy. However, realworld data is often imbalanced, often giving bias to training. By using data reduction techniques inspired by those used for image data, results show that data reduction on the larger class only can maintain and even improve AUC-ROC scores 0.483% with as much as 39% of the dataset removed. Dataset fusion is also explored as a means to improve the generalisability of the classi er. Results of experimentation show a decrease in AUC-ROC of as much as 48% with the full dataset, and 33% with the reduced dataset. This shows that by attempting to increase the available data, overall performance is decreased as the model struggles to specialise. Also, data reduction on a fused dataset can minimise the detrimental e ects of too much data. Class density is also calculated to be a metric that measures class balance and data su ciency, which is vital for determining whether a dataset may be reduced to improve classi cation performance prior to any model training. The above investigations show that the exclusion of some data may improve model performance, as well as reduce training cost. The novel data reduction strategies that are introduced also give insight into which data is more important to training at the earlier and later stages. These ndings prompt us to consider not just the quantity of data, but the quality and relevance of data during both collection, where focus can be aimed at collecting most useful data, and training, where most relevant data may be applied at the most bene cial stage of training.