Reducing AI's carbon emissions with minimal data: Which strategy works best?
Efficient Data Selection and Model Optimization Reduce AI Energy Consumption
Training artificial intelligence (AI) models consumes a significant amount of electricity, largely due to the number of exponential calculations performed with large datasets. However, a new approach is being studied to address this issue, aiming to reduce energy consumption without compromising on the accuracy of the models.
This approach involves two key strategies: efficient data selection and model optimization.
Model Optimization
Model optimization includes strategies like transfer learning, where pre-trained models are reused instead of training from scratch. This approach saves both time and energy by leveraging models already trained on large datasets, eliminating the need for extensive retraining on massive data. Other optimization methods include quantization (reducing precision of calculations), shortening input and output lengths, and using smaller, specialized models instead of large all-purpose ones. Combined, these strategies have been shown to reduce AI energy consumption by up to 90% without significantly sacrificing accuracy.
Efficient Data Selection
Efficient data selection involves carefully choosing subsets of training data that are most informative or representative, which reduces unnecessary computation on redundant or low-value data. This principle involves using methods such as sampling, filtering, or prioritizing high-quality, relevant data to decrease the size of the dataset and speed up training. Using open-source AI models with transparent training data and model design also enables easier assessment and optimization of data use and energy impacts.
Additional energy-saving strategies related to training include scheduling jobs during off-peak electricity hours and combining physics-based and AI models for optimized decision-making, which reduces energy use in applications like HVAC.
In some cases, data selection can consume more energy and prove less reliable than training the entire dataset. Therefore, it is crucial to ensure the reliability of the selected data set to ensure the effectiveness of this approach.
Researchers are also investigating the use of encrypted data and images for efficient data selection, and debating two options to optimize AI processes: using high-quality, powerful calculation models or smaller, less energy-consuming models.
Currently, text generation models and text and image classification are the most popular AI models available on Hugging Face. However, researchers are comparing the efficiency of various AI models, including energy-efficient ones like Vit/b, with those that are most used, like Eva-L.
The work is still under study, and it is expected that these strategies will contribute substantially to lowering the energy footprint of AI model training, making AI more sustainable for the future.
References:
- Canal U: AI Session - Oumayma Haddaji - Random sampling for energy-efficient training of machine learning models
- Canal U: AI Session - Tiago Da Silva Barros - Small is sufficient: reducing the AI energy consumption through model selection
- Andrea Asperti et al: A Survey on Variational Autoencoders from a Green AI Perspective
- University of the French Riviera research team
- LAAS (Laboratory for Analysis and Architecture of Systems) CNRS
- The strategy of model optimization in AI, such as transfer learning and quantization, has the potential to reduce energy consumption by up to 90%, through techniques like reusing pre-trained models, reducing precision of calculations, and using smaller models.
- Efficient data selection in AI is achieved by carefully choosing the most informative or representative subsets of training data, which helps to reduce unnecessary computation and speed up training, thus contributing to energy savings.
- In addition to efficient data selection and model optimization, researchers are exploring other avenues to minimize AI energy consumption, like scheduling jobs during off-peak electricity hours, using encrypted data for secure data selection, and debating the merits of high-quality, powerful calculation models versus smaller, less energy-consuming models.
