According to researchers from USP and Unicamp, the question of whether larger artificial intelligence (AI) is a guarantee of better performance depends on the specific task. In several applications, more efficient and specialized systems may require less data and computational resources while providing the necessary level of performance.
This discussion does not mean that large models have lost their significance. While Marcelo Finger criticizes the growing dependence of systems on data and computing power, mathematician Marcos Eduardo Valle points to a gap between small models and large deep learning systems, seeing potential for studying intermediate solutions.
For Marcelo Finger, one of the main achievements of recent progress in AI is the expansion of machines' ability to recognize patterns in various types of information, including speech, images, and sounds. This breakthrough gained new momentum in 2017 with the appearance of the Transformer architecture, originally developed for language translation, which became the basis for modern large language models.
However, Finger notes the limitation of this approach: the Transformer has to repeatedly re-examine input data, and this process grows quadratically with the increase in input data size. Simply put, the more information the model needs to analyze, the faster the required computation increases, which raises processing costs and the resources needed to work with increasingly large models.
Besides computational power, Finger mentions resources such as energy, water, and heat dissipation associated with equipment operation. He also believes that after the popularization of large language models, the development in this area has become overly focused on this approach, as some research began following this direction once AI entered the market.
Marcos Eduardo Valle also sees limits in this process, suggesting that hardware development may be approaching stagnation. He refers to the progress of Graphics Processing Units (GPUs) as a factor that allowed for increased available power for deep learning but argues: 'This hardware issue seems stalled, and I believe we will also hit a ceiling on this limit; perhaps these giant models are stagnating.'
Large general-purpose models have an obvious advantage—they can handle a wide variety of situations. Nevertheless, according to Valle, this does not make them the optimal choice for every task. He gives examples from medicine and agriculture, where the volume of data can be incomparable to the data used to train large language models. In medicine, for instance, obtaining the necessary information often depends on professional assessments and involves more complex processes.
Under such conditions, a solution designed for a specific task may utilize available data better. Instead of trying to study a huge number of possibilities, the model can integrate known characteristics of the problem itself. However, this specialization has a condition: the specific model is based on certain assumptions about the problem being solved. If these assumptions prove incorrect in a real situation, a generalist model may be more advantageous due to its ability to process more options. As he emphasized: 'So, you cannot try to fit it into another task for which it was not designed.'
Thus, the choice is not only about model size; the type and volume of available data, the purpose of application, and the specifics of the problem itself are also important.
A project conducted by Finger during the Covid-19 pandemic demonstrates the application of a specialized approach under data scarcity. The goal was to analyze a person's voice to detect signs of respiratory failure and assist in identifying patients who might require hospitalization. The objective was not to detect coronavirus from the voice.
The initial task was to identify respiratory failure, a problem that can arise for various reasons and was linked to the most severe cases of Covid-19 during the pandemic. The first hurdle was data acquisition: the team collected information on approximately 600 patients over two months, but nearly a third of the recordings had to be discarded due to recording issues. Audio recordings were made in hospital settings with a high amount of noise, including coughing, equipment operation, television, and people moving.
Instead of simply removing these sounds, the team decided to use the ambient noise to give the data greater similarity. Researchers collected sound samples from service areas and added these noises to the recordings. This was done so that the system would not learn a characteristic unrelated to the study: the difference between a recording made in a hospital and one made at home. By making the signals more similar, the team aimed to make the model focus on vocal features related to respiratory failure.
According to Finger, the first version achieved 91% accuracy. After changes to the algorithm and new analyses, this figure rose to 96.5%, and then to 99% for identifying respiratory failure in the studied context. The result was presented by the researcher and pertains to the context studied by the team; the tool had a clearly defined task: detecting signs of respiratory failure.
A more efficient solution does not necessarily have to be created from scratch. Valle mentions methods that allow leveraging knowledge obtained from models trained on large databases. One of these is transfer learning, where a model trained on a large dataset is subsequently adapted for a more specific task. Another method is knowledge distillation, which allows using the knowledge of a larger model to build a smaller one.
This possibility shows that large models and specialized solutions do not necessarily have to compete. Valle cites the possibility of using a large language model to interact with a user and then passing the information to a smaller, specialized model responsible for a specific task. In such an architecture, different systems can perform different functions: the generalist model handles communication, and the specialized solution processes information according to the application's needs.
Valle's own work is based on this idea. The mathematician studies deep learning models that incorporate geometric and algebraic information to improve performance and reduce data requirements in certain applications. One example presented by the researcher relates to surveillance cameras. If a system needs to recognize suspicious movement, changing the camera's position should not cause it to stop recognizing that movement. The approach takes into account the geometry of the problem, allowing it to include characteristics that remain relevant even when the perspective changes. According to Valle, such a strategy can make models more efficient and, ultimately, trained on less data.
There are also advantages in operating such systems. Valle argues that more compact solutions can show good performance with lower costs, requiring less processing, energy, and computational resources. Depending on the application, they can be run locally, avoiding data transmission to the cloud.
Finger sees a trend in the concentration of attention on large models. In his opinion, researchers and companies tend to direct efforts toward technologies that are in the spotlight. He noted: 'The field of computation is guided more by fashion than by high fashion.'
Valle takes a less binary position. He believes that large models still have room for development but also sees potential in the space between traditional small models and giant deep learning systems. 'I think the trend is to move in this gap, to create a model that is neither too big nor too small. Thus, I think there is a lot to research in medium-sized models.'