{"id":95488,"date":"2023-07-27T15:07:12","date_gmt":"2023-07-27T13:07:12","guid":{"rendered":"https:\/\/www.hiig.de\/?p=95488"},"modified":"2024-01-31T15:35:42","modified_gmt":"2024-01-31T14:35:42","slug":"inside-hugging-face","status":"publish","type":"post","link":"https:\/\/www.hiig.de\/en\/inside-hugging-face\/","title":{"rendered":"Inside Hugging Face"},"content":{"rendered":"\n

To understand the dynamics of current open-source machine learning research, one platform is of central importance: Hugging Face. For accessing state-of-the-art models, Hugging Face is the place to be. In this post, I analyse how models on this platform changed during the last years and what organisations are active on Hugging Face.<\/strong><\/p>\n\n\n\n

Hugging Face, not OpenAI<\/strong><\/h2>\n\n\n\n

While much of recent reporting on AI is focused on OpenAI, the company behind ChatGPT, another company that is much more relevant to the everyday life of a Machine Learning (ML) developer, receives much fewer attention: Hugging Face<\/a>. Hugging Face is a platform that hosts ML models including Large Language Models (LLMs). Literally everyone can upload and download their models on this platform and it is one of the main drivers for democratising research in ML. Therefore, understanding what actors and models are on Hugging Face, tells us much about current open-source research in ML. <\/p>\n\n\n\n

The company behind the emoji<\/strong><\/h2>\n\n\n\n

Hugging Face is a young company that was founded in 2017 in New York City. Its original business model was providing a chatbot. However, nowadays it is mostly known as a platform that hosts ML models. At first only language models but in the recent months vision, speech and other modalities were added. Besides free hosting, it also provides some paid services such as a hub to deploy models. Hugging Face is currently rated at two billion dollars<\/a> and recently partnered with Amazon Web Services<\/a>.<\/p>\n\n\n\n

Models are becoming larger<\/strong><\/h2>\n\n\n\n

Contributions to Hugging Face are constantly increasing and repository sizes are growing over time (Fig 1). This means that increasingly large models are uploaded, which mirrors recent developments in AI research. Many of the models are provided by companies, universities, or non-profits but most come from entities that provide no information (\u201cOther\u201d). Manual inspection shows that it includes most often individuals who uploaded one or several models but also organisations that simply did not add their information.<\/p>\n\n\n\n

\"\"\/
Fig 1 Uploaded models over time and their repository sizes in Gigabyte (n = 151,296). Y-axis in log scale.<\/em><\/figcaption><\/figure>\n\n\n\n

What types of organisations are active on Hugging Face?<\/strong><\/h2>\n\n\n\n

Who uploads the largest models and whose models are most wanted? The biggest group in terms of uploaded models is \u201cOther\u201d (Fig 2). The remaining groups, except of \u201cclassroom\u201d, are fairly close to each other. This means that universities, non-profits, and companies contribute about equally on Hugging Face.<\/p>\n\n\n\n

The recent surge in ML research is mostly driven by Big Tech and elite universities because training large models is extremely resource-intensive in terms of compute and money. Accordingly, one would expect that most large models come from companies or universities and it is also their models that are downloaded the most. But this is only partly the case. Models by companies and universities are most wanted. However, while models uploaded by companies are also relatively large, models by universities are not. The largest models are uploaded by non-profits, however, this is mostly due to a few models by LAION<\/a> and Big Science<\/a>, whereas each model is more than hundred gigabytes.<\/p>\n\n\n\n

\"\"\/
Fig 2 Uploaded models, repository size, and count of downloads by (self-assigned) organisation type. X-axis log scale.<\/em><\/figcaption><\/figure>\n\n\n\n

Upload and download are not proportional<\/strong><\/h2>\n\n\n\n

What are the individual organisations that contribute the most on Hugging Face? For universities, the University of Helsinki<\/a>, who provides translation models for various language combinations, is leading by far (Fig 3). However, when it comes to downloads, the Japanese Nara Institute of Science and Technology (NAIST)<\/a> is leading. This is mostly due to MedNER<\/a>, their named entity recognition model for medical documents that is also among the top 10 of the most downloaded models on Hugging Face. German universities are among the organisations with the highest model count but also the most downloads.<\/p>\n\n\n\n

The non-profit repositories with the most models should not be trusted too much, since three of them are entertained by a single PhD student. However, the organisations in this category with the highest download rate are well-known non-profit organisations like LAION and BigCode, that contribute a lot to the open-source community.<\/p>\n\n\n\n

Little surprising, Big Tech companies lead the industry category with respect to upload counts. However, it is neither Google nor Facebook who have the highest download rate but Runway with their Stable Diffusion model<\/a>.<\/p>\n\n\n\n

The top 10 of most downloaded models is led by Wave2Vec<\/a>, a speech recognition model. It is surprising that even though Hugging Face was long known as a platform for unimodal language models, many of the most downloaded models are bimodal for text and either vision or speech.<\/p>\n\n\n\n

\"\"\/
Fig 3 Organisations with the highest upload\/download count.<\/em><\/figcaption><\/figure>\n\n\n\n

Open-source LLMs are on the rise<\/strong><\/h2>\n\n\n\n

Latest since the launch of ChatGPT, LLMs found their way into public attention. However, in line with a recently leaked Memo<\/a> from a Google-developer, open-source is becoming a serious competitor for LLMs. Hugging Face started a leader board<\/a> for open LLMs and some of them reach performance close to ChatGPT.<\/p>\n\n\n\n

Model size, measured in Parameters, increased in recent years drastically (Fig 4). While in 2021 only few models reached 3 billion parameters, currently there are many more and larger open models, reaching almost 70 billion parameters.<\/p>\n\n\n\n

The organisation type that uploads the most LLMs is \u201cOther\u201d. However, this is mostly driven by a few actors, who uploaded sometimes even more than 15 models each. The group that uploads the second most LLMs are companies and they upload LLMs with an average of 10 billion parameters. Universities on the other side do not only upload the fewest LLMs but also the smallest on average.<\/p>\n\n\n\n

\"\"
Fig 4 Parameter count of Large Language Models (LLMs) over time and by organisation type (retrieved from Hugging Face\u2019s LLM Leaderboard).<\/em><\/figcaption><\/figure>\n\n\n\n

Large models, large emissions. Small models, small emissions<\/strong><\/h2>\n\n\n\n

Good news first: training most of the models emitted less CO2<\/sub> than streaming in 4K quality for one hour. The bad news is that only a bit more than 1% of the models at Hugging Face provide this information. 1706 of these 1846 models that include emission information are from \u201cOther\u201d. However, 99.5% of the emissions come from Non-Profit models. This means that training only a few models lead to the biggest share of emissions. This emphasises that the major part of emissions comes from large projects and not from smaller ones<\/a>. But it also emphasises that it needs more transparency and emissions must be documented more consistently.<\/p>\n\n\n\n

\"\"\/
Fig 5 Training emissions (CO2<\/sub>\/kg) for each model and by organisation type. Selection is restricted to repositories that include their emissions in their model card (n = 1846). Pie chart displays the share of repositories that included this information by organisation type. X-axis log scale.<\/em><\/figcaption><\/figure>\n\n\n\n

Summing up<\/strong><\/h2>\n\n\n\n

Hugging Face is the main platform for distributing pre-trained ML models. Without a platform like this, most smaller AI projects would not have access to state-of-the-art models. In a way Hugging Face reflects the current trend in ML research: The field is dominated by a few actors who have the resources to train increasingly large models. However, even though training large models is restricted to actors with sufficient resources, Hugging Face enables projects with less resources to make use of these models. Moreover, even though large companies draw most attention, there is a vivid base of small- and mid-sized projects producing constant output and only small emissions.<\/p>\n\n\n\n

Method & limitations<\/strong><\/h2>\n\n\n\n

All data was scraped beginning of July 2023. As progress happens fast in ML research, the numbers might not be fully accurate already. For information on the models, Hugging Face\u2019s own API was used. Information on the organisations was retrieved with a custom scraper. Users on Hugging Face have the option to upload information sheets, cards as they call them. These cards provide information on organisations, datasets, or models and they are the main source that informs this article. All information on organisations is self-assigned and there are a lot of empty repositories and fake-organisations. For example, while Hugging Face offers more than 260,000 models according to their search engine, only about 150,000 of these repositories are larger than ten megabytes. Since ML models require much memory, it is unlikely that these repositories actually contain models. I believe that the general trend is correct but some details might be inaccurate. The detailed information on individual organisations were retrieved manually and to the best of my knowledge.<\/p>\n\n\n\n

All data can be viewed via https:\/\/github.com\/SamiNenno\/Inside-Huggingface<\/a><\/em><\/p>\n