Sunday Links: Meta, Risk, and Model Collapse

This week, Meta almost dominates the headlines with its continuing push in open source and a model that is now officially "risky" in the EU. Let's get into the links:

• Open Source AI Is the Path Forward. On Tuesday, Meta's CEO, Mark Zuckerberg, published a lengthy open letter outlining his belief that open-source was essential to both the success and safety of AI (the letter is well worth a read). It also contained clear statements of intent that Meta would continue to open-source its generative AI models for the foreseeable future. The business reasons behind this are worth a post on their own, but suffice it to say it means the level of competition in the LLM space will remain very high. The company is also leaning hard into partnerships with providers to run/host llama models. I agree with Zuckerberg's two core points: 1) that a world in which the obvious power of AI is available only to a few large providers would be an extremely negative one, open source spreads the economic good far more widely, and 2) in the long haul, while open models can enable bad actors, spreading knowledge and scrutinizing systems in public will be far safer than hoping a few closed source systems never leak or are compromised. Having said this, we must all invest heavily in putting in place the right controls and validation against harm.
• Meta unleashes its most powerful AI model, Llama 3.1, with 405B parameters. Hot on the heels of Mark Zuckerberg's letter, Meta released its most powerful open-source generative AI model. The model is already comparing extremely well to the best available models (from OpenAI and Google) and exceeding some benchmarks. (Here is the release post with benchmark scores.) Meta has also updated the open source terms to make them more friendly for commercial use and fine tunes than before. This throws down the gauntlet to other model providers, and Meta clearly plans to continue investing. The company also released a detailed research paper on the training process, which is very valuable in itself.
• ... and now perhaps we know why Meta has decided not to release its most powerful models in the EU. The EU AI Act defines an AI model to be high impact (and hence a potential systemic risk) as follows: "A general-purpose AI model shall be presumed to have high impact capabilities pursuant to paragraph 1, point (a) when the cumulative amount of computation used for its training measured in floating point operations is greater than 10(^25)." Llama 3.1 405B used 3.8 x 10(^25) flops in pre-training and so exceeds this limit. No doubt Meta was unwilling to enter the regulatory debate about what should now happen, given they are not monetizing the model. It's likely rather surprising to many that we have already hit the cap in the AI Act (and likely Meta is not the only one to hit it). The result is LLama 3.1 likely won't be available in the EU anytime soon. Meta has also announced it is stopping training models in the EU due to regulatory concerns and a direct request from the European privacy watchdog. This also means EU data will not be used in model training for the time being. If this ultimately results in a better agreement on opting in/out or data usage rules, perhaps this is a good thing. If, on the other hand, it merely means that the most popular AI models in the world know less about EU culture, then that's much less of a win.
• OpenAI announces SearchGPT, its AI-powered search engine. In a not entirely surprising move, OpenAI has released demos of a new search interface that uses ChatGPT to search the web and summarize answers in much the way Perplxity does. The service is in Beta for now and will be available to around 10,000 users. Assuming results are high quality, this seems like it will be an obviously useful service (search, summarize, allow exploration of the results) that all search engines will need going forward. Meta.AI and Google's own search-integrated AI, as well as perplexity, will be similar. I think it also shows OpenAI's ambition to become the primary UI layer for personal internet usage. On average, Internet users conduct 3-4 searches daily. Capturing a significant share of that flow is the holy grail of a consumer-facing Internet product.
• AI models collapse when trained on recursively generated data. To escape all the corporate moves, we'll wrap up with a more scientific paper. This study, published in Nature Communications, looks at what happens to answer quality in LLMs if they recursively train on their own outputs. The researchers conclude that model degradation ultimately sets in, and answer quality deteriorates rapidly. The team also highlights the potential dangers of the public Internet, both being a place for much new AI-generated content and a common source of training data. While it's useful to have a study confirm this, this really shouldn't be a surprise to anyone. Multiple iterations of inference and training are almost guaranteed to wash out nuanced and then ultimately any reality in results. What makes the model work well is real-world feedback on answers - either from human supervision, simulation, or real-world reactions.

Wishing you a happy Sunday!