What Is Pathways Language Model
Why do we need it. How does it work. Where is it used.
Hey reader, welcome to the š„ free edition š„ of my weekly newsletter. I write about ML concepts, how to build ML products, and how to thrive at work. You can learn more about me here. Feel free to send me your questions and Iām happy to offer my thoughts. Subscribe to this newsletter to receive it in your inbox every week.
Google Research recently announced the Pathways Language Model (PaLM). It's a 540-billion parameter model for language modeling. How does it work? And why is it important?
First of all, what is āPathwaysā?
Google Research introduced the concept of Pathways in Oct 2021. It's a new AI architecture to allow a single model to handle many tasks.
Today's machine learning systems tend to overspecialize at individual tasks. It means that you have to build individual models for every single task. And these individual models cannot do anything else. The Pathways architecture is an attempt towards building a single model that can do many tasks.
Just like how our brain has multiple capabilities that can be called upon as needed, the goal is to build an AI model that has multiple capabilities. We can use these capabilities as and when needed. We can even stitch together these capabilities to perform more complex tasks.
They achieved a key milestone by developing the new Pathways system to orchestrate distributed computation for accelerators. You can read more about it here.
What exactly is PaLM?
PaLM is a model that can perform language-related tasks. Examples including language understanding, summarization, explaining jokes, translation, question answering, code completion, and more.
To take a step further, it's a dense decoder-only transformer model with 540 billion parameters. It has been trained with the Pathways system using 6,144 chips. Thatās a lot of chips! The training dataset contains a combination of English and multilingual datasets including documents, books, Wikipedia, conversations, and GitHub code.
What can PaLM do?
PaLM can perform a variety of tasks that require human intelligence:
Distinguishing cause and effect: In any given text, it can understand what's the cause and what's the effect.
Reasoning: It can follow chain of thought and make deductions.
Explaining a joke: Jokes have been a uniquely human endeavor. But now we have a system that can read the given text and explain the joke to us. Iām sure there are many people out there who could use it.
Generating code: You can describe a task in plain english and it will generate the corresponding code for it. You can then run this program, all without ever having to write any code.
Fixing compilation errors: Computers have been identifying errors for a long time now, but they weren't able to fix them. PaLM can look at a code written by someone and fix the errors by understanding what the code is trying to achieve.
Why is it important?
Building PaLM is an amazing feat of research and engineering. It allows people to use the Pathways architecture to build more models. As they've mentioned in their article, the vision for the Pathways system is to "enable a single AI system to generalize across thousands or millions of tasks, to understand different types of data, and to do so with remarkable efficiency". This provides a roadmap to build AI models that can generalize across many different domains and tasks.
šš„ Two new episodes on the Infinite ML pod
Denis Rothman: We talk NLP, metahuman AI, the importance of practice in machine learning, why use transformers, working with customers, interviewing people, and thriving at work. You can listen to it on Apple Podcasts, Spotify, and Google Podcasts.
Margaretta Colangelo: We talk about the state of AI in healthcare, 2021 milestones in this field, cancer research, AI-power drug discovery, the process of writing, and building a career. You can listen to it on Apple Podcasts, Spotify, and Google Podcasts.
š Job Opportunities in AI
Check out this job board for the latest opportunities in AI. It features a list of open roles in Machine Learning, Data Science, Computer Vision, and NLP at startups and big tech.
šš»āāļø šš»āāļø How would you rate this weekās newsletter?
You can rate this newsletter to let me know what you think. Your feedback will help make it better.