Machine Learning Times
Machine Learning Times
EXCLUSIVE HIGHLIGHTS
Video – Credit Models, Microfinance, and Improving the Lives of Families in the Developing World
 Event: Machine Learning Week 2021 Keynote: Credit Models, Microfinance, and...
Video – Identifying Program Effectiveness for Survivors of Human Trafficking from Muneeb Alam of QuantumBlack
 Event: Machine Learning Week 2021 Keynote: Identifying Program Effectiveness for Survivors...
Video – How to Use AI Ethically from Natalia Modjeska of Omdia
 Event: Machine Learning Week 2021 Keynote: How to Use AI...
Video – Alexa On The Edge – A Case Study in Customer-Obsessed Research from Susanj of Amazon
 Event: Machine Learning Week 2021 Keynote: Alexa On The Edge...
SHARE THIS:

3 weeks ago
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World’s Largest and Most Powerful Generative Language Model

 
Originally published in Microsoft Research Blog, Oct 11, 2021

We are excited to introduce the DeepSpeed- and Megatron-powered Megatron-Turing Natural Language Generation model (MT-NLG), the largest and the most powerful monolithic transformer language model trained to date, with 530 billion parameters. It is the result of a research collaboration between Microsoft and NVIDIA to further parallelize and optimize the training of very large AI models.

As the successor to Turing NLG 17B and Megatron-LM, MT-NLG has 3x the number of parameters compared to the existing largest model of this type and demonstrates unmatched accuracy in a broad set of natural language tasks such as:

  • Completion prediction
  • Reading comprehension
  • Commonsense reasoning
  • Natural language inferences
  • Word sense disambiguation

The 105-layer, transformer-based MT-NLG improved upon the prior state-of-the-art models in zero-, one-, and few-shot settings and set the new standard for large-scale language models in both model scale and quality.

Large-scale language models

Transformer-based language models in natural language processing (NLP) have driven rapid progress in recent years fueled by computation at scale, large datasets, and advanced algorithms and software to train these models.

Language models with large numbers of parameters, more data, and more training time acquire a richer, more nuanced understanding of language. As a result, they generalize well as effective zero– or few-shot learners, with high accuracy on many NLP tasks and datasets. Exciting downstream applications include summarization, automatic dialogue generation, translation, semantic search, and code autocompletion. It’s no surprise that the number of parameters in state-of-the-art NLP models have grown at an exponential rate.

To continue reading this article, click here.

Leave a Reply