Machine Learning Times
Machine Learning Times
A University Curriculum Supplement to Teach a Business Framework for ML Deployment
    In 2023, as a visiting analytics professor...
The AI Playbook: Providing Important Reminders to Data Professionals
 Originally published in DATAVERSITY. This article reviews the new...
Decode the Algorithm: Navigate the World of Machine Learning in Business with ‘The AI ​​Playbook’
  This article reviews the new book, The AI Playbook, by...
To Deploy Machine Learning, You Must Manage Operational Change—Here Is How UPS Got It Right
 Originally published in Harvard Data Science Review. For more...

1 year ago
Productizing Large Language Models

Originally posted on, Sept 21, 2022. 

Large Language Models (LLMs) are known for their near-magical ability to learn from very few examples — as little as zero — to create language wonders. LLMs can chat, write poetry, write code, and even do basic arithmetic. However, the same properties that make LLMs magical also make them challenging from an engineering perspective.

At Replit we have deployed transformer-based language models of all sizes: ~100m parameter models for search and spam, 1-10B models for a code autocomplete product we call GhostWriter, and 100B+ models for features that require a higher reasoning ability. In this post we’ll talk about what we’ve learned about building and hosting large language models.


Any sufficiently advanced bullshit is indistinguishable from intelligence, or so the LLM thought. LLMs are super suggestible — in fact, the primary way to interact with LLMs is via “prompting.” Basically, you give the LLM a string of text and it generates a response, mostly in text form although some models can also generate audio or even images. The problem is, you can prompt the LLM with nonsense and it will generate nonsense. Garbage in, garbage out. Also, LLMs tend to get stuck in loops, repeating the same thing over and over again, since they have a limited attention span when dealing with some novel scenarios that were not present during training.

To continue reading this article, click here.

10 thoughts on “Productizing Large Language Models

  1. Pingback: Productizing Large Language Models « Machine Learning Times ✔️ Autocomp

Leave a Reply