Researchers have developed a breakthrough in training language models with the creation of datasets like TinyStories and CodeTextbook. These datasets were used to train small language models of around 10 million parameters, resulting in the generation of fluent narratives and high-quality content. By carefully selecting publicly-available data and filtering it based on educational value, researchers were able to train a more capable SLM named Phi-1.
The process involved repetitive filtering of content and the development of a prompting and seeding formula to ensure high-quality data for training. The resulting dataset, CodeTextbook, mimicked the approach of a teacher breaking down difficult concepts for students, making it easier for language models to read and understand.
To address potential safety challenges, developers undertook a multi-layered approach in training the Phi-3 models, including additional examples and feedback, assessment testing, and manual red-teaming. They also utilized tools available in Azure AI to build more secure and trustworthy applications.
While small language models have limitations compared to larger models in-depth knowledge retrieval, they are still valuable for certain tasks. Large language models excel in complex reasoning over vast amounts of data, making them ideal for applications like drug discovery.
Companies can offload specific tasks to small models if the complexity is minimal, such as summarizing documents, generating copy, or powering support chatbots. Microsoft has implemented suites of models where large models act as routers, directing queries to small models for less computing-intensive tasks.
It is important to understand the strengths and weaknesses of different model sizes, as small language models are uniquely positioned for edge computing and device-based tasks. While there may always be a gap between small and large models, progress continues to be made in advancing language model capabilities.
Overall, the research into small language models represents a significant step forward in AI development, with the potential for a wide range of applications across various industries.
