small language model

• why we need SLMs?
  • LLMs are large and need more computational power to run and train
  • SLMs contain less than 10B⇐ params whereas LLMs have 100B+
  • SLMs are fast and efficient and can be deployed on small devices like phones, laptops, IoTs and can be fine tuned for specific tasks like healthcare, legal, customer support etc
  • On device privacy

1. Dataset

• tiny stories dataset - https://huggingface.co/datasets/roneneldan/TinyStories generated by GPT-3.5 and 4 - https://arxiv.org/pdf/2305.07759
• the goal of this dataset is to have as small dataset as possible while having all the concise information to teach something to the model
• so our task is to use this dataset which contains many tiny stories in english for 3-5 year old children, can we successfully teach english to our SLM which is going to be only 15M params - if yes, then our goal is achieved!

Goal: can we construct a language model with just 10-15M params and which produces coherent text?

2. Data pre-processing

3. Assembling the model architecture

4. Setting up the SLM training pipeline

5. Pre-training the SLM

6. Running inference

Links:

202608280808