RMSNorm

تسوية الجذر التربيعي للمتوسط

تبسيط لتسوية الطبقة يُعيد قياس المتجهات بجذرها التربيعي للمتوسط دون طرح المتوسط الحسابي. أسرع حوسبياً وفعّالة تجريبياً بالقدر ذاته لاستقرار التدريب. اعتمدها LLaMA بدلاً من LayerNorm.

A simplification of LayerNorm that rescales vectors by their root mean square without subtracting the mean. Faster and empirically as effective for training stability. Adopted by LLaMA instead of LayerNorm.

Also translated asتسوية RMS، التسوية بالجذر التربيعي

First appears in this corpus in: Layer Normalization (2016)

Appears in these papers

DeepSeek-V3 Technical Report2024 in the sky ✦
Gemma: Open Models Based on Gemini Research and Technology2024 in the sky ✦
Scaling Language Models: Methods, Analysis & Insights from Training Gopher2022 in the sky ✦
Layer Normalization2016 in the sky ✦
Llama 2: Open Foundation and Fine-Tuned Chat Models2023 in the sky ✦
The Llama 3 Herd of Models2024 in the sky ✦
LLaMA: Open and Efficient Foundation Language Models2023 in the sky ✦
LLaMA: Open and Efficient Foundation Language Models2023 in the sky ✦
Mistral 7B2023 in the sky ✦