Data Parallelism

توازي البيانات

استراتيجية تدريب تُوزّع دُفعات البيانات على عدة وحدات معالجة رسومية، كل منها تحتفظ بنسخة من النموذج وتُجمع التدرجات بعد كل خطوة. تُستخدم مع توازي النموذج في تدريب LLaMA.

A training strategy that distributes data batches across multiple GPUs, each holding a copy of the model and aggregating gradients after each step. Used alongside model parallelism in LLaMA training.

Also translated asالتوازي في البيانات، توزيع مصفوفات البيانات عبر المعالجات، معالجة البيانات بالتوازي، معالجة حزم المدخلات المختلفة بالتزامن

First appears in this corpus in: Deep Speech 2: End-to-End Speech Recognition in English and Mandarin (2015)

Appears in these papers

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model2022 in the sky ✦
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model2022 in the sky ✦
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin2015 in the sky ✦
DeepSeek-V3 Technical Report2024 in the sky ✦
The Falcon Series of Open Language Models2023 in the sky ✦
The Falcon Series of Open Language Models2023 in the sky ✦
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning2023 in the sky ✦
Gemma: Open Models Based on Gemini Research and Technology2024 in the sky ✦
Gemma: Open Models Based on Gemini Research and Technology2024 in the sky ✦
Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation2016 in the sky ✦
Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation2016 in the sky ✦
Scaling Language Models: Methods, Analysis & Insights from Training Gopher2022 in the sky ✦
Scaling Language Models: Methods, Analysis & Insights from Training Gopher2022 in the sky ✦
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures2018 in the sky ✦
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour2017 in the sky ✦
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour2017 in the sky ✦
The Llama 3 Herd of Models2024 in the sky ✦
The Llama 3 Herd of Models2024 in the sky ✦
LLaMA: Open and Efficient Foundation Language Models2023 in the sky ✦
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism2019 in the sky ✦
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism2019 in the sky ✦
OPT: Open Pre-Trained Transformer Language Models2022 in the sky ✦
OPT: Open Pre-Trained Transformer Language Models2022 in the sky ✦
PaLM: Scaling Language Modeling with Pathways2022 in the sky ✦
PaLM: Scaling Language Modeling with Pathways2022 in the sky ✦
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity2022 in the sky ✦
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity2022 in the sky ✦
Robust Speech Recognition via Large-Scale Weak Supervision2022 in the sky ✦
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models2019 in the sky ✦
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models2019 in the sky ✦