All-Reduce
الاختزال الجماعي
عملية اتصال جماعية في الحوسبة المتوازية تجمع (مثلاً بالجمع) مصفوفات من عدة معالجات وتوزّع النتيجة على جميع المعالجات. أساسية لتوازي البيانات في تدريب الشبكات العصبية العميقة.
A collective communication operation in parallel computing that combines (e.g. sums) matrices from multiple processors and distributes the result to all processors. Essential for data parallelism in deep neural network training.
Also translated asاختزال شامل
First appears in this corpus in: Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017)
Appears in these papers
- The Falcon Series of Open Language Models2023in the sky ✦
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour2017in the sky ✦
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism2019in the sky ✦
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models2019in the sky ✦
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models2019in the sky ✦