Pretraining

التدريب المسبق

المرحلة الأولى الكثيفة لتدريب النموذج على كميات هائلة من البيانات العامة لتعلم قواعد اللغة الأساسية.

Pretraining

Also translated asالتهيئة الأولية الشاملة، التأهيل القبلي للنموذج

First appears in this corpus in: Reducing the Dimensionality of Data with Neural Networks (2006)

Appears in these papers

BEiT: BERT Pre-Training of Image Transformers2021 in the sky ✦
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2018 in the sky ✦
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2018 in the sky ✦
Chronos: Learning the Language of Time Series2024 in the sky ✦
Learning Transferable Visual Models from Natural Language Supervision2021 in the sky ✦
Constitutional AI: Harmlessness from AI Feedback2022 in the sky ✦
Representation Learning with Contrastive Predictive Coding2018 in the sky ✦
DeBERTa: Decoding-Enhanced BERT with Disentangled Attention2020 in the sky ✦
Reducing the Dimensionality of Data with Neural Networks2006 in the sky ✦
Reducing the Dimensionality of Data with Neural Networks2006 in the sky ✦
A Fast Learning Algorithm for Deep Belief Nets2006 in the sky ✦
A Fast Learning Algorithm for Deep Belief Nets2006 in the sky ✦
Extracting and Composing Robust Features with Denoising Autoencoders2008 in the sky ✦
Extracting and Composing Robust Features with Denoising Autoencoders2008 in the sky ✦
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World2017 in the sky ✦
ELECTRA: Pre-Training Text Encoders as Discriminators Rather Than Generators2020 in the sky ✦
Deep Contextualized Word Representations2018 in the sky ✦
Deep Contextualized Word Representations2018 in the sky ✦
The Falcon Series of Open Language Models2023 in the sky ✦
The Falcon Series of Open Language Models2023 in the sky ✦
Explaining and Harnessing Adversarial Examples2015 in the sky ✦
Flamingo: a Visual Language Model for Few-Shot Learning2022 in the sky ✦
Finetuned Language Models Are Zero-Shot Learners2022 in the sky ✦
Improving Language Understanding by Generative Pre-Training2018 in the sky ✦
Language Models Are Unsupervised Multitask Learners2019 in the sky ✦
Language Models Are Few-Shot Learners2020 in the sky ✦
GPT-4 Technical Report2023 in the sky ✦
GPT-4 Technical Report2023 in the sky ✦
Group Normalization2018 in the sky ✦
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017 in the sky ✦
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017 in the sky ✦
ImageNet: A Large-Scale Hierarchical Image Database2009 in the sky ✦
Training Language Models to Follow Instructions with Human Feedback2022 in the sky ✦
Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes2019 in the sky ✦
LIMA: Less Is More for Alignment2023 in the sky ✦
LIMA: Less Is More for Alignment2023 in the sky ✦
Visual Instruction Tuning2023 in the sky ✦
LoRA: Low-Rank Adaptation of Large Language Models2021 in the sky ✦
Masked Autoencoders Are Scalable Vision Learners2022 in the sky ✦
Momentum Contrast for Unsupervised Visual Representation Learning2020 in the sky ✦
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer2022 in the sky ✦
Open X-Embodiment: Robotic Learning Datasets and RT-X Models2024 in the sky ✦
Open X-Embodiment: Robotic Learning Datasets and RT-X Models2024 in the sky ✦
OPT: Open Pre-Trained Transformer Language Models2022 in the sky ✦
OPT: Open Pre-Trained Transformer Language Models2022 in the sky ✦
REALM: Retrieval-Augmented Language Model Pre-Training2020 in the sky ✦
REALM: Retrieval-Augmented Language Model Pre-Training2020 in the sky ✦
Rectified Linear Units Improve Restricted Boltzmann Machines2010 in the sky ✦
Representation Learning: A Review and New Perspectives2013 in the sky ✦
RETRO: Improving Language Models by Retrieving from Trillions of Tokens2022 in the sky ✦
RoBERTa: A Robustly Optimized BERT Pretraining Approach2019 in the sky ✦
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control2023 in the sky ✦
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019 in the sky ✦
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019 in the sky ✦
Toolformer: Language Models Can Teach Themselves to Use Tools2023 in the sky ✦
Universal Language Model Fine-Tuning for Text Classification2018 in the sky ✦
Universal Language Model Fine-Tuning for Text Classification2018 in the sky ✦
An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale2020 in the sky ✦
VQA: Visual Question Answering2015 in the sky ✦
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision2023 in the sky ✦
Distributed Representations of Words and Phrases and Their Compositionality2013 in the sky ✦
Understanding the Difficulty of Training Deep Feedforward Neural Networks2010 in the sky ✦
XLNet: Generalized Autoregressive Pretraining for Language Understanding2019 in the sky ✦
You Only Look Once: Unified, Real-Time Object Detection2015 in the sky ✦