Tokenization

تجزئة النصوص

عملية تفكيك وتقسيم النص المسترسل إلى وحدات لغوية صغيرة جاهزة للتحويل الرقمي.

Tokenization

Also translated asترميز الوحدات اللغوية، تقسيم النص الرقمي

First appears in this corpus in: ELIZA — A Computer Program for the Study of Natural Language Communication (1966)

Appears in these papers

Neural Machine Translation of Rare Words with Subword Units2016 in the sky ✦
Chronos: Learning the Language of Time Series2024 in the sky ✦
Chronos: Learning the Language of Time Series2024 in the sky ✦
Evaluating Large Language Models Trained on Code2021 in the sky ✦
DeepSeek-V3 Technical Report2024 in the sky ✦
DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter2019 in the sky ✦
ELIZA — A Computer Program for the Study of Natural Language Communication1966 in the sky ✦
ELIZA — A Computer Program for the Study of Natural Language Communication1966 in the sky ✦
Emergent Abilities of Large Language Models2022 in the sky ✦
A Generalist Agent2022 in the sky ✦
A Generalist Agent2022 in the sky ✦
Gemini: A Family of Highly Capable Multimodal Models2023 in the sky ✦
Gemini: A Family of Highly Capable Multimodal Models2023 in the sky ✦
Gemma: Open Models Based on Gemini Research and Technology2024 in the sky ✦
Scaling Language Models: Methods, Analysis & Insights from Training Gopher2022 in the sky ✦
Scaling Language Models: Methods, Analysis & Insights from Training Gopher2022 in the sky ✦
MusicLM: Generating Music From Text2023 in the sky ✦
OPT: Open Pre-Trained Transformer Language Models2022 in the sky ✦
OPT: Open Pre-Trained Transformer Language Models2022 in the sky ✦
A Time Series Is Worth 64 Words: Long-Term Forecasting with Transformers2023 in the sky ✦
Textbooks Are All You Need2023 in the sky ✦
RoBERTa: A Robustly Optimized BERT Pretraining Approach2019 in the sky ✦
Toolformer: Language Models Can Teach Themselves to Use Tools2023 in the sky ✦