tokenization

Tokenization is the process of breaking down text or data into smaller units called tokens. In natural language processing, this typically means splitting text into words, subwords, or symbols that can then be analyzed or processed by algorithms. In data security or payments, tokenization refers to replacing sensitive information (like credit card numbers) with non-sensitive equivalents (tokens) to protect the original data.