The text discusses the challenges of using one-hot encoding for handling large categorical data and introduces a solution through the use of embeddings, addressing memory requirements and computational complexity. It details methods for reducing memory footprint, including dimension reduction, hashing, and the quotient-remainder trick, as well as their implementation in TensorFlow. The author also shares their experiments and results, emphasizing the effectiveness of various methods.
“`html
Memory-Efficient Embeddings
Creating smaller models with a new kind of embedding layer
Embeddings
Embeddings are real-valued vectors of the categories that are short and should capture some meaning of the categories. The real numbers are learned during training.
Model size
Embeddings are great, but if you have a lot of categories, you have to store a bunch of real numbers for each of them. The size of this table can get out of hand very easily.
Simple Tricks to Reduce the Embedding Table Size
There are several ways to make this table smaller but usually, there is a trade-off. Reducing the table size reduces your model’s performance.
Compositional Embeddings
Let us now turn to the paper Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems by Shi et al. In their paper, they describe a method to save memory that is better than the hashing trick.
Implementation in TensorFlow
Before we implement this logic in TensorFlow, let us all get on the same page. Let us recap how you can use embedding layers in TensorFlow:
Performance Results
The authors used the Criteo Ad Kaggle Competition dataset for click-through rate prediction to test their model. It has 13 dense features and 26 categorical features. They used two different neural network architectures to test their method: DCM and DLRM.
My experiments
I also played around with it by building a simple recommender for the MovieLens 20M dataset. This dataset is full of movie ratings by users in the form of a star rating from 1 to 5.
Conclusion & Discussion
In this article, we have seen that embeddings are a neat tool to encode categorical data. However, these embeddings can result in huge models that can be hard to deploy, so it is good to know measures to reduce the embedding table size.
Discover how AI can redefine your company
If you want to evolve your company with AI, stay competitive, use for your advantage Memory-Efficient Embeddings.
Spotlight on a Practical AI Solution
Consider the AI Sales Bot from itinai.com/aisalesbot designed to automate customer engagement 24/7 and manage interactions across all customer journey stages.
Discover how AI can redefine your sales processes and customer engagement
Explore solutions at itinai.com.
“`
List of Useful Links:
- AI Lab in Telegram @aiscrumbot – free consultation
- Memory-Efficient Embeddings
- Towards Data Science – Medium
- Twitter – @itinaicom
