The article details the development of a semantic search engine for emojis, aiming to address the limitations of existing emoji search methods by incorporating both textual and visual information. The author outlines the challenges encountered and the strategies employed, ultimately creating a search engine that effectively navigates the overlap between two traditionally distinct modalities: images and text.
### **Find The Sentiment You’re Looking For 🔍🤔😀🚀**
#### Semantic search over emojis for “halloween” using a custom emoji search engine.
If you’ve ever used Google Docs, or Slack, you may have noticed that when you type a “:” immediately followed by another character, a list of emojis pops up:
Since I discovered this, I’ve been making major use out of the feature. I add emojis into way more of my messages, blog posts, and other written works than I ever imagined I would. I actually got so accustomed to this means of adding emojis that I installed Rocket — a free app that brings the same emoji searchability to all text boxes and text editors on the computer. It’s a game changer.
But as I’ve used these emoji search engines more and more, I’ve noticed a frustrating limitation: all of the searches are based on the exact text in your query and in the name and description of the emoji. Essentially, you need to search for something incredibly precisely for any results to show up.
Here’s an example: if we search for “audio”, not a single result shows up:
This isn’t because the set of emojis is lacking in the audio category. If we were to type in “music” or “speaker”, we would get a long list of results. Instead, it has to do with the fact that the specific string of text “audio” does not show up in the name or textual description associated with any of the emojis.
This relatively minor inconvenience bothered me so much that I decided to build this:
By “this”, I mean an open-source semantic emoji search engine, with both UI-centric and CLI versions. The Python CLI library can be found here, and the UI-centric version can be found here. You can also play around with a hosted (also free) version of the UI emoji search engine online here.
#### Command line version of the Semantic Emoji Search Engine
Building this was not as simple or straightforward as I initially hoped. It took a lot of experimentation, and a lot of ideas I thought were quite clever fell essentially flat. But in the end, I was able to create an emoji search engine that works fairly well.
#### **What is an Emoji**
#### The Data
#### Emojis versus Images and Text
#### Bridging the Modality Gap
#### Using the Emoji Search Engine
The resulting search engine isn’t perfect, but it does a decent job at incorporating textual and visual information. Because using a cross-encoder is more computationally expensive (and higher latency), this is reserved for the pared-down set of candidates. I use the distilroberta-base checkpoint with the CrossEncoder class from the Sentence Transformers library.
When all of these steps are combined, this is the result:
Again, it isn’t perfect. But it’s not bad!
#### Using the Emoji Search Engine
There are three ways to use this emoji search engine: hosted (free), locally via UI (open source), or locally via command line (also open source). All three options are quite easy!
#### **Online**
Head over to [try.fiftyone.ai/datasets/emojis](https://try.fiftyone.ai/datasets/emojis), sign in (it’s free), and click on the emoji button in the menu above the grid of images. That’s it!
#### **Locally via the UI**
If you want to perform emoji searches locally with the same visual interface, you can do so with the Emoji Search plugin for FiftyOne.
First, install FiftyOne:
“`bash
pip install fiftyone
“`
Then download the Emoji Search plugin and install its requirements:
“`bash
fiftyone plugins download https://github.com/jacobmarks/emoji-search-plugin
fiftyone plugins requirements @jacobmarks/emoji_search –install
“`
Launch the FiftyOne App:
“`bash
fiftyone app launch
“`
Click on the “browse operations” text, search for “emoji”, and click on the entry “Create Emoji Dataset”. This will download the high-resolution images of the emojis, along with embeddings and all other relevant data. At the top left of the app, click in the “Select dataset” box and select “Emojis”. Now you should see the same UI as in the hosted version.
#### **Locally via the CLI**
Finally, you can search via the command line using the Emoji Search Python CLI library. Install the package from GitHub repository with:
“`bash
pip install git+https://github.com/jacobmarks/emoji_search.git
“`
Then you can start searching using the emoji-search command, followed by the text query (with or without quotation marks).
“`bash
emoji-search beautiful sunset
“`
The first search you perform will download embeddings to your device if necessary.
All three versions support copying an emoji to clipboard with pyperclip. In the UI, click on the image for an emoji, and you’ll see a copy button appear in the menu. In the CLI, pass the -c argument to copy the top result to clipboard.
#### **Conclusion**
Emojis might seem like a silly subject to obsess over. And in practice, the utility of a semantic emoji search engine over lexical emoji search may be somewhat limited. The real value in this endeavor is in understanding the boundaries and overlaps between two modalities we traditionally think of as distinct: images and text. Emojis sit squarely in this intersection and as such, they allow us to probe the strengths and weaknesses — the capabilities and limitations of today’s multimodal models.
The Semantic Emoji Search Engine I ended up building is far from perfect. Frankly, emojis have subjectivity, connoting different things for different people, that is impossible to precisely bottle up. But going back to the motivating example, when I type in “an audio player”, I get some solid results:
I’ll end with a quote from Nancy Gibbs, a Professor at the Harvard Kennedy School and former managing editor for TIME magazine:
What makes emojis special is the fact that [they have] helped millions express themselves better than even the wide array of words in the Oxford dictionary [could].
Nancy Gibbs
Note: All images in article created by the author unless otherwise noted
**How to Build a Semantic Search Engine for Emojis** was originally published in **Towards Data Science** on **Medium**, where people are continuing the conversation by highlighting and responding to this story.
If you want to evolve your company with AI, stay competitive, use for your advantage **How to Build a Semantic Search Engine for Emojis**.
**Discover how AI can redefine your way of work. Identify Automation Opportunities: Locate key customer interaction points that can benefit from AI.
Define KPIs: Ensure your AI endeavors have measurable impacts on business outcomes.
Select an AI Solution: Choose tools that align with your needs and provide customization.
Implement Gradually: Start with a pilot, gather data, and expand AI usage judiciously.
For AI KPI management advice, connect with us at [hello@itinai.com](mailto:hello@itinai.com). And for continuous insights into leveraging AI, stay tuned on our [Telegram](https://t.me/itinainews) or [Twitter](https://twitter.com/itinaicom).
**Spotlight on a Practical AI Solution:**
Consider the AI Sales Bot from [itinai.com/aisalesbot](https://itinai.com/aisalesbot) designed to automate customer engagement 24/7 and manage interactions across all customer journey stages.
**Discover how AI can redefine your sales processes and customer engagement. Explore solutions at [itinai.com](https://itinai.com).**
