Product Category Memo #9: Vector databases
Analysis of the landscape of vector database products
Welcome to Infinite Curiosity, a weekly newsletter that explores the intersection of Artificial Intelligence and Startups. Tech enthusiasts across 200 countries have been reading what I write. Subscribe to this newsletter for free to receive it in your inbox every week:
The category of vector database products has exploded in the last 12 months. AI application developers need an efficient way to look through huge amounts of data and extract knowledge from it. Vector databases play a central role in this. To get us in the mood, I asked DALL-E to generate an impressionist oil painting of vector databases:
In this post, we’ll talk about:
What is a vector database
Why do we need vector database products
What does a good vector database look like
What products are competing in this market
What factors drive pricing
How these products acquire customers
Before we start, this post uses many concepts discussed in the previous post. If you haven't read it, give it a quick read before proceeding.
Alright let’s dive in.
What is a vector database?
The amount of unstructured data being gathered is increasing everyday. Operating on raw data is slow and expensive. Why slow? Because it's just a lot of data you have to go through. Why expensive? Because your cloud compute bills increase in proportion to the size of the data.
You need to convert this raw data into numerical form so that AI algorithms can extract information from it. This numerical form is called a vector (short for vector embedding).
These vectors need to be stored somewhere so that you can work on them. A vector database is a database that's built specifically to work with vectors.
Why do we need vector database products?
A vector database can index vectors, make them searchable, and retrieve them quickly as needed. This is hugely valuable for AI applications, especially the ones that need search capability.
Let's say you offer a product that stores text, images, videos, and audio for your customers. You might eventually gather terabytes of data. When a user wants to look for something, they want to type "show me all the snippets where I've talked about cryptography". The snippet can refer to a paragraph, an image, a portion of a video file, or a portion of an audio file. If you conduct a brute force search every time a user asks a question like this, it will take forever to retrieve all the relevant snippets. And it's expensive for you because you'll consume of a lot of compute power. That's where vector databases come in.
They can convert all the raw data to vectors and keep it ready. This allows you to retrieve the answers instantly and show it to your user.
What does a good vector database look like?
It should offer the following capabilities:
Ability to index the vectors: This is a given, but needs to be said. A good vector database allows the user to index the vectors such that they can be used for efficient search and retrieval.
Ability to scale: In AI, developers usually have to deal with enormous quantities of data. It means that the number of vectors they have to deal with is large. And it keeps getting larger. The vector database needs to be able to scale horizontally. What is "horizontal" scaling? When a single server can't handle the load, you have to add more servers to handle it. This allows you to distribute the load. And the database should work seamlessly across these multiple servers. This is done using data sharding. For the sake of completion, "vertical" scaling refers to making the individual servers bigger (as opposed to adding more servers of the same capacity).
Filtering: Conducting a search on the entire database every single time is not very efficient. Based on the input, you should conduct the search on a subset of the database that's relevant to the input. This allows you to speed up the search. But determining which items to include in the subset is a time-consuming process. You can either pre-filter the vectors to create the subset by parsing the metadata (slower, more accurate). Or you can do post-filtering by applying the nearest neighbor search to get the list (faster, less accurate). A good vector database combines the goodness of both pre-filtering and post-filtering to allow the user to conduct a search on a subset of the database. This allows it to be fast and accurate.
Equipped with APIs and SDKs: A good vector database is equipped with APIs and SDKs. This allows developers to use the database across a variety of applications, languages, and environments.
Handling multiple users: The database should be able to handle multiple users. But it should do so without creating a new database for every user.
Isolation: If you take an action on one collection in the database, it should not be visible to the overall system. The action could be querying, inserting, deleting, or modifying the data. If should only be allowed if the admin gives the permission.
Availability: We've been talking about handling vector searches. Let's say we have many users who keep inputting queries. And we need our vector database to handle all those searches in parallel. Our vector database system is designed to handle only so many queries at once. After a certain point, the searches will get backed up. Each new query has to wait in line. If this happens, it would lead to poor user experience. The vector database needs to be available at all times to all users. How do we do this? By creating replicas of our vector database system (including all the servers in our horizontally scaled setup).
Storage optimization: For vector databases to work well, they need to run in-memory (i.e. RAM). But RAM is expensive. As the size of database increases, you can store it all on RAM but it quickly becomes prohibitively expensive. How do we resolve it? By storing the full resolution database on a disk (inexpensive, but slower) and storing the compressed vector index on the RAM (expensive, but faster). This allows you to quickly run the initial search in memory, choose the relevant subset, and then conduct the search on the disk with only the relevant subset.
Ability to tune: Different applications tend to leverage different aspects of the vector database. Some applications need to handle high insert rate where as other applications need to handle high query rate. A good vector database allows the user to tune the database to their needs.
Good user interface: It's important for the admin to have full control and clear visibility into the database system. A good vector database product offers this to the users.
What products are competing in this market?
Here are the companies in this market:
Zilliz (powered by the open source vector database Milvus)
What factors drive pricing?
Here are the factors that determine the pricing of a product:
Number of compute units per database
Number of replicas
Amount of storage
Availability across different zones
Number of projects
Number of environments
Uptime guarantee requirements
Data retention requirements
Frequency of backups
How these products acquire customers?
These products are oriented towards developers. The customer acquisition process is mainly centered on technical channels.
Open-source adoption: This has been a popular channel in this category. Products aim to get individual developers to adopt it by providing an open source package. That's how a product makes its way into an organization.
Freemium model: Companies like Pinecone have taken this approach. They offer a free tier for developers to try. That's how a product makes its way into an organization. Once it's in, you can approach them directly to sign a bigger deal.
Slack/Discord communities: This is a critical customer acquisition channel. There needs to be a strong developer community around a product, especially since this is a relatively newer category. The strength of this community will dictate how many practitioners adopt a new database.
Content marketing: Companies in this category have been putting out good technical content. The goal is to share knowledge about the technology. And establish yourself as the go-to destination for any questions related to this topic. This attracts developers to your company and your product.
If you are getting value from this newsletter, consider subscribing for free and sharing with your friends: