MongoDB and Qdrant would be great additions to this test :) Great video! Thanks

🎯 Key Takeaways for quick navigation: 00:00 🌐 *Overview of Semantic Search with OpenAI Embeddings* - Introduction to three setups for semantic search using OpenAI embeddings: Pinecone, FAISS, and pgvector. - Overview of the project "dry" aimed at solving repetitive questions in chatbots using semantic search. - Explanation of the schema, involving generating vectors, searching the vector database, and storing metadata for incoming messages. 02:26 🧠 *Understanding Semantic Search and Embeddings* - Definition of embeddings models, with a focus on OpenAI's text embeddings ada002. - Explanation of how embeddings models generate dense vectors representing words, sentences, or paragraphs. - Clarification of how the multi-dimensional space in the model reflects the semantic meaning of different strings. 05:24 🍍 *Pinecone Integration* - Introduction to Pinecone as a fully managed Vector database. - Demonstration of setting up a project, specifying environment variables, and creating a Pinecone index. - Implementation of Pinecone integration in a Google Colab notebook, including generating vectors, saving them, and performing similarity searches. 16:24 💲 *Evaluation of Pinecone* - Efficiency assessment of Pinecone as a managed service for vector storage. - Low effort needed for integration with Pinecone due to available Python and TypeScript examples. - Consideration of costs, with the expectation of a $70/month plan being sufficient for a pet project but potentially expensive for larger-scale commercial use. 18:23 📚 *FAISS Integration* - Introduction to FAISS as a library for efficient similarity search of dense vectors. - Noting nuances of FAISS, including the need to manage large indexes and separate metadata storage. - Implementation of FAISS integration in a Google Colab notebook, demonstrating vector generation, saving, and similarity searches. 28:23 💲 *Evaluation of FAISS* - Recognition of FAISS's efficiency in similarity search with proper usage and fine-tuning. - Intermediate effort level due to additional considerations for metadata storage and potential index management. - Acknowledgment of potentially lower costs as an open-source package but with considerations for high RAM requirements. 29:18 🐘 *pgvector Integration* - Introduction to pgvector as an extension for PostgreSQL, allowing searching by vectors. - Setting environment variables and connecting to a PostgreSQL database using Superbase. - Creation of a table with columns for vector ID, text, embeddings, and metadata, including dimensionality. 34:00 💲 *Evaluation of pgvector* - Recognition of pgvector's efficiency with potential for fine-tuning in usage. - Intermediate effort level, considering the need for an additional database for metadata storage. - Acknowledgment of potential costs, including high RAM requirements and potential additional expenses for hosting metadata. 34:15 📊 *pgvector Database Operations* - Introduction to functions for counting rows and deleting rows by ID in pgvector. - Demonstration of adding test messages to the table, generating embeddings, and saving vectors with metadata. - Usage of functions to search for top relevant messages based on cosine distance and manual filtering. 35:58 🧾 *Testing Similarity Search with pgvector* - Implementation and explanation of the get top relevant messages function in pgvector. - Testing similarity search on three test messages with varying relevance scores. - Display of results, showcasing the closest match with a similarity score of 0.106. 36:41 🔄 *Searching and Storing Similar Messages in pgvector* - Introduction to the gets similar and store function, aiming to search for similar messages and store new messages. - Execution of the function on a list of 91 messages from a Telegram chat. - Confirmation of matches found and the addition of new messages to the pgvector database. 38:13 💼 *Evaluation of pgvector* - Efficiency rating of pgvector with three stars, indicating its capability for the intended purpose. - Effort assessment, noting a moderate complexity in integration, especially for users familiar with SQL queries. - Consideration of potential costs, suggesting minimal impact if an existing PostgreSQL database is already part of the architecture. Made with HARPA AI

Thanks for a very concise and informative video!

Very useful. Thank you.

Great video thanks!

Yes I was specifically thinking about chatbot response speed

Very helpful

Добрий день! Дякую за дуже корисне та своєчасне відео. Маю питання, чи доцільно зберігати ембедінги в pgvector, а для similarity пошуку використовувати faiss?

Quite useful and good comparison. It would have been easier to see using English sentences though. Well done!

Thank you for the great explanation. How does MongoDB fare in this regard?

🎯 Key Takeaways for quick navigation: 00:00 📋 *Introduction to Semantic Search and Comparison of Vector Stores* - Introduction to the video's topic: Semantic search and comparison of vector stores. - Explanation of the project "dry" aimed at creating a chatbot for repetitive questions. - Overview of how semantic search can solve the problem of repetitive queries. 02:00 🤖 *Understanding Semantic Search and Embeddings Models* - Explanation of semantic search and embeddings models. - Description of the OpenAI embeddings model (text embeddings ada002). - How embeddings models generate dense vectors for text strings. 05:24 🌲 *Pinecone: Fully Managed Vector Database* - Introduction to Pinecone as a fully managed vector database. - Setting up a Pinecone project and environment variables. - Demonstrating the integration with Pinecone for semantic search. 09:18 🌟 *Pinecone Efficiency, Effort, and Cost Evaluation* - Evaluation of Pinecone's efficiency for semantic search. - Discussion of the low effort required for integration with Pinecone. - Consideration of Pinecone's cost, especially for commercial projects. 20:06 🧊 *Facebook Faiss: Efficient Similarity Search* - Introduction to Facebook Faiss as an efficient similarity search library. - Setting up environment variables for Faiss and pidgey vector. - Integration of Faiss with the ability to store metadata. 26:30 ⚙️ *Faiss Efficiency, Effort, and Cost Evaluation* - Evaluation of Faiss's efficiency for semantic search. - Discussion of the initial integration effort and the need for metadata storage. - Consideration of costs associated with Faiss, including RAM requirements. 29:32 🐘 *PGVector: PostgreSQL Extension for Vector Search* - Introduction to PGVector as a PostgreSQL extension for vector search. - Overview of using PGVector to search by vectors in PostgreSQL. - Setting up PostgreSQL for PGVector integration. 29:44 🧩 *Setting Up Environment Variables and Database Connection* - Setting environment variables for API keys, connection string, similarity limit, and top K records. - Explanation of options to obtain a PostgreSQL database. - Introduction to using Superbase as an online service for PostgreSQL. 31:16 🧬 *Integration with OpenAI API and PostgreSQL Database* - Installation of the OpenAI package and confirmation of interaction with the OpenAI API. - Establishing a connection to the PostgreSQL database and creating a test table. - Installing the PGVector extension for PostgreSQL. 34:28 🛠️ *Generating Embeddings, Storing Data, and Testing Similarity Search* - Explanation of functions for generating embeddings, creating vector IDs, and storing data in the table. - Testing similarity search using cosine distance and filtering results. - Applying the "get similar and store" function to a list of test messages. 38:13 💼 *PGVector Efficiency, Effort, and Cost Evaluation* - Evaluation of PGVector's efficiency in the context of the project. - Discussion of the complexity of integration compared to Pinecone. - Consideration of cost implications, including using existing PostgreSQL databases and managed services. Made with HARPA AI

When using a hosted pgvector postgre service, would the cost be more like 2.5 stars?

Can you do a video with FAISS and Any DB (Postgres)?

5:14 Pinecone 18:24 FIASS 29:17 pgvector

I wish you did not use langchain!

Which is fastest?

Nice one mate. Just wish you did not use the russian text. As I do not understand russian, the results do not make any sense to me. But I am just believing... rather than knowing and seeing with my own eyes.