This tutorial truly stands out from the rest! After struggling with coding along other KZfaq tutorials on GPT-index and langchain, I finally stumbled upon this gem, and behold, my code worked like a charm. Thanks for sharing your expertise and making it easy to follow. You're a lifesaver!

Thank you, Shweta. I was able to get something working based on your code. Please note that lots of libraries are now outdated (gpt_index is now llama_index). My complete code (works 100%!): from langchain import OpenAI import sys import os from llama_index import SimpleDirectoryReader,GPTListIndex,GPTVectorStoreIndex,LLMPredictor,PromptHelper,ServiceContext from llama_index import StorageContext, load_index_from_storage def create_index(path): max_input = 4096 tokens = 200 chunk_size = 600 #for LLM, we need to define chunk size max_chunk_overlap = 20 prompt_helper = PromptHelper(max_input,tokens,max_chunk_overlap,chunk_size_limit=chunk_size) #define prompt llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-ada-001",max_tokens=tokens)) #define LLM docs = SimpleDirectoryReader(path).load_data() #load data service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper) vectorIndex = GPTVectorStoreIndex.from_documents( docs, service_context=service_context ) vectorIndex.storage_context.persist(persist_dir="storage") return vectorIndex def answerMe(): storage_context = StorageContext.from_defaults(persist_dir="storage") index = load_index_from_storage(storage_context) query_engine = index.as_query_engine() q = "What is the best plan?" print(q) print('------------') response = query_engine.query(q) print(response) #create_index('data') answerMe()

GPTIndex Medium link : medium.com/@shweta-lodha/create-chatbot-based-on-the-data-feed-by-you-gpt-index-openai-python-772291576bee LlamaIndex Medium link: shweta-lodha.medium.com/create-chatbot-based-on-the-data-feed-by-you-gpt-index-llamaindex-openai-3efd7abe3ed9 For PromptHelper and documents error: kzfaq.info/get/bejne/e5x2nJWLu9GzYYk.html

Thank you so much for this! You made something very intimidating to me super easy to understand. I am very grateful for the time and effort your put into this video. Thanks a lot!

Thank you Swetha. It can’t get any better than your explanation with show and tell 👍

thanks for your contribution, it's easy to understand for a beginner like me. You can continue to make the next video to guide the question and answer continuously like chatgpt.

Very good explanation. You speak very calm and make audience to follow all step in detail. Keep good work

Please cover UI part as well in one of the coming video. Thank you so much in advance.

Thank you. Is there a way to capture user inputs like contact details (email, name etc)? That would be really useful.

This is really useful, Shweta! Thank you so much for making such an awesome content.

Hi Shweta, Awesome video, great learning. Thanks. Just one concern - when we grant access to ChatGpt to our custom data stored in our machine, then is there a risk that the data can be copied/ used by users of ChatGPT or Open AI or otherwise?

Ms. Shweta, your tutorials are superb!! Stands out first from the rest.

Outstanding preparation and presentation. Thanks so much!

Your instructions are really good, thank you so much Shweta Ji

@shweta-lodha, wonderful video and article. Thank You. Please keep up the good work. One qs: is there a way to implement the same solution using a web based solution - angular, react etc.

I noticed that in the implementation shown in the video, the GPT model is being called every time a user prompt is received, and this can be expensive in a real-world scenario where the application is serving multiple users concurrently. Each request to the GPT model requires a certain number of OpenAI tokens, and this can quickly add up and become expensive.

ImportError Traceback (most recent call last) Cell In[29], line 1 ----> 1 from gpt_index import SimpleDirectoryReader, GPTListIndex, GPTSimpleVectorIndex, LLMPredictor, PromptHelper 2 from langchain import OpenAI 3 import sys What should I do for this error???

Question about execution flow: Hi Sweta - Thanks for a very helpful tutorial. Trying to wrap my head around the execution flow. For example, createvectorIndex probably sends the document to openAI servers, the servers return embeddings that are then stored locally. During answerMe - is the prompt sent back to openAI for embeddings? Does answerMe again send document embeddings back to openAI servers? How much is done in the local process (on PC) vs how much is done by openAI servers? Thanks.

Hi Shweta - I am planning to follow this tutorial, it looks amazing, I was wondering if you know if the local data we used is kept and is local only or if any of the data gets make public or goes back to OpenAI?

@Shweta this is brilliant! as usual top notch like any of your other videos, thank you so much for taking time doing this. One question here, can you please provide the differences in the parameter if the same had to be achieved through Azure Open AI with openai.api_base, openai.api_type, openai.api_version, deployment_name - I have been trying to meddle with this but no luck so far..

Great Video, you are inspiring me to start learning how to code. I am doing this as a project to see if I can follow along withyou. Thank you.

awesome !! very inspiring, thanks a lot for your work !

Thank you so much Shweta! You got me so passionate on the topic. After completing your tutorial how do I move this into a chatbot that I can bring to my app? I’m stuck

This looks like a brilliant tutorial thank you. Please excuse the silly question, but where are you editing your code, I opened IDLE but it must be some other editor/console? Thank you again.

Hello, Shweta Lodha! Great job. I really love your videos. And I have a quick question: Do I have to have a paid GPT plan in order for the code to work?

Hii, I am facing this error RetryError: RetryError[] while calling answerMe function, can you please help me out .

Hello Shweta Lodha, I have a couple of questions. Firstly, do you have any ideas on how we can reduce the cost of tokens? It would be really helpful if you could make a video explaining any potential solutions. Additionally, as a learner, I'm facing issues with expired APIs. Do you have any suggestions on how I can resolve this problem? Thank you in advance for your help.

Hi Shweta, that was a great tutorial! However, I have a question. Just like you used the custom data from the local disk here, How can we use the data from aws/elasticsearch?. I have a huge database (About 20 million records) which our employees access via querying on elasticsearch, and If I wish to create a custom chatbot trained on that data, how would one achieve that?

Hi, thank you for the video! I have tried this code to answer questions based on data about a very specific product. It answers well, however, it answers general questions as well (such as "What is Earth?") having no information at all about this in the files provided. How can I make this code answer based on the information that I provided only?

Amazing tutorial, thank you!!

Exactly what I need! thank you

Thank you very much. I really hope you will have more useful videos like that.

I have implemented code, works perfectly, Thanks for that, I have question, it is just providing answer from context, Is gpt_index is all about context ? can not get general information like, who is trump or any other information? if there is way please help me with in this code

I have not found such a detailed explanation of Open AI based Chat bot, Thankyou for Sharing. Can you please make a video about how i can feed my NLP based ML model to create a Chat bot based on ChatGPT.

Thank you for this great video!

Very nicely presented its a marvel

Thank you. Very useful.

Hi Shweta, You did a great job. Can you please help me how to set that environment on which your are doing code.

wonderful. you made it look so simple.

having data in a file and real time embeddings vs embeddings in a db for chatbot for an application (provides information about an application)?

For me, this code was throwing an error. If you update the code by first importing ServiceContext from gpt_index (along with the other stuff you were already importing) and then update the code by adding the following lines, '-----#load data docs = SimpleDirectoryReader(path).load_data() service_context=ServiceContext.from_defaults(llm_predictor=llmPredictor,prompt_helper=PromptHelper) #create vector index vectorIndex=GPTSimpleVectorIndex.from_documents(documents=docs,service_context=service_context) --- it should solve your issue. P.S: I used this code from another one of your videos :))

A tutor that is looking for the result of a func without calling the func :))))))!!!!!

Thankyou for this detailed tutorial, so you asked 2 question from it, can you please share how much openai credits does it used in this complete operation? that would be really helpful

Thank you Swetha for this easy and well-formed tutorial! Is there a way to connect this bot to a WordPress site? I have created a custom bot explained in this tutorial which uses a dataset from a knowledgebase site. Now I'd like to connect that chatbot to that site so that users can directly communicate there, in addition, to seeing articles in a standard way.

HI! This is a very helpful tutorial. I have a question: How would you bring this to a website? I mean, creating a chat website where you can ask questions and the bot answers based on your custom data. Is it possible? Thanks in advance :)

Very good tutorial, one question if have some logs which contains sensitive data like ip etc, if I feed the logs to the llm will my data move out of my system to the llm model provider's server. If so is there a security threat. I have a presentation to make on using llm to my company and for sure this question will rise, any feedback will be appreciated

hi shweta, does this take structured data like csv or excel for manipulate any data?

Hello, this tutorial is really amazing but i was trying to replicate the same work but got an error when executing the function createVectorIndex saying that the function __init__() in GPTSimpleVectorIndex got an unexpected keyword argument 'documents' did anyone get the same issue ?

That's a great video thanks for sharing this. I have a question if you don't mind. How different is this method rather than using Open AI APIs for chat completion, embedding and completion? Thanks.

hi shweta your tutorial is really helpful and your knowledge on open Ai. i have also emailed you please reply

Thanks a lot great video , just want to know if all my data is stored in sharepoint how can i take them all into the same mechanizm u did thanks again for this great tutorial

Very good video thank you. I must have blinked and it was over! Which specific function uses langchain?

This was awesome, thank you so much!

This helped! Thank you

Hi Shweta, Very useful tutorial and I followed the same but m always getting the response None for any question, However I kept the document at same place where you have mentioned. Could you please help me out, what could be the reason for the same.

madam, I am from India, after doing tons of research on ChatGPT API ,this is the best! I am working on Linux platform and have Jupyter notebook, will it work in that env?

I appreciate your video very much!

Thank you 🙏🏽 ❤

Hi Shweta, this tutorial is amazing! I have one question after running the bot, on my OpenAI usage I am getting text-davinci requests as well as text-embedding-ada-002-v2. Any thoughts on why I am getting the davinci requests?

Thank you for this wonderful video🎉. I have a question when I was trying it. I run the codes to create vector index,but I cannot find it in the OSDisk. There is no error😂

also getting this as well Output exceeds the size limit. Open the full output data in a text editor

@Shweta, Awesome video and Thanks for sharing.. I have data in CSV which has many text columns and many rows. I want to build chat application based on the data. Can you please let me know how can i implement this .

Hi, thanks for the great video. I'm a beginner. How do I proceed now that gpt_index has been replaced by llama_index? I'm getting the error "No module named 'gpt_index'

there is no longer GPTKeywordVectorIndex. There is GPTSImpleKeywordTableIndex() are they the same?

This is helpful, but how will I get it to the openai version, is there any way to upload my knowledge base to open ai, so that it will give me answer according to my needs?

Hello and thanks for the video. Very useful. Is it possible to use a sphinx generated website (essentially a handful of html pages) as the resource documents instead of a plan txt file?

Any plans for coming up with part 2 for this with custom data that needs to be continuously indexed in the background?

hi Shweta, i tried this code on my end but in { vectorIndex.save_to_disk('vectorIndex.json') } is giving me an error so i tried { index.storage_context.persist('vectorIndex') } this is not creating a vectorIndex.json file on my system its creating a folder name vectorIndex.json inside which i has 4 json file docsstore, graph_store, index_store and vector_store.. can you pls tell me where i am going wrong

Hi shweta, this is very helpful tutorial. I tried this code but after asking question, it is not responding...I waited for 15 minutes...still no response.

Hi Shweta, amazing job, hope you can help me: when running vectorIndex = create_index('Knowledge'), I get an error message: Output exceeds the size limit. Open the full output data in a text editor. maybe you know why?

Thank you !

HI Shweta Lodha! First... Thanks a lot for the tutorial! :) I have a problem with an error. Can You help me? I wrote the same code and use the same data. But when I try to run the code, this error appear: TypeError: __init__() got an unexpected keyword argument 'documents'. I Printed the variable and it contains the book text... The error is on the line: vectorIndex = GPTSimpleVectorIndex(documents=docs,llm_predictor=llmPredictor,prompt_helper=prompt_helper) I looked for it on internet but I didn't understand why there's a problem with the arg "documents"... :/

Thanks a lot . Can we use llama_index instead of gpt_index ?

could this learn from new unlabeled questions provided by the user ? Is this a trainable chatbot which could learn new things every time user asks a new question

appreciate for your work, just wonder if this method works for extracting specific number ? like i want to extract some price for mutilple products

Hi @shweta, can this bot integrate with any database..?

thanks for this!

Thank you for the tutorial, it was great! How would I deploy this app to the internet onto my own custom domain?

hey, any tips on how to fine tune a model based on a very large pdf document without the " " to split prompt/resolution? I thought maybe have a script break down in every question mark? Or is there some other way?

Love it!!!!!!!!!!!!!

Hi shweta please tell which extensions you have used in VS code for this video

ValueError: One of documents or index_struct must be provided. what i need to do with this error?

i got this error "TypeError: BaseGPTIndex.__init__() got an unexpected keyword argument 'documents'". dont know if i will get any assistance what the heck.

Hi, I done it with sucess! really thank you! However its possible to start one chat as the last part of the video and save where it stop and come back in other moment?

Is my own data imported to openai system in this case? I don't want to breach the company's data confidentiality code.

Hi, just want to ask if the data that I will feed will be exposed to public or to openAI? Is this safe for business? Thank you.

I got error gpt_index not found when I import it

i am getting this error while running code: Output exceeds the size limit. Open the full output data in a text editor--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[17], line 1 ----> 1 vectorIndex = createVectorIndex('Chatbot') Cell In[14], line 16, in createVectorIndex(path) 13 docs = SimpleDirectoryReader(path).load_data() 15 #create vector index ---> 16 vectorIndex = GPTSimpleVectorIndex(documents=docs,llmPredictor=llmPredictor,prompt_helper=prompt_helper) 17 vectorIndex.save_to_disk('vectorIndex.json') 18 return vectorIndex

def _initialize_persistent_process(prompt: str) -> pexpect.spawn: ^^^^^^^^^^^^^ AttributeError: module 'pexpect' has no attribute 'spawn'...getting this error while importing the packages..please help

Hlo mam i have used the new Llam_Index but it also not working

I am a beginner When using gpt chat, do I have to get the API from it, or can I get it from anywhere?

getting this error on line PromptHelper(): chunk_overlap_ratio must be a float between - and 1

Hii need your help. I have followed exact same steps but facing below issue. I have given pip install gpt_index and got response same as shown in your video then when I am giving from gpt_index import SinpleDirectoryReader code then I am getting error Module not found “gpt_index” I tried uninstalling, re-installing and checking on chatgpt but failed for everything. Plz plz plz plz help

Hi thank you for a great video. Is there a way we can combine the code and add gradio ui ?

thanks for the amazin video, can it read podf file instead of txt?

How to restrict the bot to only search for answers based on data provided or how to make it generate an error if the question asked is outside the available data

Hi when i run this code it is showing that the documents argument is unexpected

Hi, it seems I can’t find VectorSimpleIndex, has it been replaced by VectorStoreIndex?

How to add this to a website

How do we load multiple data, and will the script and OpenAI remember the prior conversation?