Thank you! :)

Hey thank you! I also have the Table of Content on the README of this github repo if that could be helpful in navigating the different topics! github.com/onlyphantom/llm-python

@@SamuelChan I am trying to run the above code but i m getting error - Entering new LLMChain chain... Prompt after formatting: You had one job : Fintech CEO and you dont have to be sarcastic Traceback (most recent call last): File "gpt2.py", line 15, in print(llm_chain.run("Fintech CEO")) File "\AppData\Local\Programs\Python\Python38\lib\site-packages\langchain\chains\base.py", line 114, in run return self({self.input_keys[0]: text})[self.output_keys[0]] File "j\AppData\Local\Programs\Python\Python38\lib\site-packages\langchain\chains\base.py", line 87, in __call__ outputs = self._call(inputs) File "\AppData\Local\Programs\Python\Python38\lib\site-packages\langchain\chains\llm.py", line 63, in _call response = self.llm(prompt, **kwargs) File "\AppData\Local\Programs\Python\Python38\lib\site-packages\langchain\llms\huggingface_hub.py", line 95, in __call__ response = self.client(inputs=prompt, params=_model_kwargs) File "j\AppData\Local\Programs\Python\Python38\lib\site-packages\huggingface_hub\inference_api.py", line 213, in __call__ raise NotImplementedError( NotImplementedError: text/html; charset=utf-8 output type is not implemented yet. You can pass `raw_response=True` to get the raw `Response` object and parse the output by yourself. I have tried using different models but keeep getting the same error.Please help

You’re welcome! 😊

If you are having memory issues, you can change the cache location: export TRANSFORMERS_CACHE=/whatever/path/you/want I have a more detailed video explaining this here: kzfaq.info/get/bejne/p6eme9ZnurTGYp8.html You will need something that can fit your RAM, otherwise you’ll have to do the inference “off machine”. Hope it’s helpful!

Hey thanks a lot! 😊 For databases with LangChain I have a video dedicated to that LangChain + OpenAI to chat w/ (query) own Database / CSV! kzfaq.info/get/bejne/fOBgiq2I38yygH0.html So the idea with langchain is that it’s quite agnostic to the LLM of your choice, you specify it in the call to your chain to use open source LLMs on huggingface instead of OpenAI and the rest of the code pretty much stays the same! :)

It doesn’t work off a “credit” system no. It does require you to sign up but that’s more so to prevent abuse of its inference api / service than to limit you.

No, not at the moment but it should be straightforward! The following video might also be helpful when working with a database! kzfaq.info/get/bejne/fOBgiq2I38yygH0.html

LLM are by default non-deterministic so you’ll get different results given the same input - you can try tinkering with different settings like temperature etc; or try using a newer version of the model / larger LLMs

If you can find a second hand 3090 card, or a 4070 card. Both offer great value for GPU deep learning work anyway and sufficient VRAM. Run your LLM on your an open source model hosted on your machine kzfaq.info/get/bejne/p6eme9ZnurTGYp8.html

Might be what you’re looking for: LangChain + OpenAI to chat w/ (query) own Database / CSV! kzfaq.info/get/bejne/fOBgiq2I38yygH0.html You could use GPT-3 or just swap it out to any LLM of your choice (look at my other videos on how to load in a locally hosted model for that!) Good luck! :)

I think the candidates are Llama, Alpaca, Vicuna and Dolly? Having said that, I'm sure my answer would be outdated when I check back in 3-6 months. Meta's OPT-175B is also another choice of course. "We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop" (huggingface.co/docs/transformers/model_doc/opt) and "OPT-175B is a GPT-3 equivalent model trained by Meta. It is by far the largest pretrained language model available with 175 billion parameters" (ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/). I'm in the middle of recording a tutorial of running locally hosted LLM offline that uses OPT-30B, and that itself is a 60GB model. OPT-66B is ~130GB. You need a server with quite substantial RAM and memory, and in the first few times I run it while having my video recording software in the background, the process would crash :( May have to rent a cloud server, run it on that, and then do my screencasting and audio on my linux box separately. But that substantially increase the scope and accessibility of the tutorial so I'm not really keen on that too - viewers won't be able to reproduce the tutorial.

@@SamuelChan I think out of the above options Dolly is the only one with Apache 2. Havent got around to trying it out though to see if it is any good. I am waiting for access to GPT4 API and it is quite frustrating to not have a quality LLM API access. Look forward to your next video

@@moreshk did I misunderstand your quality LLM comment? You do not consider gpt3 or gpt3.5 good enough? (hence waiting on the gpt4 api) On open models, to me its a matter of time. I'm not a gambler but if I have to pin down a "most likely outcome", my bet is that there would be two or more open source LLM (not just source code, but also open in its instruction tuning etc) competitive or surpassing proprietary ones. Meta's OPT 175b paper is really worth a read (link in the comment above). I might be wrong of course, but to me that's probabilistically most likely than the alternative outcome. I'd be surprised that a closed source, proprietary one dominates the market in 24 months with no comparable / competitive alternatives.

@@SamuelChan I am not an expert, so whatever I saw should be taken with a grain of salt. I am only getting my feet wet atm by following online tutorials such as yours. GPT 3 and 3.5 are good by themselves, but there are specific use cases that seem to need GPT4 API. Teenage AGI is an example, there is another where someone loads many large PDFs and reads data off that. Both will work with 3.5 but the responses will be of lower quality I am told

@@moreshk No I really appreciate it. Thanks for sharing your observations! LLM is still in its early formative years and so I get what you're saying. Been reading a ton of documentation and papers -- fully expecting half of them to be redundant in 12 months, so I'm just as much a beginner in that I'm learning and building on the go (I'm lucky I'm self-employed, so I get to learn and iterate on a self-determined pace) and then sharing these mini projects in these tutorial forms.

Seems like there are several promising projects on huggingface so I would say yeah it’s definitely possible. You’re doing image to text through the inference API; huggingface.co/models?pipeline_tag=image-to-text

I cover this throughout the LLM series! :) github.com/onlyphantom/llm-python for the table of content. Part 2,3,6,7,8,9 all demonstrates how you would build LLM apps and workflows with your own data in your own particular domain. Here's part 2 video: kzfaq.info/get/bejne/er9_iMKLlc6lp6M.html -- you can work your way through from part 1 to part 14 too; The entire LLM series is ~8 hours long.

You can use gpt2 for question answering by “prompting” it first. A good video that explains this is in my Cohere tutorial kzfaq.info/get/bejne/q7hpZtyFs63YgGg.html My Guidance tutorial also touches on this technique kzfaq.info/get/bejne/oZp1ncZkx7K1mYk.html

Thank for your response. Can I use gpt2 + langchain to make question answering over my document using "promting"? Do you have any video about it or do you any tutorial about it?

gpt2 is a text completion model at heart. You can “trick” it into Q&A by doing a few Q&A examples in your prompt so that it continues to emulate that style in its response. The Cohere tutorial I linked to above shows how I approach it :)

@@SamuelChan Thank you so much Samuel for your responding. I will do it.

Not sure what's a good prompt but this is a nice follow up video: kzfaq.info/get/bejne/p6eme9ZnurTGYp8.html and I plan on releasing one with Llama 2 but I just got back from a very long holiday in Japan so would take a while to settle down at work etc before I have time making videos again!

hey, yeah you could do that with GPT-4. That would allow you to ask for a description of the images you upload. "We’ve created GPT-4, a large multimodal model (accepting image and text inputs, emitting text outputs) that, while less capable than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks." www.videogamer.com/cdn-cgi/image/width=845,height=807,fit=crop,quality=80,format=auto,onerror=redirect,metadata=none/wp-content/uploads/image-278.png openai.com/research/gpt-4

I was looking into Bert and all , to be honest i am very new to this area. I am still learning. Could you just help me out?

Hey Nikhil, huggingface’s API is free for a vast majority of use cases until you start running up some big volume, so I’d say that’s the simplest way to get set up. The alternative, if you don’t want to sign up for a huggingface account, is to run the model locally. You’ll download the model (let’s say GPT-2 or a BERT based model) and then run the model offline on your own machine’s GPU (hopefully). Here’s a video tutorial in the LLM series that goes over that process: kzfaq.info/get/bejne/p6eme9ZnurTGYp8.html

What about it is not working? Might need more details to be able to reproduce your problem and attempt some troubleshooting!

@@SamuelChan can I send you a message or email you? I could even send a notebook with the exact code and way I envision it working

I’m currently on a months long holiday in Japan (currently in Sapporo, just got here from Hakodate) so I don’t have my work devices with me. I can take a look at it when im back! If you need some urgent help, you could also look at booking an expert by the hour from collective.supertype.ai I work with them and they’re a lovely bunch and very experienced with these kind of LLM consulting work :)

@@SamuelChan Thank you!

GPT-2 is open source, so you could download and host this model yourself and not have licensing issues. You can use as much as you need as long as it's being served from your home server / cloud server. Running on HuggingFace API though you'll need to consult their commercial terms based on your usage patterns.

@@SamuelChan My PC will not be able to handle running an LLM with its hardware. But I can just download the models onto Google Colab's free tier, right? Like what you did

@@Coms_Productions yeah of course; you can also find a cloud gpu (Paperspace, Gradient, Kaggle, Colab) or something like Modal -- most of these have a great free tier to try out. For longer term usage, buying a graphics card with enough VRAM (3060 is a popular choice) might still be way more economical though -- there are plenty of second hand ones from the crypto mining bust (ex mining rigs). Then get enough RAM on your CPU and turn it into a simple home server.

@@SamuelChan Thank you so much 😄, can finally play with these models

You’ll need to do better in the prompt. It is a text completion model so you’ll want to “trick” it into doing Q&A by feeding it examples of Q&A. Some call this prompt engineering or prompt design. You can watch my videos on Cohere and Guidance where I explain this in greater depth. kzfaq.info/get/bejne/q7hpZtyFs63YgGg.html kzfaq.info/get/bejne/oZp1ncZkx7K1mYk.html

Your understanding is right. For audio to text you can use Whisper. I have a video on that coming up in 3 days (in final stages of tuning out the idea for bg noise removal etc)! Come back on Monday!

i think you should try "from langchain_community" instead?

No. But if you are interested in running it locally check out my video on this (large language models running downloaded from huggingface to your machine) specifically: kzfaq.info/get/bejne/p6eme9ZnurTGYp8.html If you have a CUDA-compatible card though, you just need to add a device parameter and its underlying model (PyTorch) will be smart enough to take advantage of your GPU. The video I linked to explained it in more details!

@@SamuelChan Okay. Can we run all the models from huggingface hub free version? or can only run light weight models and need to download heavy models like Llama2 when we are using free version in huggingface?

Yeah definitely not all models have a huggingface inference API, you may need to install transformers and host it on your machine if there is no hosted version! For Llama2 model specifically you also need to request and be granted access to the weights too in addition to what I mentioned above! :) ai.meta.com/resources/models-and-libraries/llama-downloads/

Cool idea! The first business I started (2014) was a Slack chatbot, sold 2 years later after a lot of hustle. Built several chatbots during the time running it, never on FB Messenger though.

@@SamuelChan it would be great if you can make a tutorial on this, im sure people would need it for customer service or general inquiry for their companies!

If you’re using python (I assume you are) this is very simple by combining what you already do with another google sheets library. Watch my google api video to get a sense of how working with Google’s api look like. It’s maybe an extra 5 lines of code! :)

@SamuelChan you say it's not hard but for me it sure will be, I don't know anything about code

Maybe take a look at the docs developers.google.com/sheets/api/quickstart/python I also have some videos on working with Google’s client library + OAuth if prefer a step by step instructions in video. We are all beginners at some point and had to start somewhere! :) you got this!

​@SamuelChan ok thank you very much, so the easiest way is with face hugging? you don't know any other image to text? hugging face is the best?

With image to text, do you mean OCR? Py-Tesseract perhaps? You wouldn’t need huggingface at all

Hey you can’t add it as a variable directly into the code. You have to set it as an environment variable (EXPORT or the SET in shell) or set it using something like .env and load_dotenv(). The error message says you can also pass huggingfacehub_api_token as a named parameter (python is case sensitive, so make sure you name it as such). But keep in mind that’s not a python variable you create (it seems like you’re doing that) but as a parameter you passed into the function call :)

@@SamuelChan Thanks. I had it in .env, the problem was upper case. Thanks again

Both requires an OpenAI account and not open source models :)

I’ll give that a try!

Set a higher penalty for repetition? Dial it all the way up to penalise repetition and see if that’s where the problem is?

@@SamuelChan Thanks, but, what do you mean? 😅 I'm kinda "new". O by the way, when I don't use "Verbose=True" I don't get the "Error in on_chain_start callback: 'name'" message, but the repetition keeps the same . You were just a good guy. I'm not sure if you're aware of the fact that you're a customer service agent. You're a customer service agent. You're a customer service agent. You're a customer service agent. You're a customer service agent. You're a customer service agent. You're a customer service agent. You're a customer service agent. You . You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were just . You were just a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good guy. You were a good . You're the one who's going to get the job done. You're the one who's going to get the job done. You're the one who's going to get the job done. You're the one who's going to get the job done. You're the one who's going to get the job done. You're the one who's going to get the job done. You

Line 21 is where you set the model parameters :) github.com/onlyphantom/llm-python/blob/main/05_hf.py You want to more harshly penalise the repetition, to discourage the behavior (of repeating the same thing). Might also be interesting to tune the temperature = higher produces more unpredictable, random results, lower produces safer guesses.

@@SamuelChan I reduce the max_lenght to 50, with the first print runs pretty well, but then it duplicates again. 🤷 Error in on_chain_start callback: 'name' Prompt after formatting: You had one job 😡! You're the customer service agent and you didn't have to be sarcastic > Finished chain. . You were just a good guy. I'm not sure if you're aware of the fact that you're a customer service agent. Error in on_chain_start callback: 'name' Prompt after formatting: You had one job 😡! You're the politician and you didn't have to be sarcastic > Finished chain. . You were just a good guy. You were just a good guy. You were just a good guy. You were just a good guy. You were Question. Why the "Error in on_chain_start callback: 'name'" message when using "Verbose=True"?

Can you raise a GitHub issue on that Verbose=True behavior if you suspect there is an issue? That will remind me to check on it when I have time this week (Currently away from my home pc)!

I'm not sure I follow here. GPT-2 is open source, and huggingface costs nothing except if you choose to upgrade to a paid plan. Whole thing is as free as it gets, without requiring an openai api key

@@SamuelChan There are API limits.

Have you run into any API limits? Ive never run into personally and the docs says it may happen only under “very heavy use cases” citing an example of 10k simultaneous requests. How heavy were your workflows before you start hitting limits?

​@@SamuelChan There is info even in the pricing. PRO version is limited to 1 milion characters. So the free one is also limited, probably up to 300k char.

​@@patrykkosik9729300k per what? Per month? Per account??

If it’s a work enquiry, email might be better! You get my email by going to the Channel section and click on “show email” or whatever it’s called.