I work with Langchain on a daily basis now, and so often I find myself asking; do I really need a whole LLM framework for this? At this point, the assistant I am writing, will likely be more stable rewritten in pure Python. The deeper and more complex the application becomes, the more of a risk Langchain seems to become to keeping it maintainable. But even at less complex levels, if I want to do this:
1. Have a huge dataset of documents.
2. Want to ask questions and have an LLM chat conversation based on these documents.
3. Be able to implement tools like math, wiki or Google search on top of the retrieval.
4. Implement memory management for longer conversations.
Its still a lot more straightforward to maintain it in Python. The only thing where it becomes interesting is having agents execute async, which is not that easy replicate, but at the moment agents are not that helpful. Not trying to diss Langchain too much here, because its such an awesome framework, but I can't help seeing past it other than just being a helpful tool to understand LLM's and LLM programming for now.
The most important aspect of langchain is NOT using OpenAI for the LM.
The most useful aspect of using langchain is to use it with Galpaca (or vicuna/koala/etc) to spin up an assistant for your home.
This way, you can push all of your files through it - even petabytes or terabytes of files, at a fraction of the cost - and have it organize things for you. No privacy problems, no extreme costs, just ease of use, low latency, offline, blazingly fast beauty. That's at least the trajectory.
Meta may soon release an improvement to Galactica similar to Galpaca (GeorgiaTech attempt) more officially (perhaps with more multimodal focus), which will likely improve upon the llama based models even further.
ChatGPT is just one model among many here, and it's not even the first to use RLHF (Deepmind, as usual, was a bit earlier).
The simple task of downloading Redpamajas/thePile/etc and getting a vector db for it locally, and enhancing it with local files effectively brings a local Google to everyone, and it may only require a decent spinning disk HD for the DB storage with the typical langchain LLM setup to have a completely local 'jarvis'-like assistant. (Sure, I know some people care about 'news'-like info that requires connectivity, but most things don't)
The vast majority of people building LM apps with (or without) LangChain are using OpenAI.
I sincerely hope local LM tech like Galpaca (or vicuna/koala/etc) succeed but I don't understand why we are collectively pretending they are currently anywhere near gpt-3.5-turbo both in terms of speed and quality. Honestly the local models feel more like first generation BERT/GPT-1 models that have been fine-tuned for QA using RLHF.
The models now are probably closer to GPT-2, and gets very close to GPT-3 with decent hardware (larger param models) and effort.
They're certainly worse when it comes to 'knowledge' or memorized parts of the training data, but a decent bit of this is ameliorated by having "the pile" + other data locally available for reference.
There are some issues that arise from not having decent priors due to that lack of knowledge, which may or may not be important for the given task.
A (perhaps somewhat bad) example may be: if you ask me "what is a good one-liner in bash for parsing xml from a stream", I may give you an answer using xmlstarlet. However, this may not be the best answer - since Xalan can handle XSLT version 3, but xmlstarlet can't (XSLTv3 handles streams).
So if looking up information in the database, some things may be slightly missed like that - but this behavior would be close to what ChatGPT offers (ChatGPT is quite awful in this way most of the time).
You are right that it would miss GPT-4 by a good bit though in these cases, but most people aren't using GPT-4 for this anyway.
Ultimately both can be used. OpenAI can do things you really want it for, such as a programming assistant, or things that may require much more "reasoning" (not a proper word, but conveys the message) capabilities.
Local models can do the really useful base work of completely re-organizing or re-encoding files to free up space if you set it to do so, integrating with a HomeAssistant system, setting up a HomeAssistant system if you want one, answering your vocal 'Alexa/Siri'-like questions completely offline, setting up backup solutions for all your computers, setting up servers that perform more tasks that you may want - essentially a complete personal assistant. OpenAI shouldn't be needed for this, and it is highly desired to not have them do any of this (due to costs and the number of credentials it would give to them).
It's more or less an advanced personal project to stay on top of the LLM learning curve, rather than just being exposed to news and press releases. I also have an appetite for further wrapping my mind around all of this. I already work in the AI space as web-developer on the B2B & enterprise side of things. My opinion here is that there are going to be loads of use-cases and necessary plugins, which for privacy, legal and security reasons need a proprietary solution and won't be able to interface with any third party APIs, plugins or frameworks.
I think that is a phase, which will come to pass, eventually most of the plugins and apps which are going to be popular, will likely run their own models or use open source models. Because the API calls for complex applications to OpenAI are currently far from economical. In the end you will have to charge the user, and that is going to be the crux. As a sole developer, doing loads of experiments on less than 50 documents, I am already crossing 50$ in API calls within a month. I can already run llama.cpp, but its just not good enough; but the cost would effectively be 0$ (not counting my hardware).
We cannot give my company's information to OpenAI/MS. No legal paperwork will change this. This information is so important, it is only on offline computers.
I've experimented with LangChain for my chatbot as well, but ultimately, I resorted to using custom Python. Here are a few issues I faced with LangChain:
- By developing your own solutions, you can engineer specific components that would be provided by LangChain to better suit your use case. For example, by fine-tuning to your use case you can have better results with converation history, context and summarization better by prompt engineering. If you look at prompts within langchain they are pretty basic.
- LangChain is designed around the idea that an entire chat logic resides within a single "REPL loop." In my use case, I had a single-page web app frontend, a standard web "RESTful" backend, and a separate chat service. Different parts of the information are stored and managed by these components. Using LangChain would have forced me to consolidate all logic into the chat service, which doesn't align with the overall architecture of my system beyond just the chat functionality.
Please note that I'm not a LangChain expert, so my assessment might not be entirely accurate about its capabilities. However, based on my evaluation, LangChain introduced too many constraints in comparison to what it provided.
This, I just pretty much ended up using the basic LLMChain and do my own custom flow. The built in agents are close to useless for anything but a toy project; it is simply way too unreliable.
We've been building LLM chains for over a year and found that whilst for simple use-cases it's easy enough to grab a couple of APIs. However, having a Notebook experience built for iterating and collaborating whilst managing the complexity is something we've seen companies care deeply about.
1. Have a huge dataset of documents.
2. Want to ask questions and have an LLM chat conversation based on these documents.
3. Be able to implement tools like math, wiki or Google search on top of the retrieval.
4. Implement memory management for longer conversations.
Its still a lot more straightforward to maintain it in Python. The only thing where it becomes interesting is having agents execute async, which is not that easy replicate, but at the moment agents are not that helpful. Not trying to diss Langchain too much here, because its such an awesome framework, but I can't help seeing past it other than just being a helpful tool to understand LLM's and LLM programming for now.