ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sentry-sdk 1.22.2 requires urllib3<2.0.0, but you have urllib3 2.0.2 which is incompatible
…many many lines
raise ValueError("%r is not a directory" % (package_path,))
ValueError: 'build/py3k/scipy' is not a directory
Converting to Python3 via 2to3... /tmp/pip-install-hkb_4lh7/scipy_088b20410aca4f0cbcddeac86ac7b7b1/build/py3k/scipy/signal/fir_filter_design.py
[end of output]
note: This error originates from a subprocess, and is
likely not a problem with pip.
error: metadata-generation-failed
Someone is thinking "this is part of learning the language," but I think it's just bad design.
python3 -m venv venv
. venv/bin/activate
Also, `python` is usually Python 2.7. If it is, I advise removing it from your system unless you have a strong reason to keep it.
All that nonsense vs docker pull / docker run
From a linux perspective, I wouldn't blindly suggest the average reader to purge Python 2.7 from their system, as it might drag core parts of the WM with it. Consider aliasing, or better yet, relying on modern venv tools such as Conda instead.
(base) vid@kk:~/D/ai/tortoise-tts$ python3 -m venv venv
(base) vid@kk:~/D/ai/tortoise-tts$ . venv/bin/activate
(venv) (base) vid@kk:~/D/ai/tortoise-tts$ python -m pip install -r ./requirements.txt
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting rotary_embedding_torch
Using cached rotary_embedding_torch-0.2.3-py3-none-any.whl (4.5 kB)
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 196, in <module>
setup_package()
File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 147, in setup_package
from numpy.distutils.core import setup
ModuleNotFoundError: No module named 'numpy'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[notice] A new release of pip available: 22.3.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
(venv) (base) vid@kk:~/D/ai/tortoise-tts$
This isn't the first time venv didn't work for me, then there's anaconda, miniconda, and a bunch of other things that add env and directories. I don't really know what any of them do, and -I don't want to- I'm not an expert on every app on my system, but I can use nearly all of them without pain. (remember this is about ease of use)
Oh yeah, and python 2 vs python 3. <rolls eyes>
It's very much the "works for me" experience from the old days. There's no good learning from it, except dependencies suck and python systems aren't good at them.
I think when releasing anything that includes dependencies that span the operating system, it's just good engineering to use a container approach. Otherwise you're just causing a lot of discomfort in the world for no good reason.
It's funny because chatgpt would give me an answer to this in a few moments, but I'm locked out for a week because it can't figure out where I am.
Now I'm spending my Sunday morning setting up a dockerfile for tortoise-tts. At least I will learn something reusable from that. I guess I will create a PR for it, though it seems the author isn't tending the repo anymore.
It's too bad the ecosystem seems to be so messy, because Python seems like the best language for general utilities.
The comment does not really make sense. It sounds like pyenv wasn't setup correctly on your system (needs to be added to your bash_profile etc). The typical setup is to put pyenv first in PATH so it takes precedence.
It may be easier to use pipx which will auto manage the virtual envs of end user apps for you.
Python is very fragile to deploy and run on your own machine.
You should not be seeing these dependency conflict issues if you install each project in its own virtual environment.
If you just want them to be easily installed you can just use pipx (`pipx install my-package`) which will manage the virtual environment automatically.
Making a full blown Docker image for it is overkill 99% of the time. Virtual environments serve the same purpose while being much faster and lighter weight.
Say what you will about the old school way of manually building and copying shit around, at least when something breaks I don't have to spend a couple hours keelhauling a bloated toolchain in a debugger for mutiny.
While consumers are happy to get their data mined to avoid paying, businesses are the opposite: willing to pay a lot to avoid feeding data to MSFT/GOOG/META.
They may give assurances on data protection (even here GitHub copilot TOS has sketchy language around saving down derived data), but can’t get around fundamental problem that their products need user interactions to work well.
So it seems with BigTechLLM there’s inherent tension between product competitiveness and data privacy, which makes them incompatible with enterprise.
Biz ideas along these lines: - Help enterprises set up, train, maintain own customized LLMs - Security, compliance, monitoring tools - Help AI startups get compliant with enterprise security - Fine tuning service
I can see a self-hosted LLM being akin to a company’s ship-mind. Anyone can ask questions, order analyses, etc, so long as you are a member of the company. No two LLM’s will be exactly the same - and that’s ok.
https://fractalverse.net/explore-to-sleep-in-a-sea-of-stars/...
It has worked before like in case of Linux and can work again.
Right now, you can't pay a lot and get a local LLM with similar performance to GPT-4.
Anything you can run on-site isn't really even close in terms of performance.
The ability to finetune to your workplaces terminology and document set is certainly a benefit, but for many usecases that doesn't outweigh the performance difference.
https://lmsys.org/blog/2023-03-30-vicuna/
https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
In this case it appears to be using RetrievalQA from LangChain, which I think is this prompt here: https://github.com/hwchase17/langchain/blob/v0.0.176/langcha...
Use the following pieces of context to answer the question at the end. If you don't
know the answer, just say that you don't know, don't try to make up an answer.
{context}
Question: {question}
Helpful Answer:The LLM will absolutely lie if it doesn't know and you haven't made it perfectly clear that you'd rather it did not do that.
LLMs seem to be trying to give answers that make you happy. A good lie will make you happy. Unless it understands that you will not be happy with a lie.
Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them.
What is taught indirectly though is level of certainty, so if you get LLM's to rationalise their answers you tend to get more reliable evidence based answers.
Bottom line, teaching a monolithic model what it means to not know something with certainty, is difficult and not currently done. You'll likely get a lot of false negatives.
- [GitHub - e-johnstonn/BriefGPT: Locally hosted tool that connects documents to LLMs for summarization and querying, with a simple GUI.](https://github.com/e-johnstonn/BriefGPT)
- [GitHub - go-skynet/LocalAI: Self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware. No GPU required. LocalAI is a RESTful API to run ggml compatible models: llama.cpp, alpaca.cpp, gpt4all.cpp, rwkv.cpp, whisper.cpp, vicuna, koala, gpt4all-j, cerebras and many others!](https://github.com/go-skynet/LocalAI)
- [GitHub - paulpierre/RasaGPT: RasaGPT is the first headless LLM chatbot platform built on top of Rasa and Langchain. Built w/ Rasa, FastAPI, Langchain, LlamaIndex, SQLModel, pgvector, ngrok, telegram](https://github.com/paulpierre/RasaGPT)
- [GitHub - imartinez/privateGPT: Interact privately with your documents using the power of GPT, 100% privately, no data leaks](https://github.com/imartinez/privateGPT)
- [GitHub - reworkd/AgentGPT: Assemble, configure, and deploy autonomous AI Agents in your browser.](https://github.com/reworkd/AgentGPT)
- [GitHub - deepset-ai/haystack: Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-4, ChatGPT and alike). Haystack offers production-ready tools to quickly build complex question answering, semantic search, text generation applications, and more.](https://github.com/deepset-ai/haystack)
- [PocketLLM « ThirdAi](https://www.thirdai.com/pocketllm/)
- [GitHub - imClumsyPanda/langchain-ChatGLM: langchain-ChatGLM, local knowledge based ChatGLM with langchain ｜ 基于本地知识库的 ChatGLM 问答](https://github.com/imClumsyPanda/langchain-ChatGLM)
"ggml_new_tensor_impl: not enough space in the context's memory pool (needed 18296202768, available 18217606000)"
I want to have the memory part of langchain down, vector store + local database + client to chat with an LLM (gpt4all model can be swapped with OpenAI api just switching the base URL)
https://github.com/aldarisbm/memory
It's still got ways to go, if someone wants to help let me know :)
The process roughly is:
Ingestion:
- Process embeddings for your documents (from text to array of numbers)
- Store your documents in a Vector DB
Query time:
- Process embeddings for the query
- Find documents similar to the query using distance from other docs in the Vector db
- Construct prompt with format:
""" Answer question using this context: {DOCUMENTS RETRIEVED}
Question: {question} Answer: """
Is that correct? Now, my question is, can the models be swapped easily? Or that requires a complete recalculation of the embedding (and new ingestion)?
This project could help me create a personal AI which answers any questions to my life, finances or knowledge...
https://www.youtube.com/watch?v=A3F5riM5BNE
Also has a suggestion of a few alternative models to use.
> Name Quant method Bits Size RAM required Use case
WizardLM-7B.GGML.q4_0.bin q4_0 4bit 4.2GB 6GB 4bit.
WizardLM-7B.GGML.q4_1.bin q4_0 4bit 4.63GB 6GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
WizardLM-7B.GGML.q5_0.bin q5_0 5bit 4.63GB 7GB 5-bit. Higher accuracy, higher resource usage and slower inference.
WizardLM-7B.GGML.q5_1.bin q5_1 5bit 5.0GB 7GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.
WizardLM-7B.GGML.q8_0.bin q8_0 8bit 8GB 10GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.
> Name Quant method Bits Size RAM required Use case
wizard-vicuna-13B.ggmlv3.q4_0.bin q4_0 4bit 8.14GB 10.5GB 4-bit.
wizard-vicuna-13B.ggmlv3.q4_1.bin q4_1 4bit 8.95GB 11.0GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
wizard-vicuna-13B.ggmlv3.q5_0.bin q5_0 5bit 8.95GB 11.0GB 5-bit. Higher accuracy, higher resource usage and slower inference.
wizard-vicuna-13B.ggmlv3.q5_1.bin q5_1 5bit 9.76GB 12.25GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.
wizard-vicuna-13B.ggmlv3.q8_0.bin q5_1 5bit 16GB 18GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.
> Name Quant method Bits Size RAM required Use case
VicUnlocked-30B-LoRA.ggmlv3.q4_0.bin q4_0 4bit 20.3GB 23GB 4-bit.
VicUnlocked-30B-LoRA.ggmlv3.q4_1.bin q4_1 5bit 24.4GB 27GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
VicUnlocked-30B-LoRA.ggmlv3.q5_0.bin q5_0 5bit 22.4GB 25GB 5-bit. Higher accuracy, higher resource usage and slower inference.
VicUnlocked-30B-LoRA.ggmlv3.q5_1.bin q5_1 5bit 24.4GB 27GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.
VicUnlocked-30B-LoRA.ggmlv3.q8_0.bin q8_0 8bit 36.6GB 39GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.
Copied of some of The-Bloke's model descriptions on huggingface. With 16G you can run practically all 7B and 13B versions. With shared GPU+CPU inference, one can also offload some layers onto a GPU (not sure if that makes the initial RAM requirement smaller), but you do need CUDA of course.
It looks like you can ask a question and the model will use its combined knowledge of all your documents to figure out the answer. It looks like it isn't fine-tuned or trained on all the documents, is that right? How is each document turned into an embedding, and then how does the model figure out which documents to consult to answer the question?
Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?
Also, the local model used makes a big difference. Right now wizard-mega and manticore are the best ones to use. I run the 16b ggml versions in an M2 Pro and it takes about 30 seconds to “warm up” and produce some quality responses.
What would be the difference in user experience or information retrieval performance between the two?
My impression is it saves work on the dataset transformation and compute for fine tuning, so it must be less performant. Is there a reason to prefer the strategy here other than ease of setup?
In general, Qdrant is a real DB, not a library and that's a huge difference.
Why? Why can't I define any directory (my existing Obsidian vault, for example) as the source directory?