i built a pc that has a crop ton of processing power, but i know nothing about the software side of things/
thoughts? prayers? concerns? comments? @$%&'s to give?
- Install linux on it https://ubuntuhandbook.org/index.php/2024/04/install-ubuntu-24-04-desktop/
- Install ollama https://ollama.com/download/linux
- Install Open WebUI https://docs.openwebui.com/getting-started/quick-start/
- Install stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui
- Spent a coupple of weeks learning how to configure it so you can get a chat running and a image generator
Do you actually need the webui stuff or can you just use ollama on the command line?
Ollama can be run from CLI.
It’s just an optional interface. There’s the build in console. There’s other 3rd party TUIs too.
You can run it from the command line but you will not have tools and the formatting will be unpleasant.
Processing (cpu) doesn’t really matter as much as gpu, and generally the constraint is gpu memory on consumer grade machines. Processing via nvidia chips has become the standard, which is a huge part of why they have become the single most valuable company on the planet, though you can use cpu you’ll find the performance almost unbearably slow.
Ollama is the easiest option, but you can also use option and pytorch (executorch), vllm, etc
You can download your model through huggingface or sometimes directly from the lab’s website
It’s worth learning the technical side but ollama genuinely does an excellent job and takes a ton off your plate
Not entirely sure what you mean by “Limitation Free”, but here goes.
First thing you need is a way to actually run a LLM. For me I’ve used both Ollama and Koboldcpp.
Ollama is really easy to set up and has it’s own library of models to pull from. It’s a CLI interface, but if all you’re wanting is a locally hosted AI to ask silly questions to, that’s the one. Something of note for any locally hosted LLM, they’re all dated. So none of them can tell you about things like local events. They’re data is current as of when the model was trained. Generally a year or longer ago. If you wanted up to date news you could use something like DDGS and write a python script that calls Ollama. At any rate.
Koboldcpp. If your “limitation free” is more spicy roleplay, this is the better option. It’s a bit more work to get going, but has tons of options to let you tweak how your models run. You can find .gguf models at Hugging Face, load em up and off you go. kobold’s UI is kinda mid, and though is more granular than ollama, if you’re really looking to dive into some kinda role play or fantasy trope laden adventure, SillyTavern has a great UI for that and makes managing character cards easier. Note that ST is just a front end, and still needs Koboldcpp (or another back end) running for it to work.
Models. Your “processing power” is almost irrelevant for LLMs. Its your GPUs VRAM that matters. A general rule of thumb is to pick a model that has a download size 2-4GB smaller than your available VRAM. If you got 24G VRAM, you can probably run a model that’s 22G in download (Roughly a 32B Model depending on the quant).
Final notes, I could have misunderstood and this whole question was about image gen, hah. InvokeAI is good for that. Models can be found on CivitAI (Careful it’s… wild). I’ve also heard good things about ComfyUI but never used it.
GL out there.
thanks! this helps a lot! ill have to learn what it means first but ill definitely try it!
What do you mean by “make” what do you want it to do that you aren’t getting.
Maybe some existing model via ollama - llama-uncensored?
Do you need to add context with some specific set of data, should it be retrieval based or tuned or cross trained?
Does it even need to be an llm? What are you trying to actually achieve?
i want to make my own chatbot that can also act without my input, be able to create emails, and do online jobs, and make its own decisions, things like that
Most of the options mentioned in this thread won’t act independent of your input. You’d need some kind of automation software. n8n has a community edition that you can host locally in a docker container. You can link it to an LLM API and emails, excel sheets etc. As for doing “online jobs” I’m not sure what that means, but at the point where you’re trying to get a single AI to interact with the web and make choices on it’s own, you’re basically left coding it all yourself in python.
i mean like actual jobs a person could do online, like commissions with art programs, or administration jobs for software companies, basically it would mimic a person online
If someone with a home computer and very little knowledge of AI could setup an AI that could do admin jobs for software companies … Why wouldn’t the software companies do exactly that themselves rather than outsource work?
I think you’re massively overestimating what a LLM is capable of.
Install Linux
Install llmstudio
Profit
I was going to mention this one. LMStudio is much better than ollama.
LLMstudio is local AI on easy mode.
