i built a pc that has a crop ton of processing power, but i know nothing about the software side of things/

thoughts? prayers? concerns? comments? @$%&'s to give?

  • Grenfur@pawb.social
    link
    fedilink
    arrow-up
    8
    ·
    edit-2
    5 hours ago

    Not entirely sure what you mean by “Limitation Free”, but here goes.

    First thing you need is a way to actually run a LLM. For me I’ve used both Ollama and Koboldcpp.

    Ollama is really easy to set up and has it’s own library of models to pull from. It’s a CLI interface, but if all you’re wanting is a locally hosted AI to ask silly questions to, that’s the one. Something of note for any locally hosted LLM, they’re all dated. So none of them can tell you about things like local events. They’re data is current as of when the model was trained. Generally a year or longer ago. If you wanted up to date news you could use something like DDGS and write a python script that calls Ollama. At any rate.

    Koboldcpp. If your “limitation free” is more spicy roleplay, this is the better option. It’s a bit more work to get going, but has tons of options to let you tweak how your models run. You can find .gguf models at Hugging Face, load em up and off you go. kobold’s UI is kinda mid, and though is more granular than ollama, if you’re really looking to dive into some kinda role play or fantasy trope laden adventure, SillyTavern has a great UI for that and makes managing character cards easier. Note that ST is just a front end, and still needs Koboldcpp (or another back end) running for it to work.

    Models. Your “processing power” is almost irrelevant for LLMs. Its your GPUs VRAM that matters. A general rule of thumb is to pick a model that has a download size 2-4GB smaller than your available VRAM. If you got 24G VRAM, you can probably run a model that’s 22G in download (Roughly a 32B Model depending on the quant).

    Final notes, I could have misunderstood and this whole question was about image gen, hah. InvokeAI is good for that. Models can be found on CivitAI (Careful it’s… wild). I’ve also heard good things about ComfyUI but never used it.

    GL out there.