Comment on Is there anyway to use LLMs without supporting the billionaires/data centers but without also entirely upgrading your computer?
You are viewing one comment and its replies. View all comments
You are viewing one comment and its replies. View all comments
Comments
Sure, there are open-weight models and many of them are very usable on quite limited hardware, some can even run on phones which is ridiculous. Don’t expect paid-service-level speed or abilities or even free-tier of those same services where it’s running on datacenter hardware, but running locally on reasonable consumer hardware it’s certainly possible to get close to that, at least sometimes, in some situations. YMMV depending on your configuration and your needs. Most of the good open-weight models are in fact originally released by billionaire asshole companies who have the resources to actually put down these foundations (but a lot of the best open-weight releases are coming from China, which is its own sort of concern), but how and why they get released is not really the important part for us because once they’re released, they’re released forever and people can take them and adjust them and re-train them for all kinds of different purposes. You just have to be aware of where they came from. On their own, they do not read your disk or phone home (they are not capable of it without an external client enabling and allowing them to use tools at all). They are just text input, and text output (or images/audio input using mmproj techniques, or stable diffusion for image/audio generation, those are totally different cans of worms and outside the scope of this, but it can be done) They’re ultimately just text generators, but some of them are trained to general particular formats of text that tell an external software when it should try to use tools.
Personally, I found ollama the easiest way to get started, but I’m sort of a minimalist and started out very casual (and in some ways still am). Another place that looked nice to start out with was the desktop application “Jan”. OpenWebUI was relentlessly recommended to me but I found it a bit buggy and unnecessarily resource intensive and ultimately dropped it. I tried Librechat also and it was a little better but ultimately I had pretty similar problems with it. (A lot of stuff in the AI field is vibe-coded slopware, and it really shows sometimes, be warned and be aware) Capek is a very minimal alternative web chat frontend that I’ve kind of settled on personally. I find minimalism is the best way I can think of to avoid all the feature-bloated insecure AI slop. I don’t know if it’s actually helping or a good strategy, but it’s the strategy I’ve chosen.
Technically yes, but pushed too far it slows down dramatically to the point of not really being useful anymore, unless you’re careful about how you do it. Key things to look for:
Hope that helps, have fun, be safe!
Is this answer AI-generated?
Nah, it’s similar to how I write. Probably just an autistic tech nerd like the rest of us XD
Sign in to comment. Sign in