1 Like
I recommend trying serge.chat. it is a docker container that allows you to run a ton of different AI language models locally on your computer.
2 Likes
Appreciate it Batvin, added to the list!
There’s also GT4All if you want a native GUI though serge seems to support a lot more models.
1 Like
“… Microsoft is including AI in their Office Suite.” - Gah! I am still flustered with the LAST time they tried this!!
1997 - “It looks like you’re typing a letter. Would you like help?” - Clippy
If Microsoft ever gets a hold of virtual reality, we’ll certainly witness the return of Microsoft Bob!
1 Like
Open source AI is like an infinite time sink to learn right now and it’s been a real struggle parsing the signal from the noise.
Strictly from what i’ve learned so far… Open Assistant seems just be interested in selling generic SaaS products.
What hardware will be required to run the models?​
The current smallest (Pythia) model is 12B parameters and is challenging to run on consumer hardware, but can run on a single professional GPU. In future there may be smaller models and we hope to make progress on methods like integer quantisation which can help run the model on smaller hardware.
The leading models they’re offering, falcon, oasst, pythia, galactica are made by other people, are already quantized to run on “smaller hardware” available here and the method for quantizing for off-the-shelf Nvidia GPUs is here. You don’t need an expensive cloud server running a Discord bot to use these models, you can run them at home on Linux right now (or on a much cheaper cloud server).
What I think Linux desperately needs is a project that makes these easier to use similar to GPT4All but utilizing proper GPU quantized models, not just the CPU ggml models. On the cloud API front things seem to be going pretty well with things like langflow and Flowise, and most self-hostable AI projects tend to have good APIs though they can get buried a bit under all the SaaS marketting.
1 Like