[Finished] Sunday drop-in workshop: Install your own AI on Linux 🗜

Linux Discussion
1

Sunday drop-in workshop: Install your own AI on Linux

ai0
ai01024×171 238 KB

I’m running a drop-in 4hr workshop on Jitsi this Sunday for helping people learn about and install their own AI LLMs on Linux. These’ll be for text chat and text-to-image though i’m happy to share how to run STT and TTS in Python if anyone’s interested.

ALL experience levels are welcome from brand new Linux users to gurus of CLI.

If you just want to check out the software instead of installing it, i’ll be doing screen share demos and links to live demos. There’ll also be plenty of room for questions.

Workshop format will be adhoc depending on who arrives and when, i’m expecting to be helping people 1 on 1 as they come in with free chat but if there’s a lot of people i’ll be doing a guided installation that everyone does together.

Schedule: 2023-07-09T18:00:00Z→2023-07-09T22:00:00Z

[[ Jitsi link ]]

What hardware do I need?

  • For text chat (basic): 4GB of CPU ram or higher, no GPU needed
  • For text chat (advanced): Nvidia GPU with >=6GB of VRAM and CPU ram
  • For text-to-image: Nvidia GPU with >=6GB of VRAM and CPU RAM

Can I use an AMD GPU instead of Nvidia?

I don’t have experience with running AI on AMD but there’s options in some projects incl’ text-chat and text-to-image that make AMD GPU’s work, usually at the cost of needing double the CPU RAM. Happy to help you troubleshoot.

How else can I prepare?

If you’re running an Nvidia GPU, make sure the drivers from Nvidia are installed along with the cuda toolkit.

sudo apt install nvidia-cuda-toolkit

Install the Nvidia driver from your package manager:

Ubuntu Guide: 2 Ways to Install Nvidia Driver on Ubuntu 22.04 (GUI & Command Line)
Debian Guide: How to Clean Install NVIDIA Drivers on Debian 11

If everything is installed correctly, you should see your card after running this command in a terminal:

nvidia-smi

How do I get help outside the workshop?

Ask questions here, post in Help Desk, or @Ulfnic in TuxDigital Telegram / Matrix (note: primary TuxDigital chat is Discord) or Linux Saloon Telegram.

ai
ai833×165 167 KB

images in this post were rendered locally using stable-diffusion on a GeForce RTX 3060

2 Likes
2

I’m definitely going to be there. If there are any distro limitations or other requirements (such as already having Nvidia drivers installed) please let us know ahead of time. If there is anything else that can be done ahead of time by attendees to make things easier for you just post them. I installed Kubuntu 23.10 yesterday just to have some AIs running in Linux.

1 Like
3

Running late?

1 Like
4

Had a client call run long, needed a few extra mins :stuck_out_tongue: Link is up!

1 Like
5
6
1 Like
7
8

Note: video shows using the ggml version. You want gptq for GPU.

1 Like
9
1 Like
10

Nous Hermes version if anyone didn’t catch it.

I’ve been dragging my feet on Llama2 but Hermes was my intro and it’s very noticeably better than anything else self-hosted i’ve tried.

Problem is I really miss the 8k context versions the community worked out for Llama1 so I found the thread requesting this model in 13B on TheBloke’s Discord and asked about a possible SuperHOT-8K-GPTQ version.

What I got was this screenshot between the founder of NousResearch (maker of the model above) and Enrico Shippole who’s another major AI contributor.

image
image766×782 60.2 KB

Since then Enrico Shippole finished those 8k datasets for Llama2 so it’s over to Teknium, then TheBloke can roll a Nous-Hermes-8K-GPTQ version.

BUT it seems between Orca and TheBloke as of 3 days ago we’re getting our first Lama2-8K-GPTQ model in ggml and 13B!! (is it because I asked for a 8K-GPTQ Hermes in specifically 13B? No idea but man i’m happy lol). Going to run 8k Orca till I can try Hermes at 8k. Great stuff!!

1 Like
11

Awwww Man I can’t believe I missed this I hope this was great !

12

It was great. Hate you missed it. Ulfnic walked me right through the textgen installation.

1 Like