Thread

7B parameters models work fine with 16 GB RAM, getting ~3 word/s on Ryzen 5 5600X. Running on GPU helps a lot even with a low end one: getting ~12 word/s on RX6400 with 4 GB GDDR6. I was not able to run models with more parameters, limited by RAM.