Free Open-Source Artificial Intelligence

You can keep an eye on p9. The aim is to make a decent (maybe even good) quality local copilot LLM.

Specifically on what LLM to use, I've been meaning to try Starcoder, but can't vouch for how good it is. In general I've found Vicuna-13B pretty good at generating code.

As for general recommendations, I'd say the main determinant will be if you can afford the hardware requirements to locally host - I presume you're familiar with the fact that you'll (usually) need roughly 2x the number of parameters in VRAM (e.g. 7B parameters means 14GB of VRAM). Techniques like quantization to 8-bits halve the requirement, with the more extreme 4-bit quantization halving them again (at the expense of generation quality).

And if you don't have enough VRAM, there's always llama.cpp - I think that list of supported models is outdated, and it supports way more than those.

On the "what software to use for self-hosting" I've quite liked FastChat, they even have a way to run an OpenAI API compatible server, which will be useful if your tools expect OpenAI.

Hope this is helpful!

There is a bit of a conundrum here: in order to run a model that is any good in coding you want it to have a lot of parameters (the more the better) but also since it’s code and not some spoken language - precision matters here. Home hardware like 3090 is able to run ~30b models, but there is a catch - it just fits and only in quantized form = with 4x worse precision typically. Unless we see some breakthrough here that makes inference of huge models possible at full precision - the hosted AI will always be better for coding. Not saying such breakthrough is impossible though - quite the opposite in my opinion.