Local LLM

July 29, 2024

Backend:

• https://github.com/ggerganov/ggml
• https://github.com/ggerganov/whisper.cpp
• https://github.com/tinygrad/tinygrad
• https://github.com/ml-explore/mlx

Compute:

• https://lambdalabs.com/

Inference:

• https://github.com/ggerganov/llama.cpp
• https://github.com/NVIDIA/TensorRT-LLM
• https://github.com/vllm-project/vllm
• https://github.com/mudler/LocalAI
• https://fireworks.ai/ (SAAS)
• https://replicate.com/ (SAAS)
• https://groq.com/ (SAAS)
• https://www.anyscale.com/ (SAAS)

Distribution:

• https://github.com/Mozilla-Ocho/llamafile
• https://github.com/replicate/cog

Applications:

• https://github.com/nomic-ai/gpt4all
• https://github.com/ollama/ollama
  • https://github.com/ollama/ollama-python
• https://github.com/lmstudio-ai/lms
• https://github.com/OpenInterpreter/open-interpreter
• https://github.com/janhq/jan

Coding assistant:

• https://github.com/continuedev/continue

Databases:

• https://github.com/facebookresearch/faiss
• https://github.com/lancedb/lancedb

RAG:

• https://github.com/weaviate/Verba

Utilities:

• https://github.com/urchade/GLiNER
• https://github.com/1rgs/jsonformer
• https://github.com/stanfordnlp/dspy
• https://github.com/openlit/openlit

NLP:

• https://github.com/stanfordnlp/stanza