Promptfoo

Unit tests for your prompts

LLM / RAG frameworkPrompt testingMITRuns locallyTypeScriptBeginner

OSAI Pulse ⓘ ★★★★★★★★★★ /100 signals tracked

🐳 Docker pulls 📦 PyPI downloads / month 📦 npm downloads / month 🚀 Latest release ·

Visit website →View on GitHub

What is Promptfoo?

Promptfoo runs your prompts against test cases across models, catching regressions before they reach production, and includes red-teaming checks.

Why people choose Promptfoo

Regression tests for prompts
Compares models side by side
Built-in security red-teaming

Promptfoo at a glance

Category	LLM / RAG framework
Type	Prompt testing
License	MIT
Runs locally	Yes
Built with	TypeScript
Skill level	Beginner
Best for	teams shipping prompts to production

Open-source alternatives to Promptfoo

Other open-source llm / rag framework tools worth comparing:

LangChainCompose chains, tools and agents

LlamaIndexThe data framework for RAG

HaystackProduction pipelines for search and RAG

DSPyProgram — not prompt — language models

txtaiAll-in-one embeddings database

RAGFlowDeep-document-understanding RAG

Semantic KernelMicrosoft's enterprise agent framework

GraphRAGRAG that builds a knowledge graph first

LiteLLMOne API for 100+ LLM providers

InstructorReliable structured outputs from LLMs

OutlinesGuarantee valid JSON and grammars

GuidanceInterleave control and generation

LLMWareEnterprise RAG with small specialised models

FlashRankUltra-light reranking for better RAG

Sentence TransformersThe standard way to make embeddings

RagasMeasure whether your RAG is any good

LangfuseSee what your LLM app actually did

PhoenixTrace, evaluate and debug LLM apps

Promptfoo head-to-head

Promptfoo vs LangChain Promptfoo vs LlamaIndex Promptfoo vs Haystack Promptfoo vs DSPy Promptfoo vs txtai Promptfoo vs RAGFlow Promptfoo vs Semantic Kernel Promptfoo vs GraphRAG Promptfoo vs LiteLLM Promptfoo vs Instructor Promptfoo vs Outlines Promptfoo vs Guidance Promptfoo vs LLMWare Promptfoo vs FlashRank Promptfoo vs Sentence Transformers Promptfoo vs Ragas Promptfoo vs Langfuse Promptfoo vs Phoenix

FAQ

Is Promptfoo free?

Promptfoo is free and open-source (MIT license), so you can use, self-host and modify it at no cost.

Can I run Promptfoo locally?

Yes. Promptfoo is designed to run on your own machine or server, keeping your data private.

What is the best alternative to Promptfoo?

Popular open-source alternatives include LangChain, LlamaIndex, Haystack. See the comparisons above to choose.

Explore more open-source AI

Browse the full directory of open-source AI tools, models and projects — updated daily.

Browse all tools →