docetl
A system for agentic LLM-powered data processing and ETL
DocETL is an agent skill and framework for building LLM-powered data processing pipelines that transform structured and unstructured data at scale. It allows developers to describe ETL operations in natural language (e.g., classify, filter, summarize) and automatically orchestrates map-reduce workflows, optimizes prompts and model selection, and parallelizes execution across datasets. The skill integrates with Claude Code to let AI assistants generate complete pipelines from task descriptions, while also offering Python and YAML APIs for scripting and production use. DocETL handles the complexity of chaining LLM calls, tuning for accuracy and cost, and returning queryable tabular results.
Key Features
Use Cases
- 01Classifying and summarizing customer support tickets by category using LLM-powered map-reduce
- 02Extracting structured data from unstructured documents like contracts, reports, or research papers
- 03Building data transformation pipelines in Jupyter notebooks with automatic cost optimization
- 04Processing large volumes of text data (reviews, feedback, transcripts) with parallelized LLM calls
- 05Prototyping ETL workflows interactively in DocWrangler before deploying to production
- 06Letting Claude Code generate complete data processing pipelines from task descriptions
Related Skills
View moresuperpowers
An agentic skills framework & software development methodology that works.
skills
Skills for Real Engineers. Straight from my .agents directory.
skills
Public repository for Agent Skills
ponytail
Makes your AI agent think like the laziest senior dev in the room. The best code is the code you never wrote.
docetl — FAQ
What is DocETL?+
DocETL is both an agent skill for Claude Code and a standalone framework for building LLM-powered ETL pipelines. It orchestrates data transformations using natural language operations like map, reduce, and filter, automatically optimizing for accuracy and cost.
How do I install DocETL for Claude Code?+
Install via pip (`pip install docetl`), then run `docetl install-skill` to add the skill to Claude Code. After installation, describe your data processing task to Claude and it will generate the pipeline.
Which AI clients does DocETL work with?+
DocETL includes a native skill for Claude Code. For ChatGPT or the Claude app, copy the prompt from docetl.org/llms-full.txt before describing your task. It can also be used standalone via Python API or YAML.
Do I need API keys to use DocETL?+
Yes, you need an LLM provider API key (OpenAI, Anthropic, etc.). Set it as an environment variable like `export OPENAI_API_KEY=your_key` before running pipelines.
Is DocETL free?+
DocETL itself is free and MIT licensed. You pay only for LLM API calls made during pipeline execution; the framework tracks total cost after each run.
Can I use DocETL without Claude Code?+
Absolutely. DocETL offers a full Python API for notebooks and scripting, a YAML interface for low-code pipelines via CLI, and DocWrangler UI for visual development.
How do I install docetl?+
Open the source repository on GitHub and follow its README. docetl is a skill — MCP Agents Market links you directly to the official repo.
Is docetl free?+
docetl is an open-source project hosted on GitHub. Check the repository for its license and any usage requirements.