How to Build a Feedback Loop Into Any AI Skill

Illustration of a sunny kitchen table with a laptop showing an AI chat beside a cork board of sticky-note rules

Updated November 2026 · Rewritten on November 13 for the new Skills features in Claude, ChatGPT and Gemini.

A feedback loop turns your corrections into written rules, so an AI skill stops repeating the same mistakes. Below is a simple version that works in any app, plus the built-in tools that now handle part of it.

The short versionKeep a short “lessons learned” file next to your skill. Tell the AI to read it before each task and to suggest new rules after each one. You approve the rules. In Claude you can now edit a skill in place, and ChatGPT and Gemini now use the same skill format.

Why Do AI Skills Stop Getting Better?

An AI skill is a saved set of instructions for one job, such as writing a newsletter section or summarizing a meeting. You write it once and reuse it. The trouble is that it stays exactly as you wrote it on day one.

So you correct the same things again and again. You swap “purchase” for “buy”, you ask for the summary to come first, you remind it your readers are beginners. The AI forgets those fixes unless you write them down where it will read them next time.

That is the whole idea of a feedback loop: each correction becomes a rule, and the rules travel with the skill. Anthropic’s advice for writing skills puts it in one line.

Anthropic skill best-practices docs showing the run validator, fix errors, repeat pattern
“Run validator, fix errors, repeat”, from Anthropic’s skill best practices. Screenshot: Beginners in AI, November 13, 2026.

What Does the Research Say?

Feedback loops, by the numbers

~20%

average gain when a model critiques and revises its own work, across 7 tasks

Self-Refine, 2023

91%

coding test score for an AI that writes itself lessons from feedback and tries again

Reflexion, 2023

5 of 6

Anthropic skills that triggered more reliably after their descriptions were tuned

Anthropic

HumanEval coding test, first-try pass rate

GPT-4 alone

Reflexion agent with written lessons

Figures as reported in the Reflexion paper (Shinn et al., 2023). One coding benchmark, not a measure of every task.

The Reflexion result is the closest research match to a lessons file. The agent wrote a short note about what went wrong, kept it, and read it before the next try.

One warning: a 2023 study found that models asked to check their own reasoning, with nothing from outside to go on, struggled to fix their answers and sometimes made them worse. The loop works because you supply the correction. The AI writes the rule down; you decide whether it is right.

Build the Loop in Four Steps

Step 1: Start a lessons file

Make a plain text file called lessons.md. It can start almost empty. This template gives it sections to grow into:

lessons.md template
# Lessons Learned: [skill name]
Last updated: [date]

## Always do
- [Example: Define every technical term the first time it appears]
- [Example: Put the main answer in the first two sentences]

## Never do
- [Example: Don't write "purchase"; write "buy"]
- [Example: Don't open with "In today's world"]

## Format rules
- [Example: Numbered lists for steps, bullets for everything else]
- [Example: A table whenever you compare three or more things]

## About the audience
- [Example: Readers are beginners; give a concrete example before the theory]

## Mistakes to check for
- [Example: The FAQ section is often missing from first drafts]

## Change log
- [date]: started this file

Step 2: Tell the skill to use it

Add these lines to the skill’s instructions (or a project’s instructions). They make the AI read the file first and propose new rules last. Our CLEAR prompting guide covers writing the rest of the instructions.

Add to your instructions
Before you start any task, read the Lessons Learned file and follow every rule in it.

When you finish, suggest 2 or 3 specific additions to the Lessons Learned file, based on what went well, what went badly, and any corrections I made. Quote the correction, then write the rule.

Step 3: Review the session

At the end of any session where you corrected the AI, paste this. It turns your corrections into rules you can copy straight into the file:

End-of-session review
Look back over this chat. List every correction I made to your work. For each one, write a one-line rule I could add to my Lessons Learned file so you never need that correction again. Skip anything that only applied to this one task.

Read the suggestions before you accept them. Keep the ones that match what you want, and add anything it missed. A rule like “define technical words on first use” is worth keeping; a rule about one odd task isn’t.

Step 4: Keep the file short

More rules aren’t always better. Anthropic suggests keeping a skill’s main file under 500 lines, and the docs for Claude Code, its coding tool, say instruction files longer than about 200 lines “reduce adherence”. Every few weeks, merge duplicate rules and delete ones you no longer need.

How Do You Update a Skill in Claude Now?

Claude changed most here since April. A Claude skill is a folder with a SKILL.md file inside, and custom skills now work on every plan, including Free, as long as code execution (Claude running small programs for you) is turned on. You manage them under Customize > Skills.

You no longer need to download, edit and re-upload a file to change a skill. While you chat, the skill file opens beside the conversation. Highlight the text you want changed, click Edit with Claude, and type the change. You can also ask Claude in the chat to update the skill.

Claude Help Center section explaining Edit with Claude for skills
Editing a skill in place, from Anthropic’s Help Center. Screenshot: Beginners in AI, November 13, 2026.

That means your lessons file can live inside the skill folder. Here is a small skill that points to one:

SKILL.md with a lessons file
---
name: beginner-explainers
description: Writes short plain-English explainers of AI terms for beginners. Use when asked to explain an AI word or idea.
---

# Beginner explainers

Read lessons.md in this skill before writing, and follow every rule in it.

## Steps
1. Answer in the first two sentences.
2. Give one everyday example.
3. Define any other technical word you use.
4. Keep it under 150 words.

The name must use lowercase letters, numbers and hyphens, and match the folder name. Keep the description under 200 characters (Claude’s upload limit) and say when the skill should be used; that is the part Claude reads to decide whether to load it. The full format rules are in the Agent Skills specification.

To make one without writing files, describe the job in a chat and ask Claude to build a skill. It uses a built-in skill-creator to write the SKILL.md for you. Since March, skill-creator can also write tests for a skill and compare two versions side by side, judging the results without knowing which is which. It’s the lessons-file loop with test scores attached.

Memory, Projects and Skills: Which Should Hold Your Lessons?

Illustration of an index-card box of rules beside a laptop showing an AI chat
A lessons file is a card box for your AI. Illustration made with Gemini in the style of Edward Hopper.

Claude now has three places that can remember things, and they suit different kinds of lessons.

FeatureWhat it holdsBest forHow you edit it
SkillInstructions for one kind of task, loaded only when neededRules for a repeat jobEdit with Claude, or ask Claude to update it
ProjectInstructions and files that are always loaded in that projectBackground for ongoing workSet project instructions; add or swap files in project knowledge
MemoryFacts about you, saved as topics while you chatYour preferences across every chatSettings > Your Claude > Manage
Claude Code (CLAUDE.md + auto memory)Your written rules, plus notes Claude saves from your correctionsCoding and file workEdit the file, or type /memory

Memory is on by default on Free, Pro, Max and Ultra, and you can read, edit or delete each topic it saves. It is good for general preferences, but it doesn’t replace a lessons file for a specific task, because you can’t see exactly when it applies. In Claude Code, auto memory has a type called feedback for “corrections you give Claude”, which is a lessons file Claude keeps for you. See our Claude memory settings guide, Projects guide and CLAUDE.md guide for the details.

Illustration of a lamplit desk with a ring binder of rules growing new pages beside a laptop
Every correction becomes a page in the binder. Illustration made with Gemini in the style of Kay Nielsen.

What We Learned Running Our Own Loop

We run this site this way. When our editor corrects the AI that helps produce these pages, the correction becomes a short written rule that every later session reads first. On November 13, 2026 the file held 81 rules, written between April and November.

Month (2026)New rulesExample rule from that month
April11Never bulk-edit many published pages in one run
May21Run every new post through a check for AI writing habits
June to September18Check a partner’s link in their page code, not their word
October14Put every copyable prompt in a box with a Copy button
November (to the 13th)17Take screenshots at double resolution so they stay sharp

From our fileThe rules came in bursts, when we started something new (May was a busy launch month). The best rules say why, so the AI can apply them to cases we never listed. And some rules replaced older ones, so pruning matters as much as adding.

It also shows the limit of a loop, which only learns what you correct. Our October 10 newsletter edition covered Anthropic’s test agents doing jobs nobody asked for, and the fix there was another written rule saying where the job ends.

Feedback Loops in ChatGPT and Gemini

Both now offer skills in the open SKILL.md format (in ChatGPT, on Business, Enterprise, Healthcare and Edu plans), so a lessons file works there too. Custom GPTs and Gems, the older tools, are being retired.

ChatGPTGemini
SkillsSkills for Business, Enterprise, Healthcare and Edu, under Plugins > SkillsGemini Skills for personal accounts 18+, under Customize > Skills
Old featureCustom GPTs retire on December 11, 2026; their instructions become a skill in a pluginGems become read-only on November 17, 2026 and are removed on December 17
Where to keep lessonsA Project’s instructions and files, or inside a skillA file inside the skill (download, add the file, then Replace skill)

If you built your loop inside a Custom GPT or a Gem, move it now. Copy the instructions and your lessons file somewhere safe before the dates above. For the general idea beyond any one app, see our AI feedback loop guide, and for loops that run on their own, the Ralph loop.

Beginners in AI community on Skool: learn AI in fourteen days, master it in 30, one small win a day, zero experience needed

Making It a Habit

  • End every real session with the review prompt. It takes a couple of minutes.
  • Write the reason with the rule. “Define jargon, because readers are beginners” beats “define jargon”.
  • Retest after a model update. Anthropic warns a skill that worked last month can behave differently today, and some rules stop being needed.
  • Prune once a month. Merge duplicates and delete rules you no longer need.

New to building skills? Start with our 10 AI skills every beginner should build, then add a lessons file to the one you use most. If you’re new to AI altogether, how to use AI and what is artificial intelligence cover the basics, and our free AI courses list has more.

Frequently Asked Questions

Do I need to code to build a feedback loop?

No. A lessons file is plain text, and the instructions are two sentences. Claude can write the skill file for you when you describe the job in a chat.

Does Claude learn from my corrections on its own?

Partly. Claude’s memory saves preferences as topics you can edit, and Claude Code’s auto memory saves corrections. A lessons file is still the clearest way to control exactly which rules a specific skill follows.

How long should the lessons file be?

Short enough that every rule still matters. Anthropic suggests keeping a skill’s main file under 500 lines; most lessons files work best far shorter. Prune duplicates every few weeks.

What happens to my Custom GPT or Gemini Gem?

Custom GPTs retire on December 11, 2026 and can be migrated to a plugin, where the instructions become a skill. Gems turn read-only on November 17 and are converted to Gemini Skills, then removed on December 17. Save your instructions and lessons file first.

Can the AI grade its own work instead of me?

Not well. Research shows models struggle to correct themselves with no outside feedback, and sometimes get worse. Let the AI suggest rules, but you decide which ones go in.

Made for regular people, by a real person

One AI tool, tip or update a day in plain English, including tricks like this that make the tools you already use work better. Curated and edited by a human.

Free forever. Unsubscribe anytime.

Sources

You May Also Like