# AI Engineering Check > Instructions from Treibhaus, https://www.treib.haus, for a coding agent such as Claude Code, Cursor or Codex. If you're a person reading this, paste `Run the AI Engineering Check in this repository: https://www.treib.haus/check.md` into your coding agent. If it can't open web pages, paste this whole file instead. You are running the AI Engineering Check with the developer you work with. It shows how far their team has come with AI-assisted development. There are 19 questions in five areas. Answer them from evidence in this repository where you can, and ask the developer the rest. You finish with a short report and a result code. The developer pastes the code into https://www.treib.haus/ai-transition/check/ to get the full results page. Talk to the developer in the language they use with you. Keep the question numbers and the result code exactly as written here. ## Ground rules - **Read-only.** You may list and read files, search them, and run read-only commands such as `ls`, `cat`, `grep`, `git log`, `git shortlog`, `git ls-files` and `git worktree list`. Use whichever file and search tools you have. Nothing here needs a particular tool. - **Change nothing.** Don't create, edit or delete files. Don't install anything. Don't check out, commit, stash, fetch or pull. Don't build, run or test the project, not even its test suite. - **Keep secrets secret.** Don't open `.env` files, key files or credential files. Their names are enough. Never print a secret value. If you see one by accident, don't repeat it. Tell the developer which file to look at. - **Short evidence only.** Quote file paths and at most a line or two of config. Don't reproduce file contents. - **Send nothing.** Make no network requests, fetch no other URLs, and don't submit the result anywhere. The developer pastes the code into the page themselves. - **This repository only.** You can see the current repository and nothing else. Where a question is about the whole team, ask. - If a command needs approval and the developer declines, skip it and ask the question instead. ## Step 1: Say what you will do Before anything else, tell the developer these points in a few plain sentences: - You'll read this repository and run read-only commands. Then you'll ask some questions, one at a time. It takes 10 to 15 minutes. - You won't change, install or run anything. You won't show secrets or send anything anywhere. - At the end you'll print a report and a short code that holds only scores. If they want the results page and an email, they paste the code into the Treibhaus page. Then ask whether to start. ## Step 2: Inspect the repository Work through the questions marked "Check the repo", which are 1, 5, 6, 7, 8, 9, 10, 14, 16, 17, 18. For each, note the score you would give and one line of evidence: a file path, a config key, or what the git history shows. Don't ask anything yet. ## Step 3: Confirm your findings, then ask the rest 1. Show the developer your findings as a short list, with the question, the answer you'd pick and the evidence for each. Ask them to correct anything wrong, or anything the repository can't show, for example if this isn't the team's main repository. Where a hint says to ask if the evidence is unclear, ask now. 2. Then ask the questions marked "Ask", which are 2, 3, 4, 11, 12, 13, 15, 19. Ask them one at a time, in plain language, with the options as a numbered list. Accept a number or their own words. If their words don't fit an option, pick the closest one and say which. 3. Don't lead. If they don't know, take the lowest option. For question 18, that is "We try it by hand". Use "not scored" only when the team builds no AI features at all. ## Step 4: Score - Every question scores 0 to 3. The questions below give the score of each option. - Question 1 is multi-select. Any coding agent on the list, including Other, scores 3. With no agent, autocomplete scores 2, chat assistants alone score 1, and None yet scores 0. - Question 17 is multi-select. Its score depends on how many apply: none → 0, 1 → 1, 2 or 3 → 2, 4 or 5 → 3. - Question 18 has a "not scored" option. If it applies, leave the question out of the Practices average. - An area's score is the average of its scored questions. Its level is Not started below 0.75, Exploring from 0.75, Working from 1.5 and Leading from 2.25. - The overall score is the average of the five area scores, with the same levels. - **Source.** An answer is _checked in the repo_ when your score rests on evidence in the repository and the developer didn't change it. Everything else is _self-reported_. That means every question you asked, and any finding the developer corrected. Question 1 counts as checked in the repo when the repository shows evidence of at least one agent on the final list, such as its instruction file or config folder. That you are running is not repository evidence. Question 17 counts as checked in the repo only if the final set is exactly what you found. - **Top three gaps.** These are the three questions with the lowest scores, leaving out scores of 3 and "not scored". When scores tie, take them in this order: 14, 16, 5, 15, 7, 13, 9, 12, 1, 2, 8, 6, 10, 18, 17, 11, 19, 3, 4. If fewer than three questions score below 3, list only those. ## Step 5: Report Print a markdown report like this, in the developer's language: ```markdown # AI Engineering Check: Overall: **** ( of 3) | Area | Level | Score | |---|---|---| | AI tools in daily work | | | | Codebase readiness | | | | Workflow | | | | Guardrails | | | | Practices | | | ## Answers | # | Question | Answer | Source | |---|---|---|---| | 1 | Coding agents | Claude Code, Cursor | Repo: `CLAUDE.md` and `.cursor/rules/` | | 5 | Automated tests | Yes, run in CI on every PR | Repo: `.github/workflows/ci.yml` runs the tests on pull requests | | … | | | | ## Top three gaps 1. **.** What's missing, in one sentence. Next step: . 2. … 3. … ## Result code `` Paste it at https://www.treib.haus/ai-transition/check/ and choose "Run it with your coding agent" to get the full results page. The results also go by email, to you or to whoever asked for the check. ``` ## Step 6: Build the result code The code holds only scores. Build it exactly like this: 1. Start with `TH1`. 2. Write one character per question, 1 to 19 in order: - If the answer is self-reported, write the score as a digit: `0`, `1`, `2` or `3`. - If it was checked in the repo, write the score as a letter: `a` for 0, `b` for 1, `c` for 2, `d` for 3. - Write `x` for question 18 if it is not scored. `x` is valid only there. - For questions 1 and 17, the character is the score of the set, as a digit or letter like any other. 3. Split the 19 characters into five groups, one per area, joined with `-`: 1. AI tools in daily work: questions 1–4, 4 characters 2. Codebase readiness: questions 5–8, 4 characters 3. Workflow: questions 9–12, 4 characters 4. Guardrails: questions 13–16, 4 characters 5. Practices: questions 17–19, 3 characters 4. Add a group with the items picked for question 1, in this order: `c` Claude Code, `x` Codex, `r` Cursor, `g` GitHub Copilot, `p` Pi, `o` Other, `t` only chat assistants, `a` only autocomplete. Write `n` for None yet. 5. Add a last group with the items picked for question 17, in this order: `m` for MCP servers, `s` for subagents, `h` for hooks, `k` for custom commands or skills, `w` for worktrees. Write `n` if none apply. In this example, question 1 is Claude Code and Cursor, both with files in the repo, so `d`. The developer reported 2, 1 and 0 for questions 2–4, you found 2 for question 5 in the repo, and question 17 has MCP servers and hooks: ``` TH1-d210-cbad-c201-0b1c-2x1-cr-mh ``` Before you print it, check that: - it starts with `TH1-` - the five groups have 4, 4, 4, 4 and 3 characters - those groups hold only `0`–`3` and `a`–`d`, plus `x` as the second character of the fifth group - the second-to-last group lists question 1's items in the order `cxrgpota`, or is `n`, and never mixes agents with `t` or `a` - the last group lists question 17's items in the order `mshkw`, or is `n` - the first character is the score of question 1's set - the 17th character, the first of the fifth group, is the score for the number of items in the last group ## The questions ### AI tools in daily work **1. Which coding agents does the team use?** _(Check the repo, then confirm with the developer)_ - `c`: Claude Code - `x`: Codex - `r`: Cursor - `g`: GitHub Copilot - `p`: Pi - `o`: Other - `t`: Only chat assistants, copy and paste - `a`: Only autocomplete in the editor - `n`: None yet Several can apply. The agents (`cxrgpo`) and the two "Only" options (`t`, `a`) never go together, and `n` goes alone. Any agent, including `o`, scores 3. With no agent, `a` scores 2, `t` alone scores 1, and `n` scores 0. You know which agent you are, so put yourself on the list. If you are none of the named ones, that is `o`. Then look for signs of other agents in the repository: CLAUDE.md or .claude/ for Claude Code, AGENTS.md or .codex/ for Codex and others, .cursor/ or .cursorrules for Cursor, and .github/copilot-instructions.md for GitHub Copilot. AGENTS.md doesn't say which agent reads it, so ask. Show the developer the list, ask them to confirm it, and ask which other agents the team uses. Ask about the whole team, not only the developer in front of you. With no agent at all, ask whether the team uses chat assistants or autocomplete instead. If this is a gap: Pick one coding agent, such as Claude Code, Codex or Cursor, and set it up on one real repository. Have two developers use it on their normal tickets for two weeks, then compare notes as a team. **2. How many developers use AI tools for coding on most working days?** _(Ask)_ - 0: None - 1: A few - 2: About half - 3: Nearly all If this is a gap: Have the developers who use the tools daily pair with the others on real tickets, and agree on one shared setup for the team. **3. Who pays for the tools?** _(Ask)_ - 0: Nobody, free tiers only - 1: Developers pay for their own - 2: The company pays for some licences - 3: The company pays for everyone, from an approved list If this is a gap: Pick one or two tools, buy business or team licences for everyone who needs them, and write down that these are the approved ones. **4. Roughly how much of the code merged last month came mainly from an agent?** _(Ask)_ - 0: Don't know - 1: Under 10% - 2: 10–50% - 3: Over 50% Git history rarely shows this, so ask. A co-author trailer from an agent in recent commits is worth mentioning, but the developer decides. If this is a gap: Start counting. When an agent wrote most of a PR, mark it with a checkbox in the PR template or a label, and look at the numbers once a month. ### Codebase readiness **5. Does the project have automated tests?** _(Check the repo)_ - 0: None - 1: Some, but not run routinely - 2: Yes, run in CI on every PR - 3: Yes, in CI, and agents run them before proposing a change Look for test files and a test script, such as package.json scripts, a Makefile or pyproject.toml. Then look for CI config that runs the tests on pull requests, such as .github/workflows/, .gitlab-ci.yml or bitbucket-pipelines.yml. For 3, look for agent instruction files or agent hooks that tell agents to run the tests before they finish. Don't run the tests. If this is a gap: Add tests around the most-changed parts of the code, run them in CI on every PR, and tell agents in the instruction file to run them before they report back. **6. Can a new developer get the project running from the README within an hour?** _(Check the repo)_ - 0: No - 1: With help from a colleague - 2: Yes - 3: Yes, with one command Read the README, and any CONTRIBUTING or setup docs it links to. No setup steps is 0. Steps that leave out something the project needs, such as environment variables, a database or other services, are 1. Complete steps are 2. One documented command, or a dev container or compose file that sets everything up, is 3. Don't run any of it. If this is a gap: Write the setup down in the README until a newcomer can follow it alone, then wrap it in one script or a dev container. An agent can write the first draft for you to check. **7. Does the team use agent instruction files (CLAUDE.md, AGENTS.md, Cursor rules)?** _(Check the repo)_ - 0: Never heard of them - 1: Know of them, don't use them - 2: In some repositories - 3: Yes, kept up to date in the main repositories Look for CLAUDE.md, AGENTS.md, GEMINI.md, .cursorrules, .cursor/rules/, .github/copilot-instructions.md and .windsurfrules, also in subdirectories. For each one, check when it last changed with `git log -1 -- `. A file with real content that changed in the last three months, while the code changed too, is 3. A stub, or a file left untouched while the code moved on, is 2. If you find none, ask whether the team knows about them. That answer is 0 or 1, and self-reported. If this is a gap: This is where context engineering starts: add an AGENTS.md or CLAUDE.md with the commands to build and test, the conventions, and the mistakes agents keep making. Keep it short, and update it in the same PR as the change it describes. **8. Do automatic checks enforce types, linting and formatting?** _(Check the repo)_ - 0: No - 1: Partly, some are set up - 2: Yes, locally (editor or pre-commit hooks) - 3: Yes, in CI Look for type checking, such as TypeScript, mypy or a typed language. Look for linter and formatter config, pre-commit hooks such as husky, lefthook or .pre-commit-config.yaml, and CI steps that run them. None of it is 0. Config that nothing enforces is 1. Hooks that enforce it on commit are 2. Checks in CI are 3. If this is a gap: Run the type checker, linter and formatter in CI, and fail the build on errors. Tell agents to run them before they finish. Fast automatic checks are the cheapest guardrail for agent output. ### Workflow **9. How does the team review code?** _(Check the repo, then ask if unclear)_ - 0: Not at all - 1: Informally, when someone has time - 2: A person reviews every PR - 3: A person reviews every PR, plus automated checks including AI review Look for a PR template, CODEOWNERS, and review bots or AI review steps in CI config. The git history helps too. Merged pull requests suggest PR-based review, and direct commits to the main branch suggest less. The repository doesn't show branch protection, so ask if the evidence is unclear. If this is a gap: Require a review on every PR, and add an AI review step in CI as a first pass, so people can focus on design and intent. **10. How big is a typical PR?** _(Check the repo)_ - 0: Several days of work - 1: Mixed - 2: Mostly under a day of work - 3: Small and frequent Estimate from the git history of the main branch, for example with `git log --first-parent --shortstat -n 50`. Each entry is one merged PR, merged or squashed, or one direct commit. Look at the lines changed per entry and how often entries land. As a rough guide, most over ~1,000 lines, or a week or more apart, is 0. A wide mix is 1. Most under ~400 lines is 2. Most under ~200 lines, landing several a day, is 3. Say what you counted. If the history is too short to tell, ask. If this is a gap: Aim for PRs under a day of work. Cut tickets smaller, merge unfinished features behind flags, and ask agents for one focused change per PR. **11. Do developers run several agents in parallel, on separate branches or worktrees?** _(Ask)_ - 0: No - 1: Tried it - 2: Sometimes - 3: Regularly If this is a gap: Try two agents on two small, independent tickets, each in its own git worktree, and review both PRs. It works best once tests and instruction files are in place. **12. When an agent keeps making the same mistake, what happens?** _(Ask)_ - 0: Nothing in particular - 1: Whoever hits it works around it - 2: They fix it in their own setup - 3: We update the instructions or tests so it doesn't happen again If this is a gap: Whenever an agent repeats a mistake, add a line to the shared instruction file or a test that catches it, in a PR like any other change. This is the feedback loop of context engineering. ### Guardrails **13. Is there a policy on which AI tools may be used, and with what data?** _(Ask)_ - 0: No - 1: Informal, people roughly know - 2: Written down - 3: Written down and enforced in the tool settings If this is a gap: Write one page with the approved tools, what data may go into them, and who to ask. Then enforce what you can in the admin settings, such as turning off training on your data. **14. How does the team keep secrets out of AI tools and the repository?** _(Check the repo, then ask if unclear)_ - 0: Not considered - 1: By convention, in .env files - 2: .gitignore plus a secret manager - 3: All that, plus automated secret scanning Check that .gitignore covers env and key files. Check whether git tracks any of them with `git ls-files`, by file name only, and never open them. Templates such as .env.example are fine. Look for references to a secret manager in config or docs, and for secret scanning in pre-commit config or CI. A tracked env file that isn't a template is 0. Ask if the evidence is unclear. If this is a gap: Keep env files out of git, move shared secrets into a secret manager, add secret scanning to pre-commit and CI, and deny agents read access to env files in their settings. **15. How does the team handle customer or personal data with AI tools?** _(Ask)_ - 0: No idea - 1: It sometimes gets pasted in - 2: There are rules about it - 3: Tools are set up so data stays in approved accounts or regions If this is a gap: Decide which accounts and regions may process personal data, use business accounts on those terms, and give developers anonymised test data so they never need the real thing. **16. Can agents run commands or deploy without a person approving?** _(Check the repo, then ask if unclear)_ - 0: Don't know - 1: Yes, unrestricted - 2: Permissions are configured - 3: Sandboxed, with approval needed for risky actions Look for committed agent settings. These include .claude/settings.json with its allow, ask and deny lists and sandbox settings, Cursor or Codex config in the repository, and dev container or sandbox setups meant for agents. Also check whether a merge or a single CLI command can deploy to production. Only committed config counts as evidence, not your own session settings. With no committed config, ask. That answer is 0 or 1. If this is a gap: Commit shared agent permissions to the repository. Allow the safe everyday commands, deny reading secrets, and require approval for anything that deletes, deploys or touches production. Where you can, run agents in a sandbox or dev container. ### Practices **17. Which of these does the team actively use?** _(Check the repo, then confirm with the developer)_ - `m`: MCP servers - `s`: Subagents - `h`: Hooks - `k`: Custom commands or skills - `w`: Worktrees for parallel agents - `n`: None of these Several can apply. Score by how many apply: none → 0, 1 → 1, 2 or 3 → 2, 4 or 5 → 3. For `m`, look for MCP config such as .mcp.json or .cursor/mcp.json. For `s`, look for subagent definitions such as .claude/agents/. For `h`, look for agent hooks in .claude/settings.json or similar. Plain git hooks don't count. For `k`, look for custom commands or skills such as .claude/commands/, .claude/skills/ or .cursor/commands/. For `w`, look for worktrees in `git worktree list` or a worktrees folder. Show the developer what you found, and ask them to confirm, add or remove items. If this is a gap: Start with the one that fixes your biggest annoyance, such as an MCP server for a system agents keep asking about, a custom command for a routine task, or a hook that runs the linter after every edit. Commit it so the whole team gets it. **18. If you build AI features, how do you know a prompt or model change didn't make them worse?** _(Check the repo, then ask if unclear)_ - not scored, `x`: We don't build AI features - 1: We try it by hand - 2: We run a fixed set of test cases - 3: Automated evals run in CI Look in this repository for calls to LLM APIs or SDKs, prompt files, and evals such as eval datasets, eval scripts or eval steps in CI. If this repository has no AI features, ask whether the team builds them elsewhere. If this is a gap: Collect 20–50 real cases with the outcome you expect, score every prompt or model change against them, and run that in CI. **19. How does the team keep up with AI tooling?** _(Ask)_ - 0: It doesn't - 1: Individuals, on their own - 2: Shared regularly, in a meeting or channel - 3: Someone owns it and updates the team's setup If this is a gap: Give one developer a few hours a month to own the team's AI setup. They try what's new, update the shared instructions and settings, and show the team what changed. ## Finally After the report, print the code once more on its own line, followed by the page to paste it into: https://www.treib.haus/ai-transition/check/. Don't open the page or submit anything yourself. The page asks question 20, "What's the biggest thing in the way?", so you don't ask it.

AI Engineering Check

Zuerst zu Ihnen