Three practices for maintaining a shared AI skill library

Imagine a colleague finding a skill called “Operations Reporting: Concise Analysis for Leadership” and using it to prepare a management report on a proposed project. The result looks respectable, until you notice that it has considered none of the legal questions this particular decision depends on. A closer look reveals that its original assignment was to report daily parking-space occupancy for the back-office team.

Admittedly, the example exaggerates. A better description would help, but even a clearly labelled skill needs to establish whether its method fits the assignment. Colleagues should be able to reuse a useful way of working without having to reconstruct all the assumptions its creator took for granted.

I would build three practices into how we introduce and maintain a shared skill library.

1. Start with a short interview.

The skill should explain its purpose and ask for the information needed to do the work. For the parking report, that means the location, reporting period and occupancy data. A request for a management decision paper should trigger clarification before drafting starts. Confirm the intended task and any limits with the user, using information already supplied. An “I understand” checkbox tells us little about whether the assignment fits.

2. Test where it should stop.

In addition to assessing its output quality, test an assignment it should handle, one it should decline, and one where it needs clarification. An instruction to recognise its limits is a starting point; we need to check whether it does. Keep these cases so that changes can be checked against work the previous version handled successfully.

3. Review without waiting for complaints.

Someone may quietly repair a poor result and move on. Where admitting to AI assistance feels uncomfortable, even that use may remain invisible. Voluntary feedback alone will not tell us whether the skill is working well.

Give each shared skill a maintainer who reruns those tests after material changes and at agreed intervals. Tasks with more serious consequences if they go wrong warrant closer review. As new uses expose problems, add those cases to the tests.

Before sharing a skill with colleagues

For a first review, I would ask the maintainer to demonstrate the following:

• Scope: Can the skill explain what it does, who it is for and which assignments fall outside its scope?
• Opening questions: Does it establish the purpose and required inputs, without asking again for information the user has already supplied?
• Boundary tests: Are there retained examples of work it should complete, decline and clarify, with an agreed basis for judging the results?
• Maintenance: Is someone responsible for reviewing it, with a review date and a check after material changes?
• Learning from use: When a new problem becomes visible, is it turned into a test case before the revised skill is shared?

Keep the record proportionate to the work. A routine occupancy report needs less scrutiny than a paper informing a consequential management decision; both need a way to detect when the assignment no longer fits.