Skip to content

Commit 9987855

Browse files
rouaultrouault
authored
Merge pull request #4717 from OSGeo/backport-4710-to-9.8
[Backport 9.8] Doc: Add AI tool policy 'human in the loop'
2 parents d228deb + ca8938b commit 9987855

4 files changed

Lines changed: 186 additions & 1 deletion

File tree

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<!-- Feel free to remove check-list items aren't relevant to your change -->
22

3-
- [ ] AI/LLM (Copilot, ChatGPT, Claude or something similar) supported my development of this PR
3+
- [ ] AI (Copilot or something similar) supported my development of this PR. See our [policy about AI tool use](https://proj.org/community/ai_tool_policy.html). Use of AI tools *must* be indicated.
44
- [ ] Closes #xxxx
55
- [ ] Tests added
66
- [ ] Added clear title that can be used to generate release notes

‎docs/source/community/ai_tool_policy.rst‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
.. _ai_tool_policy:
2+
3+
================================================================================
4+
AI/LLM tool policy
5+
================================================================================
6+
7+
Summary
8+
-------
9+
10+
PROJ's policy is that contributors can use whatever tools they would like to
11+
craft their contributions, but **there must be a human in the loop**.
12+
Contributors must read and review all AI (Artificial Intelligence) /
13+
Large Language Model (LLM)-generated code or text before they ask other project
14+
members to review it. The contributor is always the author and is fully
15+
accountable for their contributions.
16+
17+
Warning
18+
-------
19+
20+
It is not currently a settled question within the PROJ community whether code
21+
created by LLMs is acceptable in the PROJ code base at all. This document only
22+
addresses the interaction aspects of LLM content, and should not be viewed as the
23+
project taking a position on whether or not LLM code contributions are acceptable.
24+
25+
Policy
26+
------
27+
28+
PROJ's policy is that contributors can use whatever tools they would like to
29+
craft their contributions, but there must be a **human in the loop**.
30+
Contributors must read and review all AI (Artificial Intelligence) / Large Language Model (LLM)-generated code
31+
or text before they ask other project members to review it.
32+
The contributor is always the author and is fully accountable for their contributions.
33+
Contributors should be sufficiently confident that the contribution is high enough
34+
quality that asking for a review is a good use of scarce maintainer time, and they
35+
should be **able to answer questions about their work** during review.
36+
37+
We expect that new contributors will be less confident in their contributions,
38+
and our guidance to them is to **start with small contributions** that they can
39+
fully understand to build confidence. We aspire to be a welcoming community
40+
that helps new contributors grow their expertise, but learning involves taking
41+
small steps, getting feedback, and iterating. Passing maintainer feedback to an
42+
LLM doesn't help anyone grow, and does not sustain our community.
43+
44+
Contributors **must be transparent and label contributions that
45+
contain substantial amounts of tool-generated content**, and always mention it.
46+
The pull request and issue templates contain a checkbox for that purpose.
47+
Failure to do so, or lies when asked by a reviewer, will be considered as a violation.
48+
Our policy on labeling is intended to facilitate reviews, and not to track which parts of
49+
PROJ are generated. Contributors should note tool usage in their pull request
50+
description, commit message, or wherever authorship is normally indicated for
51+
the work. For instance, use a commit message trailer like Assisted-by: <name of
52+
code assistant>. This transparency helps the community develop best practices
53+
and understand the role of these new tools.
54+
55+
This policy includes, but is not limited to, the following kinds of
56+
contributions:
57+
58+
- Code, usually in the form of a pull request
59+
- RFCs or design proposals
60+
- Issue or security vulnerability reporting
61+
- Comments and feedback on pull requests
62+
63+
Details
64+
-------
65+
66+
To ensure sufficient self review and understanding of the work, it is strongly
67+
recommended that contributors write PR descriptions themselves (if needed,
68+
using tools for translation or copy-editing), in particular to avoid over-verbose
69+
descriptions that LLMs are prone to generate. The description should explain
70+
the motivation, implementation approach, expected impact, and any open
71+
questions or uncertainties to the same extent as a contribution made without
72+
tool assistance.
73+
74+
An important implication of this policy is that it bans agents that take action
75+
in our digital spaces without human approval, such as the GitHub `@claude`
76+
agent. Similarly, automated review tools that
77+
publish comments without human review are not allowed. However, an opt-in
78+
review tool that **keeps a human in the loop** is acceptable under this policy.
79+
As another example, using an LLM to generate documentation, which a contributor
80+
manually reviews for correctness and relevance, edits, and then posts as a PR,
81+
is an approved use of tools under this policy.
82+
83+
Extractive Contributions
84+
------------------------
85+
86+
The reason for our "human-in-the-loop" contribution policy is that processing
87+
patches, PRs, RFCs, comments, issues, security alerts to PROJ is not free --
88+
it takes a lot of maintainer time and energy to review those contributions! Sending the
89+
unreviewed output of an LLM to open source project maintainers *extracts* work
90+
from them in the form of design and code review, so we call this kind of
91+
contribution an "extractive contribution".
92+
93+
Our **golden rule** is that a contribution should be worth more to the project
94+
than the time it takes to review it. These ideas are captured by this quote
95+
from the book `Working in Public <https://press.stripe.com/working-in-public>`__ by Nadia Eghbal:
96+
97+
When attention is being appropriated, producers need to weigh the costs and
98+
benefits of the transaction. To assess whether the appropriation of attention
99+
is net-positive, it's useful to distinguish between *extractive* and
100+
*non-extractive* contributions. Extractive contributions are those where the
101+
marginal cost of reviewing and merging that contribution is greater than the
102+
marginal benefit to the project's producers. In the case of a code
103+
contribution, it might be a pull request that's too complex or unwieldy to
104+
review, given the potential upside.
105+
106+
-- Nadia Eghbal
107+
108+
109+
Prior to the advent of LLMs, open source project maintainers would often review
110+
any and all changes sent to the project simply because posting a change for
111+
review was a sign of interest from a potential long-term contributor. While new
112+
tools enable more development, it shifts effort from the implementor to the
113+
reviewer, and our policy exists to ensure that we value and do not squander
114+
maintainer time.
115+
116+
Handling Violations
117+
-------------------
118+
119+
If a maintainer judges that a contribution doesn't comply with this policy,
120+
they should paste the following response to request changes:
121+
122+
.. code-block:: text
123+
124+
This PR does not appear to comply with our policy on tool-generated content,
125+
and requires additional justification for why it is valuable enough to the
126+
project for us to review it. Please see our developer policy on
127+
AI-generated contributions:
128+
https://proj.org/community/ai_tool_policy.html
129+
130+
The best ways to make a change less extractive and more valuable are to reduce
131+
its size or complexity or to increase its usefulness to the community. These
132+
factors are impossible to weigh objectively, and our project policy leaves this
133+
determination up to the maintainers of the project, i.e. those who are doing
134+
the work of sustaining the project.
135+
136+
If/or when it becomes clear that a GitHub issue or PR is off-track and not
137+
moving in the right direction, maintainers should apply the `extractive` label
138+
to help other reviewers prioritize their review time.
139+
140+
If a contributor fails to make their change meaningfully less extractive,
141+
maintainers may lock the conversation and/or close the pull request/issue/RFC.
142+
In case of repeated violations of our policy, the PROJ project reserves itself
143+
the right to ban temporarily or definitely the infringing person/account.
144+
145+
Copyright
146+
---------
147+
148+
Artificial intelligence systems raise many questions around copyright that have
149+
yet to be answered. Our policy on AI tools is similar to our copyright policy:
150+
Contributors are responsible for ensuring that they have the right to
151+
contribute code under the terms of our license, typically meaning that either
152+
they, their employer, or their collaborators hold the copyright. Using AI tools
153+
to regenerate copyrighted material does not remove the copyright, and
154+
contributors are responsible for ensuring that such material does not appear in
155+
their contributions. Contributions found to violate this policy will be removed
156+
just like any other offending contribution. If a reviewer has doubts about the
157+
legal aspects of a contribution, they may ask the contributor to provide more
158+
details on the origins of a particular piece of code.
159+
160+
Credits for this document
161+
-------------------------
162+
163+
This document is a quasi direct adaptation from the
164+
`LLVM software "AI Tool Use Policy" <https://github.com/llvm/llvm-project/blob/main/llvm/docs/AIToolPolicy.md>`__,
165+
and due credits go to its original authors: Reid Kleckner, Hubert Tong and
166+
"maflcko"
167+
168+
.. below is an allow-list for spelling checker.
169+
170+
.. spelling:word-list::
171+
Reid
172+
Kleckner
173+
Hubert
174+
Tong
175+
maflcko
176+
LLM
177+
unreviewed
178+
Eghbal
179+
implementor

‎docs/source/community/contributing.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,11 @@ that you have a GitHub account. Familiarity with
2323
`issues <https://guides.github.com/features/issues/>`__ and the `GitHub
2424
Flow <https://guides.github.com/introduction/flow/>`__ is an advantage.
2525

26+
AI Tool Use Policy
27+
------------------
28+
29+
See :ref:`ai_tool_policy`.
30+
2631
Help a fellow PROJ user
2732
-------------------------
2833

‎docs/source/community/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ contributor the community is always very welcoming.
1313

1414
channels
1515
contributing
16+
ai_tool_policy
1617
code_contributions
1718
code_of_conduct
1819
service_providers

0 commit comments

Comments
 (0)