Skip to content

Commit 7415a42

Browse files
feat: add community flag and classification details (#16)
Thank you for building this! I was working on an action myself, but this one seems to work pretty well, so I thought I would add a few things I had in mind: ### Copy change I have softened the text because this tool isn't foolproof and will make errors. People shouldn't feel judged too harshly; we're simply analyzing their events to identify potential patterns similar to automations. ### Community flags I've added the [community flag](https://github.com/MatteoGabriele/agentscan/blob/main/data/verified-automations-list.json) by querying the JSON file in the AgentScan repo. Let me know what you think 🙏
1 parent a1553ce commit 7415a42

File tree

1 file changed

+50
-5
lines changed

1 file changed

+50
-5
lines changed

src/detect-agent.ts

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
1-
import { identifyReplicant } from "voight-kampff-test";
1+
import { identifyReplicant, getClassificationDetails } from "voight-kampff-test";
22
import { setOutput } from "./utils";
33

44
const jsonMode = process.argv.includes("--json");
5+
const log = jsonMode ? console.error : console.log;
56

67
const { GITHUB_TOKEN, PR_AUTHOR } = process.env;
78
if (!GITHUB_TOKEN || !PR_AUTHOR) {
@@ -44,13 +45,55 @@ const { classification, score, flags } = identifyReplicant({
4445
events,
4546
});
4647

47-
const isAgent = classification === "automation";
48+
type AutomationListItem = {
49+
username: string;
50+
reason: string;
51+
createdAt: string;
52+
issueUrl: string;
53+
};
54+
55+
let verifiedAutomationList: AutomationListItem[] = []
56+
57+
try {
58+
const automationJSONPath = 'data/verified-automations-list.json'
59+
const response = await fetch(
60+
`https://api.github.com/repos/matteogabriele/agentscan/contents/${automationJSONPath}`,
61+
{ headers },
62+
);
63+
64+
if (!response.ok) {
65+
throw new Error(`Failed to fetch verified automations list: ${response.status}`);
66+
}
67+
68+
const data = await response.json();
69+
70+
if ("content" in data) {
71+
const content = Buffer.from(data.content, "base64").toString("utf-8");
72+
verifiedAutomationList = JSON.parse(content)
73+
}
74+
} catch (error) {
75+
log("Could not fetch verified automations list");
76+
}
77+
78+
const verifiedAutomation: AutomationListItem | undefined = verifiedAutomationList.find(
79+
(account) => account.username === PR_AUTHOR,
80+
);
81+
82+
const communityFlagDetails = {
83+
label: "Flagged by community",
84+
description:
85+
"This account has been flagged as potentially automated by the community.",
86+
}
87+
88+
const hasCommunityFlag: boolean = !!verifiedAutomation;
89+
const details = hasCommunityFlag ? communityFlagDetails : getClassificationDetails(classification);
90+
4891
const flagsTable = flags
4992
.map((f) => `| ${f.label} | ${f.points > 0 ? "+" : ""}${f.points} | ${f.detail} |`)
5093
.join("\n");
51-
const comment = `### 🤖 Automated account detected
94+
const comment = `### ${details.label}
5295
53-
[@${PR_AUTHOR}](https://github.com/${PR_AUTHOR}) has been flagged as a likely automated account.
96+
[@${PR_AUTHOR}](https://github.com/${PR_AUTHOR}) ${details.description}
5497
5598
**Classification:** \`${classification}\` (score: ${score})
5699
@@ -60,12 +103,14 @@ ${flagsTable}
60103
61104
<sub>Analyzed ${events.length} public events via <a href="https://www.npmx.dev/package/voight-kampff-test">voight-kampff-test</a></sub>`;
62105

63-
const log = jsonMode ? console.error : console.log;
106+
64107
log(`Classification: ${classification} (score: ${score})`);
65108
for (const flag of flags) {
66109
log(` [${flag.points > 0 ? "+" : ""}${flag.points}] ${flag.label}: ${flag.detail}`);
67110
}
68111

112+
const isAgent = classification === "automation";
113+
69114
if (jsonMode) {
70115
console.log(JSON.stringify({ classification, score, isAgent, comment }));
71116
} else {

0 commit comments

Comments
 (0)