You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Many source code analysis tools use comments in a special format to mark it up. This is an important part of the Python ecosystem, but there is still no single standard around it. This library offers such a standard.
21
+
Many source code analysis tools use specially formatted comments to annotate code. This is an important part of the Python ecosystem, but there is still no single standard for it. This library proposes one.
22
22
23
23
24
24
## Table of contents
@@ -32,33 +32,33 @@ Many source code analysis tools use comments in a special format to mark it up.
32
32
33
33
## Why?
34
34
35
-
In the Python ecosystem, there are many tools dealing with source code: linters, test coverage collection systems, and many others. Many of them use special comments, and as a rule, the style of these comments is very similar. Here are some examples:
35
+
The Python ecosystem includes many source code tools, such as linters, coverage tools, and formatters. Many of them use special comments, and their syntax is often very similar. Here are some examples:
But you know what? *There is no single standard for such comments*.
45
45
46
-
The internal implementation of reading such comments is also different. Someone uses regular expressions, someone uses even more primitive string processing tools, and someone uses full-fledged parsers, including the Python parser or even written from scratch.
46
+
Tools also parse these comments differently. Some tools use regular expressions, others rely on simple string processing, and still others use full-fledged parsers, including the Python parser or even written from scratch.
47
47
48
-
As a result, as a user, you need to remember the rules by which comments are written for each specific tool. And at the same time, you can't be sure that things like double comments (when you want to leave 2 comments for different tools in one line of code) will work in principle. And as the creator of such tools, you are faced with a seemingly simple task — just to read a comment — and find out for yourself that it suddenly turns out to be quite difficult, and there are many possible mistakes.
48
+
As a result, as a user, you need to remember the rules by which comments are written for each specific tool. And at the same time, you can't be sure that things like double comments (when you want to leave two comments for different tools on the same line of code) will work in principle. And as the creator of such tools, you are faced with a seemingly simple task — just to read a comment — and discover that it is surprisingly tricky, and there are many possible mistakes.
49
49
50
-
This is exactly the problem that this library solves. It describes a [simple and intuitive standard](https://xkcd.com/927/) for action comments, and also offers a ready-made parser that creators of other tools can use. The standard offered by this library is based entirely on a subset of the Python syntax and can be easily reimplemented even if you do not want to use this library directly.
50
+
This is exactly the problem that this library solves. It describes a [simple and intuitive standard](https://xkcd.com/927/) for action comments, and also offers a ready-made parser that creators of other tools can use. The standard offered by this library is based entirely on a subset of Python syntax and can be easily reimplemented even if you do not want to use this library directly.
51
51
52
52
53
53
## The language
54
54
55
-
So, this library offers a language for action comments. Its syntax is a subset of Python syntax, but without Python semantics, as full-fledged execution does not occur. The purpose of the language is simply to provide the developer with the content of the comment in a convenient way, if it is written in a compatible format. If the comment format is not compatible with the parser, it is ignored.
55
+
So, this library offers a language for action comments. Its syntax is a subset of Python syntax, but without Python semantics, because no code is executed. The goal of the language is to expose the contents of a comment in a convenient form when the comment is written in a compatible format. If the comment format is not compatible with the parser, it is ignored.
56
56
57
-
From the point of view of the language, any meaningful comment can consist of 3 elements:
57
+
From the point of view of the language, any meaningful comment can consist of three elements:
58
58
59
-
-**Key**. This is usually the name of the specific tool for which this comment is intended, but in some cases it may be something else. This can be any string allowed as an [identifier](https://docs.python.org/3/reference/lexical_analysis.html#identifiers) in Python.
60
-
-**Action**. The short name of the action that you want to link to this line. Also, only the allowed Python identifier.
61
-
-**List of arguments**. These are often some kind of identifiers of specific linting rules or other arguments associated with this action. The list of possible data types described below.
59
+
-**Key**. This is usually the name of the tool the comment is intended for, but in some cases it may be something else. This can be any valid [Python identifier](https://docs.python.org/3/reference/lexical_analysis.html#identifiers).
60
+
-**Action**. A short name for the action associated with this line. Again, this must be a valid Python identifier.
61
+
-**List of arguments**. These are often some kind of identifiers of specific linting rules or other arguments associated with this action. The possible data types are described below.
62
62
63
63
Consider a comment designed to ignore a specific mypy rule:
64
64
@@ -67,7 +67,8 @@ Consider a comment designed to ignore a specific mypy rule:
67
67
└-key-┘└action┴-arguments┘
68
68
```
69
69
70
-
> ↑ The key here is the word `type`, that is, what you see before the colon. The action is the `ignore` word, that is, what comes before the square brackets, but after the colon. Finally, the list of arguments is what is in square brackets, in this case, there is only one argument in it: `error-code`.
70
+
71
+
> ↑ The key here is the word `type`, that is, what you see before the colon. The action is the `ignore` word, that is, what comes before the square brackets, but after the colon. Finally, the list of arguments is what appears in square brackets. In this case, it contains only one argument: `error-code`.
71
72
72
73
Simplified writing is also possible, without a list of arguments:
73
74
@@ -76,50 +77,50 @@ Simplified writing is also possible, without a list of arguments:
76
77
└-key-┘└action┘
77
78
```
78
79
79
-
> ↑ In this case, the parser assumes that there is an argument list, but it is empty.
80
+
> ↑ In this case, the parser treats this as an empty argument list.
80
81
81
-
The number of arguments in the list is unlimited, they can be separated by commas. Here are the valid data types for arguments:
82
+
There can be any number of arguments; they can be separated by commas. Here are the valid data types for arguments:
82
83
83
84
-[Valid Python identifiers](https://docs.python.org/3/reference/lexical_analysis.html#identifiers). They are interpreted as strings.
84
-
- Two valid Python identifiers, separated by the `-` symbol, like this: `error-code`. There can also be any number of spaces between them, they will be ignored. Interpreted as a single string.
85
+
- Two valid Python identifiers, separated by the `-` symbol, like this: `error-code`. There can be any number of spaces between them; they will be ignored. Interpreted as a single string.
- Any other Python-compatible code. This is disabled by default, but you can force the mode of reading such code and get descriptions for any inserts of such code in the form of [`AST`objects](https://docs.python.org/3/library/ast.html#ast.AST), after which you can somehow process it yourself.
91
+
- Any other Python expressions. This is disabled by default, but you can enable parsing of such code and receive such arguments as [`AST`nodes](https://docs.python.org/3/library/ast.html#ast.AST), after which you can somehow process it yourself.
91
92
92
-
The syntax of all these data types is completely similar to the Python original (except that you can't use multi-line writing options). Over time, it is possible to extend the possible syntax of `metacode`, but this template will always be supported.
93
+
The syntax of these data types matches Python syntax (except that you can't use multi-line writing options). Over time, it is possible to extend the syntax of `metacode`, but this core syntax will always be supported.
93
94
94
-
There can be several comments in the `metacode`format. In this case, they should be interspersed with the `#` symbol, as if each subsequent comment is a comment on the previous one. You can also add regular text comments, they will just be ignored by the parser if they are not in `metacode` format:
95
+
A single line can contain multiple `metacode`comments. In this case, they should be separated by the `#` symbol, effectively chaining comments on the same line. You can also add regular text comments, they will just be ignored by the parser if they are not in `metacode` format:
95
96
96
97
```python
97
98
# type:ignore# <- This is a comment for mypy! # fmt: off # <- And this is a comment for Ruff!
98
99
```
99
100
100
-
If you scroll through this text [above](#why) to the examples of action comments from various tools, you may notice that the syntax of most of them (but not all) is it can be described using `metacode`, and if not, it can be easily adapted to `metacode`. Read on to learn how to use a ready-made parser in practice.
101
+
If you look back at the examples of action comments from various tools[above](#why), you may notice that the syntax of most of them (but not all) can be described using `metacode`, and the rest can usually be adapted with minor changes. Read on to learn how to use the provided parser in practice.
101
102
102
103
103
104
## Installation
104
105
105
-
Install it:
106
+
You can install [`metacode`](https://pypi.org/project/metacode/) with `pip`:
106
107
107
108
```bash
108
109
pip install metacode
109
110
```
110
111
111
-
You can also quickly try out this and other packages without having to install using [instld](https://github.com/pomponchik/instld).
112
+
You can also use [`instld`](https://github.com/pomponchik/instld) to quickly try out this package and others without installing them.
112
113
113
114
114
115
## Usage
115
116
116
-
The parser offered by this library is just one function that is imported like this:
117
+
The library exposes a single parser function:
117
118
118
119
```python
119
120
from metacode import parse
120
121
```
121
122
122
-
To use it, you need to extract the text of the comment in some third-party way (preferably, but not necessarily, without the `#` symbol at the beginning) and pass it, and the expected key must also be passed as the second argument. As a result, you will receive a list of the contents of all the comments that were parsed:
123
+
To use it, you need to extract the comment text however you like (ideally without the leading `#`, though this is not required) and pass it to the function, along with the expected key as the second argument. The function returns a list of all parsed comments:
> ↑ Please note that you are transmitting a key, which means that the result is returned filtered by this key. This way you can read only those comments that relate to your tool, ignoring the rest.
140
+
> ↑ Please note that you are passing a key, which means that the result is filtered by that key. This way you can read only those comments that relate to your tool, ignoring the rest.
140
141
141
-
By default, an argument in a comment must be of one of the strictly allowed types. However, you can enable reading of arbitrary other types, in which case they will be transmitted in the [`AST`node](https://docs.python.org/3/library/ast.html#ast.AST) format. To do this, pass `allow_ast=True`:
142
+
By default, an argument in a comment must be of one of the strictly allowed types. However, you can enable parsing of arbitrary expressions, in which case they will be returned as [`AST`nodes](https://docs.python.org/3/library/ast.html#ast.AST). To do this, pass `allow_ast=True`:
#> [ParsedComment(key='key', command='action', arguments=[<ast.BinOp object at 0x102e44eb0>])]
146
147
```
147
148
148
-
> ↑ If you do not pass `allow_ast=True`, a `metacode.errors.UnknownArgumentTypeError` exception will be raised. When processing an argument, you can also raise this exception for an AST node of a format that your tool does not expect.
149
+
> ↑ If you do not pass `allow_ast=True`, a `metacode.errors.UnknownArgumentTypeError` exception will be raised. When processing an argument, you can also raise this exception for an AST node in a form your tool does not support.
149
150
150
-
> ⚠️ Be careful when writing code that analyzes the AST. Different versions of the Python interpreter can generate different AST based on the same code, so don't forget to test your code (for example, using [matrix](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations) or [tox](https://tox.wiki/)) well. Otherwise, it is better to use standard `metacode` argument types.
151
+
> ⚠️ Be careful when writing code that analyzes the AST. Different versions of the Python interpreter can generate different ASTs for the same code, so don't forget to test your code (for example, using [matrix](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations) or [tox](https://tox.wiki/)) well. Otherwise, it is better to use standard `metacode` argument types.
151
152
152
-
You can allow your users to write keys in any case. To do this, pass `ignore_case=True`:
153
+
You can allow your users to write keys in any letter case. To do this, pass `ignore_case=True`:
Well, now we can read the comments. But what if we want to record? There is another function for this: `insert()`:
167
+
Well, now we can read the comments. But what if we want to write them? There is another function for this: `insert()`:
167
168
168
169
```python
169
170
from metacode import insert, ParsedComment
170
171
```
171
172
172
-
You send the comment you want to insert there, as well as the current comment (empty if there is no comment, or starting with `#` if there is), and you get a ready-made new comment text:
173
+
Pass the comment you want to insert, as well as the current comment (empty if there is no comment, or starting with `#` if there is), and get the resulting comment text:
If you are writing your Python-related tool not in Python, as is currently fashionable, but in some other language, such as Rust, you may want to adhere to the `metacode` standard for machine-readable comments, however, you cannot directly use the ready-made parser described [above](#usage). What to do?
194
+
If you are writing your Python-related tool in some other language, such as Rust, you may want to adhere to the `metacode` standard for machine-readable comments; however, you cannot directly use the ready-made parser described [above](#usage). What can you do in that case?
195
+
194
196
195
-
The proposed `metacode` language is a syntactic subset of Python. The original `metacode` parser allows you to read arbitrary arguments written in Python as AST nodes. The rules for such parsing are determined by the specific version of the interpreter that `metacode` runs under, and they cannot be strictly standardized, since [Python syntax](https://docs.python.org/3/reference/grammar.html) is gradually evolving in an unpredictable direction. However, you can use a "safe" subset of the valid syntax by implementing your parser based on this [`EBNF`](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) grammar:
197
+
The proposed `metacode` language is a syntactic subset of Python. The original `metacode` parser allows you to read arbitrary arguments written in Python as AST nodes. Such parsing depends on the Python version under which `metacode` runs, and it cannot be strictly standardized, since [Python syntax](https://docs.python.org/3/reference/grammar.html) is gradually evolving in an unpredictable direction. However, you can use a "safe" subset of the valid syntax by implementing your parser based on this [`EBNF`](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) grammar:
196
198
197
199
```
198
200
line ::= element { "#" element }
@@ -218,4 +220,4 @@ number_literal ::= ? python-style number ?
218
220
complex_literal ::= ? python-style complex number ?
219
221
```
220
222
221
-
If you suddenly implement your ready-made open-source parser of this grammar in a language other than Python, please [let me know](https://github.com/mutating/metacode/issues). This information can be added to this text.
223
+
If you implement an open-source parser of this grammar in a language other than Python, please [let me know](https://github.com/mutating/metacode/issues). This information can be added to this README.
0 commit comments