Add Basilisk to conformance results

conformance/README.md

@@ -81,7 +81,7 @@ Note that some type checkers may not run on some platforms. If a type checker fa
 
 Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.
 
-[Conformance results](https://htmlpreview.github.io/?https://github.com/python/typing/blob/main/conformance/results/results.html) are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, zuban, ty, and pycroscope. It is the goal and desire to add additional type checkers over time.
+[Conformance results](https://htmlpreview.github.io/?https://github.com/python/typing/blob/main/conformance/results/results.html) are reported and summarized for each supported type checker. Currently, results are reported for basilisk, mypy, pyrefly, pyright, zuban, ty, and pycroscope. It is the goal and desire to add additional type checkers over time.
 
 ## Adding a New Test Case
 

conformance/pyproject.toml

@@ -3,6 +3,7 @@ name = "typing-conformance"
 version = "0.1.0"
 requires-python = "==3.12.*"
 dependencies = [
+    "basilisk-python",
     "jinja2",
     "markdown",
     "mypy",

conformance/results/basilisk/aliases_explicit.toml

@@ -0,0 +1,27 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+aliases_explicit.py:67:9: error: Type alias `GoodTypeAlias2` is not generic and cannot be parameterized [aliases_implicit]
+aliases_explicit.py:68:9: error: Type alias `GoodTypeAlias3` is not generic and cannot be parameterized [aliases_implicit]
+aliases_explicit.py:69:9: error: Too many type arguments for `GoodTypeAlias4`: expected 1, got 2 [aliases_implicit]
+aliases_explicit.py:70:9: error: Too many type arguments for `GoodTypeAlias8`: expected 1, got 2 [aliases_implicit]
+aliases_explicit.py:71:9: error: Invalid type argument for `ParamSpec` parameter in `GoodTypeAlias9` [aliases_implicit]
+aliases_explicit.py:79:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias1` [aliases_implicit]
+aliases_explicit.py:80:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias2` [aliases_implicit]
+aliases_explicit.py:81:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias3` [aliases_implicit]
+aliases_explicit.py:82:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias4` [aliases_implicit]
+aliases_explicit.py:83:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias5` [aliases_implicit]
+aliases_explicit.py:84:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias6` [aliases_implicit]
+aliases_explicit.py:85:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias7` [aliases_implicit]
+aliases_explicit.py:86:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias8` [aliases_implicit]
+aliases_explicit.py:87:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias9` [aliases_implicit]
+aliases_explicit.py:88:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias10` [aliases_implicit]
+aliases_explicit.py:89:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias11` [aliases_implicit]
+aliases_explicit.py:90:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias12` [aliases_implicit]
+aliases_explicit.py:91:1: error: Invalid type expression as right-hand side of `TypeAlias` for `BadTypeAlias13` [aliases_implicit]
+aliases_explicit.py:100:5: error: Type alias `ListAlias` is not generic and cannot be parameterized [aliases_implicit]
+aliases_explicit.py:101:6: error: Cannot instantiate union type alias `ListOrSetAlias` [aliases_implicit]
+aliases_explicit.py:102:5: error: Type alias `ListOrSetAlias` is not generic and cannot be parameterized [aliases_implicit]
+"""

conformance/results/basilisk/aliases_implicit.toml

@@ -0,0 +1,35 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+aliases_implicit.py:76:9: error: Type alias `GoodTypeAlias2` is not generic and cannot be parameterized [aliases_implicit]
+aliases_implicit.py:77:9: error: Type alias `GoodTypeAlias3` is not generic and cannot be parameterized [aliases_implicit]
+aliases_implicit.py:77:9: error: `GoodTypeAlias3` cannot be subscripted; it has no free type parameters [generics_defaults_specialization]
+aliases_implicit.py:78:9: error: Too many type arguments for `GoodTypeAlias4`: expected 1, got 2 [aliases_implicit]
+aliases_implicit.py:78:9: error: Too many type arguments for `GoodTypeAlias4`; expected at most 1, got 2 [generics_defaults_specialization]
+aliases_implicit.py:79:9: error: Too many type arguments for `GoodTypeAlias8`: expected 1, got 2 [aliases_implicit]
+aliases_implicit.py:79:9: error: Too many type arguments for `GoodTypeAlias8`; expected at most 1, got 2 [generics_defaults_specialization]
+aliases_implicit.py:80:9: error: Invalid type argument for `ParamSpec` parameter in `GoodTypeAlias9` [aliases_implicit]
+aliases_implicit.py:81:9: error: Type argument `str` does not satisfy bound `float` of TypeVar `TFloat` in `GoodTypeAlias12` [aliases_implicit]
+aliases_implicit.py:106:9: error: Variable `BadTypeAlias1` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:107:9: error: Variable `BadTypeAlias2` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:108:9: error: Variable `BadTypeAlias3` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:109:9: error: Variable `BadTypeAlias4` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:110:9: error: Variable `BadTypeAlias5` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:111:9: error: Variable `BadTypeAlias6` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:112:9: error: Variable `BadTypeAlias7` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:113:9: error: Variable `BadTypeAlias8` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:114:9: error: Variable `BadTypeAlias9` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:115:5: error: Invalid type expression in annotation for parameter `p10` [annotations_forward_refs]
+aliases_implicit.py:115:10: error: Variable `BadTypeAlias10` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:116:5: error: Invalid type expression in annotation for parameter `p11` [annotations_forward_refs]
+aliases_implicit.py:116:10: error: Variable `BadTypeAlias11` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:117:10: error: Variable `BadTypeAlias12` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:118:5: error: Invalid type expression in annotation for parameter `p13` [annotations_forward_refs]
+aliases_implicit.py:118:10: error: Variable `BadTypeAlias13` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:119:5: error: Invalid type expression in annotation for parameter `p14` [annotations_forward_refs]
+aliases_implicit.py:119:10: error: Variable `BadTypeAlias14` is not a valid type and cannot be used as an annotation [aliases_implicit]
+aliases_implicit.py:133:6: error: Cannot instantiate union type alias `ListOrSetAlias` [aliases_implicit]
+aliases_implicit.py:135:5: error: Type alias `ListOrSetAlias` is not generic and cannot be parameterized [aliases_implicit]
+"""

conformance/results/basilisk/aliases_newtype.toml

@@ -0,0 +1,21 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+aliases_newtype.py:11:1: error: Argument to `UserId` ("user") is not compatible with its base type `int` [aliases_newtype]
+aliases_newtype.py:12:1: error: Cannot assign a literal value directly to `UserId`; use `UserId(value)` to create a `UserId` instance [aliases_newtype]
+aliases_newtype.py:12:1: error: Type mismatch: `u1` is annotated `UserId` (userid) but assigned int [assignment_compatibility]
+aliases_newtype.py:18:1: error: `UserId` is a `NewType`, not an instance of `type`; `NewType()` does not return a class object [aliases_newtype]
+aliases_newtype.py:23:1: error: `UserId` is a `NewType` and cannot be used as the second argument to `isinstance`; `NewType` types are not runtime classes [aliases_newtype]
+aliases_newtype.py:26:1: error: Class `UserIdDerived` cannot subclass `UserId` which is a `NewType` [aliases_newtype]
+aliases_newtype.py:35:12: error: `NewType` name `BadName` does not match the variable name `GoodName` [aliases_newtype]
+aliases_newtype.py:41:1: error: Variable `nt1`: `NewType` cannot be used as a generic type [aliases_newtype]
+aliases_newtype.py:47:15: error: Invalid base type for `NewType` `BadNewType1`: cannot use a union type as a `NewType` base [aliases_newtype]
+aliases_newtype.py:50:15: error: Invalid base type for `NewType` `BadNewType2`: cannot use a TypeVar-parameterized generic as a `NewType` base [aliases_newtype]
+aliases_newtype.py:52:15: error: Invalid base type for `NewType` `BadNewType3`: cannot use a Protocol class as a `NewType` base [aliases_newtype]
+aliases_newtype.py:54:15: error: Invalid base type for `NewType` `BadNewType4`: cannot use `Literal` as a `NewType` base [aliases_newtype]
+aliases_newtype.py:61:15: error: Invalid base type for `NewType` `BadNewType5`: cannot use a `TypedDict` class as a `NewType` base [aliases_newtype]
+aliases_newtype.py:63:15: error: `NewType` takes exactly 2 arguments (3 given) for `BadNewType6` [aliases_newtype]
+aliases_newtype.py:65:15: error: Invalid base type for `NewType` `BadNewType7`: cannot use `Any` as a `NewType` base [aliases_newtype]
+"""

conformance/results/basilisk/aliases_recursive.toml

@@ -0,0 +1,18 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+aliases_recursive.py:19:1: error: Type mismatch: `j4` is annotated `Json` (json) but assigned dict[str, int | Unknown] [assignment_compatibility]
+aliases_recursive.py:20:1: error: Type mismatch: `j5` is annotated `Json` (json) but assigned list[int | Unknown] [assignment_compatibility]
+aliases_recursive.py:38:1: error: Type mismatch: `t6` is annotated `RecursiveTuple` (recursivetuple) but assigned tuple[int, tuple[str, int], tuple[int, tuple[int, list[int]]]] [assignment_compatibility]
+aliases_recursive.py:39:1: error: Type mismatch: `t7` is annotated `RecursiveTuple` (recursivetuple) but assigned tuple[int, list[int]] [assignment_compatibility]
+aliases_recursive.py:50:1: error: Type mismatch: `m7` is annotated `RecursiveMapping` (recursivemapping) but assigned dict[str, list[int]] [assignment_compatibility]
+aliases_recursive.py:51:1: error: Type mismatch: `m8` is annotated `RecursiveMapping` (recursivemapping) but assigned dict[str, str | int | list[int]] [assignment_compatibility]
+aliases_recursive.py:52:1: error: Type mismatch: `m9` is annotated `RecursiveMapping` (recursivemapping) but assigned dict[str, str | int | dict[str, str | int | list[int]]] [assignment_compatibility]
+aliases_recursive.py:63:1: error: Type mismatch: `g3` is annotated `GenericTypeAlias1[str]` (generictypealias1[str]) but assigned list[str | list[float]] [assignment_compatibility]
+aliases_recursive.py:69:1: error: Type mismatch: `g6` is annotated `GenericTypeAlias2[str, int]` (generictypealias2[str, int]) but assigned list[list[int | list[str | int | list[float]]] | str] [assignment_compatibility]
+aliases_recursive.py:72:1: error: Type alias `RecursiveUnion` creates a cyclical reference [aliases_recursive]
+aliases_recursive.py:75:1: error: Type alias `MutualReference1` creates a cyclical reference [aliases_recursive]
+aliases_recursive.py:75:63: error: Type alias `MutualReference2` creates a cyclical reference [aliases_recursive]
+"""

conformance/results/basilisk/aliases_type_statement.toml

@@ -0,0 +1,32 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+aliases_type_statement.py:17:1: error: Cannot access attribute `bit_count` on type alias `GoodAlias1` [generics_syntax_scoping]
+aliases_type_statement.py:19:1: error: Cannot call type alias `GoodAlias1`: type aliases are not callable [generics_syntax_scoping]
+aliases_type_statement.py:23:7: error: Cannot access attribute `other_attrib` on type alias `GoodAlias1` [generics_syntax_scoping]
+aliases_type_statement.py:26:7: error: Cannot use type alias `GoodAlias1` as a base class [generics_syntax_scoping]
+aliases_type_statement.py:31:22: error: Cannot use type alias `GoodAlias1` in `isinstance` [generics_syntax_scoping]
+aliases_type_statement.py:37:6: error: Invalid type expression in `type BadTypeAlias1` alias [aliases_type_statement]
+aliases_type_statement.py:38:6: error: Invalid type expression in `type BadTypeAlias2` alias [aliases_type_statement]
+aliases_type_statement.py:39:6: error: Invalid type expression in `type BadTypeAlias3` alias [aliases_type_statement]
+aliases_type_statement.py:40:6: error: Invalid type expression in `type BadTypeAlias4` alias [aliases_type_statement]
+aliases_type_statement.py:41:6: error: Invalid type expression in `type BadTypeAlias5` alias [aliases_type_statement]
+aliases_type_statement.py:42:6: error: Invalid type expression in `type BadTypeAlias6` alias [aliases_type_statement]
+aliases_type_statement.py:43:6: error: Invalid type expression in `type BadTypeAlias7` alias [aliases_type_statement]
+aliases_type_statement.py:44:6: error: Invalid type expression in `type BadTypeAlias8` alias [aliases_type_statement]
+aliases_type_statement.py:45:6: error: Invalid type expression in `type BadTypeAlias9` alias [aliases_type_statement]
+aliases_type_statement.py:46:6: error: Invalid type expression in `type BadTypeAlias10` alias [aliases_type_statement]
+aliases_type_statement.py:47:6: error: Invalid type expression in `type BadTypeAlias11` alias [aliases_type_statement]
+aliases_type_statement.py:48:6: error: Invalid type expression in `type BadTypeAlias12` alias [aliases_type_statement]
+aliases_type_statement.py:49:6: error: Invalid type expression in `type BadTypeAlias13` alias [aliases_type_statement]
+aliases_type_statement.py:53:6: error: PEP 695 `type` statement uses old-style TypeVar `V` [generics_syntax_scoping]
+aliases_type_statement.py:58:6: error: PEP 695 `type` statement uses old-style TypeVar `T1` [generics_syntax_scoping]
+aliases_type_statement.py:68:1: error: Type argument `str` is not compatible with type parameter `S` bound `int` in type alias `RecursiveTypeAlias2` [generics_syntax_scoping]
+aliases_type_statement.py:70:1: error: Type argument `int` is not compatible with type parameter `T` bound `str` in type alias `RecursiveTypeAlias2` [generics_syntax_scoping]
+aliases_type_statement.py:73:6: error: Circular type alias definition: `RecursiveTypeAlias3` references itself [generics_syntax_scoping]
+aliases_type_statement.py:75:6: error: Circular type alias definition: `RecursiveTypeAlias4` references itself with different type arguments [generics_syntax_scoping]
+aliases_type_statement.py:79:6: error: Circular type alias definition: `RecursiveTypeAlias6` is part of a circular alias chain [generics_syntax_scoping]
+aliases_type_statement.py:80:6: error: Circular type alias definition: `RecursiveTypeAlias7` is part of a circular alias chain [generics_syntax_scoping]
+"""

conformance/results/basilisk/aliases_typealiastype.toml

@@ -0,0 +1,29 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+aliases_typealiastype.py:32:7: error: `TypeAliasType` object `GoodAlias1` has no attribute `other_attrib` [aliases_typealiastype]
+aliases_typealiastype.py:40:5: error: Incorrect type arguments for `GoodAlias5`: expected 4, got 3 [aliases_typealiastype]
+aliases_typealiastype.py:43:13: error: Type variable `S` used in `BadAlias1` is not declared in `type_params` [aliases_typealiastype]
+aliases_typealiastype.py:44:13: error: Type variable `S` used in `BadAlias2` is not declared in `type_params` [aliases_typealiastype]
+aliases_typealiastype.py:45:57: error: `type_params` for `BadAlias3` must be a literal tuple expression [aliases_typealiastype]
+aliases_typealiastype.py:46:13: error: Type alias `BadAlias4` references itself, creating a circular dependency [aliases_typealiastype]
+aliases_typealiastype.py:47:13: error: Type alias `BadAlias5` references itself, creating a circular dependency [aliases_typealiastype]
+aliases_typealiastype.py:48:13: error: Type alias `BadAlias6` references itself, creating a circular dependency [aliases_typealiastype]
+aliases_typealiastype.py:49:13: error: Type alias `BadAlias7` references itself, creating a circular dependency [aliases_typealiastype]
+aliases_typealiastype.py:52:40: error: Invalid type expression in `TypeAliasType` for `BadAlias8` [aliases_typealiastype]
+aliases_typealiastype.py:53:40: error: Invalid type expression in `TypeAliasType` for `BadAlias9` [aliases_typealiastype]
+aliases_typealiastype.py:54:42: error: Invalid type expression in `TypeAliasType` for `BadAlias10` [aliases_typealiastype]
+aliases_typealiastype.py:55:42: error: Invalid type expression in `TypeAliasType` for `BadAlias11` [aliases_typealiastype]
+aliases_typealiastype.py:56:42: error: Invalid type expression in `TypeAliasType` for `BadAlias12` [aliases_typealiastype]
+aliases_typealiastype.py:57:42: error: Invalid type expression in `TypeAliasType` for `BadAlias13` [aliases_typealiastype]
+aliases_typealiastype.py:58:42: error: Invalid type expression in `TypeAliasType` for `BadAlias14` [aliases_typealiastype]
+aliases_typealiastype.py:59:42: error: Invalid type expression in `TypeAliasType` for `BadAlias15` [aliases_typealiastype]
+aliases_typealiastype.py:60:42: error: Invalid type expression in `TypeAliasType` for `BadAlias16` [aliases_typealiastype]
+aliases_typealiastype.py:61:42: error: Invalid type expression in `TypeAliasType` for `BadAlias17` [aliases_typealiastype]
+aliases_typealiastype.py:62:42: error: Invalid type expression in `TypeAliasType` for `BadAlias18` [aliases_typealiastype]
+aliases_typealiastype.py:63:42: error: Invalid type expression in `TypeAliasType` for `BadAlias19` [aliases_typealiastype]
+aliases_typealiastype.py:64:42: error: Invalid type expression in `TypeAliasType` for `BadAlias20` [aliases_typealiastype]
+aliases_typealiastype.py:66:14: error: Type alias `BadAlias21` references itself, creating a circular dependency [aliases_typealiastype]
+"""

conformance/results/basilisk/aliases_variance.toml

@@ -0,0 +1,10 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+aliases_variance.py:24:16: error: Variance incompatibility in base class `ClassA`: `T_co` is covariant but the corresponding type parameter is invariant [generics_variance]
+aliases_variance.py:28:16: error: Variance incompatibility in base class `A_Alias_1`: `T_co` is covariant but the corresponding type parameter is invariant [generics_variance]
+aliases_variance.py:32:16: error: Variance incompatibility in base class `A_Alias_2`: `T_co` is covariant but the corresponding type parameter is invariant [generics_variance]
+aliases_variance.py:44:16: error: Variance incompatibility in base class `B_Alias_1`: `T_contra` is contravariant but the corresponding type parameter is invariant [generics_variance]
+"""

conformance/results/basilisk/annotations_coroutines.toml

@@ -0,0 +1,6 @@
+conformant = "Pass"
+conformance_automated = "Pass"
+errors_diff = """
+"""
+output = """
+"""