Improve Step 5 of overload call evaluation.

Makes two modifications to step 5 of overload call evaluation:
1. When materializing arguments to check whether we can
   eliminate overloads, skip arguments that have the same
   parameter type in all overloads.
2. When we still have multiple overloads after the materialization
   filter, try to find a return type that all materializations of
   all other return types are assignable to.

Updates the conformance tests as well. All type checkers already do
(1). Mypy and pyrefly pass the conformance test for (2). I believe
mypy has a heuristic that approximates the new rule; pyrefly
implements it exactly.

Author: rchen152

conformance/results/mypy/overloads_evaluation.toml

@@ -14,10 +14,10 @@ Line 161: Unexpected errors ['overloads_evaluation.py:161: error: No overload va
 Line 162: Unexpected errors ['overloads_evaluation.py:162: error: Expression is of type "Any", not "Literal[0, 1]"  [assert-type]']
 Line 205: Unexpected errors ['overloads_evaluation.py:205: error: Argument 1 to "expand_tuple" has incompatible type "tuple[int, int | str]"; expected "tuple[int, int]"  [arg-type]']
 Line 206: Unexpected errors ['overloads_evaluation.py:206: error: Expression is of type "int", not "int | str"  [assert-type]']
-Line 265: Unexpected errors ['overloads_evaluation.py:265: error: Expression is of type "list[Any]", not "Any"  [assert-type]']
-Line 281: Unexpected errors ['overloads_evaluation.py:281: error: Expression is of type "list[Any]", not "Any"  [assert-type]']
-Line 303: Unexpected errors ['overloads_evaluation.py:303: error: Expression is of type "Any", not "float"  [assert-type]']
-Line 347: Unexpected errors ['overloads_evaluation.py:347: error: Expression is of type "list[Any]", not "Any"  [assert-type]']
+Line 268: Unexpected errors ['overloads_evaluation.py:268: error: Expression is of type "list[Any]", not "Any"  [assert-type]']
+Line 284: Unexpected errors ['overloads_evaluation.py:284: error: Expression is of type "list[Any]", not "Any"  [assert-type]']
+Line 306: Unexpected errors ['overloads_evaluation.py:306: error: Expression is of type "Any", not "float"  [assert-type]']
+Line 350: Unexpected errors ['overloads_evaluation.py:350: error: Expression is of type "list[Any]", not "Any"  [assert-type]']
 """
 output = """
 overloads_evaluation.py:38: error: All overload variants of "example1_1" require at least one argument  [call-overload]
@@ -46,8 +46,8 @@ overloads_evaluation.py:161: note:     def expand_enum(x: Literal[Color.BLUE]) -
 overloads_evaluation.py:162: error: Expression is of type "Any", not "Literal[0, 1]"  [assert-type]
 overloads_evaluation.py:205: error: Argument 1 to "expand_tuple" has incompatible type "tuple[int, int | str]"; expected "tuple[int, int]"  [arg-type]
 overloads_evaluation.py:206: error: Expression is of type "int", not "int | str"  [assert-type]
-overloads_evaluation.py:265: error: Expression is of type "list[Any]", not "Any"  [assert-type]
-overloads_evaluation.py:281: error: Expression is of type "list[Any]", not "Any"  [assert-type]
-overloads_evaluation.py:303: error: Expression is of type "Any", not "float"  [assert-type]
-overloads_evaluation.py:347: error: Expression is of type "list[Any]", not "Any"  [assert-type]
+overloads_evaluation.py:268: error: Expression is of type "list[Any]", not "Any"  [assert-type]
+overloads_evaluation.py:284: error: Expression is of type "list[Any]", not "Any"  [assert-type]
+overloads_evaluation.py:306: error: Expression is of type "Any", not "float"  [assert-type]
+overloads_evaluation.py:350: error: Expression is of type "list[Any]", not "Any"  [assert-type]
 """

conformance/results/pyrefly/callables_annotation.toml

@@ -14,10 +14,10 @@ ERROR callables_annotation.py:29:8-9: Unexpected keyword argument `a` [unexpecte
 ERROR callables_annotation.py:29:8-11: Expected 2 more positional arguments [bad-argument-count]
 ERROR callables_annotation.py:29:13-14: Unexpected keyword argument `b` [unexpected-keyword]
 ERROR callables_annotation.py:35:8-9: Expected 0 positional arguments, got 1 [bad-argument-count]
-ERROR callables_annotation.py:55:5-18: `Callable` requires exactly two arguments but 1 was found [bad-specialization]
+ERROR callables_annotation.py:55:5-18: Expected 2 arguments for `Callable`, got 1 [bad-specialization]
 ERROR callables_annotation.py:56:14-17: Callable types can only have `ParamSpec` in this position, got `int` [bad-specialization]
 ERROR callables_annotation.py:57:18-23: Expected a type form, got instance of `list[type[int]]` [not-a-type]
-ERROR callables_annotation.py:58:5-28: `Callable` requires exactly two arguments but 3 was found [bad-specialization]
+ERROR callables_annotation.py:58:5-28: Expected 2 arguments for `Callable`, got 3 [bad-specialization]
 ERROR callables_annotation.py:59:15-18: Invalid position for `...` [invalid-argument]
 ERROR callables_annotation.py:91:7-15: `() -> str` is not assignable to variable `cb3` with type `(int, ...) -> str` [bad-assignment]
 ERROR callables_annotation.py:93:7-15: `(*, a: int) -> str` is not assignable to variable `cb3` with type `(int, ...) -> str` [bad-assignment]

conformance/results/pyrefly/directives_no_type_check.toml

@@ -4,8 +4,6 @@ errors_diff = """
 """
 output = """
 ERROR directives_no_type_check.py:15:14-16: `Literal['']` is not assignable to `int` [bad-assignment]
-ERROR directives_no_type_check.py:29:7-17: Argument `Literal[b'invalid']` is not assignable to parameter `a` with type `int` in function `func1` [bad-argument-type]
-ERROR directives_no_type_check.py:29:19-31: Argument `Literal[b'arguments']` is not assignable to parameter `b` with type `str` in function `func1` [bad-argument-type]
 ERROR directives_no_type_check.py:32:6-8: Missing argument `a` in function `func1` [missing-argument]
 ERROR directives_no_type_check.py:32:6-8: Missing argument `b` in function `func1` [missing-argument]
 """

conformance/results/pyrefly/enums_member_values.toml

@@ -3,12 +3,9 @@ conformance_automated = "Pass"
 errors_diff = """
 """
 output = """
-ERROR enums_member_values.py:21:12-43: assert_type(int, Literal[1]) failed [assert-type]
-ERROR enums_member_values.py:22:12-41: assert_type(int, Literal[1]) failed [assert-type]
-ERROR enums_member_values.py:26:16-50: assert_type(int, Literal[1, 3]) failed [assert-type]
 ERROR enums_member_values.py:30:16-51: assert_type(int, Literal[1, 2, 3]) failed [assert-type]
 ERROR enums_member_values.py:54:12-46: assert_type(Any, Literal[1]) failed [assert-type]
 ERROR enums_member_values.py:68:12-42: assert_type(int, Literal[1]) failed [assert-type]
-ERROR enums_member_values.py:78:5-10: Enum member `GREEN` has type `str`, must match the `_value_` attribute annotation of `int` [bad-assignment]
+ERROR enums_member_values.py:78:5-10: Enum member `GREEN` has type `Literal['green']`, must match the `_value_` attribute annotation of `int` [bad-assignment]
 ERROR enums_member_values.py:85:24-29: `int` is not assignable to attribute `_value_` with type `str` [bad-assignment]
 """

conformance/results/pyrefly/generics_paramspec_components.toml

@@ -1,10 +1,5 @@
-conformant = "Partial"
-notes = """
-Does not reject usage of args/kwargs for out-of-scope ParamSpec
-"""
-conformance_automated = "Fail"
+conformance_automated = "Pass"
 errors_diff = """
-Line 30: Expected 1 errors
 """
 output = """
 ERROR generics_paramspec_components.py:17:25-33: `ParamSpec` **kwargs is only allowed in a **kwargs annotation [invalid-annotation]
@@ -13,6 +8,8 @@ ERROR generics_paramspec_components.py:20:23-29: `ParamSpec` *args is only allow
 ERROR generics_paramspec_components.py:23:5-24:13: `ParamSpec` *args and **kwargs must be used together [invalid-param-spec]
 ERROR generics_paramspec_components.py:23:46-52: `ParamSpec` *args is only allowed in an *args annotation [invalid-annotation]
 ERROR generics_paramspec_components.py:26:5-27:13: `ParamSpec` *args and **kwargs must be used together [invalid-param-spec]
+ERROR generics_paramspec_components.py:30:25-31: Expected a type form, got instance of `ParamSpecArgs` [not-a-type]
+ERROR generics_paramspec_components.py:30:43-51: Expected a type form, got instance of `ParamSpecKwargs` [not-a-type]
 ERROR generics_paramspec_components.py:35:18-24: `ParamSpec` *args is only allowed in an *args annotation [invalid-annotation]
 ERROR generics_paramspec_components.py:36:20-28: `ParamSpec` **kwargs is only allowed in a **kwargs annotation [invalid-annotation]
 ERROR generics_paramspec_components.py:38:5-39:13: `ParamSpec` *args and **kwargs must be used together [invalid-param-spec]
@@ -22,8 +19,7 @@ ERROR generics_paramspec_components.py:51:11-12: Expected 0 positional arguments
 ERROR generics_paramspec_components.py:60:28-34: Keyword-only parameter `s` may not appear after ParamSpec args parameter [bad-function-definition]
 ERROR generics_paramspec_components.py:70:10-30: Expected *-unpacked P.args and **-unpacked P.kwargs [invalid-param-spec]
 ERROR generics_paramspec_components.py:72:10-27: Expected 1 more positional argument [bad-argument-count]
-ERROR generics_paramspec_components.py:83:13-14: Unexpected keyword argument `x` in function `foo` [unexpected-keyword]
-ERROR generics_paramspec_components.py:83:13-16: Expected 1 more positional argument in function `foo` [bad-argument-count]
+ERROR generics_paramspec_components.py:83:13-14: Expected argument `x` to be positional in function `foo` [unexpected-keyword]
 ERROR generics_paramspec_components.py:98:20-23: Argument `Literal['A']` is not assignable to parameter `a` with type `int` in function `twice` [bad-argument-type]
 ERROR generics_paramspec_components.py:98:25-26: Argument `Literal[1]` is not assignable to parameter `b` with type `str` in function `twice` [bad-argument-type]
 """

conformance/results/pyrefly/generics_self_usage.toml

@@ -4,13 +4,13 @@ Does not implement some restrictions on where Self can be used
 """
 conformance_automated = "Fail"
 errors_diff = """
-Line 82: Expected 1 errors
 Line 87: Expected 1 errors
 """
 output = """
 ERROR generics_self_usage.py:73:14-18: `Self` must appear within a class [invalid-annotation]
 ERROR generics_self_usage.py:73:23-27: `Self` must appear within a class [invalid-annotation]
 ERROR generics_self_usage.py:76:6-10: `Self` must appear within a class [invalid-annotation]
+ERROR generics_self_usage.py:82:9-37: `Self` cannot be used when `self` has an explicit TypeVar annotation [invalid-annotation]
 ERROR generics_self_usage.py:103:15-19: `Self` must appear within a class [invalid-annotation]
 ERROR generics_self_usage.py:105:12-16: `Self` must appear within a class [invalid-annotation]
 ERROR generics_self_usage.py:105:12-16: Invalid base class: `Self` [invalid-inheritance]

conformance/results/pyrefly/protocols_modules.toml

@@ -1,13 +1,7 @@
-conformant = "Partial"
-notes = """
-Fails one subtyping example of protocol modules
-"""
-conformance_automated = "Fail"
+conformance_automated = "Pass"
 errors_diff = """
-Line 25: Unexpected errors ['`Module[_protocols_modules1]` is not assignable to `Options1` [bad-assignment]']
 """
 output = """
-ERROR protocols_modules.py:25:17-36: `Module[_protocols_modules1]` is not assignable to `Options1` [bad-assignment]
 ERROR protocols_modules.py:26:17-36: `Module[_protocols_modules1]` is not assignable to `Options2` [bad-assignment]
 ERROR protocols_modules.py:48:18-37: `Module[_protocols_modules2]` is not assignable to `Reporter2` [bad-assignment]
 ERROR protocols_modules.py:49:18-37: `Module[_protocols_modules2]` is not assignable to `Reporter3` [bad-assignment]

conformance/results/pyrefly/qualifiers_final_annotation.toml

@@ -1,13 +1,5 @@
-conformant = "Partial"
-notes = """
-Does not reject Final inside ClassVar.
-Issues with NamedTuple interaction.
-"""
-conformance_automated = "Fail"
+conformance_automated = "Pass"
 errors_diff = """
-Line 107: Expected 1 errors
-Line 131: Unexpected errors ['Expected first item to be a string literal [invalid-argument]', 'Expected first item to be a string literal [invalid-argument]']
-Line 133: Unexpected errors ['Unexpected keyword argument `x` in function `N.__new__` [unexpected-keyword]', 'Unexpected keyword argument `y` in function `N.__new__` [unexpected-keyword]']
 """
 output = """
 ERROR qualifiers_final_annotation.py:16:7-12: Expected a type argument for `Final` [invalid-annotation]
@@ -22,17 +14,16 @@ ERROR qualifiers_final_annotation.py:67:9-17: Cannot set field `ID7` [read-only]
 ERROR qualifiers_final_annotation.py:71:8-11: Cannot assign to variable `RATE` because it is marked final [bad-assignment]
 ERROR qualifiers_final_annotation.py:81:1-18: Cannot set field `DEFAULT_ID` [read-only]
 ERROR qualifiers_final_annotation.py:94:5-17: `BORDER_WIDTH` is declared as final in parent class `ClassC` [bad-override]
+ERROR qualifiers_final_annotation.py:107:5-11: `Final` may not be nested inside `ClassVar` [invalid-annotation]
 ERROR qualifiers_final_annotation.py:108:19-27: `ClassVar` may not be nested inside `Final` [invalid-annotation]
 ERROR qualifiers_final_annotation.py:118:9-19: `Final` is not allowed in this context [invalid-annotation]
 ERROR qualifiers_final_annotation.py:121:14-19: `Final` is only allowed on a class or local variable annotation [invalid-annotation]
 ERROR qualifiers_final_annotation.py:121:14-30: `Final` is not allowed in this context [invalid-annotation]
-ERROR qualifiers_final_annotation.py:131:23-24: Expected first item to be a string literal [invalid-argument]
-ERROR qualifiers_final_annotation.py:131:33-34: Expected first item to be a string literal [invalid-argument]
-ERROR qualifiers_final_annotation.py:133:3-4: Unexpected keyword argument `x` in function `N.__new__` [unexpected-keyword]
-ERROR qualifiers_final_annotation.py:133:8-9: Unexpected keyword argument `y` in function `N.__new__` [unexpected-keyword]
+ERROR qualifiers_final_annotation.py:134:2-7: Missing argument `x` in function `N.__new__` [missing-argument]
+ERROR qualifiers_final_annotation.py:134:2-7: Missing argument `y` in function `N.__new__` [missing-argument]
 ERROR qualifiers_final_annotation.py:134:3-4: Unexpected keyword argument `a` in function `N.__new__` [unexpected-keyword]
-ERROR qualifiers_final_annotation.py:135:3-4: Unexpected keyword argument `x` in function `N.__new__` [unexpected-keyword]
-ERROR qualifiers_final_annotation.py:135:9-10: Unexpected keyword argument `y` in function `N.__new__` [unexpected-keyword]
+ERROR qualifiers_final_annotation.py:135:5-7: Argument `Literal['']` is not assignable to parameter `x` with type `int` in function `N.__new__` [bad-argument-type]
+ERROR qualifiers_final_annotation.py:135:11-13: Argument `Literal['']` is not assignable to parameter `y` with type `int` in function `N.__new__` [bad-argument-type]
 ERROR qualifiers_final_annotation.py:141:11-12: Cannot assign to variable `ID1` because it is marked final [bad-assignment]
 ERROR qualifiers_final_annotation.py:145:5-11: Cannot assign to variable `x` because it is marked final [bad-assignment]
 ERROR qualifiers_final_annotation.py:147:15-16: Cannot assign to variable `x` because it is marked final [bad-assignment]

conformance/results/pyrefly/qualifiers_final_decorator.toml

@@ -12,5 +12,5 @@ ERROR qualifiers_final_decorator.py:81:9-15: `method` is declared as final in pa
 ERROR qualifiers_final_decorator.py:86:9-15: `@final` should only be applied to the implementation of an overloaded function. [invalid-overload]
 ERROR qualifiers_final_decorator.py:95:9-15: `method` is declared as final in parent class `Base4` [bad-override]
 ERROR qualifiers_final_decorator.py:118:9-15: `method` is declared as final in parent class `Base5_2` [bad-override]
-ERROR qualifiers_final_decorator.py:125:1-7: Decorator `@final` can only be used on methods. [invalid-decorator]
+ WARN qualifiers_final_decorator.py:125:1-7: Decorator `@final` can only be used on methods. [invalid-decorator]
 """

conformance/results/pyrefly/typeddicts_alt_syntax.toml

@@ -5,7 +5,7 @@ errors_diff = """
 output = """
 ERROR typeddicts_alt_syntax.py:23:1-14: Expected valid functional typed dictionary definition [invalid-argument]
 ERROR typeddicts_alt_syntax.py:27:45-46: Expected first item to be a string literal [invalid-argument]
-ERROR typeddicts_alt_syntax.py:31:27-38: Expected string literal "BadTypedDict3" [invalid-argument]
+ WARN typeddicts_alt_syntax.py:31:27-38: Expected string literal "BadTypedDict3" [name-mismatch]
 ERROR typeddicts_alt_syntax.py:35:72-83: Unrecognized keyword argument `other` in typed dictionary definition [invalid-argument]
 ERROR typeddicts_alt_syntax.py:41:1-7: Expected valid functional typed dictionary definition [invalid-argument]
 ERROR typeddicts_alt_syntax.py:41:30-38: Unrecognized keyword argument `name` in typed dictionary definition [invalid-argument]