Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: refactor sem analysis of the await-not-async on generators and list comprehension #18152

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

fix: refactor sem analysis of the await-not-async on generators and list comprehension #18152

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
d3b2d9d
fix: refactor sem analysis of the await-not-async on generators and l…
josetapadas Nov 13, 2024
ed2937a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 13, 2024
c97bce7
chore: added unit test
josetapadas Nov 13, 2024
ada4720
chore: refactor to remove function for simple check
josetapadas Nov 13, 2024
43fc65a
fix: fixed the await outside coroutine within list comprehensions error
josetapadas Nov 13, 2024
e3004a1
chore: error type is not syntax but await-not-async
josetapadas Nov 13, 2024
e7ef1f6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 13, 2024
0c82fb9
fix: added await-outside-coroutine checks also to list and dict compr…
josetapadas Nov 14, 2024
8e8298e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 14, 2024
0fee8d6
chore: remove artefact from update data
josetapadas Nov 18, 2024
95c5730
chore: fail consistently for the new case also on the set comprehension
josetapadas Nov 18, 2024
48148ac
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions mypy/message_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ def with_additional_msg(self, info: str) -> ErrorMessage:

ASYNC_FOR_OUTSIDE_COROUTINE: Final = '"async for" outside async function'
ASYNC_WITH_OUTSIDE_COROUTINE: Final = '"async with" outside async function'
AWAIT_OUTSIDE_FUNCTION: Final = '"await" outside function'
AWAIT_OUTSIDE_COROUTINE: Final = '"await" outside coroutine ("async def")'

INCOMPATIBLE_TYPES_IN_YIELD: Final = ErrorMessage('Incompatible types in "yield"')
INCOMPATIBLE_TYPES_IN_YIELD_FROM: Final = ErrorMessage('Incompatible types in "yield from"')
Expand Down
56 changes: 50 additions & 6 deletions mypy/semanal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,7 @@
set_callable_name as set_callable_name,
)
from mypy.semanal_typeddict import TypedDictAnalyzer
from mypy.traverser import has_await_expression
from mypy.tvar_scope import TypeVarLikeScope
from mypy.typeanal import (
SELF_TYPE_NAMES,
Expand Down Expand Up @@ -6073,24 +6074,59 @@ def visit_type_application(self, expr: TypeApplication) -> None:
if analyzed is not None:
expr.types[i] = analyzed

def _check_await_outside_coroutine(
self, expr: ListComprehension | SetComprehension | DictionaryComprehension
) -> None:
if not has_await_expression(expr):
return

if not self.is_func_scope() or not self.function_stack[-1].is_coroutine:
self.fail(
message_registry.AWAIT_OUTSIDE_COROUTINE,
expr,
code=codes.AWAIT_NOT_ASYNC,
serious=True,
)

def visit_list_comprehension(self, expr: ListComprehension) -> None:
if any(expr.generator.is_async):
if not self.is_func_scope() or not self.function_stack[-1].is_coroutine:
self.fail(message_registry.ASYNC_FOR_OUTSIDE_COROUTINE, expr, code=codes.SYNTAX)
self.fail(
message_registry.ASYNC_FOR_OUTSIDE_COROUTINE,
expr,
code=codes.SYNTAX,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really like the use of codes.SYNTAX vs codes.AWAIT_NOT_ASYNC here. Prior to this PR AWAIT_NOT_ASYNC was only used once.

As far as I understand, the purpose of using a special code was to support "specific" environments: e.g. IPython allows top-level awaits, so it should be easy to disable such a check when checking IPython snippets.

I see two possible resolutions here:

  1. Declare that we only type-check true python code. Then codes.AWAIT_NOT_ASYNC can be safely removed altogether and replaced with codes.SYNTAX
  2. Declare that we want to support such environments. Then all these checks become more complicated: when at module scope (note it isn't same as not self.is_func_scope()), produce codes.AWAIT_NOT_ASYNC, otherwise produce codes.SYNTAX. This can also be extracted into a helper like def require_async_scope(self, message). This logic is close to what visit_await_expr was doing before.

I somewhat prefer 2 as that isn't that much work to do, but can live with 1 either.

Disclaimer: please note I'm not a mypy maintainer and just try to help sometimes. You'd better ask for a maintainer's opinion here: @brianschubert do you have any preference?

Copy link
Contributor

@sterliakov sterliakov Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OTOH 2 can also be misleading and deserves an explicit mention in the docs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review and explanation, I understand your point so thank you for clarifying it. Will then wait for any guidance on how to proceed on any of these two options.

Copy link

@jakkdl jakkdl Nov 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is an important enough problem to change our stance on allowing top-level awaits, so you're probably safer to go ahead with option 2 for now. A bare await outside async has always been a syntax error in python, so whatever reason the original PR had for not making that use codes.SYNTAX should apply here as well.

serious=True,
)

self._check_await_outside_coroutine(expr)

expr.generator.accept(self)

josetapadas marked this conversation as resolved.
Show resolved Hide resolved
def visit_set_comprehension(self, expr: SetComprehension) -> None:
if any(expr.generator.is_async):
if not self.is_func_scope() or not self.function_stack[-1].is_coroutine:
self.fail(message_registry.ASYNC_FOR_OUTSIDE_COROUTINE, expr, code=codes.SYNTAX)
self.fail(
message_registry.ASYNC_FOR_OUTSIDE_COROUTINE,
expr,
code=codes.SYNTAX,
serious=True,
)

self._check_await_outside_coroutine(expr)

expr.generator.accept(self)

def visit_dictionary_comprehension(self, expr: DictionaryComprehension) -> None:
josetapadas marked this conversation as resolved.
Show resolved Hide resolved
if any(expr.is_async):
if not self.is_func_scope() or not self.function_stack[-1].is_coroutine:
self.fail(message_registry.ASYNC_FOR_OUTSIDE_COROUTINE, expr, code=codes.SYNTAX)
self.fail(
message_registry.ASYNC_FOR_OUTSIDE_COROUTINE,
expr,
code=codes.SYNTAX,
serious=True,
)

self._check_await_outside_coroutine(expr)

with self.enter(expr):
self.analyze_comp_for(expr)
Expand Down Expand Up @@ -6166,10 +6202,18 @@ def visit_await_expr(self, expr: AwaitExpr) -> None:
# We check both because is_function_scope() returns True inside comprehensions.
# This is not a blocker, because some enviroments (like ipython)
# support top level awaits.
self.fail('"await" outside function', expr, serious=True, code=codes.TOP_LEVEL_AWAIT)
elif not self.function_stack[-1].is_coroutine:
self.fail(
'"await" outside coroutine ("async def")',
message_registry.AWAIT_OUTSIDE_FUNCTION,
expr,
serious=True,
code=codes.TOP_LEVEL_AWAIT,
)
elif (
not self.function_stack[-1].is_coroutine
and self.scope_stack[-1] != SCOPE_COMPREHENSION
):
self.fail(
message_registry.AWAIT_OUTSIDE_COROUTINE,
expr,
serious=True,
code=codes.AWAIT_NOT_ASYNC,
Expand Down
25 changes: 24 additions & 1 deletion test-data/unit/check-async-await.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1013,13 +1013,15 @@ async def foo(x: int) -> int: ...

# These are allowed in some cases:
top_level = await foo(1) # E: "await" outside function [top-level-await]
crasher = [await foo(x) for x in [1, 2, 3]] # E: "await" outside function [top-level-await]
crasher = [await foo(x) for x in [1, 2, 3]] # E: "await" outside coroutine ("async def") [await-not-async] \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More explicit tests would be really helpful here. Consider at least checking all of the following in different scopes (module level, non-async function body and async function body at least; the former two should produce the same errors and async function should allow all of those):

(await x for x in xs)  # OK
[await x for x in xs]  # E
{await x for x in xs}  # E
{0: await x for x in xs}  # E
{await x: 0 for x in xs}  # E

(x async for x in xs)  # OK
[x async for x in xs]  # E
{x async for x in xs}  # E
{x: 0 async for x in xs}  # E

(x for x in await xs)  # E
[x for x in await xs]  # E
{x for x in await xs}  # E
{x: 0 for x in await xs}  # E

There's already testAsyncForOutsideCoroutine for the 2nd block, the 3rd one is probably not interesting (follows regular rules), but the 1st one isn't exercised yet.

# E: "await" outside function [top-level-await]

def bad() -> None:
# These are always critical / syntax issues:
y = [await foo(x) for x in [1, 2, 3]] # E: "await" outside coroutine ("async def") [await-not-async]
async def good() -> None:
y = [await foo(x) for x in [1, 2, 3]] # OK

josetapadas marked this conversation as resolved.
Show resolved Hide resolved
[builtins fixtures/async_await.pyi]
[typing fixtures/typing-async.pyi]

Expand Down Expand Up @@ -1080,3 +1082,24 @@ class Launcher(P):

[builtins fixtures/async_await.pyi]
[typing fixtures/typing-async.pyi]

[case testAwaitOutsideCoroutine]
def foo():
yield 0

def bar():
(await x for x in foo())

(await x for x in xs) # OK
[await x for x in xs] # E: "await" outside coroutine ("async def")
{await x for x in xs} # E: "await" outside coroutine ("async def")
{0: await x for x in xs} # E: "await" outside coroutine ("async def")
{await x: 0 for x in xs} # E: "await" outside coroutine ("async def")

(x for x in await xs) # E: "await" outside coroutine ("async def")
[x for x in await xs] # E: "await" outside coroutine ("async def")
{x for x in await xs} # E: "await" outside coroutine ("async def")
{x: 0 for x in await xs} # E: "await" outside coroutine ("async def")

[builtins fixtures/async_await.pyi]
[typing fixtures/typing-async.pyi]
Loading