Behavioral differences between the with-statement and `ExitStack.enter_context`

[gottadiveintopython]

Bug report

Bug description:

I noticed a behavioral difference between the (async) with statement and (Async)ExitStack. The with statement correctly recognizes context manager methods (__enter__, __exit__, __aenter__, and __aexit__) even when they are defined via __slots__, whereas (Async)ExitStack does not.

class SlotsCM:
    __slots__ = ("__enter__", "__exit__", )

    def __init__(self):
        self.__enter__ = self.__exit__ = lambda *__: None


# This works
with SlotsCM():
    pass


# This doesn't.
from contextllib import ExitStack
with ExitStack() as stack:
    stack.enter_context(SlotsCM())

Traceback (most recent call last):
  File "xxxx.py", line 16, in <module>
    stack.enter_context(SlotsCM())
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "......../contextlib.py", line 530, in enter_context
    result = _enter(cm)
TypeError: 'member_descriptor' object is not callable

Furthermore, both (async) with-statement and (Async)ExitStack don't suppot those special dunder methods stored in __dict__. You can verify from this repository:
https://github.com/gottadiveintopython/invest-context-manager

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-144386: Add support for descriptors in ExitStack and AsyncExitStack #144420
• gh-144386: Update equivalent code for "with", "async with" and "async for" #144472
• [3.14] gh-144386: Update equivalent code for "with", "async with" and "async for" (GH-144472) #144945
• [3.13] gh-144386: Update equivalent code for "with", "async with" and "async for" (GH-144472) #144946

[tanloong]

contextlib.ExitStack.enter_context(cm) calls type(cm).__enter__(cm). The problem looks like that type(cm) does not recognize those __enter__ and __exit__ methods defined by __slots__. The document says type() generally returns the same object as returned by object.__class__, which throws the same TypeError:

class SlotsCM:
    __slots__ = (
        "__enter__",
        "__exit__",
    )

    def __init__(self):
        self.__enter__ = self.__exit__ = lambda *__: None

s = SlotsCM()
s.__enter__(s) # This works.
type(s).__enter__(s) # TypeError: 'member_descriptor' object is not callable
s.__class__.__enter__(s) # TypeError: 'member_descriptor' object is not callable

[gottadiveintopython]

I don't fully understand the error message, but yes, the cause of the issue seems to be that (Async)ExitStack.enter_(async)_context() looks up dunder methods on the class rather than on
the instance.

[tanloong]

I am not familiar with the member_descriptor neither, it looks like a concept in C. A printing shows it refers to the __enter__ object reserved by __slots__.
Also I found that calling __get__(s) on the member descriptor returns the same function object as s.__enter__.

s = SlotsCM()
print(type(s.__enter__)) # <class 'function'>
print(type(s.__class__.__enter__)) # <class 'member_descriptor'>
print(id(s.__class__.__enter__.__get__(s, SlotsCM)) == id(s.__enter__)) # True

I tried to modify the contextlib.ExitStack.enter_context() code to use s.__class__.__enter__.__get__(s). It works, the TypeError is gone:

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b646fabca0..2bb72bae536 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -524,6 +524,10 @@ def enter_context(self, cm):
         try:
             _enter = cls.__enter__
             _exit = cls.__exit__
+            if not callable(_enter):
+                _enter = _enter.__get__(cm, cls)
+            if not callable(_exit):
+                _exit = _exit.__get__(cm, cls)
         except AttributeError:
             raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
                             f"not support the context manager protocol") from None

[tanloong]

I just found that there is also a difference between the actual with-statement behavior and its documented semantically equivalent code. The documentted equivalent code throws the same TypeError as contextlib.ExitStack.enter_context().

The following code:

with EXPRESSION as TARGET:
    SUITE

is semantically equivalent to:

manager = (EXPRESSION)
enter = type(manager).__enter__
exit = type(manager).__exit__
value = enter(manager)
hit_except = False

try:
    TARGET = value
    SUITE
except:
    hit_except = True
    if not exit(manager, *sys.exc_info()):
        raise
finally:
    if not hit_except:
        exit(manager, None, None, None)

PEP 343 explains with statement with generally the same code which also throws the TypeError in this usecase.

[picnixz]

Instead of using callable() I would suggest that maybe we first try to check if the enter/exit methods are descriptors or not by checking the presence of __get__. I think we have similar approaches with partial/partialmethod but I am not entirely sure

[tanloong]

@picnixz Thanks for the suggestion! This is the modified change I made following the partialmethod.

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b646fabca0..fd798796e7d 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -524,6 +524,10 @@ def enter_context(self, cm):
         try:
             _enter = cls.__enter__
             _exit = cls.__exit__
+            if (_enter_get := getattr(_enter, "__get__", None)) is not None:
+                _enter = _enter_get(cm, cls)
+            if (_exit_get := getattr(_exit, "__get__", None)) is not None:
+                _exit = _exit_get(cm, cls)
         except AttributeError:
             raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
                             f"not support the context manager protocol") from None

[picnixz]

This looks correct but I am not very fond of having a huge try-catch here. Likewise it does not seem right to catch the attribute error that could be raised by __get__ so you should move the descriptor handling after the try-catch.

[tanloong]

OK here is the new change:

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b646fabca0..5b5dc48affa 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -527,6 +527,10 @@ def enter_context(self, cm):
         except AttributeError:
             raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
                             f"not support the context manager protocol") from None
+        if (_enter_get := getattr(_enter, "__get__", None)) is not None:
+            _enter = _enter_get(cm, cls)
+        if (_exit_get := getattr(_exit, "__get__", None)) is not None:
+            _exit = _exit_get(cm, cls)
         result = _enter(cm)
         self._push_cm_exit(cm, _exit)
         return result

[picnixz]

Yes, something like that. Though I think you can write something more readable as:

if hasattr(_enter, "__get__"):
    _enter = _enter.__get__(cm, cls)

[tanloong]

Thank you for the suggestion! I updated the change. But I found that the change breaks some contextlib tests:

Running Debug|x64 interpreter...
..................................................EE......E..F..............................
======================================================================
ERROR: test_dont_reraise_RuntimeError (Lib.test.test_contextlib.TestExitStack.test_dont_reraise_RuntimeError)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1151, in test_dont_reraise_RuntimeError
    es_ctx.enter_context(second())
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
    result = _enter(cm)
TypeError: _GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_enter_context (Lib.test.test_contextlib.TestExitStack.test_enter_context)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 786, in test_enter_context
    stack.enter_context(cm)
    ~~~~~~~~~~~~~~~~~~~^^^^
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
    result = _enter(cm)
TypeError: TestBaseExitStack.test_enter_context.<locals>.TestCM.__enter__() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_exit_exception_explicit_none_context (Lib.test.test_contextlib.TestExitStack.test_exit_exception_explicit_none_context) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1002, in test_exit_exception_explicit_none_context
    with cm():
         ~~^^
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 996, in my_cm_with_exit_stack
    stack.enter_context(my_cm())
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
    result = _enter(cm)
TypeError: _GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given

======================================================================
FAIL: test_exit_exception_with_correct_context (Lib.test.test_contextlib.TestExitStack.test_exit_exception_with_correct_context)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1055, in test_exit_exception_with_correct_context
    stack.enter_context(gets_the_context_right(exc4))
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
    result = _enter(cm)
TypeError: _GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1060, in test_exit_exception_with_correct_context
    self.assertIs(exc, exc4)
    ~~~~~~~~~~~~~^^^^^^^^^^^
AssertionError: TypeError('_GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given') is not Exception(4)

----------------------------------------------------------------------
Ran 92 tests in 0.493s

FAILED (failures=1, errors=3)

I found that the __enter__ object defined by a regular class method defination also has a __get__ attribute:

class CM:
    def __init__(self):
        pass
    def __enter__(self):
        pass
    def __exit__(self, *exc_info):
        pass

class SlotsCM:
    __slots__ = ("__enter__", "__exit__")
    def __init__(self):
        self.__enter__ = self.__exit__ = lambda *__: None

cm = CM()
cls = type(cm)
_enter = cls.__enter__
print(getattr(_enter, "__get__", None)) # <method-wrapper '__get__' of function object at 0x00000154BDFB4650>

s = SlotsCM()
cls = type(s)
_enter = cls.__enter__
print(getattr(_enter, "__get__", None)) # <method-wrapper '__get__' of member_descriptor object at 0x000002C78844D5B0>

I tried to add the callable() check and this passes existing tests:

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b5dc48affa..fbc4de9277b 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -527,10 +527,14 @@ def enter_context(self, cm):
         except AttributeError:
             raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
                             f"not support the context manager protocol") from None
+        if not callable(_enter) and hasattr(_enter, "__get__"):
+            _enter = _enter.__get__(cm, cls)
+        if not callable(_exit) and hasattr(_exit, "__get__"):
+            _exit = _exit.__get__(cm, cls)
         result = _enter(cm)
         self._push_cm_exit(cm, _exit)
         return result

[picnixz]

Urgh, I hate working with descriptors. I actually wonder what we did on the C side for __enter__ as a slot to work. This may require more subtle checks

[tanloong]

I am not good at C but dis.dis() shows there is a LOAD_SPECIAL to handle the special methods.

class SlotsCM:
    __slots__ = ("__enter__", "__exit__")
    def __init__(self):
        self.__enter__ = self.__exit__ = lambda *__: None

def func():
    with SlotsCM():
        pass

import dis
from pprint import pprint
pprint(dis.dis(func))

#    8           RESUME                   0
#
#    9           LOAD_GLOBAL              1 (SlotsCM + NULL)
#                CALL                     0
#                COPY                     1
#                LOAD_SPECIAL             1 (__exit__)
#                SWAP                     2
#                SWAP                     3
#                LOAD_SPECIAL             0 (__enter__)
#                CALL                     0
#        L1:     POP_TOP
#
#   10   L2:     NOP
#
#    9           LOAD_CONST               0 (None)
#                LOAD_CONST               0 (None)
#                LOAD_CONST               0 (None)
#                CALL                     3
#                POP_TOP
#                LOAD_CONST               0 (None)
#                RETURN_VALUE
#        L3:     PUSH_EXC_INFO
#                WITH_EXCEPT_START
#                TO_BOOL
#                POP_JUMP_IF_TRUE         2 (to L6)
#        L4:     NOT_TAKEN
#        L5:     RERAISE                  2
#        L6:     POP_TOP
#        L7:     POP_EXCEPT
#                POP_TOP
#                POP_TOP
#                POP_TOP
#                LOAD_CONST               0 (None)
#                RETURN_VALUE
#
#   --   L8:     COPY                     3
#                POP_EXCEPT
#                RERAISE                  1
# ExceptionTable:
#   L1 to L2 -> L3 [2] lasti
#   L3 to L4 -> L8 [4] lasti
#   L5 to L7 -> L8 [4] lasti
# None