mirror of
https://github.com/gcc-mirror/gcc.git
synced 2026-05-06 23:25:24 +02:00
d07b9e7fe4
This will allow us to extend atomic waiting functions to support a possible future 64-bit version of futex, as well as supporting futex-like wait/wake primitives on other targets (e.g. macOS has os_sync_wait_on_address and FreeBSD has _umtx_op). Before this change, the decision of whether to do a proxy wait or to wait on the atomic variable itself was made in the header at compile-time, which makes it an ABI property that would not have been possible to change later. That would have meant that std::atomic<uint64_t> would always have to do a proxy wait even if Linux gains support for 64-bit futex2(2) calls at some point in the future. The disadvantage of proxy waits is that several distinct atomic objects can share the same proxy state, leading to contention between threads even when they are not waiting on the same atomic object, similar to false sharing. It also result in spurious wake-ups because doing a notify on an atomic object that uses a proxy wait will wake all waiters sharing the proxy. For types that are known to definitely not need a proxy wait (e.g. int on Linux) the header can still choose a more efficient path at compile-time. But for other types, the decision of whether to do a proxy wait is deferred to runtime, inside the library internals. This will make it possible for future versions of libstdc++.so to extend the set of types which don't need to use proxy waits, without ABI changes. The way the change works is to stop using the __proxy_wait flag that was set by the inline code in the headers. Instead the __wait_args struct has an extra pointer member which the library internals populate with either the address of the atomic object or the _M_ver counter in the proxy state. There is also a new _M_obj_size member which stores the size of the atomic object, so that the library can decide whether a proxy is needed. So for example if linux gains 64-bit futex support then the library can decide not to use a proxy when _M_obj_size == 8. Finally, the _M_old member of the __wait_args struct is changed to uint64_t so that it has room to store 64-bit values, not just whatever size the __platform_wait_t type is (which is a 32-bit int on Linux). Similarly, the _M_val member of __wait_result_type changes to uint64_t too. libstdc++-v3/ChangeLog: * config/abi/pre/gnu.ver: Adjust exports. * include/bits/atomic_timed_wait.h (_GLIBCXX_HAVE_PLATFORM_TIMED_WAIT): Do not define this macro. (__atomic_wait_address_until_v, __atomic_wait_address_for_v): Adjust assertions to check that __platform_wait_uses_type is true. * include/bits/atomic_wait.h (__waitable): New concept. (__platform_wait_uses_type): Different separately for platforms with and without platform wait. (_GLIBCXX_HAVE_PLATFORM_WAIT): Do not define this macro. (__wait_value_type): New typedef. (__wait_result_type): Change _M_val to __wait_value_type. (__wait_flags): Remove __proxy_wait enumerator. Reduce range reserved for ABI version by the commented-out value. (__wait_args_base::_M_old): Change type to __wait_args_base. (__wait_args_base::_M_obj, __wait_args_base::_M_obj_size): New data members. (__wait_args::__wait_args): Set _M_obj and _M_obj_size on construction. (__wait_args::_M_setup_wait): Change void* parameter to deduced type. Adjust bit_cast to work for types of different sizes. (__wait_args::_M_load_proxy_wait_val): Remove function, replace with ... (__wait_args::_M_setup_proxy_wait): New function. (__wait_args::_S_flags_for): Do not set __proxy_wait flag. (__atomic_wait_address_v): Adjust assertion to check that __platform_wait_uses_type is true. * src/c++20/atomic.cc (_GLIBCXX_HAVE_PLATFORM_WAIT): Define here instead of in header. Check _GLIBCXX_HAVE_PLATFORM_WAIT instead of _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT. (__platform_wait, __platform_notify, __platform_wait_until): Add unused parameter for _M_obj_size. (__spin_impl): Adjust for 64-bit __wait_args_base::_M_old. (use_proxy_wait): New function. (__wait_args::_M_load_proxy_wait_val): Replace with ... (__wait_args::_M_setup_proxy_wait): New function. Call use_proxy_wait to decide at runtime whether to wait on the pointer directly instead of using a proxy. If a proxy is needed, set _M_obj and _M_obj_size to refer to its _M_ver member. Adjust for change to type of _M_old. (__wait_impl): Wait on _M_obj unconditionally. Pass _M_obj_size to __platform_wait. (__notify_impl): Call use_proxy_wait to decide whether to notify on the address parameter or a proxy (__spin_until_impl): Adjust for change to type of _M_val. (__wait_until_impl): Wait on _M_obj unconditionally. Pass _M_obj_size to __platform_wait_until. Reviewed-by: Tomasz Kamiński <tkaminsk@redhat.com>