Skip to content
Snippets Groups Projects
New look: Off

AudioLayer - Deadlock between audio loop and main thread

    • Closed Issue created by Mohamed Chibani @mchibani1

    A deadlock may occur between the pulse audio thread and daemon main thread because of the lock inversion of callMutex_ (call class) and pa_threaded_mainloop private mutex. See backtrace in the comments.

    Edited

    Designs

      Child items ...

      2. Activity

      • Mohamed Chibani assigned to @mchibani1

        assigned to @mchibani1

      • Mohamed Chibani changed title from AudioLayer - Dead lock between audio loop and main thread to AudioLayer - Deadlock between audio loop and main thread

        changed title from AudioLayer - Dead lock between audio loop and main thread to AudioLayer - Deadlock between audio loop and main thread

      • Mohamed Chibani
        Mohamed Chibani @mchibani1 ·
        Author Developer

        Deadlock backtrace of the involved threads:

        Thread 2 (Thread 0x7f2f6d5dd700 (LWP 2776611)):
#0  futex_lock_pi (private=0, abstime=0x0, futex_word=0x55910d127fb0) at ../sysdeps/nptl/futex-internal.h:412
#1  __pthread_mutex_lock_full (mutex=0x55910d127fb0) at ../nptl/pthread_mutex_lock.c:419
#2  0x00007f2f77da7822 in pa_mutex_lock () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so
#3  0x000055910b8e437c in jami::PulseMainLoopLock::PulseMainLoopLock(pa_threaded_mainloop*) (this=0x7f2f6d5d93b0, loop=0x55910d196fe0) at pulselayer.cpp:54
#4  0x000055910b8e6839 in jami::PulseLayer::stopStream(jami::AudioDeviceType) (this=0x55910d1637f0, type=jami::AudioDeviceType::CAPTURE) at pulselayer.cpp:457
#5  0x000055910b463620 in jami::AudioDeviceGuard::~AudioDeviceGuard() (this=0x7f2f3405a5c0, __in_chrg=<optimized out>) at manager.cpp:2474
#6  0x000055910b47cbb6 in std::default_delete<jami::AudioDeviceGuard>::operator()(jami::AudioDeviceGuard*) const (this=0x7f2f34027d98, __ptr=0x7f2f3405a5c0) at /usr/include/c++/9/bits/unique_ptr.h:81
#7  0x000055910b4768a6 in std::unique_ptr<jami::AudioDeviceGuard, std::default_delete<jami::AudioDeviceGuard> >::~unique_ptr() (this=0x7f2f34027d98, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:292
#8  0x000055910b8c81ac in jami::AudioInput::~AudioInput() (this=0x7f2f34027ae0, __in_chrg=<optimized out>) at audio_input.cpp:63
#9  0x000055910b6430d9 in __gnu_cxx::new_allocator<jami::AudioInput>::destroy<jami::AudioInput>(jami::AudioInput*) (this=0x7f2f34027ae0, __p=0x7f2f34027ae0) at /usr/include/c++/9/ext/new_allocator.h:153
#10 0x000055910b64301f in std::allocator_traits<std::allocator<jami::AudioInput> >::destroy<jami::AudioInput>(std::allocator<jami::AudioInput>&, jami::AudioInput*) (__a=..., __p=0x7f2f34027ae0) at /usr/include/c++/9/bits/alloc_traits.h:497
#11 0x000055910b642ce7 in std::_Sp_counted_ptr_inplace<jami::AudioInput, std::allocator<jami::AudioInput>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (this=0x7f2f34027ad0) at /usr/include/c++/9/bits/shared_ptr_base.h:557
#12 0x000055910b43aab0 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (this=0x7f2f34027ad0) at /usr/include/c++/9/bits/shared_ptr_base.h:155
#13 0x000055910b437b29 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (this=0x7f2f6d5d9608, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730
#14 0x000055910b474d1c in std::__shared_ptr<jami::AudioInput, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (this=0x7f2f6d5d9600, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169
#15 0x000055910b5af426 in std::__shared_ptr<jami::AudioInput, (__gnu_cxx::_Lock_policy)2>::reset() (this=0x7f2f08008ee8) at /usr/include/c++/9/bits/shared_ptr_base.h:1287
#16 0x000055910b5ac5b0 in jami::AudioRtpSession::stop() (this=0x7f2f08008b00) at audio_rtp_session.cpp:226
#17 0x000055910b541082 in jami::SIPCall::stopAllMedia() (this=0x7f2f08003f90) at sipcall.cpp:2056
#18 0x000055910b542c08 in jami::SIPCall::onIceNegoSucceed() (this=0x7f2f08003f90) at sipcall.cpp:2386
#19 0x000055910b545b6b in jami::SIPCall::<lambda(bool)>::<lambda()>::operator()(void) const (__closure=0x7f2edc01c2d0) at sipcall.cpp:2952
#20 0x000055910b549cab in jami::<lambda()>::operator()(void) (this=0x7f2edc01c2d0) at /home/mchibani/workdir/ring-linux/daemon/src/manager.h:1069
#21 0x000055910b54d20b in std::_Function_handler<void(), jami::runOnMainThread(Callback&&) [with Callback = jami::SIPCall::initIceMediaTransport(bool, std::optional<jami::IceTransportOptions>)::<lambda(bool)>::<lambda()>]::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/9/bits/std_function.h:300
#22 0x000055910b43813c in std::function<void ()>::operator()() const (this=0x7f2edc186050) at /usr/include/c++/9/bits/std_function.h:688
#23 0x000055910b52d2fc in jami::ScheduledExecutor::loop() (this=0x55910d10e6c0) at scheduled_executor.cpp:137
#24 0x000055910b52c714 in jami::ScheduledExecutor::<lambda()>::operator()(void) const (__closure=0x55910d10b5a8) at scheduled_executor.cpp:32
#25 0x000055910b52e9be in std::__invoke_impl<void, jami::ScheduledExecutor::ScheduledExecutor()::<lambda()> >(std::__invoke_other, jami::ScheduledExecutor::<lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#26 0x000055910b52e973 in std::__invoke<jami::ScheduledExecutor::ScheduledExecutor()::<lambda()> >(jami::ScheduledExecutor::<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#27 0x000055910b52e920 in std::thread::_Invoker<std::tuple<jami::ScheduledExecutor::ScheduledExecutor()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x55910d10b5a8) at /usr/include/c++/9/thread:244
#28 0x000055910b52e8f6 in std::thread::_Invoker<std::tuple<jami::ScheduledExecutor::ScheduledExecutor()::<lambda()> > >::operator()(void) (this=0x55910d10b5a8) at /usr/include/c++/9/thread:251
#29 0x000055910b52e8da in std::thread::_State_impl<std::thread::_Invoker<std::tuple<jami::ScheduledExecutor::ScheduledExecutor()::<lambda()> > > >::_M_run(void) (this=0x55910d10b5a0) at /usr/include/c++/9/thread:195
#30 0x00007f2f783bade4 in  () at /lib/x86_64-linux-gnu/libstdc++.so.6
#31 0x00007f2f78915609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#32 0x00007f2f780a8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f2f69d99700 (LWP 2776619)):
#0  __lll_lock_wait (futex=futex@entry=0x7f2f08004080, private=0) at lowlevellock.c:52
#1  0x00007f2f78918131 in __GI___pthread_mutex_lock (mutex=0x7f2f08004080) at ../nptl/pthread_mutex_lock.c:115
#2  0x000055910b42a8af in __gthread_mutex_lock(__gthread_mutex_t*) (__mutex=0x7f2f08004080) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:749
#3  0x000055910b435032 in __gthread_recursive_mutex_lock(__gthread_recursive_mutex_t*) (__mutex=0x7f2f08004080) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:811
#4  0x000055910b43551c in std::recursive_mutex::lock() (this=0x7f2f08004080) at /usr/include/c++/9/mutex:106
#5  0x000055910b437eb2 in std::lock_guard<std::recursive_mutex>::lock_guard(std::recursive_mutex&) (this=0x7f2f69d95100, __m=...) at /usr/include/c++/9/bits/std_mutex.h:159
#6  0x000055910b470d68 in jami::Call::isSubcall() const (this=0x7f2f08003f90) at call.h:306
#7  0x000055910b45a8cc in jami::Manager::hasCurrentCall() const (this=0x55910c1de5a0 <jami::Manager::instance()::instance>) at manager.cpp:973
#8  0x000055910b5bb8a1 in jami::AudioLayer::notifyIncomingCall() (this=0x55910d1637f0) at audiolayer.cpp:179
#9  0x000055910b5bbd99 in jami::AudioLayer::getToPlay(jami::AudioFormat, unsigned long) (this=0x55910d1637f0, format=..., writableSamples=882) at audiolayer.cpp:216
#10 0x000055910b8e6b60 in jami::PulseLayer::writeToSpeaker() (this=0x55910d1637f0) at pulselayer.cpp:487
#11 0x000055910b8ee694 in std::__invoke_impl<void, void (jami::PulseLayer::*&)(), jami::PulseLayer*&>(std::__invoke_memfun_deref, void (jami::PulseLayer::*&)(), jami::PulseLayer*&) (__f=@0x55910d0ef1b0: (void (jami::PulseLayer::*)(class jami::PulseLayer * const)) 0x55910b8e69ce <jami::PulseLayer::writeToSpeaker()>, __t=@0x55910d0ef1c0: 0x55910d1637f0) at /usr/include/c++/9/bits/invoke.h:73
#12 0x000055910b8ee11f in std::__invoke<void (jami::PulseLayer::*&)(), jami::PulseLayer*&>(void (jami::PulseLayer::*&)(), jami::PulseLayer*&) (__fn=@0x55910d0ef1b0: (void (jami::PulseLayer::*)(class jami::PulseLayer * const)) 0x55910b8e69ce <jami::PulseLayer::writeToSpeaker()>) at /usr/include/c++/9/bits/invoke.h:95
#13 0x000055910b8eda2f in std::_Bind<void (jami::PulseLayer::*(jami::PulseLayer*))()>::__call<void, unsigned long&&, 0ul>(std::tuple<unsigned long&&>&&, std::_Index_tuple<0ul>) (this=0x55910d0ef1b0, __args=...) at /usr/include/c++/9/functional:400
#14 0x000055910b8eca92 in std::_Bind<void (jami::PulseLayer::*(jami::PulseLayer*))()>::operator()<unsigned long, void>(unsigned long&&) (this=0x55910d0ef1b0) at /usr/include/c++/9/functional:484
#15 0x000055910b8eb21a in std::_Function_handler<void (unsigned long), std::_Bind<void (jami::PulseLayer::*(jami::PulseLayer*))()> >::_M_invoke(std::_Any_data const&, unsigned long&&) (__functor=..., __args#0=@0x7f2f69d954e0: 3528) at /usr/include/c++/9/bits/std_function.h:300
#16 0x000055910b9f0399 in std::function<void (unsigned long)>::operator()(unsigned long) const (this=0x55910d1bdce0, __args#0=3528) at /usr/include/c++/9/bits/std_function.h:688
#17 0x000055910b9ef8c5 in jami::AudioStream::<lambda(pa_stream*, size_t, void*)>::operator()(pa_stream *, size_t, void *) const (__closure=0x0, bytes=3528, userdata=0x55910d1bdcc0) at audiostream.cpp:92
#18 0x000055910b9ef8f9 in jami::AudioStream::<lambda(pa_stream*, size_t, void*)>::_FUN(pa_stream *, size_t, void *) () at audiostream.cpp:93
#19 0x00007f2f7a91a91e in  () at /lib/x86_64-linux-gnu/libpulse.so.0
#20 0x00007f2f77d92ee0 in pa_pdispatch_run () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so
#21 0x00007f2f7a8fd5f3 in  () at /lib/x86_64-linux-gnu/libpulse.so.0
#22 0x00007f2f77d95947 in  () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so
#23 0x00007f2f77d9872b in  () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so
#24 0x00007f2f77d98aea in  () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so
#25 0x00007f2f77d9937a in  () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so
#26 0x00007f2f7a912ba3 in pa_mainloop_dispatch () at /lib/x86_64-linux-gnu/libpulse.so.0
#27 0x00007f2f7a912ed2 in pa_mainloop_iterate () at /lib/x86_64-linux-gnu/libpulse.so.0
#28 0x00007f2f7a912f70 in pa_mainloop_run () at /lib/x86_64-linux-gnu/libpulse.so.0
#29 0x00007f2f7a92111d in  () at /lib/x86_64-linux-gnu/libpulse.so.0
#30 0x00007f2f77da872c in  () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-13.99.so
#31 0x00007f2f78915609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#32 0x00007f2f780a8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

        The pulse-audio thread (thread 9) will call the playback callback (writeToSpeaker) while locking its private mutex, then in this callback, we try to lock callMutex_ in isSubCall() method. In daemon main thread (thread 2), we try to lock the private mutex of the pulse-audio thread while locking callMutex_ in onNegoDone() callback.

        Edited by Mohamed Chibani
      • Mohamed Chibani added SprintTo review label

        added SprintTo review label

      • Mohamed Chibani mentioned in commit 8dcbbc38

        mentioned in commit 8dcbbc38

      • Mohamed Chibani added SprintDone label and removed SprintTo review label

        added SprintDone label and removed SprintTo review label

      • Sébastien Blin removed SprintDone label

        removed SprintDone label

      • Sébastien Blin closed

        closed

      Please register or sign in to reply