Jami recivies SIGABRT while connecting to network

Can't reproduce, but can you send the first lines of the daemon logs? It contains gnutls + pjsip version.

cc @mchibani1, did you saw this?

No, first time, I will try to look at it.

Requested versions from console output:

[1634687224.251|    2|manager.cpp       :798  ] Using PJSIP version 2.11 for x86_64-unknown-linux-gnu
[1634687224.251|    2|manager.cpp       :799  ] Using GnuTLS version 3.6.16
[1634687224.251|    2|manager.cpp       :800  ] Using OpenDHT version 2.3.0
[1634687224.251|    2|manager.cpp       :801  ] Using FFmpeg version 4.0.0-2480-g1cff5df87
[1634687224.251|    2|manager.cpp       :804  ] Using Libgit2 version 1.3.0
[1634687224.251|    2|sipvoiplink.cpp   :659  ] Using SIP nameserver: 192.168.16.1
[1634687224.251|    2|sipvoiplink.cpp   :657  ] Error printing SIP nameserver: ai_family not supported
[1634687224.252|    2|sipvoiplink.cpp   :753  ] SIPVoIPLink@0x55f92ea38c40
[1634687224.252|    2|manager.cpp       :820  ] Configuration file path: /home/ildar/.var/app/net.jami.Jami/config/jami/dring.yml
[1634687224.254|   78|system_codec_container.cpp:197  ] Can't find a usable accelerated H265/HEVC codec, disabling.
[1634687224.254|   78|system_codec_container.cpp:232  ] Encoders found: H264 VP8 MP4V-ES H263-1998 opus G722 speex speex speex PCMA PCMU
[1634687224.255|   78|system_codec_container.cpp:233  ] Decoders found: H264 VP8 MP4V-ES H263-1998 opus G722 speex speex speex PCMA PCMU
[

The only big difference I see is GnuTLS is old, (we use 3.7.1 on our released versions), however it should be ok (I checked that because the issue seems to occurs in gnutls_handshake). I can't reproduce the issue on my devices/builds/releases from jami.net

If Manjaro supports snap, is the snap also crashing?

For what I see you have a IceTransport with one component:

1634310323.017|  142|ice_transport.cpp :712  ] [ice:0x7ffedc1b7700] TCP connection pairs ([comp id] local [type] <-> remote [type]):
 [1] 192.168.16.178:55192 [prflx]  <-> 192.168.16.182:14206 [host]

and it sends data to this component (jami::IceTransport::getRemoteAddress(unsigned int) const (this=0x7ffedc0bd6d0, comp_id=1)) so it should not be out of range.

Also we don't manage flatpak, so don't know what are the daemon's commit used? another possibility can be to build with libasan, (like uses CXXFLAGS="-fsanitize=address" ./configure --enable-debug=yes --disable-shared).

Also in gdb, may be useful:

Uses up until #5
p compCount_ (it should be 1)
p iceDefaultRemoteAddr_

Also sipvoiplink.cpp :657 ] Error printing SIP nameserver: ai_family not supported is anormal. What does _res contains after https://linux.die.net/man/3/res_init in a flatpak env?

Unfortunately, I do not have enough knowledge either in debugging or in a flatpack. I invited the flatpack maintainer to this discussion, I hope he can help.

This is really IceTransport, if I disconnect the second machine with the jami running, the jami from the flatpack starts without problems, but as soon as I connect another client to the network, the it crashes.

Sorry for the wait but I'm busy with studying for uni.

I can reproduce this with:
daemon: 01594813f594a1029c7b78e0dbca246f372a0df9
libclient: e47fa5830e962bf95014a586fb65d6cb80b14dd2
qt-client: a79ea5fd

Compiling jami-daemon with CXXFLAGS="-fsanitize=address" ./configure --enable-debug=yes --disable-shared broke linking jami-libclient so I had to build with CXXFLAGS="-fPIC -fsanitize=address".

Now I fail at linking qt-client with jami-libclient.so:

/usr/lib/gcc/x86_64-unknown-linux-gnu/10.2.0/../../../../x86_64-unknown-linux-gnu/bin/ld: /app/lib/libringclient.so: undefined reference to `pa_context_get_source_info_list'
/usr/lib/gcc/x86_64-unknown-linux-gnu/10.2.0/../../../../x86_64-unknown-linux-gnu/bin/ld: /app/lib/libringclient.so: undefined reference to `av_buffersink_get_sample_rate'
/usr/lib/gcc/x86_64-unknown-linux-gnu/10.2.0/../../../../x86_64-unknown-linux-gnu/bin/ld: /app/lib/libringclient.so: undefined reference to `git_diff_tree_to_tree'
...

gdb gives me (only --enable-debug=yes):

#0  0x00007ffff54b87f5 in raise () at /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff54a1855 in abort () at /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fffec56d406 in std::__replacement_assert(char const*, int, char const*, char const*)
    (__file=__file@entry=0x7fffecfc2a58 "/usr/include/c++/10.2.0/bits/stl_vector.h", __line=__line@entry=1063, __function=__function@entry=0x7fffecfcbf40 "std::vector<_Tp, _Alloc>::const_reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) const [with _Tp = jami::IpAddr; _Alloc = std::allocator<jami::IpAddr>; std::vector<_"..., __condition=__condition@entry=0x7fffecfc7e68 "__builtin_expect(__n < this->size(), true)") at /usr/include/c++/10.2.0/x86_64-unknown-linux-gnu/bits/c++config.h:457
#3  0x00007fffec5d13df in std::vector<jami::IpAddr, std::allocator<jami::IpAddr> >::operator[](unsigned long) const (this=<optimized out>, __n=<optimized out>) at /usr/include/c++/10.2.0/bits/stl_vector.h:1061
#4  std::vector<jami::IpAddr, std::allocator<jami::IpAddr> >::operator[](unsigned long) const (this=<optimized out>, this=<optimized out>, __n=<optimized out>) at /usr/include/c++/10.2.0/bits/stl_vector.h:1061
#5  jami::IceTransport::Impl::getDefaultRemoteAddress(unsigned int) const (this=<optimized out>, compId=<optimized out>) at ice_transport.cpp:1094
#6  0x00007fffec5d1426 in jami::IceTransport::getRemoteAddress(unsigned int) const (this=0x7ffe70005010, comp_id=1) at /usr/include/c++/10.2.0/bits/unique_ptr.h:421
#7  0x00007fffec5d1629 in jami::IceTransport::send(unsigned int, unsigned char const*, unsigned long) (this=0x7ffe70005010, compId=1, buf=buf@entry=0x7ffe6000db3b "\026\003\003", len=len@entry=192)
    at ice_transport.cpp:1681
#8  0x00007fffec5e8616 in jami::IceSocketEndpoint::write(unsigned char const*, unsigned long, std::error_code&) (ec=..., len=192, buf=0x7ffe6000db3b "\026\003\003", this=0x7ffe80000b80)
    at /usr/include/c++/10.2.0/bits/shared_ptr_base.h:1324
#9  jami::IceSocketEndpoint::write(unsigned char const*, unsigned long, std::error_code&) (this=0x7ffe80000b80, buf=0x7ffe6000db3b "\026\003\003", len=192, ec=...) at peer_connection.cpp:145
#10 0x00007fffec778583 in jami::tls::TlsSession::TlsSessionImpl::sendRaw(void const*, unsigned long) (this=0x7ffe800015f0, buf=0x7ffe6000db3b, size=192) at ./security/tls_session.cpp:898
#11 0x00007fffec77894b in jami::tls::TlsSession::TlsSessionImpl::sendRawVec(iovec const*, int) (iovcnt=<optimized out>, iov=<optimized out>, this=0x7ffe800015f0) at ./security/tls_session.cpp:909
#12 operator() (iovcnt=<optimized out>, iov=<optimized out>, t=0x7ffe800015f0, __closure=0x0) at ./security/tls_session.cpp:605
#13 _FUN(gnutls_transport_ptr_t, giovec_t const*, int) () at ./security/tls_session.cpp:606
#14 0x00007fffe836a309 in _gnutls_io_write_flush () at /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#15 0x00007fffe836b598 in _gnutls_handshake_io_write_flush () at /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#16 0x00007fffe836e380 in _gnutls_send_change_cipher_spec () at /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#17 0x00007fffe8367f4d in _gnutls13_handshake_server () at /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#18 0x00007fffe8372ccb in gnutls_handshake () at /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#19 0x00007fffec77655e in jami::tls::TlsSession::TlsSessionImpl::handleStateHandshake(jami::tls::TlsSessionState) (this=0x7ffe800015f0, state=jami::tls::TlsSessionState::HANDSHAKE)
    at ./security/tls_session.cpp:1154
#20 0x00007fffec77b6fc in std::function<jami::tls::TlsSessionState (jami::tls::TlsSessionState)>::operator()(jami::tls::TlsSessionState) const
    (__args#0=jami::tls::TlsSessionState::HANDSHAKE, this=0x7ffe60000c28) at /usr/include/c++/10.2.0/bits/std_function.h:617
#21 jami::tls::TlsSession::TlsSessionImpl::process() (this=0x7ffe800015f0) at ./security/tls_session.cpp:1554
#22 0x00007fffec5cab8d in std::function<void ()>::operator()() const (this=0x7ffe677fafe0) at /usr/include/c++/10.2.0/bits/std_function.h:617
#23 jami::ThreadLoop::mainloop(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>) (this=0x7ffe80001960, tid=..., setup=..., process=..., cleanup=...) at threadloop.cpp:38
#24 0x00007fffec5cb499 in std::__invoke_impl<void, void (jami::ThreadLoop::*)(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>), jami::ThreadLoop*, std::reference_wrapper<std::thread::id>, std::function<bool ()>, std::function<void ()>, std::function<void ()> >(std::__invoke_memfun_deref, void (jami::ThreadLoop::*&&)(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>), jami::ThreadLoop*&&, std::reference_wrapper<std::thread::id>&&, std::function<bool ()>&&, std::function<void ()>&&, std::function<void ()>&&)
    (__f=<optimized out>, __t=<optimized out>, __f=<optimized out>, __t=<optimized out>) at /usr/include/c++/10.2.0/bits/refwrap.h:337
#25 std::__invoke<void (jami::ThreadLoop::*)(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>), jami::ThreadLoop*, std::reference_wrapper<std::thread::id>, std::function<bool ()>, std::function<void ()>, std::function<void ()> >(void (jami::ThreadLoop::*&&)(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>), jami::ThreadLoop*&&, std::reference_wrapper<std::thread::id>&&, std::function<bool ()>&&, std::function<void ()>&&, std::function<void ()>&&) (__fn=<optimized out>) at /usr/include/c++/10.2.0/bits/invoke.h:95
#26 std::thread::_Invoker<std::tuple<void (jami::ThreadLoop::*)(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>), jami::ThreadLoop*, std::reference_wrapper<std::thread::id>, std::function<bool ()>, std::function<void ()>, std::function<void ()> > >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>) (this=<optimized out>)
    at /usr/include/c++/10.2.0/thread:264
#27 std::thread::_Invoker<std::tuple<void (jami::ThreadLoop::*)(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>), jami::ThreadLoop*, std::reference_wrapper<std::thread::id>, std::function<bool ()>, std::function<void ()>, std::function<void ()> > >::operator()() (this=<optimized out>) at /usr/include/c++/10.2.0/thread:271
#28 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (jami::ThreadLoop::*)(std::thread::id&, std::function<bool ()>, std::function<void ()>, std::function<void ()>), jami::ThreadLoop*, std::reference_wrapper<std::thread::id>, std::function<bool ()>, std::function<void ()>, std::function<void ()> > > >::_M_run() (this=<optimized out>) at /usr/include/c++/10.2.0/thread:215
#29 0x00007ffff5894a74 in execute_native_thread_routine () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#30 0x00007ffff54654d2 in start_thread () at /usr/lib/x86_64-linux-gnu/libpthread.so.0
#31 0x00007ffff557d323 in clone () at /usr/lib/x86_64-linux-gnu/libc.so.6

I only find this about: ai_family not supported

building with gnutls 3.7.2 leads to the same crash

Hi, I did spent some time on this issue. To be honest I have no clue what causing the crash. We have this same code running for thousands of calls both in CI test bench and production, and never reproduce it. As I understand, you are building the project from source code, so just make sure, when you update the code, to force the rebuild of the contrib code (in daemon/contrib/native), or may be just do a fresh checkout in an empty folder. Sorry I could not be more helpful. BR.

I only use ffmpeg and pjproject from daemon/contrib everything else comes from the Flatpak runtime or is build separately. GnuTLS for example is included in the flatpak runtime. Maybe this is a build system issue?

continued in jami-daemon#667 (closed)

assigned to @mchibani1

added SprintTo review label

hi, we might have a patch for this, it's under review. Since we are not able the reproduce the crash, we will need your help for validation. We will let know when ready

FYI @FuchtelJockel @akhilman thanks

I added https://review.jami.net/c/jami-daemon/+/19796 and TLS handshakes work now. But accepting audio/video calls results in a crash.

[1637076056.238|  175|sipcall.cpp       :1544 ] [call:2065138378183698] onAnswered()
[1637076056.386|  178|ice_transport.cpp :683  ] [ice:0x7f7c2c02abd0] UDP negotiation success
[1637076056.386|  178|ice_transport.cpp :709  ] [ice:0x7f7c2c02abd0] UDP connection pairs ([comp id] local [type] <-> remote [type]):
 [1] 192.168.1.175:39352 [host]  <-> 192.168.1.147:43581 [host] 
 [2] 192.168.1.175:54159 [host]  <-> 192.168.1.147:43746 [host] 

[1637076056.386|   67|sipcall.cpp       :2455 ] [call:2065138378183698] ICE negotiation succeeded
[1637076056.386|   67|sipcall.cpp       :1893 ] [call:2065138378183698] updating negotiated media
[1637076056.387|   67|sipcall.cpp       :231  ] [call:2065138378183698] Configuring [MEDIA_TYPE_AUDIO] rtp session
[1637076056.387|   67|sipcall.cpp       :2107 ] [call:2065138378183698] stopping all medias
[1637076056.387|   67|sipcall.cpp       :1995 ] [call:2065138378183698] starting the media
/usr/include/c++/10.2.0/bits/stl_vector.h:1045: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = jami::IpAddr; _Alloc = std::allocator<jami::IpAddr>; std::vector<_Tp, _Alloc>::reference = jami::IpAddr&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.

@FuchtelJockel You mentioned an audio/video call, but I only see the audio stream. I will need more logs to investigate, can you provide the full call logs ? It's better if you add them as attached file.

Thanks

video.log I made a video call from my phone but don't have a webcam connected to my desktop.

audio.log

thanks for the validation. I will check the crash in the media soon. BR.

@FuchtelJockel hi, we have this patch under review that should fix both crashes: https://review.jami.net/c/jami-daemon/+/19809

BTW, you are seeing those crashes (and we dont) because you have the _GLIBCXX_ASSERTIONS flag in you build settings (and we don't, may be we should ).

BR

Note that if assertions are enabled for a released app it's really bad. performances will be awful.

With the last patch everything works as expected.

I read a bit about _GLIBCXX_ASSERTIONS and I think Fedora has this enabled as well. That's probably why it's also enabled in the Freedesktop SDK.

In this bug report a libstdc++ developer said the overhead is minimal:
https://bugs.launchpad.net/kicad/+bug/1838448/comments/40
https://bugs.launchpad.net/kicad/+bug/1838448/comments/42

I'm on Fedora 34 and 35 and never saw this, but for release, this should not be enabled as it will greatly decrease the perf.

https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/include/flags.yml
I will test if I can see a difference but since it's enabled by default I will probably leave it.

added SprintDone label and removed SprintTo review label

removed SprintDone label

I guess everything is ok here?

For me it's okay to leave it if you want. I'm a bit scared about performance, but it's a choice.

closed

Jami recivies SIGABRT while connecting to network

Describe your environment

Steps to reproduce

Additional information

Designs

Child items 0

Activity