test suite fails
Hello,
The test suite is not currently usable:
- Some tests fail non-deterministically (or always).
- It takes about 1 h to run, which is too long.
Reproducer:
- Run
make check
Without the above patch applied:
make check-TESTS
make[3]: Entering directory '/tmp/guix-build-libjami-20230221.0.drv-0/jami-20230221.0/daemon/test/unitTest.s ' re
make[4]: Entering directory '/tmp/guix-build-libjami-20230221.0.drv-0/jami-20230221.0/daemon/test/unitTest/g ' ag
RUNNING: ut_certstore
RUNNING: ut_account_factory
RUNNING: ut_account_archive mi
RUNNING: ut_migration
RUNNING: ut_string_utils fl
RUNNING: ut_scheduler
RUNNING: ut_map_utils
RUNNING: ut_fileutils .s
RUNNING: ut_base64 re
RUNNING: ut_video_input /g
RUNNING: ut_media_encoder xt
RUNNING: ut_media_filter
RUNNING: ut_utf8_utils
RUNNING: ut_media_decoder
RUNNING: ut_media_player
RUNNING: ut_resampler
RUNNING: ut_video_scaler -2
RUNNING: ut_media_frame
RUNNING: ut_audio_frame_resizer
RUNNING: ut_recorder
RUNNING: ut_conference
RUNNING: ut_connectionManager
RUNNING: ut_fileTransfer l:
RUNNING: ut_call
PASS: ut_utf8_utils
PASS: ut_map_utils
PASS: ut_audio_frame_resizer
PASS: ut_string_utils
PASS: ut_fileutils
PASS: ut_base64 i-
RUNNING: ut_conversationRepository
RUNNING: ut_media_negotiation fl
RUNNING: ut_conversation
RUNNING: ut_auto_answer
RUNNING: ut_hold_resume .s
RUNNING: ut_conversation_call re
PASS: ut_media_frame /g
PASS: ut_resampler xt
PASS: ut_video_scaler mi
PASS: ut_media_filter
PASS: ut_media_decoder
RUNNING: ut_conversationRequest
RUNNING: ut_conversationMembersEvent fl
RUNNING: ut_ice
RUNNING: ut_syncHistory
RUNNING: ut_revoke .s
PASS: ut_video_input re
RUNNING: ut_ice_sdp_parser /g
PASS: ut_media_player ag
RUNNING: ut_ice_media_cand_exchange sc
FAIL: ut_scheduler ):
FAIL: ut_ice_sdp_parser -2
RUNNING: ut_sip_srtp fl
RUNNING: ut_sip_basic_calls
FAIL: ut_sip_srtp
PASS: ut_media_encoder .s
RUNNING: ut_plugins re
PASS: ut_certstore /g
FAIL: ut_sip_basic_calls ag
FAIL: ut_ice
FAIL: ut_account_factory
FAIL: ut_hold_resume mi
FAIL: ut_ice_media_cand_exchange
FAIL: ut_auto_answer fl
FAIL: ut_revoke
FAIL: ut_migration
FAIL: ut_media_negotiation .s
FAIL: ut_recorder re
FAIL: ut_conversation_call /g
FAIL: ut_account_archive xt
FAIL: ut_call
FAIL: ut_plugins
FAIL: ut_fileTransfer
FAIL: ut_conversationRepository
[...] (takes a very long time)The patch at https://review.jami.net/c/jami-daemon/+/22670 has the right idea: separating quick unit tests from more expensive integration tests, but its implementation should be improved, so that we have two distinct check targets (make check for unit tests, make check-integration for integration tests).
Activity
Not much has changed on the latest commit (e246fb2e); here's a fresh run, also interrupted because it takes so ridiculously long (it's been running for more than 2 hours):
Making check in test make[1]: Entering directory '/home/maxim/src/jami/daemon/test' Making check in unitTest make[2]: Entering directory '/home/maxim/src/jami/daemon/test/unitTest' make ut_account_factory ut_account_archive ut_migration ut_certstore ut_scheduler ut_base64 ut_map_utils ut_fileutils ut_utf8_utils ut_string_utils ut_video_input ut_media_encoder ut_media_decoder ut_media_filte r ut_media_player ut_resampler ut_media_frame ut_video_scaler ut_audio_frame_resizer ut_call ut_recorder u t_conference ut_fileTransfer ut_conversationRepository ut_conversation ut_conversation_call ut_media_negot iation ut_hold_resume ut_auto_answer ut_conversationRequest ut_conversationMembersEvent ut_syncHistory ut_ ice ut_revoke ut_ice_sdp_parser ut_ice_media_cand_exchange ut_sip_basic_calls ut_sip_srtp ut_plugins ut_ro uting_table ut_bootstrap ut_swarm_conversation ut_swarm_spread ut_namedirectory make[3]: Entering directory '/home/maxim/src/jami/daemon/test/unitTest' CXX common.o CXX certstore.o CXX scheduler.o CXX media/test_media_frame.o CXX media/video/test_video_scaler.o CXX media/audio/test_audio_frame_resizer.o CXX conversation/conversationRequest.o CXX conversation/conversationMembersEvent.o CXX ice/ice_sdp_parser.o CXX ice/ice_media_cand_exchange.o CXX account_factory/testAccount_factory.o CXX account_archive/account_archive.o CXX account_archive/migration.o CXXLD ut_certstore CXXLD ut_scheduler CXX base64/base64.o CXX map_utils/testMap_utils.o CXX fileutils/testFileutils.o CXX utf8_utils/testUtf8_utils.o CXX string_utils/testString_utils.o CXX media/video/testVideo_input.o CXX media/test_media_encoder.o CXX media/test_media_decoder.o CXX media/test_media_filter.o CXX media/test_media_player.o CXX media/audio/test_resampler.o CXXLD ut_media_frame CXXLD ut_video_scaler CXXLD ut_audio_frame_resizer CXX call/call.o CXX call/recorder.o CXX call/conference.o CXX fileTransfer/fileTransfer.o CXX conversationRepository/conversationRepository.o CXX conversation/conversationcommon.o CXX conversation/conversation.o CXX conversation/call.o CXX media_negotiation/media_negotiation.o CXX media_negotiation/hold_resume.o CXX media_negotiation/auto_answer.o CXXLD ut_conversationRequest CXXLD ut_conversationMembersEvent CXX syncHistory/syncHistory.o CXX ice/ice.o CXX revoke/revoke.o CXXLD ut_ice_sdp_parser CXXLD ut_ice_media_cand_exchange CXX sip_account/sip_basic_calls.o CXX sip_account/sip_srtp.o CXX plugins/plugins.o CXX swarm/routing_table.o CXX swarm/bootstrap.o CXX swarm/swarm_conversation.o CXX swarm/swarm_spread.o CXX namedirectory/namedirectory.o CXXLD ut_account_factory CXXLD ut_account_archive CXXLD ut_migration CXXLD ut_base64 CXXLD ut_map_utils CXXLD ut_fileutils CXXLD ut_utf8_utils CXXLD ut_string_utils CXXLD ut_video_input CXXLD ut_media_encoder CXXLD ut_media_decoder CXXLD ut_media_filter CXXLD ut_media_player CXXLD ut_resampler CXXLD ut_call CXXLD ut_recorder CXXLD ut_conference CXXLD ut_fileTransfer CXXLD ut_conversationRepository CXXLD ut_conversation CXXLD ut_conversation_call CXXLD ut_media_negotiation CXXLD ut_hold_resume CXXLD ut_auto_answer CXXLD ut_syncHistory CXXLD ut_ice CXXLD ut_revoke CXXLD ut_sip_basic_calls CXXLD ut_sip_srtp CXXLD ut_plugins CXXLD ut_routing_table CXXLD ut_bootstrap CXXLD ut_swarm_conversation CXXLD ut_swarm_spread CXXLD ut_namedirectory make[3]: Leaving directory '/home/maxim/src/jami/daemon/test/unitTest' make check-TESTS make[3]: Entering directory '/home/maxim/src/jami/daemon/test/unitTest' make[4]: Entering directory '/home/maxim/src/jami/daemon/test/unitTest' RUNNING: ut_migration RUNNING: ut_account_factory RUNNING: ut_account_archive RUNNING: ut_certstore FAIL: ut_account_factory RUNNING: ut_scheduler FAIL: ut_scheduler RUNNING: ut_base64 PASS: ut_base64 RUNNING: ut_map_utils PASS: ut_map_utils RUNNING: ut_fileutils PASS: ut_fileutils RUNNING: ut_utf8_utils PASS: ut_utf8_utils RUNNING: ut_string_utils PASS: ut_string_utils RUNNING: ut_video_input PASS: ut_video_input RUNNING: ut_media_encoder PASS: ut_media_encoder RUNNING: ut_media_decoder PASS: ut_media_decoder RUNNING: ut_media_filter PASS: ut_media_filter RUNNING: ut_media_player ../../build-aux/test-driver: line 112: 6038 Segmentation fault "$@" >> "$log_file" 2>&1 FAIL: ut_media_player RUNNING: ut_resampler PASS: ut_resampler RUNNING: ut_media_frame PASS: ut_media_frame RUNNING: ut_video_scaler PASS: ut_video_scaler RUNNING: ut_audio_frame_resizer PASS: ut_audio_frame_resizer RUNNING: ut_call FAIL: ut_migration RUNNING: ut_recorder FAIL: ut_certstore RUNNING: ut_conference FAIL: ut_recorder RUNNING: ut_fileTransfer FAIL: ut_account_archive RUNNING: ut_conversationRepository FAIL: ut_call RUNNING: ut_conversation FAIL: ut_conversationRepository RUNNING: ut_conversation_call FAIL: ut_fileTransfer RUNNING: ut_media_negotiation FAIL: ut_media_negotiation RUNNING: ut_hold_resume FAIL: ut_conversation_call RUNNING: ut_auto_answer FAIL: ut_hold_resume RUNNING: ut_conversationRequest FAIL: ut_auto_answer RUNNING: ut_conversationMembersEvent FAIL: ut_conference RUNNING: ut_syncHistory FAIL: ut_conversationRequest RUNNING: ut_ice FAIL: ut_ice RUNNING: ut_revoke FAIL: ut_revoke RUNNING: ut_ice_sdp_parser PASS: ut_ice_sdp_parser RUNNING: ut_ice_media_cand_exchange FAIL: ut_conversation RUNNING: ut_sip_basic_calls PASS: ut_sip_basic_calls RUNNING: ut_sip_srtp PASS: ut_sip_srtp RUNNING: ut_plugins FAIL: ut_ice_media_cand_exchange RUNNING: ut_routing_table FAIL: ut_syncHistory RUNNING: ut_bootstrap PASS: ut_routing_table RUNNING: ut_swarm_conversation FAIL: ut_bootstrap RUNNING: ut_swarm_spread PASS: ut_swarm_spread RUNNING: ut_namedirectory FAIL: ut_conversationMembersEvent FAIL: ut_plugins FAIL: ut_namedirectory ^C (out of patience)
Latest master is ok for tests.
If a test fails, the code must be fixed.
Jami is complex and got a lot of scenarios to test and this is taking time. Some other projects do the same (cf Firefox if you run the 85k tests they have)
The main goal of the tests here is to give the feedback to the devs that everything is working. So we ran the tests everyday at 2am (it takes 2h36min to run). If something is red, devs must work on it.
Partially run the test is not really useful
-
At least for downstream users such as distributions, being able to run a test suite to ensure that integration of system components works together for a given software is very valuable.
Many projects have test flags associated with long tests, and are disabled by default. One such example is QEMU:
project('qemu', ['c'], meson_version: '>=0.63.0', default_options: ['warning_level=1', 'c_std=gnu11', 'cpp_std=gnu++11', 'b_colorout=auto', 'b_staticpic=false', 'stdsplit=false', 'optimization=2', 'b_pie=true'], version: files('VERSION')) add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true) add_test_setup('slow', exclude_suites: ['thorough'], env: ['G_TEST_SLOW=1', 'SPEED=slow']) add_test_setup('thorough', env: ['G_TEST_SLOW=1', 'SPEED=thorough'])The quick suite is the default one, which makes sense in an iterative development context (where tests should be fast otherwise no one will run it). The extended test suite can be easily turned on by setting the
SPEED=slowenvironment variable. -
I've disabled the tests for the
libjamipackage in Guix, given the original patch https://review.jami.net/c/jami-daemon/+/22670 is stale and doesn't seem to have support from jami devs.It's a bit sad in my opinion because great efforts had been spent to enable running this test suite in the Guix packaging, and it was providing some extra coverage for free to the Jami project.
Please register or sign in to reply