Hi Mathieu and Kemal,
I am having the same issue. If I map a space using G2O or even using TORO but then try to replay the database in rtabmap with G2O, I get an error at loop closure. Here is an example of my odometry message if it helps to debug:
There are some conditions that the matrices should have to be valid. You may look if your covariance matrices respect those conditions. The information matrix is the inverse of the covariance. In rtabmap, the covariance is taken from twist covariance of odometry messages.
A covariance of 46.219113711096966 is quite huge. In general, it should be in order of 10-3 to 10-6 for the linear part. Also, note that non-null diagonal-only covariance matrices should work with g2o.
By careful tuning of robot_localization, see their documentation.
On rtabmap side, we can manually fix a covariance if fixing on robot_localization side is not possible. Set odom_frame_id of rtabmap node and set also odom_tf_angular_variance and odom_tf_linear_variance parameters.