AirSLAM: An Efficient and Illumination-Robust Point-Line Visual SLAM System

Several handcrafted methods have been proposed to improve the robustness of VO and vSLAM to challenging illumination. DSO [52] models brightness changes and jointly optimizes camera poses and photometric parameters. DRMS [11] and AFE-ORB-SLAM [12] utilize various image enhancements. Some systems try different methods, such as ZNCC, the locally-scaled sum of squared differences (LSSD), and dense descriptor computation, to achieve robust tracking [14, 53, 15]. These methods mainly focus on either global or local illumination change for all kinds of images, however, lighting conditions often affect the scene differently in different areas [16]. Other related methods include that of Huang and Liu [54], which presents a multi-feature extraction algorithm to extract two kinds of image features when a single-feature algorithm fails to extract enough feature points. Kim et al. [55] employ a patch-based affine illumination model during direct motion estimation. Chen et al. [56] minimize the normalized information distance with nonlinear least square optimization for image registration. Alismail et al. [57] propose a binary feature descriptor using a descriptor assumption to avoid brightness constancy.

Compared with handcrafted methods, learning-based methods have shown better performance. Savinykh et al. [8] propose DarkSLAM, where Generative Adversarial Network (GAN) [58] is used to enhance input images. Pratap Singh et al. [59] compare different learning-based image enhancement methods for vSLAM in low-light environments. TartanVO [17], DROID-SLAM [18], and iSLAM [19] train their VO or SLAM networks on the TartanAir dataset [60], which is a large simulation dataset that contains various lighting conditions, therefore, they are very robust in challenging environments. However, they usually require good GPUs and long training times. Besides, DROID-SLAM runs very slowly and is difficult to apply to real-time applications on resource-constrained platforms. TartanVO and iSLAM are more efficient, but they cannot achieve performance as accurately as traditional vSLAM systems.

Currently, most SLAM systems still use the bag of words (BoW) [61] for loop closure detection and relocalization due to its good balance between efficiency and effectiveness [62, 4, 63]. To make the relocalization more robust to large illumination variations, Labbé et al. [20] propose the multi-session relocalization method, where they combine multiple maps generated at different times and in various illumination conditions. DXSLAM [37] uses NetVLAD [64] for the coarse image retrieval and SuperPoint with a binary descriptor for keypoint matching between the query frame and candidates.

Another similar task in the robotics and computer vision communities is the visual place recognition (VPR) problem, where many researchers handle the localization problem with image retrieval methods [64, 65]. These VPR solutions try to find images most similar to the query image from a database. They usually cannot directly provide accurate pose estimation which is needed in robot applications. Sarlin et al. address this and propose Hloc [9]. They use a global retrieval to obtain several candidates and match local features within those candidates. The Hloc toolbox has integrated many image retrieval methods, local feature extractors, and matching methods, and it is currently the SOTA system. Yan et al. [66] propose a long-term visual localization method for mobile platforms, however, they rely on other sensors, e.g., GPS, compass, and gravity sensor, for the coarse location retrieval.

In this paper, $\mathbb{R}$ represents the set of real numbers, and $\mathbb{R}^{m}$ denotes the $m$-dimensional real vector space. The transpose of a vector or matrix is written as $(\cdot)^{\top}$. For a vector $\mathbf{x}\in\mathbb{R}^{m}$, $\lVert\mathbf{x}\rVert$ represents its Euclidean norm, while $\lVert\mathbf{x}\rVert^{2}_{\Sigma}$ is shorthand for $\mathbf{x}^{\top}\Sigma\mathbf{x}$. The transformation, rotation, and translation from the $b$-coordinate system to the $a$-coordinate system are denoted by $\mathbf{T}_{ab}\in\text{SE(3)}$, $\mathbf{R}_{ab}\in\text{SO(3)}$, and $\mathbf{t}_{ab}\in\mathbb{R}^{3}$, respectively. We use $(\cdot)_{c}$ to indicate a vector in the camera frame and $(\cdot)_{w}$ to indicate a vector in the global world frame. Throughout the paper, super/subscripts may be omitted for brevity, as long as the meaning remains unambiguous within the given context.

We believe that a practical vSLAM system should possess the following features:

• •

  High efficiency. The system should have real-time performance on resource-constrained platforms.

• •

  Scalability. The system should be easily extensible for various purposes and real-world applications.

• •

  Easy to deploy. The system should be easy to deploy on real robots and capable of achieving robust localization.

Therefore, we design a system as shown in Fig. 1. The proposed system is a hybrid system as we need the robustness of data-driven approaches and the accuracy of geometric methods. It consists of three main components: stereo VO/VIO, offline map optimization, and lightweight relocalization. (1) Stereo VO/VIO: We propose a point-line-based visual odometry that can handle both stereo and stereo-inertial inputs. (2) Offline map optimization: We implement several commonly used plugins, such as loop detection, pose graph optimization, and global bundle adjustment. The system is easily extensible for other map-processing purposes by adding customized plugins. For example, we have implemented a plugin to train a scene-dependent junction vocabulary using the endpoints of line features, which is utilized in our lightweight multi-stage relocalization. (3) Lightweight relocalization: We propose a multi-stage relocalization method that improves efficiency while maintaining effectiveness. In the first stage, keypoints and line features are detected using the proposed PLNet, and several candidates are retrieved using a keypoint vocabulary trained on a large dataset. In the second stage, most false candidates are quickly filtered out using a scene-dependent junction vocabulary and a structure graph. In the third stage, feature matching is performed between the query frame and the remaining candidates to find the best match and estimate the pose of the query frame. Since feature matching in the third stage is typically time-consuming, the filtering process in the second stage enhances the efficiency of our system compared to other two-stage relocalization systems.

We transfer some time-consuming processes, e.g., loop closure detection, pose graph optimization, and global bundle adjustment, to the offline stage. This improves the efficiency of our online mapping module. In many practical applications, such as warehouse robotics, a map is typically built by one robot and then reused by others. Our system is designed with these applications in mind. The lightweight mapping and map reuse modules can be easily deployed on resource-constrained robots, while the offline optimization module can run on a more powerful computer for various map manipulations, such as map editing and visualization. The mapping robot uploads the initial map to the computer, which then distributes the optimized map to other robots, ensuring drift-free relocalization. In the following sections, we introduce our feature detection and visual odometry (VO) pipeline in Section IV and Section V, respectively. The offline optimization and relocalization modules are presented in Section VI.

With advancements in deep learning technology, learning-based feature detection methods have demonstrated more stable performance in illumination-challenging environments compared to traditional methods. However, existing point-line-based VO/VIO and SLAM systems typically detect keypoints and line features separately. While it is acceptable for handcrafted methods due to their efficiency, the simultaneous application of keypoint detection and line detection networks in VO/VIO or SLAM systems, especially in stereo configurations, often hinders real-time performance on resource-constrained platforms. Consequently, we aim to design an efficient unified model that can detect keypoints and line features concurrently.

However, achieving a unified model for keypoint and line detection is challenging, as these tasks typically require different real-image datasets and training procedures. Keypoint detection models are generally trained on large datasets comprising diverse images and depend on either a boosting step or the correspondences of image pairs for training [24, 40, 39]. For line detection, we find wireframe parsing methods [67, 51] can provide stronger geometric cues than the self-supervised models [49, 68] as they are able to detect longer and more complete lines. The wireframe includes all prominent straight lines and their junctions within the scene, providing an efficient and accurate representation of large-scale geometry and object shapes [69]. However, these methods are trained on the Wireframe dataset [69], which is limited in size with only 5,462 discontinuous images. In the following sections, we will address this challenge and demonstrate how to train a unified model capable of performing both tasks. It is important to note that in this paper, the term “line detection” refers specifically to the wireframe parsing task.

As shown in Fig. 2, we have two findings when visualizing the results of the keypoint and line detection networks: (1) Most junctions (endpoints of lines) detected by the line detection model are also selected as keypoints by the keypoint detection model. (2) The feature maps outputted by the keypoint detection model contain the edge information. Therefore, we argue that a line detection model can be built on the backbone of a pre-trained keypoint detection model. Based on this assumption, we design the PLNet to detect keypoints and lines in a unified framework. As shown in Fig. 3, it consists of the shared backbone, the keypoint module, and the line module.

Backbone: We follow SuperPoint [24] to design the backbone for its good efficiency and effectiveness. It uses 8 convolutional layers and 3 max-pooling layers. The input is the grayscale image sized $H\times W$. The outputs are $H\times W\times 64$, $\frac{H}{2}\times\frac{W}{2}\times 64$, $\frac{H}{4}\times\frac{W}{4}\times 128$, $\frac{H}{8}\times\frac{W}{8}\times 128$ feature maps.

Keypoint Module: We also follow SuperPoint [24] to design the keypoint detection header. It has two branches: the score branch and the descriptor branch. The inputs are $\frac{H}{8}\times\frac{W}{8}\times 128$ feature maps outputted by the backbone. The score branch outputs a tensor sized $\frac{H}{8}\times\frac{W}{8}\times 65$. The 65 channels correspond to an $8\times 8$ grid region and a dustbin indicating no keypoint. The tensor is processed by a softmax and then resized to $H\times W$. The descriptor branch outputs a tensor sized $\frac{H}{8}\times\frac{W}{8}\times 256$, which is used for interpolation to compute descriptors of keypoints.

Line Module: This module takes feature maps from the backbone as inputs. It consists of a U-Net-like CNN and the line detection header. We modify the U-Net [70] to make it contain fewer convolutional layers and thus be more efficient. The U-Net-like CNN is to increase the receptive field as detecting lines requires a larger receptive field than detecting keypoints. The EPD LOIAlign [51] is used to process the outputs of the line module and finally outputs junctions and lines.

Due to the training problem described in Section IV-A and the assumption in Section IV-B, we train our PLNet in two rounds. In the first round, only the backbone and the keypoint detection module are trained, which means we need to train a keypoint detection network. In the second round, the backbone and the keypoint detection module are fixed, and we only train the line detection module on the Wireframe dataset. We skip the details of the first round as they are very similar to [24]. Instead, we present the training of the line detection module.

Line Encoding: We adopt the attraction region field [51] to encode line segments. As shown in Fig. 4, for a line segment $\mathbf{l}=(\mathbf{x_{1}},\mathbf{x_{2}})$, where $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ are two endpoints of $\mathbf{l}$, and a point $\mathbf{p}$ in the attraction region of $\mathbf{l}$, four parameters and $\mathbf{p}$ are used to encode $\mathbf{l}$:

where $d$ is the distance from $\mathbf{p}$ to $\mathbf{l}$, $\theta$ is the angle between $\mathbf{l}$ and the $\mathbf{v}$-axis of the image, $\theta_{1}$ is the angle between $\mathbf{px_{1}}$ and the perpendicular line from $\mathbf{p}$ to $\mathbf{l}$, and $\theta_{2}$ is the angle between $\mathbf{px_{2}}$ and the perpendicular line. The network can predict these four parameters for point $\mathbf{p}$ and then $\mathbf{l}$ can be decoded through:

Line Prediction: The line detection module outputs a tensor sized $\frac{H}{4}\times\frac{W}{4}\times 4$ to predict parameters in (1) and a heatmap to predict junctions. For each decoded line segment by (2), two junctions closest to its endpoints will be selected to form a line proposal with it. Proposals with the same junctions will be deduplicated and only one is retained. Then the EPD LOIAlign [51] and a head classifier are applied to decide whether the line proposal is a true line feature.

Line Module Training: We use the $L1$ loss to supervise the prediction of parameters in (1) and the binary cross-entropy loss to supervise the junction heatmap and the head classifier. The total loss is the sum of them. As shown in Fig. 5, to improve the robustness of line detection in illumination-challenging environments, seven types of photometric data augmentation are applied to process training images. The training uses the ADAM optimizer [71] with the learning rate $lr=4e\text{-}4$ in the first 35 epochs and $lr=4e\text{-}5$ in the last 5 epochs.

The proposed point-line-based stereo visual odometry is shown in Fig. 6. It is a hybrid VO system utilizing both the learning-based front-end and the traditional optimization backend. For each stereo image pair, we first employ the proposed PLNet to extract keypoints and line features. Then a GNN (LightGlue [27]) is used to match keypoints. In parallel, we associate line features with keypoints and match them using the keypoint matching results. After that, we perform an initial pose estimation and reject outliers. Based on the results, we triangulate the 2D features of keyframes and insert them into the map. Finally, the local bundle adjustment will be performed to optimize points, lines, and keyframe poses. In the meantime, if an IMU is accessible, its measurements will be processed using the IMU preintegration method [72], and added to the initial pose estimation and local bundle adjustment.

Applying both learning-based feature detection and matching methods to the stereo VO is time-consuming. Therefore, to improve efficiency, the following three techniques are utilized in our system. (1) For keyframes, we extract features on both left and right images and perform stereo matching to estimate the real scale. But for non-keyframes, we only process the left image. Besides, we use some lenient criteria to make the selected keyframes in our system very sparse, so the runtime and resource consumption of feature detection and matching in our system are close to that of a monocular system. (2) We convert the inference code of the CNN and GNN from Python to C++, and deploy them using ONNX and NVIDIA TensorRT, where the 16-bit floating-point arithmetic replaces the 32-bit floating-point arithmetic. (3) We design a multi-thread pipeline. A producer-consumer model is used to split the system into two main threads, i.e., the front-end thread and the backend thread. The front-end thread extracts and matches features while the backend thread performs the initial pose estimation, keyframe insertion, and local bundle adjustment.

We use LightGlue [27] to match keypoints. For line features, most of the current VO and SLAM systems use the LBD algorithm [73] or tracking sample points to match them. However, the LBD algorithm extracts the descriptor from a local band region of the line, so it suffers from unstable line detection due to challenging illumination or viewpoint changes. Tracking sample points can match the line detected with different lengths in two frames, but current SLAM systems usually use optical flow to track the sample points, which have a bad performance when the light conditions change rapidly or violently. Some learning-based line feature descriptors [49] are also proposed, however, they are rarely used in current SLAM systems due to the increased time complexity.

Therefore, to address both the effectiveness problem and efficiency problem, we design a fast and robust line-matching method for illumination-challenging conditions. First, we associate keypoints with line segments through their distances. Assume that $M$ keypoints and $N$ line segments are detected on the image, where the $i$-th keypoint is denoted as $\mathbf{p}_{i}=(x_{i},y_{i})$ and the $j$-th line segment is denoted as $\mathbf{l}_{j}=(A_{j},B_{j},C_{j},x_{j,1},y_{j,1},x_{j,2},y_{j,2})$, where $(A_{j},B_{j},C_{j})$ are line parameters of $\mathbf{l}_{j}$ and $(x_{j,1},y_{j,1},x_{j,2},y_{j,2})$ are the endpoints. We first compute the distance between $\mathbf{p}_{i}$ and $\mathbf{l}_{j}$ through:

If $d_{ij}<3$ and the projection of $\mathbf{p}_{i}$ on the coordinate axis lies within the projections of line segment endpoints, i.e., $\min({x}_{j,1},{x}_{j,2})\leq x_{i}\leq\max({x}_{j,1},{x}_{j,2})$ or $\min({y}_{j,1},{y}_{j,2})\leq{y}_{i}\leq\max({y}_{j,1},{y}_{j,2})$, we will say $\mathbf{p}_{i}$ belongs to $\mathbf{l}_{j}$. Then the line segments on two images can be matched based on the point-matching result of these two images. For ${\mathbf{l}_{k,m}}$ on image $k$ and ${\mathbf{l}_{k+1,n}}$ on image $k+1$, we compute a score ${S}_{mn}$ to represent the confidence of that they are the same line:

where ${N}_{pm}$ is the matching number between point features belonging to ${\mathbf{l}_{k,m}}$ and point features belonging to ${\mathbf{l}_{k+1,n}}$. ${{N}_{k,m}}$ and ${{N}_{k+1,n}}$ are the numbers of point features belonging to ${\mathbf{l}_{k,m}}$ and ${\mathbf{l}_{k+1,n}}$, respectively. Then if ${S}_{mn}>\delta_{S}$ and ${N}_{pm}>\delta_{N}$, where $\delta_{S}$ and $\delta_{N}$ are two preset thresholds, we will regard ${\mathbf{l}_{k,m}}$ and ${\mathbf{l}_{k+1,n}}$ as the same line. This coupled feature matching method allows our line matching to share the robust performance of keypoint matching while being highly efficient due to that it does not need another line-matching network.

In this part, we will introduce our 3D feature processing methods, including 3D feature representation, triangulation, i.e., constructing 3D features from 2D features, and re-projection, i.e., projecting 3D features to the image plane. For 3D point processing, a 3D point is denoted as $\mathbf{X}\in\mathbb{R}^{3}$. LightGlue [27] is utilized to match keypoints between the left and right images. Successfully matched keypoints are triangulated using stereo disparity information, while unmatched keypoints are triangulated leveraging multi-frame observations. The projection of 3D points onto the image plane is modeled using either the pinhole camera model or the fisheye camera model, depending on the specific camera in use. We skip the details of 3D point processing in our system as they are similar to other point-based VO and SLAM systems [74, 4]. On the contrary, compared with 3D points, 3D lines have more degrees of freedom, and they are easier to degenerate when being triangulated. Therefore, the 3D line processing will be illustrated in detail.

Observing that the learning-based data association method used in our system is able to track two frames that have a large baseline, so different from the frame-by-frame tracking strategy used in other VO or SLAM systems, we only match the current frame with the last keyframe. We argue this strategy can reduce the accumulated tracking error.

Therefore, the keyframe selection is essential for our system. On the one hand, as described in Section V-A, we want to make keyframes sparse to reduce the consumption of computational resources. On the other hand, the sparser the keyframes, the more likely tracking failure happens. To balance the efficiency and the tracking robustness, a frame will be selected as a keyframe if any of the following conditions is satisfied:

• •

  The tracked features are less than $\alpha_{1}\cdot N_{s}$.

• •

  The average parallax of tracked features between the current frame and the last keyframe is larger than $\alpha_{2}\cdot\sqrt{WH}$.

• •

  The number of tracked features is less than $N_{kf}$.

In the above, $\alpha_{1}$, $\alpha_{2}$, and $N_{kf}$ are all preset thresholds. $N_{s}$ is the number of detected features. $W$ and $H$ respectively represent the width and height of the input image.

To improve the accuracy, we perform the local bundle adjustment when a new keyframe is inserted. $N_{o}$ latest neighboring keyframes are selected to construct a local graph, where map points, 3D lines, and keyframes are vertices and pose constraints are edges. We use point constraints and line constraints as well as IMU constraints if an IMU is accessible. Their related error terms are defined as follows.

As described in Section III-B, our map is optimized offline. Therefore, keyframes, map points, and 3D lines will be saved to the disk for subsequent optimization when the visual odometry is finished. For each keyframe, we save its index, pose, keypoints, keypoint descriptors, line features, and junctions. The correspondences between 2D features and 3D features are also recorded. To make the map faster to save, load, and transfer across different devices, the above information is stored in binary form, which also makes the initial map much smaller than the raw data. For example, on the OIVIO dataset [79], our initial map size is only about 2% of the raw data size.

This part aims to process an initial map generated by our VO module and outputs the optimized map that can be used for drift-free relocalization. Our offline map optimization module consists of the following several map-processing plugins.

In this part, we present our illumination-robust relocalization using an existing optimized map. In most vSLAM systems, recognizing revisited places typically needs two steps: (1) retrieving $N_{kc}$ keyframe candidates and (2) performing feature matching and estimating relative pose. The second step is usually time-consuming, so selecting a proper $N_{kc}$ is very important. A larger $N_{kc}$ will reduce the system’s efficiency while a smaller $N_{kc}$ may prevent the correct candidate from being recalled. For example, in the loop closing module of ORB-SLAM3 [62], only the three most similar keyframes retrieved by DBoW2 [84] are used for better efficiency. It works well as two frames in a loop pair usually have a short time interval and thus the lighting conditions are relatively similar. But for challenging tasks, such as the day/night relocalization problem, retrieving so few candidates usually results in a low recall rate. However, retrieving more candidates needs to perform feature matching and pose estimation more times for each query frame, which makes it difficult to deploy for real-time applications.

To address this problem, we propose an efficient multi-stage relocalization method to make the optimized map usable in different lighting conditions. Our insight is that if most of the false candidates can be quickly filtered out, then the efficiency can be improved while maintaining or even improving the relocalization recall rate. Therefore, we add another step to the two-step pipeline mentioned above. We next introduce the proposed multi-stage pipeline in detail.

In this section, we evaluate the performance of our PLNet. As described in Section IV-B, we follow SuperPoint [24] to design and train our backbone and keypoint detection module, and we can even use the pre-trained model of SuperPoint, therefore, we do not evaluate the keypoint detection anymore. Instead, we assess the performance of the line detection module by comparing it with SOTA systems, as it is trained with a fixed backbone, which is different from other line detectors.