A Lightweight Approach to Understand Forest Roads for New Nnergy Vehicles

3.1. Preprocessing

Review

A Lightweight Approach to Understand Forest Roads for New Nnergy Vehicles

Luping Wang ^1,*, Yuan Feng ¹, Shanshan Wang ², and Hui Wei ³

¹ Laboratory of 3D Scene Understanding and Visual Navigation, School of Mechanical Engineering, University of Shanghai for Science and Technology, No. 516 Jungong Road, Shanghai 200093, China

² Intel Asia-Pacific Research & Development Ltd., No.880 Zixing Road, Shanghai 201100, China

³ Laboratory of Algorithms for Cognitive Models, School of Computer Science, Fudan University, No. 825 Zhangheng Road, Shanghai 201203, China

* Correspondence: 15110240007@fudan.edu.cn

Received: 16 June 2024; Revised: 21 October 2024; Accepted: 24 October 2024; Published: 11 November 2024

Abstract: Scene understanding is a core issue for autonomous vehicles. However, its implementation has been thwarted by various outstanding issues, such as understanding forest roads in unknown field environments. Traditional three-dimensional (3D) point clouds or 3D estimation of fused data consume large amounts of memory and energy, making these methods less reliable in new energy vehicles with limited computational, memory, and energy resources. In this study, we propose a lightweight method to understand forest roads using a low-cost monocular camera. We extracted and clustered spatially similar texture projections based on oblique effect. Through the relative relationship between vanishing points and texture projections, contour lines can be estimated. After that, searching for the corresponding supporting evidence lines, we can segment the surface of the forest road, which can provide a decision basis for the automatic driving control system of new energy vehicles with limited resources. Unlike deep learning methods that are extremely resource-consuming, the proposed method requires no prior training, no calibration, and no internal parameters of the camera. At the same time, pure geometric reasoning makes the method robust to the ever-changing colors and lighting in the forest environment. The percentage of correctly classified pixels is compared to the ground truth. The experimental results show that the method can successfully understand forest roads and meet the requirements of autonomous navigation in forest environments for new energy vehicles with limited resources.

1. Introduction

Scene understanding is a key problem in autonomous driving. It has a multitude of unsolved problems, such as understanding forest roads in complex field environments. Understanding forest roads using a resource-constrained system with limited computation, memory, and energy resources remains a considerable challenge for new energy vehicle.

Monocular camera-based scene understanding systems have advantages in terms of energy consumption, computational efficiency, and cost control over 3D point cloud-based or multi-sensor data fusion approaches. Humans interpret semantic information of a scene through interpretable visual cues in the scene [1,2]. Although it is more difficult for humans to estimate the depth of isolated points in a scene, it is possible to utilize the relative geometric relationship cues in the scene to reason about the structure of the scene, which plays a crucial role in understanding the scene.

Unlike indoor and urban scenes, forest roads are always disturbed by soil, vegetation and weeds. As shown in the Figure 1, the rugged road surfaces are subjected to different lighting, diverse colors and unpredictable vegetation, making most of the edges and surfaces of the forest roads appear fragmented. Nonetheless, their spatial texture (2D projections with different configurations) maintains traceable geometric features in terms of location and orientation, providing a basis for understanding forest roads.

Figure 1. An unstructured forest road disturbed by soil, trees and weeds.

In this study, we present a method for understanding forest roads in a field environment without prior training, using a low-cost monocular camera. First, based on the oblique effect, we extracted clusters of tree texture projections. Then, by analyzing them in relation to vanishing points, the contour lines of forest roads were approximated. Finally, the forest road surfaces can be estimated based on the position and orientation of the corresponding evidence lines.

For the current popular deep learning method, it requires large-scale labeled data for training, and is extremely dependent on a large number of computational and storage resources, such as high-performance graphics processors (GPUs) and large-capacity storage devices and other hardware devices. These consume a lot of energy and are difficult to apply to new energy vehicles with energy constraints. Therefore, unlike machine learning methods, the method presented in this paper does not require prior training and has lower hardware requirements, which has a lower energy consumption. Interpretable geometric inference makes the proposed method robust to unexpected colors, lighting, and materials in forest environments. Understanding forest roads through monocular cameras without precise depth or knowledge of the camera’s intrinsic parameters makes it more practical and affordable.

The percentage of correctly categorized pixels was evaluated by comparing the estimated forest roads with the ground truth. The experimental results demonstrate that the proposed geometric inference-based method is able to understand forest roads, and our method is energy-saving, efficient, lightweight, and has a promising application in the visual navigation of new energy vehicles.

2. Related Work

There are some classical approaches for scene reconstruction such as SFM [3] and visual SLAM [4]. Researchers have proposed a 3D point cloud based tracking model to solve the occlusion problem in large clusters of featureless objects [5]. However, the acquisition and processing of 3D point cloud data is more complex, computationally intensive, and energy consuming, and appears to be impotent when interpreting geometric structures such as edges, textures, and spatial planes.

A great deal of research has been done to understand structured environments such as indoor or urban scenes. Significant progress has been made in areas such as indoor spatial layout and scene details through monocular cameras [6,7]. Research has also been done to propose a framework for detecting and segmenting puddles on roads in unstructured environments from dashboard camera images [8]. Some geometric models were developed to detect walkable surfaces, but these models were not validated in field environments [9–11]. Real-time pixel semantic classification and segmentation using hyperspectral images is proposed to extract, filter and approximate polygonal objects [12]. The problem of spatial layout in outdoor environments is solved by projecting at right angles to space [13,14]. However, these scene-oriented structured methods cannot work in field road environments due to their limited interpretation of natural features.

For unstructured road environments, data-driven algorithms have been adopted and extended for object recognition and semantic segmentation [15–17]. For semantic scene segmentation in unstructured environments, the researcher proposes a lightweight segmentation model improved in the DeepLabv3+ framework [18]. In addition, the researchers proposed a terrain classification framework for pixel semantic segmentation of images using an unsupervised proprioceptive classifier learned from vehicle-terrain interaction sounds [19]. However, these data-driven models do not take into account unstructured road surfaces in forest scenes well.

Some scholars have studied obstacle avoidance and traversable regions for autonomous navigation. Based on the shape and attitude of obstacles, a pure geometric model was developed to avoid obstacles [20]. Traversability maps for unstructured environments were obtained and the optimal paths were estimated based on the specific characteristics of the robot [21,22]. Another approach uses irregular angle projections to model unstructured paths in field environments but cannot handle forest road without angle projections [23,24]. Most of these methods focus only on categorizing traversable or non-traversable areas and therefore cannot understand forest roads.

For natural scenes, a method for detecting vanishing points is proposed that combines contour-based vanishing point edges [25]. Some researchers have also proposed a dense semantic segmentation algorithm based on resolution pyramid and heterogeneous feature fusion [26]. There have been attempts to perform semantic segmentation in unstructured scenes by stereo visual odometry [27]. In order to cope with changes in the environmental terrain, an ensemble category semantic segmentation method has been proposed to obtain different categories, such as sky and obstacles in off-road environments [28]. However, these methods lack edge resolution and are difficult to apply to forest roads.

Since most methods are designed for scene understanding of structured paths and require prior training, it is necessary to propose an energy-saving, efficient, and lightweight method that is able to understand unstructured paths in forest environments by means of pure geometric modeling without the need for prior training and external equipment, so as to satisfy the requirements of resource-constrained new energy vehicles to achieve autonomous navigation in field environments.

3. Understanding a Forest Road

In the case of forest roads, the surface, although always fragmented, still has a particular spatial texture. These textures are projected as two-dimensional projections with different shapes that can be considered as visual cues. By analyzing the position and orientation of these texture projections, their spatial morphology can be inferred, thus contributing to the understanding of such forest roads.

First, lines and vanishing points (VPs) are detected [29,30] as follows:

$lin{e}_i=\left[p_{\mathrm{1}}^i,p_{\mathrm{2}}^i\right]=\left[x_{\mathrm{1}}^i,y_{\mathrm{1}}^i,x_{\mathrm{2}}^i,y_{\mathrm{2}}^i\right];i\in \left[\mathrm{1},\mathbb{N}\right],$ (1)

$VP=\left[P_{\mathrm{1}};P_{\mathrm{2}};P_{\mathrm{3}};\right]$ (2)

where $\mathbb{N}$ is the number of lines. P₁, P₂ and P₃ are layout vanishing points.

3.2. Tree Trunk Texture Projection

Inspired by the oblique effect, neurons prefer to respond to horizontal and vertical stimuli more than to diagonal. Therefore, trees and plant bodies, which tend to grow vertically or horizontally, help estimate road layouts in field forest environments. A pair of tree bodies texture projection can be represented by a composition of two lines with geometric constraints, which can be defined as follows:

$\mathbb{T}=\{lin{e}_w,lin{e}_e\};w,e\in \left[\mathrm{1},\mathbb{N}\right]$ (3)

$d_{w,e}=distance\left(p_m^w,p_m^e\right)$ (4)

$St.min{d}_{w,e};min\mathrm{\Theta }\left(lin{e}_w,lin{e}_e\right)$ (5)

Here $\mathbb{T}$ is a pair of tree body projection that includes two lines line_w and line_e. Here, d_w,e is the Euclidean distance between two midpoints ($p_m^w=\left[\begin{array}{c}{x}_m^w,y_m^w\end{array}\right]$ and $p_m^e=\left[\begin{array}{c}{x}_m^e,y_m^e\end{array}\right]$) of line_w and line_e, respectively. A smaller distance means that the two lines are closer together, indicating better integrity of the two lines. Θ is a function to compute the angle of these two lines. Smaller angle values indicate that the two lines are more similar in orientation. The smaller the distance and angle values of the two lines indicate that this combination is more likely to be a two-dimensional projection of the spatial trunk texture.

A tree trunk texture projection has two lines. Through corresponding positions and orientations, it is possible to assign them to different clusters. Assuming that $\mathbb{Y}\left(\mathbb{T}\right)$ is a function to cluster tree trunk texture projection:

$\mathbb{Y}\left(\mathbb{T}\right)=\left\{\begin{array}{cc}\mathrm{1},& \mathrm{m}\mathrm{i}\mathrm{n}\left(x_m^w,x_m^e\right)>x_{P_{\mathrm{1}}},{\beta }_w\to \pi /\mathrm{2},{\beta }_e\to \pi /\mathrm{2}\\ \mathrm{2},& \mathrm{m}\mathrm{a}\mathrm{x}\left(x_m^w,x_m^e\right)<x_{P_{\mathrm{1}}},{\beta }_w\to \pi /\mathrm{2},{\beta }_e\to \pi /\mathrm{2}\end{array}\right.$ (6)

It indicates that a tree trunk texture projection can be assigned to different clusters, which are written as ${\mathbb{Y}}^k,k\in \mathrm{1,2}$. Accordingly, corresponding clusters can be extracted, as shown in Figure 2.

Figure 2. Preprocessing and tree texture projections. Left: input image. Second: lines. Third and Fourth: clusters of ${\mathbb{Y}}^{\mathrm{1}}$ and ${\mathbb{Y}}^{\mathrm{2}}$, respectively.

3.3. Contour Layout Lines

Because a forest road is always covered by unexpected grass, vegetation or soil, its shape appears fragmentary and irregular. Therefore, it is necessary to estimate contour layout lines that can coarsely infer the road.

Given a classified ${\mathbb{Y}}^k=\left[lin{e}_w^k,lin{e}_e^k\right]$, $p_b^{k,w}$ and $p_b^{k,e}$ means the bottom points in corresponding $lin{e}_w^k$ and $lin{e}_e^k$, respectively. The following can be modelled:

${\eta }_b^{k,w}=\mathrm{\Gamma }\left(\left[P_{\mathrm{1}},p_b^{k,w}\right],\left[{\sigma }_{min},{\sigma }_{max}\right]\right)$ (7)

${\eta }_b^{k,e}=\mathrm{\Gamma }\left(\left[P_{\mathrm{1}},p_b^{k,e}\right],\left[{\sigma }_{min},{\sigma }_{max}\right]\right)$ (8)

Here $\mathrm{\Gamma }$ is a function to find the point of intersection of two lines. ${\sigma }_{min}=\left[-W/\mathrm{2},-H/\mathrm{2}\right],$ and ${\sigma }_{min}=\left[W/\mathrm{2},-H/\mathrm{2}\right]$, in which W and H are the height and width of a monocular capture. ${\eta }_b^{k,w}$ means an intersection point of two lines, which are represented by $\left[P_{\mathrm{1}},p_b^{k,w}\right]$ and $\left[{\sigma }_{min},{\sigma }_{max}\right]$.

For all lines in cluster k = 1, the corresponding points ${\eta }_b^{k,w}$ and ${\eta }_b^{k,e}$ are computed, as shown in Figure 3 (red points in left and middle images). Then these point that has the minimum x value is determined and written as ${\eta }_b^{\mathrm{*}k=\mathrm{1}}$. This point and P₁ can shape a contour line. Similarly, for cluster k = 2, the point that has the maximal x value is described as ${\eta }_b^{\mathrm{*}k=\mathrm{2}}$, which can shape another contour line. Accordingly, two virtual lines can be modelled as following:

$\mathbb{U}=\{{\mathrm{\Psi }}^{\mathrm{1}},{\mathrm{\Psi }}^{\mathrm{2}}\};$ (9)

${\mathrm{\Psi }}^{\mathrm{1}}=\left[{\eta }_b^{\mathrm{*}k=\mathrm{1}},P_{\mathrm{1}}\right]$ (10)

${\mathrm{\Psi }}^{\mathrm{2}}=\left[{\eta }_b^{\mathrm{*}k=\mathrm{2}},P_{\mathrm{1}}\right]$ (11)

Figure 3. Contour layout lines. Left: ${\mathrm{\Psi }}^{\mathrm{1}}$. Middle: ${\mathrm{\Psi }}^{\mathrm{2}}$. Right: layout lines. Contour layout lines of a forest road can be described as two virtual lines ${\mathrm{\Psi }}^{\mathrm{1}}$ and ${\mathrm{\Psi }}^{\mathrm{2}}$.

Here $\mathbb{U}$ represents the layout lines of a forest road. ${\eta }_b^{\mathrm{*}k=\mathrm{1}}$ represents the point has the minimum x value in ${\eta }_b^{k=\mathrm{1},w}$ and ${\eta }_b^{k=\mathrm{1},e}$. Similarly, ${\eta }_b^{\mathrm{*}k=\mathrm{2}}$ represents the point has the minimum x value in ${\eta }_b^{k=\mathrm{2},w}$ and ${\eta }_b^{k=\mathrm{2},e}$. These two lines ${\mathrm{\Psi }}^{\mathrm{1}}$ and ${\mathrm{\Psi }}^{\mathrm{2}}$ can be seen as two contour lines of the road, as shown in Figure 3 (pink lines).

3.4. Evidence Surfaces

For ${\mathrm{\Psi }}^{\mathrm{1}}$, in order to determine whether there are other lines following these two lines, for $lin{e}_i$ ($j\in \mathbb{N}$), the condition can be founded:

$\mathbb{E}=\left\{lin{e}_j\right\};j\in \mathbb{N}$ (12)

$s.t.MIN\left({\delta }_j^{\mathrm{1}}\right);MIN\left({\delta }_j^{\mathrm{2}}\right);$ (13)

${\delta }_j^{\mathrm{1}}=‖\mathrm{\Theta }\left(lin{e}_j,{\mathrm{\Psi }}^{\mathrm{1}}\right)‖+‖p_m^j-{\mathrm{\Psi }}^{\mathrm{1}}‖$ (14)

${\delta }_j^{\mathrm{2}}=‖\mathrm{\Theta }\left(lin{e}_j,{\mathrm{\Psi }}^{\mathrm{2}}\right)‖+‖p_m^j-{\mathrm{\Psi }}^{\mathrm{2}}‖$ (15)

Here, $\mathbb{E}$ is a set of evidence lines for layout lines. ${\delta }_j^{\mathrm{1}}$ and ${\delta }_j^{\mathrm{2}}$ are relative relationships between $lin{e}_j$, ${\mathrm{\Psi }}^{\mathrm{1}}$ and ${\mathrm{\Psi }}^{\mathrm{2}}$, respectively. Θ is a function computing angle of two lines, and the angle representing orientation constraint. Smaller value of the angle means that $lin{e}_j$ is more likely to have similar orientation to ${\mathrm{\Psi }}^{\mathrm{1}}$. $p_m^j$ is midpoint of $lin{e}_j$. $‖p_m^j-{\mathrm{\Psi }}^{\mathrm{1}}‖$ is the distance from $p_m^j$ to ${\mathrm{\Psi }}^{\mathrm{1}}$. Smaller value of the distance means that $lin{e}_j$ is closer to ${\mathrm{\Psi }}^{\mathrm{1}}$. ${\delta }_j^{\mathrm{1}}$ means the normalized value. Hence, $lin{e}_j$ follows the above geometric constraints, which indicates that the line can be considered evidence supporting the contour line ${\mathrm{\Psi }}^{\mathrm{1}}$. Similarly, evidence lines for ${\mathrm{\Psi }}^{\mathrm{2}}$ can be extracted, as shown in Figure 4.

Figure 4. Evidence surfaces. Left: layout lines. Middle: evidence lines. Right: surfaces. Through geometric constraints, the evidence lines supporting the contour layout lines are extracted, and corresponding surfaces are approximately reshaped as a forest road.

The corresponding surfaces can be approximated by irregular, fragmented lines of evidence.

Assuming that $lin{e}_j$ is an evidence line in $\mathbb{E}$, then the following can be modelled:

$S^j={\left[\mathrm{\Gamma }\left(\left[p_m^{\eta },P_{\mathrm{1}}\right],\left[p_{\mathrm{1}}^j,P_{\mathrm{2}}\right]\right);\mathrm{\Gamma }\left(\left[p_{\mathrm{2}}^j,P_{\mathrm{2}}\right],\left[p_m^{\eta },P_{\mathrm{1}}\right]\right);p_{\mathrm{2}}^j;p_{\mathrm{1}}^j;\right]}^T,j\in \mathbb{E}$ (16)

Here $S^j$ represents a surface of an evidence line in $\mathbb{E}$. $p_m^{\eta }$ is the midpoint of virtual line [${\eta }_b^{\mathrm{*}k=\mathrm{1}}$, ${\eta }_b^{\mathrm{*}k=\mathrm{2}}$]. For all evidence lines in $\mathbb{E}$, their corresponding road surfaces can be shaped as shown in Figure 4.

4. Experimental Results

4.1. Requirement

Our research is based entirely on pure geometric reasoning, using only a low-cost monocular camera (RGB image) to understand various unstructured paths in a forest environment. Since no precise depth data or any prior training is required, our experiments were conducted on a computer with an Intel Core i7-6500 2.50 GHz CPU and 2GB of RAM, with no additional high-performance graphics processor (H-GPU), no calibration, and no internal parameters of the camera (IPC), as shown in Table 1. Obviously, the lightweight method utilizing only one monocular camera has lower energy consumption and is more suitable for new energy vehicles with energy constraints.

Table 1. Requirement list.

Condition	RGB	Depth	Training	H-GPU	Calibration	IPC
Requirement	√	×	×	×	×	×

4.2. Evaluation

There are many datasets for understanding road scenes, for example, the widely used urban driving datasets (e.g., Cityscapes [31], KITTI [32]), which are structured road scenes; there are also road datasets for outdoor environments (e.g., TAS500 [33]), but few of them contain forest roads. Our approach aims to understand unstructured roads in forested environments with a low-cost monocular camera; therefore, we build a new dataset with 253 different Unstructured Forest Road (UFR) images, which help us evaluate the effectiveness of the proposed method.

A forest road is shown in Figure 5. By comparing the estimated forest road surfaces to the ground truth, the percentage of correctly classified pixels was evaluated, as shown in Table 2. It calculates the correctly classified pixels, i.e., $IoU=\frac{TP}{TP+FP+FN}$, where TP, FP, and FN are the numbers of true positive, false positive, and false positive, and false negative pixels, respectively. The presented method can not only understand structured roads, but also can account for fragmentary road surfaces in forest environments. The results demonstrate that our algorithm has advantages in understanding such forest roads in field environments.

Figure 5. Understanding a forest road. Left: input image. Second: ground truth. Third: understanding. Compared to the ground truth, our approach can understand rough unstructured surfaces of a forest road without any prior training.

Table 2. Error of understanding unstructured forest roads.

Method	IoU
H.W. [2]	33.2%
Wang [10]	52.3%
Our method	78.7%

4.3. Comparison

Comparisons between the previous algorithms and the proposed method were made on the UFR dataset, with the results shown in Figure 6. Previous approaches approximate a scene using only spatial rectangles [2] and spatial right angles [10], but there is few complete shape of rectangles or right angles in forest environments. By contrast, our algorithm, based on assumption of contour line and evidences, has the advantages of accounting for fragmentary surfaces of forest roads that are always covered by unexpected water and side structures such as vegetation, soil, and snow in forest road environments.

Figure 6. Experimental comparisons on UFR dataset. First column: forest roads. Second column: ground truth. Third column: H.W.’s method [2]. Fourth column: Wang’s method [10]. Right column: our understanding. In contrast, our method can understand forest roads covered by water and side covers, e.g., vegetation, soil, and snow.

Furthermore, more experiments were performed on forest roads within diverse environments of different color and illumination. Since the method adopts geometric inferences, as shown in Figure 7, the approach can successfully understand forest roads, which is robust against changes in illumination and color.

Figure 7. Robustness experiment. First column: original environment. From second column to right column: changes in color and illumination, respectively. The proposed method is robust against changes in color and illumination.

Furthermore, more experiments were performed on best and worst cases, as shown in Figure 8. Since the method relies on scene line texture features, the results are worst when there are fewer lines and best when there are sufficient texture features.

Figure 8. Best and worst cases. First row: different detected lines from images. Second row: best and worst cases. Since the method relies on line extraction, the algorithm may not be able to help if there is a lack of corresponding texture features in the scene.

More experiments were compared with deep learning based approaches and the results are shown in Figure 9. The framework is end-to-end and has difficulty interpreting geometric structural features of field roads. In contrast, our algorithm can interpret geometric cues and understand field roads without prior training.

Figure 9. Experimental comparisons to deep learning methods. First column: forest ways. Second column: Duraisamy’ method [34]. Third column: our understanding. Our approach can understand forest ways without prior training.

4.4. Discussion

Experimental results show that traditional methods rely on structured features of scene roads, which makes them unsuitable for functioning in wild forest road environments full of unstructured disturbances. Deep learning methods are unable to handle complex and variable unstructured scene information (e.g., layout, location, and orientation), and are often overly complex and require high-performance GPUs, which are costly to implement in terms of energy consumption. Compared with traditional methods, the method proposed in this paper has advantages in dealing with unstructured environments without prior training, has lower energy consumption, and meets the needs of energy-constrained new energy vehicles to navigate autonomously in forest environments.

Previous methods have only used spatial rectangles and spatial right angles to approximate scenes, but forest environments rarely have complete rectangular or right-angle shapes. Inspired by the oblique effect, neurons preferred to respond to stimuli orientated horizontally and vertically rather than to angles. Thus, trees and plant bodies that tend to grow vertically or horizontally help to estimate road layouts in wild forest environments. In contrast, our algorithm, which is based on the forest vertical trunk hypothesis and the evidence hypothesis, has the advantage of utilizing trunk features on both sides of the forest road environment.

5. Conclusion

The current work proposes an energy-saving, efficient, and lightweight forest road understanding method to help new energy vehicles with limited resources to understand unstructured forest roads using a low-cost, low-power monocular camera without the need for high power consumption, high-performance computing, and any prior training. First, forest road scene edges and lines are extracted. Then, the initial contour of the forest road is estimated by the relationship between the trunk texture projection and the spatial unstructured extinction point. After searching a large number of scattered evidence lines, the forest road plane can be estimated. Unlike the data-driven method, the proposed method does not require prior training and does not need a high-performance GPU, which makes it more suitable for new energy vehicles with limited resources. Geometric reasoning also makes the proposed method robust to changes in light and color, making it more suitable for forest road environments. The estimated forest roads are compared with the true values and the percentage of correctly categorized pixels is measured. The experimental results show that the proposed method can effectively understand unstructured forest roads with low power consumption and low complexity, which has a promising application in new energy vehicle systems.

Author Contributions: L.W.: conceptualization, methodology, software, validation, writing—reviewing and editing; Y.F.: data curation, validation, writing—original draft preparation; S.W.: visualization, investigation; H.W.: supervision. All authors have read and agreed to the published version of the manuscript.

Funding: This work was supported by the NSFC Project (Project Nos. 62003212 and 61771146).

Institutional Review Board Statement: Not applicable.

Informed Consent Statement: Not applicable.

Data Availability Statement: Data will be made available on reasonable request.

Conflicts of Interest: The authors declare no competing interests.