Advances in medical imaging segmentation using deep learning (DL) have facilitated the development of a wide range of models based on different architectures. For example, the U-Net has become one of the most widely used architectures in the field. Due to its popularity, various modifications to the original U-Net architecture have been proposed, with the aim to improve the segmentation performance. Most studies utilizing ophthalmic images, such as optical coherence tomography (OCT) and scanning laser ophthalmoscopy (SLO), have employed U-Net methods and their variations for segmentation. Given the multitude of U-Net variations, selecting the optimum model for ophthalmic image segmentation may be challenging, and other factors that may impact the model’s segmentation performance, such as image complexity and its impact on model selection, have been largely unexplored. Thus, in this study, the segmentation performance of three models, including a baseline U-Net, a popular U-Net variation (U-Net++), and a segmentation architecture with a different approach (DeepLabV3), are compared, analyzing how these different methods may vary for OCT and SLO datasets with various levels of segmentation task complexity. To analyze the effect of image complexity on segmentation performance, several metrics are extracted, including the image entropy of the datasets at the pixel level, the texture level, and the global features level. The results demonstrate a relationship between the complexity of the images in a dataset and the performance of the segmentation model used for the specific task. Data complexity may serve as a metric to inform DL model selection or aid in the early design process.