Objective. Existing sparse attention networks in medical image segmentation suffer from fixed K values across modules and scales, hindering effective adaptation to the wide variability in lesion sizes and shapes. The objective of this study is to develop an adaptive, differentiable sparse attention mechanism to solve this issue for both the robustness and the segmentation accuracy of medical image analysis systems. Approach. We propose patch adaptive cut-off network (PACNet), which embeds an entropy-guided differentiable -selection (EGDK) module to learn data-dependent sparsity ratios. EGDK combines Gaussian Soft Indexing with a straight-through estimator to preserve end-to-end differentiability during optimization, addressing the gradient blocking issue in conventional discrete Top- selection. PACNet is evaluated on eight datasets spanning five imaging modalities under a unified training and evaluation protocol. Main results. PACNet achieves superior or highly competitive performance against state-of-the-art methods, obtaining the best average dice similarity coefficient (DSC) of 90.57% across the eight datasets. It improves average DSC by +1.79% over BRAU-Net++ (which also uses Top--style sparse selection). In addition, PACNet achieves strong efficiency with 6.82 M parameters and 6.02G FLOPs. Significance. The results show that adaptive and differentiable selection is more suitable than fixed or discrete sparse selection for medical image segmentation. PACNet effectively suppresses background clutter and preserves fine-grained anatomical details, providing a practical and clinically meaningful solution for robust general medical image segmentation.