In object segmentation, the existence of hard-classified-pixels limits the segmentation performance. Focusing on these hard pixels through assigning different weights to pixel loss can guide the learning of segmentation model effectively. Existing loss weight assignment methods perceive pixels hardness by current predicted information, pay less attention to past predicted information. While current studies show that samples with less improvement in predicted probability compared to the past are difficult to learn. To define hard pixels more accurately, a hardness-aware loss for object segmentation is proposed. Firstly, the metric of pixel hardness degree is defined, and a mapping function is proposed to quantitatively evaluate the hardness degree which is defined on the difference between current and past predicted probabilities. Then a new compound metric, hardness value, is defined based on hardness degree and the uncertainty. Based on the compound metric, a new loss function is proposed. Experiment results on four datasets using convolutional neural network and Transformer as the backbone models demonstrate that the proposed method effectively improves the accuracy of object segmentation. Especially, in the segmentation model based on ResNet-50, the proposed method improves mean Intersection over Union (mIoU) by almost 4.3 % compared to cross entropy on DUT-O dataset.