No, I didn't set these two flags ItemClipsToShape, ItemClipsChildrenToShape.
Then I tried setting only one flag ItemClipsToShape, and both of ItemClipsToShape, ItemClipsChildrenToShape. For each, it still check if the point is in the boundingrect() firstly, and then check the shape() which means if point is not in boundingrect(), it wouldn't bother to check shape().
So now my solution is still enlarging the boundingrect() area.