Finally I searched a while by myself, and I think I may answer my own question. Although I couldn't completely resolve my issue, I noticed that the answer is hidden in the other MouseArea events. For example, by handling the onPressed event and adding the mouse.accepted in several key locations, I could let the component take care of the scrolling when the left mouse button is pressed, whereas the right click opens a popup menu.
My conclusion is that there is no ready-to-use way, i.e there is no parameter to activate in the MouseArea itself which may resolve this kind of issue, and the solution is a good balance between activating different parameters in the different functions.