Finally I searched a while by myself, and I think I may answer my own question. Although I couldn't completely resolve my issue, I noticed that the answer is hidden in the other MouseArea events. For example, by handling the onPressed event and adding the mouse.accepted in several key locations, I could let the component take care of the scrolling when the left mouse button is pressed, whereas the right click opens a popup menu.
My conclusion is that there is no ready-to-use way, i.e there is no parameter to activate in the MouseArea itself which may resolve this kind of issue, and the solution is a good balance between activating different parameters in the different functions.
@nebulaekg It wasn't wrong, it was just the easy method. With easy comes very little customization. Now that you want to change it's behavior it becomes much harder. For simple lists you can definitely use the widget but for anything more you want a view/model and then delegates to control display.
Check into it I'm sure you'll find it's not that much harder than the widget and will give you the control you want. :)