It turns out what I wanted had a name, clipping. You can set clip: true, and that should prevent the menuitems' backgrounds from drawing outside of the menu itself. However, due to performance concerns, clip regions can only be rectangular so that did not solve the problem in this case.

The solution was to redefine the background of the menuItems like this:
(keep in mind that I'm still a noob, so while this code solves this specific problem, it might not be so great in other ways)

background: Item {
        height: root.implicitHeight 
        width: root.implicitWidth
        anchors.bottom: root.bottom
        opacity: root.activeFocus ? 0.5 : 0
        layer.enabled: true
        Rectangle {
            implicitHeight: 0.5 * root.implicitHeight
            color: "#000000"
            opacity: (root.activeFocus && (isFirst || isLast)) ? 1 : 0
            anchors.top: isFirst ? undefined : roundRect.top
            anchors.right: roundRect.right
            anchors.left: roundRect.left
            anchors.bottom: isFirst ? roundRect.bottom : undefined
            visible: opacity
        }
        Rectangle {
            id: roundRect
            radius: (isLast || isFirst) ? 15 : 0
            implicitHeight: root.implicitHeight
            color: "#000000"
            anchors.fill: parent
        }
    }