So apparently, somewhere between Qt 4.8.7 and Qt 5.11.3 there is a change in behavior for the QWidget::grabKeyboard() method in combination with modality

grabKeyboard is the ultimate "modality" and is very invasive. If you're using it (regularly) it probably means you are doing something you shouldn't be.

(or we were doing something wrong and were just lucky in 4.8.7)?

It is possible. No way to tell beside boiling it down to an MRE which can be dissected.

This seems to be a conflict between grabKeyboard() and showModal(). Previously we did widget->grabKeyboard() before moving
the widget to a new (modal) QDialog. This made the keyevents go to the wrong window in Qt 5. By moving widget->grabKeyboard() 
after adding the widget to the modal QDialog, everything seems to be OK.

Before, we were using Qt 4.8.7 and this problem did not occur. It was only until after our switch to Qt 5.11.3 (and later 5.12.0) that this problem occurred. So apparently, somewhere between Qt 4.8.7 and Qt 5.11.3 there is a change in behavior for the QWidget::grabKeyboard() method in combination with modality (or we were doing something wrong and were just lucky in 4.8.7)?

1. The flags in the first case:

windowFlags: {i=-2013204479 }

Is very suspicious, as the high-order bit is set only for a very specific flags. Not to mention that value sets a whole lot of bits that I don't believe should be set.

2. The modality:
   First case:

modality: NonModal (0)

Second case:

modality: ApplicationModal (2)

Ideas:

1. Check if the object is correct in the first case, and if there's some flag manipulation done on it, it may be wrong.
2. Check why the discrepancy between case 1 and 2 between the modality. It suggests that the focus was captured by another widget/window in the first case (as you suspected).

window->d_func()
0x083bfd20 {...}
    [QWidgetWindowPrivate]: {...}
    QObjectPrivate: {extraData=0x082a9448 {userData={ size = 0 } propertyNames={ size = 0 } propertyValues={ size = 0 } ...} ...}
    surfaceType: RasterGLSurface (2)
    windowFlags: {i=-2013204479 }
    parentWindow: 0x00000000 <NULL>
    platformWindow: 0x02313ad0 {m_data={flags={i=134279169 } geometry={ x = 0, y = 0, width = 1920, height = 1080 } fullFrameMargins=...} ...}
    visible: true
    visibilityOnDestroy: false
    exposed: true
    requestedFormat: {d=0x023b0998 {ref={...} opts={i=0 } redBufferSize=-1 ...} }
    windowTitle: Our Super Duper Cool Application
    windowIcon: {d=0x02344598 {engine=0x0236bdb8 {pixmaps={ size = 1 } } ref={...} serialNum=1 ...} }
    geometry: { x = 0, y = 0, width = 1920, height = 1080 }
    windowState: {i=4 }
    visibility: FullScreen (5)
    resizeEventPending: false
    receivedExpose: true
    positionPolicy: WindowFrameExclusive (1)
    positionAutomatic: true
    contentOrientation: PrimaryOrientation (0)
    opacity: 1.0000000000000000
    mask: {d=0x02e244f4 {Qt5Guid.dll!QRegion::QRegionData QRegion::shared_empty} {ref={atomic={_q_value=-1 } } ...} }
    minimumSize: { width = 1023, height = 841 }
    maximumSize: { width = 16777215, height = 16777215 }
    baseSize: { width = 0, height = 0 }
    sizeIncrement: { width = 0, height = 0 }
    modality: NonModal (0)
    blockedByModalWindow: true
    updateRequestPending: false
    transientParent: guarded pointer to subclass of QObject of type "QWindow"
    topLevelScreen: guarded pointer to subclass of QObject of type "QScreen"
    cursor: {d=0x023129c8 {ref={...} cshape=ArrowCursor (0) bm=0x00000000 <NULL> ...} }
    hasCursor: true
    compositing: false
    lastComposeTime: {t1=-9223372036854775808 t2=-9223372036854775808 }

while in the case where the user can enter text, we have our dialog window when we hit the mentioned breakpoint (note that blockedByModalWindow) is false:

window->d_func()
0x0899a4b8 {...}
    [QWidgetWindowPrivate]: {...}
    QObjectPrivate: {extraData=0x0885a168 {userData={ size = 0 } propertyNames={ size = 0 } propertyValues={ size = 0 } ...} ...}
    surfaceType: RasterGLSurface (2)
    windowFlags: {i=2051 }
    parentWindow: 0x00000000 <NULL>
    platformWindow: 0x08916888 {m_data={flags={i=2051 } geometry={ x = 785, y = 400, width = 350, height = 280 } fullFrameMargins=...} ...}
    visible: true
    visibilityOnDestroy: false
    exposed: true
    requestedFormat: {d=0x0237a348 {ref={...} opts={i=0 } redBufferSize=-1 ...} }
    windowIcon: {d=0x02344598 {engine=0x0236bdb8 {pixmaps={ size = 1 } } ref={...} serialNum=1 ...} }
    geometry: { x = 785, y = 400, width = 350, height = 280 }
    windowState: {i=0 }
    visibility: Windowed (2)
    resizeEventPending: false
    receivedExpose: true
    positionPolicy: WindowFrameInclusive (0)
    positionAutomatic: false
    contentOrientation: PrimaryOrientation (0)
    opacity: 1.0000000000000000
    mask: {d=0x02e244f4 {Qt5Guid.dll!QRegion::QRegionData QRegion::shared_empty} {ref={atomic={_q_value=-1 } } ...} }
    minimumSize: { width = 350, height = 280 }
    maximumSize: { width = 16777215, height = 16777215 }
    baseSize: { width = 0, height = 0 }
    sizeIncrement: { width = 0, height = 0 }
    modality: ApplicationModal (2)
    blockedByModalWindow: false
    updateRequestPending: false
    transientParent: guarded pointer to subclass of QObject of type "QWindow"
    topLevelScreen: guarded pointer to subclass of QObject of type "QScreen"
    cursor: {d=0x02312568 {ref={...} cshape=IBeamCursor (4) bm=0x00000000 <NULL> ...} }
    hasCursor: true
    compositing: false
    lastComposeTime: {t1=-9223372036854775808 t2=-9223372036854775808 }

For now, this is all I can say. I'm still in debugging modus :-) Any suggestions welcome!

My gut feeling is that you parented the linedits wrong but with no code is just guessing.