It's more of a limitation than a bug I think.
To OS, by default a window is a rectangle, no matter how you paint it, and stylesheets are just a way to paint it.
To cut out the corners you would need a mask. Something like:
auto frame = new QWidget(parent, Qt::Popup);
frame->setStyleSheet("background-color: red; border: 1px solid green; border-radius: 6px;");
QPainterPath path;
path.addRoundedRect(frame->rect(), 6, 6);
frame->setMask(path.toFillPolygon().toPolygon());
frame->show();
The problem is that the mask is a bitmap (0-1), not an image with alpha, so you get a jagged edges, but for some radii it works pretty well.