Turn off OpenGL optimization via Qt Environment Variable

[Sandra_Carney]

Qt version: 5.9
OS: CentOS 7.4
Nvidia version: 390.48-1
OpenGL version string: 4.6.0 NVIDIA 390.48

We running our software on a whitelisted system. Once we whitelist,
we can't add new binaries to run.

We have a number of applications that use Qt+OpenGL.

Some of the OpenGL software is creating mmap'ed binaries in /tmp.
They are files of the form /tmp/.gl* where the rest of the text
is randomly generated.

In the README file( http://us.download.nvidia.com/XFree86/Linux-x86/390.48/README/openglenvvariables.html#disableexecmem ),
there is a section that reads:

Disabling executable memory optimizations

By default, the NVIDIA driver will attempt to use optimizations which rely
on being able to write to executable memory. This may cause problems in
certain system configurations (e.g., on SELinux when the "allow_execmem"
boolean is disabled or "deny_execmem" boolean is enabled, and on
grsecurity kernels configured with CONFIG_PAX_MPROTECT). When possible,
the driver will attempt to detect when it is running on an unsupported
configuration and disable these optimizations automatically. If
the __GL_WRITE_TEXT_SECTION environment variable is set to 0,
the driver will unconditionally disable these optimizations.

We were hoping the messages would disappear when the variable was set.
The messages look something like:

May 15 17:46:02 <hostname> * prevented unauthorized execution of '/tmp/.gl96LhR8' by process <process>
May 15 17:46:02 <hostname> * prevented unauthorized execution of '/tmp/.gl96LhR8' by process <process>
May 15 17:46:02 <hostname> * prevented unauthorized execution of '<home>/.nv/.glNpZrDX' by process <process>
May 15 17:46:02 <hostname> * prevented unauthorized execution of '<home>/.nv/.glNpZrDX' by process <process>

If we set the variable to 0 in the environment, we see dozens and dozens of such messages
when we launch our application. If we set the variable to 1 or don't set it at
all, we see 4 such messages in the log when our application program starts up.

The strace output indirectly associates libqtgraphicaleffectsprivate.so
with the creation of the mmap'ed memory executable files.

We executed 'ldd /path/to/Qt-5.9/qml/QtGraphicalEffects/private/libqtgraphicaleffectsprivate.so'
and /usr/lib64/nvidia/libGL.so.1, /usr/lib64/nvidia/libGLX.so.0, and
/usr/lib64/nvidia/libGL_dispatch.so.0, show up.

I looked around in the Qt documentation starting with the graphical
effects stuff which lead me to a Qt environment variable which tells
Qt Quick not to use OpenGL:

QT_QUICK_BACKEND="software" /path/to/our/application

With this, the messages disappear from the logs. This is not
an option, however. We need OpenGL.

Another option to is to add our application program to an
updaters list. This tells our whitelisting software the
program is trusted to create other executables and run them.
With that, the messages disappear from the log.

However, this violates our cybersecurity requirements.
We can't add any directory a trusted list and we have
a very, very limited set of administrative programs
that are on the updaters list.

When we set the environment variable, __GL_WRITE_TEXT_SECTION=0,
and run /bin/glxgears, a standard program that is supplied by the
glx-utils package, the optimizations are prevented and hence
the complaints from our whitelisting software. Without the
environment variable, execution of /bin/glxgears results
in the messages.

We were able to reproduce the problem with one of the Qt example
programs: /path/to/Qt-5.9/examples/quickcontrols2/texteditor/texteditor.

We ran the above program while monitoring /var/log/messages.
4 messages similar to the above were generated by the whitelisting software.

When we set __GL_WRITE_TEXT_SECTION=0 and ran the same program,
dozens and dozens of these messages were generated. The environment
variable had the opposite effect from what we saw with the
/bin/glxgears program.

Is there a special way to set this variable due to locale
or UTF considerations? We are using English for locale.

Is there another way to disable these optimizatons through
a Qt environment variable other than telling Qt not to use
OpenGL via the QT_QUICK_BACKEND environment variable?

[SGaist]

Hi and welcome to devnet,

Pretty interesting use case !

Did you check whether there was already something about it in the bug report system ?

In any case, I'd recommend bringing this to the interest mailing list. You'll find there Qt's developers/maintainers. This forum is more user oriented.

[Sandra_Carney]

@SGaist I did check the bug database and found nothing there about opengl+whitelisting. I also checked for the environment variable and optimizations. There was nothing relevant.

I will send to the 'interest mailing list'.

Thankyou for your quick reply.

[Sandra_Carney]

@SGaist I performed a post to the interest mailing list as you suggested. Should I receive some sort of reply? I also tried to sign up and have not gotten a response. Is membership constrained to developers in the Qt company?

[SGaist]

No, there’s no such restriction, but you should registre before posting.

[Sandra_Carney]

@SGaist I did try registering twice. Have not heard back although the response could have gotten hung up in the SPAM filter. Have been checking that and haven't seen anything in there. I will keep trying.

[Sandra_Carney]

@Sandra_Carney OK, I am in. I was trying to use my (new) corporate email. We are now part of Abbott. The old email worked. I will now goto the next step. Thanx.