Shiboken2: need help converting args from default type to "custom"

eva00

By default Shiboken2 converts const char * to PyUnicode/PyString, but I need it to be converted to PyBytes. Software being wrapped, Scintilla/ScintillaEdit/ScintillaEditPy, stores strings as bytes, so to avoid any potential issues I'd like it to reflect this and be explicit about this by using b"string".

There are two ways to do this:

add <primitive-type> to typesystem with conversion rule from const char * to PyBytes;
add <modify-argument> along with <conversion-rule> to <modify-function>.

First won't work because there can be only one conversion native-to-target, and there already seems to be a conversion rule from const char * to PyUnicode/PyString. As for second, I can't seem to get needed result. Here's an example of typesystem modification that should do this, based on <modify-function signature="loadFromData(const uchar*,uint,const char*,QFlags<Qt::ImageConversionFlag>)"> from PySide2 typesystem_gui_common.xml:

<modify-function signature="describeProperty(const char *)" allow-thread="no">
	<modify-argument index="1">
		<replace-type modified-type="PyBytes" />
		<conversion-rule class="native">
			const char* %out = reinterpret_cast&lt;const char*>(PyBytes_AS_STRING(%PYARG_1));
		</conversion-rule>
	</modify-argument>
	<!-- <inject-code class="target"> -->
		<!-- const char* cppArg0 = reinterpret_cast&lt;const char*>(PyBytes_AS_STRING(%PYARG_1)); -->
	<!-- </inject-code> -->
</modify-function>

First, it seems that <conversion-rule> within <modify-argument> is handled differently then when in <primitive-type>: %in can't be used because it gets replaced with name, so instead argument SID-dependent %PYARG_1 must be used, which is error-prone, also %out gets replaced with name_out, which limits use of multiple unnamed arguments (compare this to nice, automatically generated cppArg0)... Because it's single type conversion, shouldn't %in and %out be enough, to avoid messing with argument SID, like in <primitive-type>? Obviously, because it's how Qt/PySide2 itself does it, I doubt there's a solution to this other than to modify source code of Shiboken2...
Second (see code examples at the bottom), by default argument conversion is added at the beginning of Call function/method block, before PyErr_Occurred(), which is used to check for potential conversion errors, but it seems that <modify-argument> is treated as <inject-code> and both are being added after PyErr_Occurred() is checked, forcing us to duplicate PyErr_Occurred() block to return in case of failure. Shouldn't <modify-argument> be treated as argument modification and processed before PyErr_Occurred()?

I have even tried using %FUNCTION_NAME(), which switches method calling to fully manual mode, but PyErr_Occurred() block still remains, even though in this case it clearly serves no purpose at all.

Analysis of Shiboken2 source code (cppgenerator.cpp::CppGenerator::writeMethodWrapper() ==> writeFunctionCalls()) shows that indeed both <conversion-rule> and <inject-code> are treated more or less equally: no matter what they are written by writeMethodCall() after PyErr_Occurred() block. writeArgumentConversion() (called from writeSingleFunctionCall()), which looks and sounds like the most appropriate place to interpret <conversion-rule> for an argument, contains statement if (argType->typeEntry()->isCustom() || argType->typeEntry()->isVarargs()) return; which likely prevents such conversion/snippet to be written in "needed" place.

The more I look at all this, the more it becomes obvious that there's no solution to my problem and the code related to this is most likely an ad-hoc/hastily implemented feature...

Here's some code to hopefully make things clearer. Code generated by default:

static PyObject* Sbk_ScintillaEditFunc_describeProperty(PyObject* self, PyObject* pyArg)
{
	ScintillaEditWrapper* cppSelf = nullptr;
	SBK_UNUSED(cppSelf)
	if (!Shiboken::Object::isValid(self))
		return {};
	cppSelf = static_cast<ScintillaEditWrapper *>(reinterpret_cast< ::ScintillaEdit *>(Shiboken::Conversions::cppPointer(SbkScintillaEditPyTypes[SBK_SCINTILLAEDIT_IDX], reinterpret_cast<SbkObject *>(self))));
	PyObject* pyResult{};
	int overloadId = -1;
	PythonToCppFunc pythonToCpp{};
	SBK_UNUSED(pythonToCpp)

	// Overloaded function decisor
	// 0: ScintillaEdit::describeProperty(const char*)
	if (Shiboken::String::check(pyArg) && (pythonToCpp = Shiboken::Conversions::isPythonToCppConvertible(Shiboken::Conversions::PrimitiveTypeConverter<const char*>(), (pyArg)))) { // DEFAULT #1
		overloadId = 0; // describeProperty(const char*)
	}

	// Function signature not found.
	if (overloadId == -1) goto Sbk_ScintillaEditFunc_describeProperty_TypeError;

	// Call function/method
	{
		const char* cppArg0; // DEFAULT #2
		pythonToCpp(pyArg, &cppArg0); // DEFAULT #2

		if (!PyErr_Occurred()) {
			QByteArray cppResult = cppSelf->describeProperty(cppArg0);
			pyResult = Shiboken::Conversions::copyToPython(reinterpret_cast<SbkObjectType *>(SbkPySide2_QtCoreTypes[SBK_QBYTEARRAY_IDX]), &cppResult);
		}
	}

	if (PyErr_Occurred() || !pyResult) {
		Py_XDECREF(pyResult);
		return {};
	}
	return pyResult;

	Sbk_ScintillaEditFunc_describeProperty_TypeError:
		Shiboken::setErrorAboutWrongArguments(pyArg, "ScintillaEditPy.ScintillaEdit.describeProperty");
		return {};
}

Code I get with <modify-argument>:

static PyObject* Sbk_ScintillaEditFunc_describeProperty(PyObject* self, PyObject* pyArg)
{
	ScintillaEditWrapper* cppSelf = nullptr;
	SBK_UNUSED(cppSelf)
	if (!Shiboken::Object::isValid(self))
		return {};
	cppSelf = static_cast<ScintillaEditWrapper *>(reinterpret_cast< ::ScintillaEdit *>(Shiboken::Conversions::cppPointer(SbkScintillaEditPyTypes[SBK_SCINTILLAEDIT_IDX], reinterpret_cast<SbkObject *>(self))));
	PyObject* pyResult{};
	int overloadId = -1;
	PythonToCppFunc pythonToCpp{};
	SBK_UNUSED(pythonToCpp)

	// Overloaded function decisor
	// 0: ScintillaEdit::describeProperty(const char*)
	if (PyBytes_Check(pyArg)) { // CHANGE #1
		overloadId = 0; // describeProperty(const char*)
	}

	// Function signature not found.
	if (overloadId == -1) goto Sbk_ScintillaEditFunc_describeProperty_TypeError;

	// Call function/method
	{
		if (!PyErr_Occurred()) {
			// describeProperty(const char*)
			// Begin code injection
			const char* name_out = reinterpret_cast<const char*>(PyBytes_AS_STRING(name)); // CHANGE #2. `name` is not defined. Must be `pyArg`.
			// End of code injection

			QByteArray cppResult = cppSelf->describeProperty(name_out); // CHANGE #3
			pyResult = Shiboken::Conversions::copyToPython(reinterpret_cast<SbkObjectType *>(SbkPySide2_QtCoreTypes[SBK_QBYTEARRAY_IDX]), &cppResult);
		}
	}

	if (PyErr_Occurred() || !pyResult) {
		Py_XDECREF(pyResult);
		return {};
	}
	return pyResult;

	Sbk_ScintillaEditFunc_describeProperty_TypeError:
		Shiboken::setErrorAboutWrongArguments(pyArg, "ScintillaEditPy.ScintillaEdit.describeProperty");
		return {};
}

Code I get with <inject-code>:

static PyObject* Sbk_ScintillaEditFunc_describeProperty(PyObject* self, PyObject* pyArg)
{
	ScintillaEditWrapper* cppSelf = nullptr;
	SBK_UNUSED(cppSelf)
	if (!Shiboken::Object::isValid(self))
		return {};
	cppSelf = static_cast<ScintillaEditWrapper *>(reinterpret_cast< ::ScintillaEdit *>(Shiboken::Conversions::cppPointer(SbkScintillaEditPyTypes[SBK_SCINTILLAEDIT_IDX], reinterpret_cast<SbkObject *>(self))));
	PyObject* pyResult{};
	int overloadId = -1;
	PythonToCppFunc pythonToCpp{};
	SBK_UNUSED(pythonToCpp)

	// Overloaded function decisor
	// 0: ScintillaEdit::describeProperty(const char*)
	if (PyBytes_Check(pyArg)) { // CHANGE #1
		overloadId = 0; // describeProperty(const char*)
	}

	// Function signature not found.
	if (overloadId == -1) goto Sbk_ScintillaEditFunc_describeProperty_TypeError;

	// Call function/method
	{
		if (!PyErr_Occurred()) {
			// describeProperty(const char*)
			// Begin code injection
			const char* cppArg0 = reinterpret_cast<const char*>(PyBytes_AS_STRING(pyArg)); // CHANGE #2
			// End of code injection

			QByteArray cppResult = cppSelf->describeProperty(cppArg0); // NO CHANGE. For some reason references undefined `cppArg0` (fully manual code injection is not definition).
			pyResult = Shiboken::Conversions::copyToPython(reinterpret_cast<SbkObjectType *>(SbkPySide2_QtCoreTypes[SBK_QBYTEARRAY_IDX]), &cppResult);
		}
	}

	if (PyErr_Occurred() || !pyResult) {
		Py_XDECREF(pyResult);
		return {};
	}
	return pyResult;

	Sbk_ScintillaEditFunc_describeProperty_TypeError:
		Shiboken::setErrorAboutWrongArguments(pyArg, "ScintillaEditPy.ScintillaEdit.describeProperty");
		return {};
}

Code I want:

static PyObject* Sbk_ScintillaEditFunc_describeProperty(PyObject* self, PyObject* pyArg)
{
	ScintillaEditWrapper* cppSelf = nullptr;
	SBK_UNUSED(cppSelf)
	if (!Shiboken::Object::isValid(self))
		return {};
	cppSelf = static_cast<ScintillaEditWrapper *>(reinterpret_cast< ::ScintillaEdit *>(Shiboken::Conversions::cppPointer(SbkScintillaEditPyTypes[SBK_SCINTILLAEDIT_IDX], reinterpret_cast<SbkObject *>(self))));
	PyObject* pyResult{};
	int overloadId = -1;
	PythonToCppFunc pythonToCpp{};
	SBK_UNUSED(pythonToCpp)

	// Overloaded function decisor
	// 0: ScintillaEdit::describeProperty(const char*)
	if (PyBytes_Check(pyArg)) { // CHANGE #1
		overloadId = 0; // describeProperty(const char*)
	}

	// Function signature not found.
	if (overloadId == -1) goto Sbk_ScintillaEditFunc_describeProperty_TypeError;

	// Call function/method
	{
		const char* cppArg0 = reinterpret_cast<const char*>(PyBytes_AS_STRING(pyArg)); // CHANGE #2

		if (!PyErr_Occurred()) {
			QByteArray cppResult = cppSelf->describeProperty(cppArg0);
			pyResult = Shiboken::Conversions::copyToPython(reinterpret_cast<SbkObjectType *>(SbkPySide2_QtCoreTypes[SBK_QBYTEARRAY_IDX]), &cppResult);
		}
	}

	if (PyErr_Occurred() || !pyResult) {
		Py_XDECREF(pyResult);
		return {};
	}
	return pyResult;

	Sbk_ScintillaEditFunc_describeProperty_TypeError:
		Shiboken::setErrorAboutWrongArguments(pyArg, "ScintillaEditPy.ScintillaEdit.describeProperty");
		return {};
}