Data lose in serial transmision to arduino

[troisiemetype]

Hi,

I've got what appears to be a problem of data lost using the QSerialPort class.

The program I'm writing is for use with a laser projector driven by an Arduino board. It basically reads the value of each pixel of an image, then sends it trough serial to the arduino which set the intensity and position of the laser in real time.

The problem here is it seems data are lost when setting a baudrate > 14400. As a picture is easily more then millions of pixels, I would like it to be set at 115200, and more if possible (the arduino could run up to 250000).

When I send data by hand trough the arduino terminal, connecting at 115200, everything is ok. But not with Qt, unless I set down baud to 14400.

Here are the concerning parts of the program.

Here the serial object is created:

    serial->setPortName(serialConfig.getString("port"));
    serial->setBaudRate(115200);
    serial->setDataBits(QSerialPort::Data8);
    serial->setParity(QSerialPort::NoParity);
    serial->setStopBits(QSerialPort::OneStop);

    if(serialConfig.getBool("flowcontrolxon")){
        serial->setFlowControl(QSerialPort::SoftwareControl);
    }

    if(serial->open(QIODevice::ReadWrite))
    {
        opened = true;
        return true;
    } else {
        opened = false;
        return false;
    }

Then The data is sent with this function:

    dataSize = dataToSend.size();
    for(int i = 0; i < dataSize; i++){
        QByteArray strToSend = QByteArray();
        strToSend.append(dataToSend.at(i));
        serial->write(strToSend);
    }

The dataToSend var is a vector<QString> populated by the part of the program dedicated to image computing.

I don't know what to do to make this work as I would like. Any advice?

[VRonin]

• Can you show us how you read the data?
• "dataToSend var is a vector<QString>" I have no idea why you are serialising an image to a vector of strings but if it works for you I'll glimpse over it

For now my suspect is that you are doing something like this: https://forum.qt.io/topic/70123/server-is-losing-bytes/2

[troisiemetype]

Hi,

Here is are the data are read:

//This function is called from the main function.
//If there are data in the rx buffer, it copies it into a line buffer.
//That way the RX buffer is kept as empty as possible.
bool serial_get_data(){
	ss.parser_state = PARSING_IDLE;
	
	//while there is data in the RX buffer.
	while (rx_tail != rx_head){

		//Get the current char in rx buffer
		char c = rx_buffer[rx_tail];

//		serial_send_pair("parsing", c );
		
		//If the char is end of line, set a "flag" (that is, char = 0),
		//initialize line pointer for next time, and call data parser
		if (c == '\n' || c == '\r'){
			line_buffer[line_counter] = 0;
			line_counter = 0;
			rx_incr(rx_tail);
			_serial_parse_data();
			return true;
		//Else chars are stored.
		} else if (c >= 'A' && c <= 'Z'){
			line_buffer[line_counter] = c;

		} else if (c >= 'a' && c <= 'z'){
			c -= 32;
			line_buffer[line_counter] = c;

		} else if (c >= '0' && c <= '9'){
			line_buffer[line_counter] = c;

		} else if (c ='-'){
			line_buffer[line_counter] = c;

		} else {
			//Every other chars are ignored.
		}

		//increment line and rx buffer pointer
		line_counter++;
		rx_incr(rx_tail);

		//Verify if we can send an XON signal to computer.
		//If yes, set a flag an enable TX interrupts.
		char queue = _serial_rx_queue();
		if ((queue < RX_FLOW_DOWN) && (ss.flow_state == XOFF_SET)){
			ss.flow_state = SET_XON;
			UCSR0B |= (1 << UDRIE0);
		}
	}

	return false;

}

/* this function parse the data string it receives
 * The string must formed like that:
 * I1X23Y45...
 * Each var (one letter) must be followed by its value.
 * The line read has been prepared before, stripped for wrong chars, and terminat with char 0.
 * The loop turns while there is available data to read
 * It's driven by parser states. For each state we should get precise chars.
 * If these chars are found, parser goes to the next state, and test chars again.
 * A normal loop should be:
 * PARSING_IDLE						We have just received a new data string.
 *
 * PARSING_VAR 			 			looking for alpha chars a-z, A-Z.
 *
 * PARSING_VALUE 					looking for number 0-9, a minus sign if number is negative.
 *									else, record the pair before to parse the next one.
 * 
 * Then it calls _serial_record_values() to populates the planner buffer.
 */
void _serial_parse_data(){

	if (planner_get_available() < 1){										// Manage the planner buffer queue.
//		serial_send_message("no buffer available");
		return;
	}

	bool is_neg;
	char c = 1;
	char line_counter = 0;
	ss.parser_state = PARSING_VAR;

	while (c != 0){
		c = line_buffer[line_counter];

//		_serial_append_value(rx_head);
//		_serial_append_value(rx_tail);

		if (ss.parser_state == PARSING_VAR){
			if (c >= 'A' && c <= 'Z'){
				ss.inVar = c;
				ss.inValue = 0;
				is_neg = 0;
				ss.parser_state = PARSING_VALUE;
//					_serial_append_string(ss.inVar);
			}
			line_counter++;

		} else if (ss.parser_state == PARSING_VALUE){
			if (c >= '0' && c <= '9'){
				ss.inValue *= 10;
				ss.inValue += (c - 48);
				line_counter++;
			} else if (c =='-'){
				is_neg = 1;
				line_counter++;
//				_serial_append_string("is neg");
			} else {
				if (is_neg == 1){
					ss.inValue = -ss.inValue;
				}
				_serial_record_pair();
				ss.parser_state = PARSING_VAR;
//				_serial_append_value(ss.inValue);
			}

		}
	}
//	_serial_append_nl();
	_serial_record_values();
//	_serial_send_go();
//	to_read_flag =0;
}

There is an ISR that copies the atmega RX register into rx_buffer. Each char of this rx_buffer is evaluated and copied as soon as possible into a line_buffer.

Then this line_buffer is parsed to look for var/value pairs, that are, if correctly formated, send to a planner that precompute the moves.

The whole project is here: arduino-projecteur-laser.

The vector of string is here to store the data before to send it. When I open an image, I first display it in the GUI. I set some values, like the projected size wanted, the distance from laser to wall and the scan speed. Then when set there's a compute button that compute angles wanted to achieve corresponding positioning given that values. The vector of string stores each line of data corresponding to a pixel, so the serial get it ready when needed.

I'm aware of what would be a more efficient solution!

[VRonin]

Looks like you are having a fight with string encoding

strToSend.append(dataToSend.at(i)); appends the UTF-8 representation of the string which can be multi-byte. try with: strToSend.append(dataToSend.at(i).toLatin1());

[troisiemetype]

Adding .toLatin1() doesn't to change anything.

I didn't have seen the link you gave before, I've just have a look at it, but I don't really understand what is it about... :/

[VRonin]

the link above refers to reading data using Qt so not your case.

• rx_buffer is a circular buffer so it might just overflow
• can you point me to the piece of code in your repository that takes care of filling rx_buffer? I'm too lazy to read through C code

[troisiemetype]

Hi,

Th rx_should not overflow: at each new char coming from the serial link, the serial ISR routines compares its size against the top value allowed before asking Xoff. And then send Xoff if needed. Plus, as soon as there is data in it, this data is transfered in a line_buffer that stores a "packet", i.e. a complete command from the Qt program.

The piece of code in the repository is serialIO.c. here is how it goes:
serial_main() is called from the main loop.
Accordingly to serial_state, it calls:
serial_get_data(), that empties the RX buffer, and skip unwanted chars.
serial_parse_data(), that parses the data received, and then stores the pairs (var + value) into the move buffer.

That said, I've discovered some things by echoing arduino states.
-Every data is received. The only problem I can see is sometimes, a newline char appears in a string of data. It appears regularly, around the 45th coordinate sent. I don't know yet where this comes from.
-If in the Qt sender function I wait for dataread (even if there is none), it seems to be more reliable: The laser behavior is closer to what it should be, and problem don't happen before several heundreds of data sent.

• If I step up the driver ISR that sets flag to ask for position update, the communication seem to be more smooth, even if it causes the motion to ask for more processor time, letting less for the serial.

This points make me think that I've got to refine the arduino side, and probably implementing some kind of realtime processing with state and priority driven mechanism...

I'll come back when it's done to see if there are progress...

Thank you for your kind help!