Keep track of number of rectangles drawn over an image using Mouse Events

duddal_

I am working with the following tool:
https://github.com/tzutalin/labelImg

I want to extend the functionality of the tool. I am almost halfway through but I would like to now know how to identify how many rectangles the user has drawn over the image.

The script to draw rectangle is this: https://github.com/tzutalin/labelImg/blob/master/libs/canvas.py

One of the ways I have been thinking about is keeping track of MousePress and MouseRelease events. But I am not sure whether this is the way to go.

But I am not entirely sure how to do this. I would appreciate any help I can get.

Thank You.

mrjj

Hi
Are those rectangle not the Shapes the code talk about ?
So they are store in self.shapes ?
That seems like a list so can you ask the list about its size and that would how many rectangles user added ?

duddal_

@mrjj Yep, thats true. But I am wondering, how to use this "self.shapes" from this script in the main script which is : https://github.com/tzutalin/labelImg/blob/master/labelImg.py

I can see that the size of the list changes when the rectangle is drawn but I want to compare this in another script as I mentioned above (link).

mrjj

@duddal_

Hi
Maybe I miss something but in labelImg.py, a canvas object is made in MainWindow
so self.canvas should give you access. (when in MainWindows scope)

duddal_

@mrjj
Sorry for the late reply. Yes, I forgot to mention it here. But as you said I can access the shapes using self.canvas.shapes.

I actually have another doubt regarding the same script but wondering whether I should open a new issue or not. What do you think?

mrjj

@duddal_
hi i think here is fine since there is ref to the script.

duddal_

Thanks @mrjj

So there is a variable called self.canvas.shapes which is a list.
Every time a rectangle is drawn over the image the list gets updated. It contains the "Shape" object for each rectangle.
For example: [<libs.shape.Shape object at 0x000002638F8FE448>, <libs.shape.Shape object at 0x000002638C89A848>]

I would like to modify the code such that it outputs the following:
[[<libs.shape.Shape object at 0x000002638F8FE448>], [<libs.shape.Shape object at 0x000002638F8FE448>, <libs.shape.Shape object at 0x000002638C89A848>]]

A list of lists so that I can compare the length of the previous and the current.

So in simple terms it should be: test = [[A], [A,B], [A,B,C]]
So that I can now compare len(test[-1]) and len(test[-2])

I have tried looking into canvas.py script but unable to find a solution.

mrjj

@duddal_
Hi
So you want a list of lists?

Wont that upset the rest of the drawing code ?

So that I can now compare len(test[-1]) and len(test[-2])

That will return the size of the lists ?

duddal_

Hello @mrjj ,

Yes, it will return the size of the lists.
I am not sure whether it will upset the drawing code, to be honest.

I have a button that changes the brightness of the image. After a rectangle is drawn on the image I push that button. And after every new rectangle is drawn I first have to save the image file (label file). So, for example, The user has drawn a bounding box, and then once the button is pushed a window appears which allows the user to save the image file. And without drawing any more rectangles the button should just change the brightness instead of asking to save the image file again. Therefore I thought if I have a list of lists of the rectangles drawn one after the other, I will be able to compare the size and then it is just the matter of comparing the previous and current size to determine whether a new rectangle has been drawn or not.

I can't quite figure out if there is another way of doing this or not.

mrjj

@duddal_

Hi
If you keep your own list, i guess nothing bad will happen :)

But if the goal is only to keep track of rectangles (shapes)
im not sure i understand why you want to keep a list to track it.

Say we have a new variable.
ShapeCountBefore=0;

Then when your SaveButton is pressed
we compare
ShapeCountBefore and Shapes.Size()
if they are the same, no new rect was added.

if ( ShapeCountBefore LESS than Shapes.Size() )
then a new rect was added. and then we
update ShapeCountBefore=Shapes.Size()

if ( ShapeCountBefore BIGGER than Shapes.Size() ) it means
rect was removed

Maybe i dont 100% understand what you want as you say drawn for the rect and not added.

duddal_

@mrjj
I am sorry for the late reply!!
So Yes I tried that and after making a few changes it worked. And now I am trying to figure out whether it is possible to save the bounding boxes (which the user draws) in the memory.

So basically as we discussed above, in order to change the brightness of the image, the image needs to be saved, as many times as the new bounding box is created.
I want to eliminate this shortcoming. For example, I want to be able to press the push button and the brightness needs to be changed even though the new bounding boxes are added without saving the image. So If a user draws new bounding boxes then by pressing the Pushbutton, the brightness needs to change, and then when all is done, the user can save the image once.

For this, I thought saving the newly drawn bounding boxes In the memory and then using them would do the trick, but not successful. So I took a look at the canvas.py script to figure out how they are moving the bounding boxes without saving the image(label file), but cannot quite understand that.

What do you think?

Hope you are healthy.
Cheers.

mrjj

Hi
Np. welcome back.
Just to be sure. when you say "bounding box", do you mean the Shapes ? ( the shape class)
Also how is the brightness adjusted now ?
Is that by done by the Shape objects ?

Also what do you mean by "how they are moving the bounding boxes without saving the image(label file)"
Where do they move or do you mean like hide ?

duddal_

@mrjj
Hello!

Yes, "bounding box" means the shape (rectangle).
Brightness is adjusted using a push-button. When an image is loaded in the tool and then the user presses the "brightness" push-button, brightness is changed by a certain percentage. And herein lies the issue: When the user draws a new shape over the image then the user needs to save ((Ctrl + S) the corresponding label file (XML format) first. If not saved, then the original image file without shapes or previous shapes is loaded (So to simplify: if any new shape is drawn, it needs to be saved first). I don't want this behavior. I want to it work such that, the brightness of the image should change even without saving it first. The user should need to save it only once at the end.
What do you mean, "by shape objects" ? What is done?
So in if a new shape is drawn, the user can just move the shape around in the image to adjust its position. It is here: https://github.com/tzutalin/labelImg/blob/master/libs/canvas.py. (Line 164)
No, I mean move (like literally from one position to another)

duddal_

@mrjj
Is it fine if I start a new chat with you?

Thanks and Take care.

mrjj

@duddal_
Hi
You mean a new post or a chat ?
I dont mind chatting but it's not good for code so forums is actually better.

Ok we are talkling about the shapes.

I looked it over but i didn't see the brightness thing.

So about the moving thing. It seems they just adjust the 4 points for the rest.
At least that's what they do to move one pixel

self.selectedShape.points[0] += QPointF(-1.0, 0)

But not sure how this relates to the brightness

duddal_

@mrjj
Hello,

You will not find the brightness thing in the aforementioned link. I have added that separately in my code. And I cannot post it here and hence wanted to chat.

Yes exactly, that's how they do the movement on the shapes. I couldn't find a way of storing the image and the shapes drawn over the image in the memory temporarily and use them.

One more thing I thought of is mentioned here (opened a new issue as the problem is different):
https://forum.qt.io/topic/124498/how-to-remove-the-confirm-save-as-dialog-box-from-qfiledialog

What do you think?

mrjj

@duddal_

Ok. well chat is not good for code. The reason I ask about it is that I don't understand what you want to do with
The Shapes and this brightness change.

. I couldn't find a way of storing the image and the shapes drawn over the image in the memory temporarily and use them.

well you can use them by looping over the list. Im not sure what you mean by storing them in memory but i aassume
something is done during saving and you want to do this without saving the image.
But Im not sure what exactly, you need help with.

you can move a shape like

            self.selectedShape.points[0] += QPointF(-1.0, 0)
            self.selectedShape.points[1] += QPointF(-1.0, 0)
            self.selectedShape.points[2] += QPointF(-1.0, 0)
            self.selectedShape.points[3] += QPointF(-1.0, 0)

i see no move function or similar

so you can loop over the shape list anbd move them by adjusting its points.
Is that what you mean by "move" ?

duddal_

@mrjj
Okay, let me frame it in another way.

Also, the "move" thing is only used as an example and is by no way related to our problem. This was just used as an example. Let's just forget about the "move" thing for now.

You said:

I'm not sure what you mean by storing them in memory but I assume something is done during saving and you want to do this without saving the image.

And this is correct.

In order to annotate (draw shapes) the user needs to follow the following steps:

Load the directory containing images.
Draw the first shape and hit "Ctrl + S" (save) and save an XML file (with base-name same as image) that contains the name of the image, the coordinates of the shape (rectangle).
The user draws a few shapes over the same image and then has to hit the save button. After this, a dialog box pops up which asks the user to select the same (point 2) file and then asks for permission to overwrite this file. And the cycle repeats itself.

In order to avoid selecting the file to save again, I can set the self.defaultSaveDir as targetDirPath as mentioned here.

At first, I thought I have to save the shapes in the memory and then use the saved shapes in order to tackle the aforementioned tackle for a more concrete solution. But this is not the correct way I guess.

But the workaround works.

duddal_

@mrjj
How are you doing?

I am working on the same thing.

With labelImg, we can move the shapes by using the left mouse click and dragging the shape. However, I want to save the file as soon as the shape is moved. There is a function saveFile(True) in labelImg.py which helps us do exactly that.

After debugging the code, I found that there is a function called labelSelectionChanged(self) in lebalImg.py which is called every time the shape is selected and/or dragged. I inserted an if-statement with the condition being that if this function is called, save the file. However, I noticed that the file keeps saving in an infinity loop (i.e. did not stop, I had to stop the program by Ctrl+C).

One more solution I tried was: check if the Qt.ClosedHandCursor is used and then use that as a condition to save the file. This too did not work.

I would like some help with this. This seems very easy but I can't get my head around this.

mrjj

Hi
Welcome back.

Saving with every move might not work well as it might happen too fast so it's called many times.
I think that was what you saw when appeared to be in an infinity loop.

Do you really want to save to file each time user adjusts the shape just a little bit?