My first algorithm / Web cam fun

holy crap, I have been working on this 8 hours almost straight!

I created an almost intelligent algorithm for my awesome class in C++. objective of this algorithm is, to recognize background from non background (in this case a tic tac toe grid).
my first attempt, was to take the average color of the background (in this case green) and declare everything with that color as background.
Sounds simple, here is what that does:

I am yet not using any threshold, the black part is always what was determined as background.
Not good at all.

A very long time later, I "taught" my algorithm how to split up the image in 2:

Somewhat better, but the (here on purpose) extreme difference in lightning creates huge differences in color output (Red, Green and Blue).

Long time later again, I made the splitting variable. Now I can just change one value and the image get split up that many times. Here with 3 now:


Here with 10:


And 30:


Pretty cool already, time to adjust the light to a more realistic background. First with 0 splitts:

Now, lets see what my algorithm can do already (20 splits):


Ok, time copy (by far not as easy as you think) this splitting "technology" to have not only vertical but also horizontal splits.
Here 1x1 split:


1x8 split:


And for shitts 20 horizontal splits:


Enough playing around, lets get rid of the white space, 5x5:


Almost, thank god i only have to change 2 numbers, 10x10:


Before we move on, just for giggles a picture as background with 10x10 split. The algorithm fails horribly, I don't blame it:


Back to the green paper, I flipped it over. You can see, the program can tell pretty decend from background and non background:


I was pretty happy when I drew the first TicTacToe grid, to try it out. I don't know if it was the colors or the thick lines, but my algorithm failed:


Time to fix this...
My first approach was, to prevent devient colors (in this case the tic tac toe grid) to be considered by the average color calculations. First I had many many errors, division by zero mostly, because
the program would look at the average color while calculating the average color; wereas some avg. color had to be there in the first place. :-P
I made it, so that it would calculate the avg. color first normal, and then again. But the seocond time, it would disregard any pixels that where different from the original avg. color.

The result sucked. In some splitted sectors, the first average color was more the color of the grid, so the second calculation disregarded the actual background color.

My next try simular to the above but instead of the first calculation, it would just start with the second right away, not even considering the overall average color of that sector. I wasted my time with this other than fixing the errors mentioned in the first approach.
It did look better, but what if the calculation started right on the grid itself...


Next, I further changed the sequence and calculations. First, it would get the avg. color of the entire frame (with 0 splitts), then while slowly adding more splitts, it would calculate a median avg. color for ever sector, roughly disergarding pixels that where much different from the median color; and finally moving to the last calculation with all the splitts and using final average colors, that had been calculated out of the median ones. Sounds confusing but anyways, it worked:


Now using a different grid (once again, i dont know if it was the color or the thickness) excellent results!

Also notice how the paper wasnt completely covering the area, but the algorithm took care of that anyways :))))
This picture shows average colors to the one above:


Now here comes a tough one, which isn't really necesairry but helped me to further improove the quality of the algorithm:


I was from now on, mostly adjusting values, mainly the 6 threshold values.


Look, how good I got it to work, (I was focusing on the yellow X):