Ensuring Specific Feature Contents in Randomly Cropped Images

/in

I am working on an application that show pictures with hidden objects in them and the user is supposed to find and click on them. It is very similar to some of the popular search-and-find games on Facebook. Here you can try it yourself. On the picture below, try to find the yellow submarine.

Good. Now try to find it again in the picture below.

That was pretty fast, wasn’t it? Well, that is a problem. It turns a fun task into a boring one of spatial memory. Now try the one below.

This was different. Cropping the image different ended up in different image puzzles. Both of the images above came from the image below.

The requirement was to create these different images from the source image via random cropping. What we know in advance are the dimensions of the source image and the dimensions of the cropping window. We also have a rectangle that holds the location in the image of the target figure. In this case the target would be the yellow submarine. The first step that I took to solve this problem was to abstract it. .NET has libraries that allows for easily cropping images if one provides a cropping rectangle. So what I need is the rectangle with the proper starting coordinates to crop it from the source image. Looking at the problem again, I realized that the problem was all about rectangles. So I forgot about images and focused on rectangles instead. Below you can see a graphic representation on how I thought about the problem.

Then I thought of the cropping rectangle as a cropping window that worked as layer on top of the image. I thought of the layer as something similar to layers as they are used in Photoshop. Manipulating the layer it was evident that there was only so much of an area that I can slide the rectangle before the target area was left out. Since the cropping window’s size is constant, I needed the starting coordinates of the rectangle. In .NET rectangles are drawn from the upper left corner coordinate. The rest of the rectangle is drawn by providing values for width and length. So I needed to find the area where, no matter where I put the cropping window, the Answer Rectangle will be included. When manipulating the rectangles, it soon becomes apparent that the rectangle can quickly be found by lining up the cropping window’s lower right corner with the answer rectangle’s lower right corner. The solution rectangle is created from the cropping window’s upper left corner and the answer rectangle’s upper-left corner. If we generate a random coordinate within this solution rectangle the answer rectangle will always be included in the new cropped image.

So I had to find the upper-left coordinate of the cropping window. Then I had to give its length by subtracting the dimensions of the cropping window by those of the answer rectangle. This is a trivial problem. The following code creates the Solution Area that appears above in orange.

var result = new Rectangle();
result.Width = cropRectangle.Width - input.Answer.Width;
result.Height = cropRectangle.Height - input.Answer.Height;
result.X = input.Answer.X - result.Width ;
result.Y = input.Answer.Y - result.Height;

The next problem was boundaries. With the Solution Area it is possible to get a random point that will draw the cropping windows outside of the image. That means that we must adapt the Solution Area to respect the image boundaries. This gives us a smaller Solution Rectangle, but one that will not crop out parts outside of the image. Below are the rectangles rearranged to create the actual Solution Rectangle, which appears in red.

Here is the full method for getting the Solution Rectangle. First we create the Solution Area. Then we check the boundaries and modify our rectangle. After we are done checking the boundaries, we end up with the Solution Rectangle that we can use to safely crop the picture and keep the answer area within it.

public Rectangle GetRange(RectangleAndSelection input, Rectangle cropRectangle)
        {
            var result = new Rectangle();
            result.Width = cropRectangle.Width - input.Answer.Width;
            result.Height = cropRectangle.Height - input.Answer.Height;
            result.X = input.Answer.X - result.Width ;
            result.Y = input.Answer.Y - result.Height;
            result = lowerBoundCorrectionX(result);
            result = lowerBoundCorrectionY(result);
            result = higherBoundCorrectionX(result, cropRectangle.Width, input.Frame.Width);
            result = higherBoundCorrectionY(result, cropRectangle.Height, input.Frame.Height);
            return result;         
        }

For illustrative purposes, I am including one of the boundary checking methods. It should be easy to produce the other ones. The rest of the boundary checking methods are variations of this one.

private Rectangle higherBoundCorrectionX(Rectangle input, int cropSize, int boundarySize)
{
            var result = input;
            var higherBoundary = result.X + result.Width + cropSize;
            if (higherBoundary >  boundarySize) 
            {
                var offset = boundarySize - higherBoundary;
                result.Width += offset;
            }
            return result;
}

After this it is just a matter of generating a random point within the Solution Rectangle, and pass in the new coordinates of the cropping window to the methods in System. Drawing to crop the image.

This example forms the core of my approach to randomly cropping images while ensuring that a specific feature remains in the newly cropped image. I am looking forward to applying it to use cases such as training/instruction or simulation applications.

This post was written by:

Hugo

Senior Engineer

For more information on this post, the technologies discussed here, or Zekiah’s system integration and software development services, please e-mail us at contact@zekiah.com