Arcimboldo Project

Johan Hesselberg, Sandra B. Fan

(CSE 557, Winter 2004)

---

Arcimboldo in a nutshell

The "Arcimboldo process" here is defined as taking an image as input and converting this into a tiled representation with the same major features using tile images from an image database was performed in the following way:

Image segmentation

The image that's going to be tiled first needs to be segmented into regions corresponding to major features in the image. The way we do this is through undersegmentation with the mean-shift algorithm as used in [2] (which the authors so kindly already implemented in C++).  During this phase we label the different segments, that is, each pixel in the image gets a label corresponding to which segment it belongs to.

We also use image segmentation for all the tiles in the database, but we don't care about the segments except the one that borders to the predefined background.

Polygon extraction

Using the labels it's easy to find the extension of every segment and convert them into polygons: take the outline of each  continuous part of the image with same label and then border pixels of these sets spans up the polygons.

Discarding undesirable polygons

The program find lots of small polygons in a picture when the picture contains lots of different colors and the different segment with the same color isn't connected. In order to lower the compuation time we discard the smallest ones; they will probably be overlapped by neighboring tiles that overshoots.

Smoothing

The curve matching procedure (see below) that will performed is highly sensitive to non-uniform noise. Because of that, we smooth out the edges but keep the major features.

Direction curve production

A good way of comparing polygonal shapes, in our case the polygonal segments in the image and the sprites from the database, is to use the turning function. The advantage of using this metric is that it's invariant under scaling, translation and rotation. For further info see [1].

Curve matching and scaling

In order to find the best sprite in the database that fits into or cover a segment in our image, we compare the segments direction curve using the turning function to each of the sprites direction curve's. At the same time, for each sprite, we scale it and compare to three different scales and takes the best fitting combination of sprite and scale according to the turning functon.

Bitmap drawing

After all these steps, we just use the starting vertex in the sprite polygon and its angle to place the sprite onto a canvas.

From here on, we can choose if we want to continue tiling and risk placing tiles on top of each other or resegment the canvas onto we just draw the sprite. The second appoach is of course more computationally expensive, but can lead to nicer results, depending on the features of the image.

Modifications

Most of the steps of this process were already in place in the original version. There were four improvements to the original implementation that we decided to focus on.

• Scaling of sprite images. The original Composite Head program had the ability to rotate and translate, but not scale the sprites. We added the ability to scale the sprites to fit the image.
• Inverse mapping of sprite images. The image shown here is from the original version of this project. Notice the small dots in the sprite images. This is a result of the forward mapping of pixels when doing translation and rotation. To correct this, we implemented an inverse mapping of the pixels by rotating, translating, and scaling first, and then recalling the color from the original image.
• Removing borders on the sprite images.Additionally, there is a blue border around the sprites. In our project, we eliminated these.
• Improving the algorithm for fitting sprites onto the image by filling in gaps better. Finally, note the gaps between the sprites in the figure. After matching the sprites to the contour of the original image, the algorithm was unable to figure out how to match sprites into the holes left over after the initial fittings. We did this by continually redrawing the edges and re-evaluating the polygons after each iteration through the database.

An Example: Hand

First, we take an image that we would like to Arcimboldo-ize. This image of a hand with a blue ring around it was also used in the original version of the program, and we have seen the resulting image, in the previous section.

Next, the program segments the image into different regions according to color. For the hand example, the segmentation looks like this:

Then, for each region, the curves of that region are matched against the database. The resulting images are in the following section.

Results

These are the results of our modified version.

Nice results

In other words, you can see what it is.

The original	A tiled version where the allowed overlap outside of a segment tiles was set pretty high (200%) but where resegmentation has been done between placing each tile.	A tiled version where the allowed overlap was set to low, but where no resegmentation was done. This is why some of the tiles are almost covered by other ones. The result still looks good though! (Can you see the cat?)

The original	A mask which we used here instead since the giraffes pattern in the original image doesn't work very well with fruits.	A fruit salad giraffe. (The blue is the outlne of the segments after  resegmentation.)

The original	Although the program didn't finish running properly on this image for coding reasons, we can see that the results it came up with really do fit the contours of the Rio very well. You can definitely see where all the segments of the Rio are.

A large contributing factor to the success of the resulting image depends on the starting image. If the image is very segmented, and the main reason why we can distinguish the object is due to its shapes and contrasting color segments, then the resulting image is easily identifiable, as evidenced by the distinct shape of the fingers in the hand, and by the tight contours of the Rio. However, if the image lacks a variety of color segments or is of one, non-specifc blob shape, then we get:

Less nice results

If you didn't know what the following image was, you'd never be able to guess!

The original	The tiled version (The blue lines are just showing were the not filled in segments are.)

Notice how the sprites fit the shape very well--even the smaller leaves are fitted. Also, this is an excellent example of the scaling at work--note that there are huge pumpkins as well as small pumpkins. Why does the result look less than satisfying, then, when the program worked so well on it? Because it's not recognizable as anything. This is due to the fact that the original image of the strawberry was not easily recognizable or distinct.

Future Directions & Conclusion

Arcimboldo's "La Primavera"	Arcimboldo's "L'autunno"

While the modified version is an improvement, there are still many issues to be resolved. Although the sprites fill the contours of the image fairly well, the results are not as satisfying as they could be; they don't quite recreate the feeling that Arcimboldo's paintings possessed. This is largely due to the fact that, despite the matching of the shapes in the image, the colors and shading do not match those of the original image. In "La Primavera," we can distinguish the woman's dress because it is completely green, we can distinguish her face because it is flesh-colored, and so forth. The effect is far more pleasing than having a jumble of randomly colored objects. Secondly, shading is a large part of why Arcimboldo's paintings work. Notice the shading in "L'autunno." Part of the reason why the fruits work so well to evoke the image of the man is that they are appropriately shaded to match the shape of the image.

We could accomplish these effects in our program in two ways. One, we could try to weigh sprites more favorably if they match the color of other sprites already in place in that region. This way, the eye can more easily recognize it as one coherent segment, and would solve the first problem. Two, we could try to select for display only sprites that match the color of the original. This would help solve the second problem, because the program may try to match shaded parts as well. However, this will not entirely fix the issue because it would be difficult to appropriately separate shaded areas, because they are not one whole segment, but rather, very gradual changes in color.

However, another important reason why our results don't quite achieve Arcimboldo's effects is due to the semantics of his paintings; he not only chose objects that followed the contours of the image, but he had an intangible, unmeasurable sense for what the human mind would find meaningful or humorous about these pictures, and knew how to use the fruits and objects in a way that would maximize the effect. For instance, although the ear is the same color as the rest of the flesh, and thus our program would have treated it as all the same object, Arcimboldo recognizes conceptually that it is an ear, and uses a single, round object to represent it. Additionally, in "La Primavera" (Italian for "spring") he uses flowers and bright colors and the image of a woman; in "L'autunno" ("autumn") he uses darker colors, harvest fruits and vegetables, a wooden basket, and an aging man. His process is not that of having a mental database of images he can draw, and then trying to put together a picture from it; he has the message that he wants to convey in mind, and then goes and finds images to represent it, with all the cultural implications included, and isn't limited to whatever he happens to find in a database.

We could replicate this effect in our program by first, conceptually segmenting our image not only by color, but by semantics as well, by hand. Secondly, we could alter the database such that we only include objects that we feel would match the semantic content of a particular image.

Given that we are using a very mechanical way of laying down our tiles and that we are only trying to match the contours of the image, our techniques have produced reasonably good results. We have extended and improved the original program and the results are have met our goals of filling in image gaps and fixing the sprites, all while having fun creating interesting, fruity pictures!

---

References

1. E.M. Arkin, L.P. Chew, D.P. Huttenlocher, K. Kedem, J.S.B. Mitchell "An Efficiently Computable Metric for Comparing Polygonal Shapes" IEEE Transactions on Pattern Analysis and Machine Intelligence. March 1991 Vol. 13 No. 3, p. 209-216., Arkin et. al
2. Dorin Comaniciu, Peter Meer. "Robust Analysis of Feature Spaces: Color Image Segmentation." CVPR97, USA.
3. Colin Zheng, Nilesh Dalvi. "Composite Head" CSE 557 Final Project. http://www.cs.washington.edu/education/courses/557/02wi/projects/final-project/nc/CSE557.htm