3D "works" in real life because the placement of our eyes combines two 2D images into one 3D image. Hold your finger at arm's length in front of you with one eye closed. Then open that eye and close the other. Notice the difference? You won't when both eyes are opened.
That's all well and good for the real world, but how to 3D images work on a flat screen? Simple: If two 2D images are taken at different angles--say, the average distance between human eyes--and we view them side-by-side while cross-eyed (with the right eye looking at the left image, and vice-versa), the image appears to pop out at us.
The trick is getting our eyes to view the correct image in order to achieve that pop-out effect, and this can be accomplished in several ways. The most iconic way is via red-and-cyan "anaglyph glasses," which work with red-and-cyan movie projector filters. The red "lens" only admits cyan light, while the cyan "lens" only admits red light. In this way, the eye only sees the light source meant for it, and the cross-eyed 3D effect is achieved without confusion or eyestrain.
Of course, the Nintendo 3DS displays its 3D images without the user needing to don a strange pair of glasses, so how does it manage to project 3D?
The Nintendo 3DS's top screen utilizes a filter called a "parallax barrier." One of the images necessary for seeing 3D is projected to the right, and the other image is projected to the left. The left image and the right image occupy alternating vertical columns of pixels, and are filtered through the parallax barrier. The barrier acts as a vent to project the images and ensure they hit your eyesight at the necessary angles to produce the desired 3D effect.
Jeff Grub used an apt comparison on Kombo.com: "Basically, it is like a pole sitting in front of a light source and your right eye can see behind the pole, but the left eye can't. Only the scale is so microscopic that your brain doesn't realize the pole is there and combines the two images as if they were one."
For the Nintendo 3DS to project its 3D illusion satisfactorily, make sure you're one to two feet away from the top screen and looking directly at it. If you look off too far to the side, the effect won't work properly.