Assignment 5

Alas, here it is! Reflections!

Good news is that I have reflections working the way I want, bad news is that I’ve incurred a lot of technical debt in the process so I’ll be working some refactoring this weekend. Truth be told, I’ve been working on it for a while (another completely codebase), but I haven’t had the time to devote to polish yet. I have some extra time this weekend, so I’m planning on going to town and getting the other codebase back up to speed. More to come on that soon

I was encountering an issue earlier where Wordpress wasn’t letting me post updates, but alas, that’s all fixed and the site is back up and running. Whoo hoo!

In other news, I just got an apartment in Pittsburgh this past weekend, so that’s really awesome! I’ll be a mile away from Apple Pittsburgh, and it’s in a great area of town. Here are some photos of the place (although the photos contain the current residents’ things ): http://www.facebook.com/media/set/?set=a.10150496899807541.364551.505467540&type=1&l=9e0ef01420

Also, I was named GCCIS College Delegate recently too, so I’ll be speaking at commencement! I’m really excited about this and it’s really huge. I can’t wait!

Remember how last week’s ray tracer was in OpenGL and GLUT? Boy, that was a mistake. Although I got the end result I was hoping for, I really shot myself in the foot for future increments. The primary reason for this was that I was attacking the problem the wrong way. Although I was able to rasterize an end result that looked visually correct, it relied on the OpenGL and GLUT calls to rasterize the end result, and I found myself having to write some janky and nefarious workarounds to make it work with GLUT that just (for lack of a better phrase) made me feel dirty.

So, on Tuesday night I decided something had to go and scrapped everything and began anew. This time, I’m taking the high road and writing everything from scratch in C++. This release has its own geometry classes (CGRay, CG3DPoint, CG3DVector, CGNormal, CGMatrix, etc.) and all that business is happening in an O-O way.

I’m happy with how it’s coming together now, and the design I’m very satisfied with. Moreover, the extra push I had to get this release working The Right Way™ will help grease the skids in the future.

Here’s a couple other improvements which I feel will help me in the future:

• The scene is now centered at the camera at 0,0,0. Now everything can be positioned relative to the camera in absolute space, which gives me some cleaner numbers to work with when manipulating objects.
• Ability to do supersampling.

Here’s the default output image:

Here’s a supersampled image, notice that it looks a little bit crisper (look closely):

I did not have time to add an extra object type for this release due to time constraints, but I’ll be working on that in the future, perhaps in terms of an .obj or .dae loader.

I’m also looking into deferring some of the processing to the GPU (CUDA) to increase rendering performance, and I plan to look at that over the break. I’ve installed the SDK and I’m dabbling with that now, and I hope to have some elements of GPU-accelerated rendering for the next release.

At any rate, there’s lots of cool stuff happening under the hood! Take a look:

For simplification purposes, a CGShape is an abstract class that (at this point) is either a CGSphere or CGRect.

There’s lots left to do though, so I’m going to capitalize on the break to get some extra polish done on the ray tracer. So far this scene rendered in 0.5 seconds. Not too bad, but I think I can do better!

PS: Every release I have for personal software projects is accompanied by a tagline and a silly image. This release was called “Not What You Expected” because my previous release ultimately had to be thrown out – something I most certainly didn’t expect

Setting up the Scene

In this assignment I began my recreation of Turner Whitted’s famous ray tracer published in his 1980 paper. The assignment was relatively straightforward, but the majority of my time was spent getting the objects to appear as they did in Whitted’s paper. The image I am trying to replicate looks like this:

So, as far as the objects are concerned and the position of the objects… not too bad! Here are the parameters I used to generate the scene:

• Size and location of two spheres:
  • Big Sphere (foreground)
    • X: -1.83
    • Y: 1.1
    • Z: 8.3
    • Size: 1.0
  • Small Sphere (background)
    • X: -0.83
    • Y: 0.35
    • Z: 7.55
    • Size: 1.0
• The size and location of the floor (conveyed by locations of vertices):
  • Vertex1
    • X: -3.0
    • Y: 0.0
    • Z: 10.0
  • Vertex2
    • X: 3.0
    • Y: 0.0
    • Z: 10.0
  • Vertex3
    • X: 3.0
    • Y: 0.0
    • Z: 0.0
  • Vertex4
    • X: -3.0
    • Y: 0.0
    • Z: 0.0
• The location and parameters of your light source:
  • X: -1.5
  • Y: 9.0
  • Z: -10.0
• The location and “lookat” of your camera:
  • Eye
    • X:  -1.65
    • Y: 0.9
    • Z: 11.2
  • Center
    • X: -1.675
    • Y: -0.005
    • Z: 0.65
  • Up
    • X: 0.0
    • Y: 1.0
    • Z: 0.0

The talk went over very well and afterwards we all made a bacon detector, and had some fun with optical flow.

The slide deck is available here [PDF] if you’d like to see it. Thanks to everyone who came!

The primary notion behind APHOG is that it first does a coarse-grained parts-based detection to suggest possible areas of interest. In this case, we were searching for hands, torsos, faces, etc.

Once the algorithm finds an area that had these features, the algorithm knew (approximately) where a body would be in relation to the feature found. For example, if a head was found, the body likely be below the detected head.

APHOG detecting the possible features of pedestrians (red bounding boxes) and suggesting possible areas of pedestrians (blue bounding boxes)

This allows the the algorithm to perform a full holistic detection (using a series of Histogram of Oriented Gradients – referred to as the “pyramid”) in designated areas of interest – which is a huge performance (and accuracy) win. We feel that it is an improvement over conventional holistic algorithms that scan the entire frame at many zoom levels and parts-based detections that have to run complex logic to determine if the geometry of detected features are connected in such a way that could be a human shape.

If you’re curious to see more how it works and the gory details (and see it in action), please see the links below!

• Final Presentation [PDF]
• Research Paper [PDF]

The iOS Tech Talk this past Thursday was a smash hit! Not only did the lab overflow with people, we covered a lot of ground with Objective-C, Xcode, Interface Builder, Cocoa Touch and parts of the iOS SDK.

Bittersweet news is that the room was so packed we couldn’t squeeze a video camera in there. So the best I can offer everyone is the slide deck as well as the sample code we wrote together. I have both an annotated version of the slides with my speaker notes, as well as a pure image form. Links for these items are at the bottom of this post.

However, since there was so much interest, I plan to host a series of Crash Courses on advanced topics in iOS as well as OS X programming such as Grand Central Dispatch, multitouch gestures, OpenGL ES, consuming network services and many more. More to come on this development soon, but if you’re interested in taking a Crash Course, please indicate what you’re interested in at the form located here.

Also, as promised, for those of you looking for more information

And as always, if you have any questions or feedback, please don’t hesitate to let me know. Thanks for everyone who came and made the night a great success!

Downloads

Source Code
Slides with Notes
Plain Slides

Your mission, should you choose to accept it: Detect all the letter i’s on a document!

Couple of things to note:

• Not all of the letter i’s will look the same (there will be some blotches of ink that make the letters look a little different)
• You won’t have knowledge of what every single letter i looks like (or ever will look like) in the universe
• You won’t have to recognize handwriting (that’s another party in and of itself)

So let’s get started! We’ll be using the following document for our character detection:

First, we can break the problem down into a bunch of subtasks, namely:

So let’s get started! I’ll be using MATLAB and the Image Processing Toolbox for my implementation in this post, but I’ll also attach a script which accomplishes the same results using Python.

Manipulating the Data

First things first, we need to read in our image. We’ll do that as follows:

im = imread(‘TextImage.tif’);
im = im2double(im);

This reads in the TextImage.tif image as a grayscale image,  and assigns each pixel to an intensity between 0 and 1 inclusive. However, we’re not quite done yet. By convention, data of interest is represented as white (all 1’s), and data which isn’t of interest being black (all 0’s) – so we’ll want to invert our image to show that the black text we just read in is of interest (e.g. white) we’ll do that as follows:

im = imadjust(im, [0 1],[1 0]);

THRESHOLD = 0.05;

im = im > THRESHOLD;

Cropping the Image

Although this is not an essential step, and the algorithm which we’ll be implementing soon will function fine without it, we’re going to restrict the area which we’ll be looking at. This helps for visualization in the end, and can be accomplished pretty quickly.

im = im(400:end,:);

This line of code cuts off everything above 400px, and leaves the rest of our image intact (end MATLAB knows means “go to the end of the image”, it’s not a variable we set earlier).

Finding Templates

The next goal we’ll have is to extract i’s from the image above that are of interest to us (computer vision junkies call these ‘templates’). I usually open up Adobe Photoshop and manually select areas which are of interest, and copy and paste the pixel coordinates. It’s important to note that we’re ‘cheating’ on some level as we have a priori knowledge of what our document already looks like – a luxury which many commercial OCR software systems don’t often have

Alas, I’ve taken some of the dirty work out for you, and found some i’s which are indicative of i’s across the entire document. Here’s my findings:

templateOne = im(510:542, 532:551);
templateOne = im2double(templateOne);

templateTwo = im(68:100, 390:409);
templateTwo = im2double(templateTwo);

templateThree = im(266:298, 899:918);
templateThree = im2double(templateThree);

templateFour = im(67:99, 144:163);
templateFour = im2double(templateFour);

templateFive = im(168:200, 875:894);
templateFive = im2double(templateFive);

So we now have five templates (patches of the letter i in the image), each template is 19 pixels wide and 32 pixels tall. Now we’re going to combine all of them into an array for easy access later.

templates(:,:,1) = templateOne;
templates(:,:,2) = templateTwo;
templates(:,:,3) = templateThree;
templates(:,:,4) = templateFour;
templates(:,:,5) = templateFive;

This creates a three dimensional array. In the first two dimensions, we store the grayscale intensity (again from 0 to 1 inclusive) of the templates we found. Each index in the third dimension represents another template (The colon (:) operator in MATLAB is shorthand for “include everything”). So imagine a cardboard box stacked with a bunch of sheets of paper of the letter i on them, roughly speaking.

Finding the rows of text

Here’s some fun debauchery. We need to detect where each row of text  begins so that we know where we can match our templates to, and this can be accomplished fairly quickly.

First, what we need to do is rotate the image 90 degrees (put it on its side), and add up the pixel values in each column. Note that now that the image is rotated 90 degrees, what used to be rows are now columns. We’re rotating the image 90 degrees as it plays nicer with MATLAB’s sum function (only adds up columns). We’ll then be storing this in an array called spaces.

spaces = sum(im’);

Next, we need to determine (another) threshold, the intensity of what each row of text should be when summed up. If the sum of each row doesn’t match a threshold, there probably isn’t text there, so it’s not the beginning of a row. We can threshold the row sums by doing the following:

ROW_SPACE_THRESHOLD = 10;
spaces = spaces > (ROW_SPACE_THRESHOLD);

Now let’s detect where changes occur by taking the derivative of spaces. This highlights where values changed from 0 to 1 and vice versa. Begins stores where each row begins (derivative is 1), and ends stores where each row ends (derivative is -1).

spaceDerivative = diff(spaces);
begins = find(spaceDerivative == 1);
ends = find(spaceDerivative == -1);

Correlating the template

Before we sink our teeth into the meat of the OCR, we first need to define thresholds for each character template we found. Roughly speaking, this is the amount of ’slack’ we’ll give to each character we find. If it’s within this ’slack’ region, the program will think the character we found is an i, otherwise it won’t be. This is useful as we’ll never know what every single i looks like (and even if we did, that algorithm would be extremely inefficient), so this defines a level of tolerance in how much a character we are analyzing can deviate from our templates of the letter i to actually be considered the letter i. We’ll be definining these using integer constants determined by experimentation.

MAX_FILTERED_I_VALUES = [ 145, 147, 147, 170, 150 ];

Each index of the MAX_FILTERED_I_VALUES corresponds to the tolerance for each template (e.g. the first number corresponds to the first template, and so on). The meaning and purpose of these templates will be explored in greater detail very soon.

Now! This is where the fun starts. First we’re going to want to iterate through each row we found. We can do that like so:

for i = 1 : (min([length(begins) length(ends)])),

Next we’re going to want to extract each row, and then shortly thereafter each character from each row.  We can get the current row this way:

currentRow = im(begins(i):ends(i),:);

This says, “hey for this particular row number (index), we only care about pixels from whatever you found from the beginning to the end”. Now we need to find the spaces in between the characters:

row_spaces = sum(currentRow);
row_spaces = row_spaces > 0.8;
row_diff = diff(row_spaces);

charBegin = find(row_diff == 1);
charEnd = find(row_diff == -1);

This is a similar approach to what we did before, in that we’re taking this row of text we found, checking to see if it matches a threshold (which I found by experimentation for you) in each column to see if there’s actually text there, and then taking the derivative of that threshold to see where each character begins and ends.

Now let’s loop through each character we found:

for j = 1 : min([length(charEnd) length(charBegin)]),

Let’s get things set up for processing:

currentChar = currentRow(:,charBegin(j):charEnd(j));
currentChar = im2double(currentChar);

possibleMach = 0;

In the above code, we’re segmenting the character from the image and converting its intensity to a double (why we’re doing this will be explained soon). We’re also creating a flag called possibleMatch to determine if we’ve actually found something that is likely to be an i.

Now for each character we found, iterate through the templates we extracted earlier to see if there’s a match. Enjoy your O(n^3) efficiency . We can iterate through each template this way (again, this is MATLAB shorthand for: every item in templates, take whatever’s in the third dimension – which is where we stored the index of our templates).

for k = 1 : length(templates(1,1,:)),

And we can correlate our above character with the template this way:

filtered = abs(imfilter(currentChar,templates(:,:,k),’corr’));
possibleMach = max(filtered(:)) > MAX_FILTERED_I_VALUES(k);

This is where the magic happens! Let’s break apart what we’re doing here:

In the first lone of code, we do something called a correlation (specified by the ‘corr’ string to MATLAB). The correlation operation multiplies each pixel in the neighborhood by the corresponding pixel in the template. The output values (in a Cartesian x,y plane) are a sum of the results. The output value will be the highest at points where the template closely matches. If the value is determined to be a possible match by matching a constant threshold (MAX_FILTERED_I_VALUES), then we have a match.

In other words, the  correlation number returned from the imfilter function is (roughly speaking) a score of how closely the character we just found matches a possible template, and if the correlation value we found is greater than the threshold which we set earlier in MAX_FILTERED_I_VALUES, we’ve got a match!

Displaying the Results

The party is over, and now it’s time to clean up.  So if we’ve found a possible match, draw a red rectangle around it:

if (possibleMach)

% Draw a red box around each character matching

finalRGB(begins(i):ends(i),charBegin(j),1) = 1.0;
finalRGB(begins(i):ends(i),charBegin(j),2) = 0.0;
finalRGB(begins(i):ends(i),charBegin(j),3) = 0.0;

finalRGB(begins(i):ends(i),charEnd(j),1) = 1.0;
finalRGB(begins(i):ends(i),charEnd(j),2) = 0.0;
finalRGB(begins(i):ends(i),charEnd(j),3) = 0.0;

finalRGB(begins(i),charBegin(j):charEnd(j),1) = 1.0;
finalRGB(begins(i),charBegin(j):charEnd(j),2) = 0.0;
finalRGB(begins(i),charBegin(j):charEnd(j),3) = 0.0;

finalRGB(ends(i),charBegin(j):charEnd(j),1) = 1.0;
finalRGB(ends(i),charBegin(j):charEnd(j),2) = 0.0;
finalRGB(ends(i),charBegin(j):charEnd(j),3) = 0.0;

break;
end

We draw a red rectangle by setting the red intensity (By MATLAB, 1st index in the third dimension of the image) to be 1 (all the way) and setting green and blue to be zero.

We’re done! Our final product now looks like this:

If you’d like to download the example source code, it can be found here:
http://chriscowdery.com/download/blog/ocr-fun.zip

Two months ago we had our first milestone release, and so far we’re really pleased with how the project has come together so far. We’ve got a lot up our sleeves, so stay tuned!

If you’re interested, here’s an excerpt from our initial blog post:

Hey and welcome to the dev blog of ‘; DROP TABLE Students; –!

Here you’ll be able to find posts relevant to our discoveries, hardships, findings and more as we carry forward with the development of our QR Code Game Framework for the 2011 SCORE Contest; spewe’ll be working on the QR Marks the Spot challenge.

This past few week we’ve been mostly keeping our heads down with configuration management, requirements and design for the project. The scope and design of the project has undergone some exciting transformations this week, and it’s really nice to see everything gaining momentum. To begin with, I’d like to share some of my findings from this week:

Workflow

A lot of exciting things have happened this week for our workflow, both internally and externally.

• We’re the proud new parents of an ActiveCollab instance, which is proving to be excellent for keeping the team on track. Not only does it tie in directly with our repository, it also gives us great metrics like burndown, schedule of deliverables, ticket tracking, and plenty of communication options. We highly recommend ActiveCollab for small to mid-sized software development teams for these reasons. Check it out!
• We’re trying really hard to be hip. We’ve got a Twitter account which we’ll be posting some of the spur-of-the-moment updates on the project, so be sure to follow us if you’re interested in some of the finer grained details.
• A classy new WordPress blog! Not only do we have a classy new theme, we’re also going to try and put some stellar content here (could happen!) as the project progresses. Glad you found us!
• We’re hopping on the Git bandwagon. We have a lot of Subversion and Mercurial vetrans on the team, but we’d like to take this opportunity to learn something new – so we’re diving head first into Git, and so far so good!

Platform

One of the requirements for our project was that it must be able to run on a mobile device; however some (polite) holy wars arose when trying to pick a specific platform to implement the project in. After some research and talking with friends, we’re currently pursuing Titanium.

Titanium has a very interesting approach to the multi-platform dilemma – it enables developers to create a rich web application using Javascript which later compiles into a native app for your device of choice. What sets Titanium apart however is that its API provides an abstract way of interfacing with platform-specific APIs; a simple call to Titanium’s API for the device’s camera (for example) will invoke different code for the Apple iPhone than it will for Google Android devices.

Is this a silver bullet for application development? Perhaps. It’s exceedingly difficult to replicate the entire breadth of functionality that a native API has to offer – but Titanium exceeded our initial expectations. With this framework, we’ll be able to rapidly prototype UI designs building off of existing Javascript libraries like MooTools and JQuery. This also let’s us do heavy data crunching on the client and the server when the time comes. Very cool.

That more or less sums up the activity this week for the team. Please don’t hesitate to leave any of your thoughts in comments. Stay tuned!

Next week we’ll be getting our hands dirty with:

• Diving more into Titanium
• Developing a WBS
• Creating wireframes for UI
• Brainstorming for some swanky names for our project

If you’re interested in the development of Seekqr, swing by the team blog at http://blog.seekqr.com/.