Archive for the ‘ Computational Cameras ’ Category


fukuoka  026

Sakura in Fukuoka

Hanami or flower viewing is commonly associated with the act of viewing cherry blossoms.

I think I’ve found my idea for my Computational Cameras final. It has an intresting history that dates back to the 18th century on how a single flower can symbolize extreme beauty and a quick death due to the fleeting nature of the cherry blossoms.

nagasaki  055

Cherry trees at Nagasaki ground zero

I’ll be implementing these in my final project by replacing it the lifespan of the cherry blossoms as you move through a given space.

So for the next few weeks I’ll be spending time at the Brooklyn Botanical Gardens to shoot pictures and video. I’m not sure yet if the petals will be falling but if not. I’ll attempt to recreate them in Processing.

Tokyo 1954

Notes to follow.

Flash cards


While most everyone is using Junaio I decided to use the AR toolkit instead. I wanted to make something like magnetic letters on a refrigerator. I got the idea for this in church where each code represents a word.

One of the first things I encountered was the words were coming out in reverse. Even though I’ve already used the pushMatrix and popMatrix command in Processing 2.04a, the video now mirrored me, but the text was still in reverse. I attribute this to a bug but I’m not so sure. To solve this annoying problem, I projected the sketch to the wall and reverse projected it. Thus the letters finally came out the right way.

I just used a short phrase composing of 8 words, but the current library does not allow the same AR image to be repeated twice. Or I may need to fiddle with the code some more.

Putting the AR cards side by side gives me a phrase.


What could this secret messgae mean?


This may give me an idea for the final project if I plan to expand this.


// Processing 2.04a + NyARToolkit 1.1.7
//pared down from Amnon Owed

import*; // for the loadPatternFilenames() function
import processing.opengl.*; // for OPENGL rendering
import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library

PFont myFont;
String the ="there";
String is ="is";
String no ="no";
String id ="I";
String there ="there";
String is2 ="is";
String only ="only";
String us ="us";

Capture cam;
MultiMarker nya;

void setup() {

size(640, 480, OPENGL);
cam = new Capture(this, 640, 480);

// create a new MultiMarker at a specific resolution (arWidth x arHeight), with the default camera calibration and coordinate system
nya = new MultiMarker(this, width, height, "camera_para.dat", NyAR4PsgConfig.CONFIG_DEFAULT);
// set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.

nya.addARMarker("4x4_1.patt", 80); //your have to print out the cooresponding pdf file and put the .patt files in data folder
nya.addARMarker("4x4_2.patt", 80);
nya.addARMarker("4x4_3.patt", 80);
nya.addARMarker("4x4_4.patt", 80);
nya.addARMarker("4x4_5.patt", 80);
nya.addARMarker("4x4_6.patt", 80);
nya.addARMarker("4x4_7.patt", 80);
nya.addARMarker("4x4_8.patt", 80);

void draw() {

background(255); // a background call is needed for correct display of the marker results;
//image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window
// flip image horizonatlly
scale(-1, 1);
translate(-cam.width, 0);
// image(cam, 0, 0, width, height);
// pushMatrix();
// scale(-1,0);
nya.detect(cam); // detect markers in the input image at the correct resolution (incorrect resolution will give assertion error)

if (nya.isExistMarker(0)) {
setMatrix(nya.getMarkerMatrix(0)); //use this marker to translate and rotate the processing drawing
translate(0, 0); //offset half the size of the cube.
textFont(myFont, 24);
text(the, 0, 0);
if (nya.isExistMarker(1)) {
translate(0, 0);
textFont(myFont, 24);
text(is, 0, 0);

if (nya.isExistMarker(2)) {
translate(0, 0);
textFont(myFont, 24);
text(no, 0, 0);

if (nya.isExistMarker(3)) {
translate(0, 0);
textFont(myFont, 24);
text(id, 0, 0);

if (nya.isExistMarker(4)) {
translate(0, 0);
textFont(myFont, 24);
text(there, 0, 0);

if (nya.isExistMarker(5)) {
translate(0, 0);
textFont(myFont, 24);
text(is2, 0, 0);

if (nya.isExistMarker(6)) {
translate(0, 0);
textFont(myFont, 24);
text(only, 0, 0);

if (nya.isExistMarker(7)) {
translate(0, 0);
textFont(myFont, 24);
text(us, 0, 0);

// popMatrix();

Ultra HD Digital 8k

It’s been a long time coming but it’s application remains to be seen.

In 2006, researchers at the NHK demonstrated in Las Vegas a transmission of Ultra High Definition Television with 22.2 surround sound. THe broadcast was from Tokyo to Osaka via an IP network running at 1Gbps. Uncompressed, the sound signal alone an at 20 Mbps while the video signal ran at 24 Gbps.

Current broadcast standards runs at MPEG-2 compressions with a maximum resolution of 1920 x 1080. Ultra HD runs at 7680 x 4320 pixels.

Developed by the NHK in Japan, ultra HD has 4000 scanning lines compared to just 1080 for the current broadcast system.

In 2007, SMPTE approves the Ultra HDTV as a standard format.

The BBC will broadcast the London Summer Olympics in ultraHD.

Each frame is equal to 33 Megapixels.

I can see this as the digital IMAX but more. The 100 degree viewing angle allows for an image that can simulate human perception. It’s quite hard to describe since the image is huge but the experience is almost realistic.

This type of imaging is a step forward to building that holodeck. The amount of detail that the resolution provides will be able to show more infomation for computers to see. Although current limitations would be enough processing power to process the data.


Camera Walk?

I’ve posted it before I think that I always wanted a holodeck. But of course this is nothing like that. So for this project I finally got the network camera working at my place in Queens and uploading the images every 15 minutes. I shot some video from a window at ITP and put the two together.

I knew I was going to use the Kinect and initially began using the depthMap and measuring in inches and using the values there to determine distance. That didn’t work for me. I decided to use the Center of Mass or CoM command to determine the position but then depth would be another problem.

Soooo for the purposes of this project I just adjusted the position of the kinect to be overhead to simplify the position.

Adding a fornt facing camera turned out to be more challenging. For some reason, OpenNI takes over all the cameras of the computer and will only want the kinect. Solution? Add another kinect.

I’d like to expand this further with head tracking such as imitating the look around an area or even creating the illusion of depth without 3D glasses.

Cameras midterm

I have no name for this as of yet but over the past few weeks I’ve been trying to recreate what I built five years ago. Sadly through the updates that Apple has made, it is no longer possible with their technology. Streaming video from one place to another seems simple enough and is actually simpler today than it was when I was building it. But getting it to place nice with Java is another thing completely.

I was using both VLC and QuickTime Broadcaster to broadcast video from one place to another without the aid of a server to be imported into a Processing applet and that didn’t turn out too well. Apple has discontinued further development on QuickTime for Java in lieu of AVFoundation. So the Quicktime that we all grew up with no longer exists. the core of AVFoundation is used to run movies in iOS, iMovie and FCPX which is why iMovie runs faster and somewhat better than FCP 7. Oh the pains of 64 bit processing.

The QT Broadcast stream, though encoded in MotionJPEG which Java recognizes, still starts with a QT header. Thus when the stream is imported into the sketch, it will not recognize it as a MotionJPEG file but as a QT streaming file. This would require a decoder to transcode it into MotionJPEG that Java accepts.

Installing a Lion server will not help. I learned that the hard way.

So currently I’m using an application called EvoCam. It’s a standalone software that recognizes practically any camera I plug in and allows a ton of options. From motion capture recording/ streaming video to grabbing stills and sending them through applescript or automator workflow. I’ve been having ftp connectivity issues to the ITP server so I’m uploading this to my personal site. So far it’s been working great save for the times of me getting disconnected on the remote end for some reason or another.

remote still

Refresh the page to get the latest image.

Once the image was finally free on the web it was easy enough to access the code in Processing. I’ve always wanted a holodeck no matter how crude it may be. I think the fact that you are walking through a given space that changes would change on how we would interact with man made images or environments. So this is a very crude version of how I would imagine things. More updates to come.

Cameras of the world 5 years from now

There are views that more cameras out there could mean two things. One is that We can finally get a grasp on what’s going on in the world. No longer can dictators and criminals hide from us. For once we can finally generate our own opinions on subjects that before, took an army of journalists to capture and analyze. We can finally have our own opinion. Then of course there’s the downside. It’s who is behind the cameras is the scary part. We already live part in that world. Our every movement is captured and stored into servers for who knows how long.

Cameras enabled it’s creators to preserve their time and space and it continues to do so today. The 2011 Japan Earthquake was so devastating that we were getting live images as the tsunami swept through the northern region. Users shared their videos of the quake as it was happening and for the first time, the word could see terrible disaster live.

I for one would like to be optimistic about where technology is leading us in terms of cameras. I long for the images of the old cities in my home. I wish I could re-create the city the way it was before World War II or even better, re-create Old Manila during the Spanish era. We would be able to take a walk into history so to speak, understand and experience the place and time where my grandparents and great-grandparents lived. Like a living holodeck based on information from the past.

Cameras are something we fear about today. But it’s something that our descendants would look for in the future.

Apple bite

Watch here


import oscP5.*;
OscP5 oscP5;

//crane[] crane;
PImage crane1;
PImage crane2;
PImage crane3;
PImage crane4;
PImage crane5;
PImage apple;

String crane= "crane1, crane2, crane3, crane4, crane5";

PVector posePosition;
boolean found;
float eyeLeftHeight;
float eyeRightHeight;
float mouthHeight;
float mouthWidth;
float nostrilHeight;
float leftEyebrowHeight;
float rightEyebrowHeight;

float[] chew = new float [5];
//float[] crane = new float [crane1, crane2, crane3, crane4, crane5];

//float chew = 0;

PVector[] meshPoints;

float poseScale;

void setup() {
size(640, 480);

for (int i =0; i < chew.length; i++) {
chew[i] += 1;

// crane = new crane [crane1, crane2, crane3, crane4, crane5];
crane1 = loadImage("crane01.JPG");
crane2 = loadImage("crane02.JPG");
crane3 = loadImage("crane03.JPG");
crane4 = loadImage("crane04.JPG");
crane5 = loadImage("crane05.JPG");
apple = loadImage("apple.jpg");

meshPoints = new PVector[66];

for (int i = 0; i < meshPoints.length; i++) {
meshPoints[i] = new PVector();

oscP5 = new OscP5(this, 8338);
oscP5.plug(this, "mouthWidthReceived", "/gesture/mouth/width");
oscP5.plug(this, "mouthHeightReceived", "/gesture/mouth/height");
oscP5.plug(this, "eyebrowLeftReceived", "/gesture/eyebrow/left");
oscP5.plug(this, "eyebrowRightReceived", "/gesture/eyebrow/right");
oscP5.plug(this, "eyeLeftReceived", "/gesture/eye/left");
oscP5.plug(this, "eyeRightReceived", "/gesture/eye/right");
oscP5.plug(this, "jawReceived", "/gesture/jaw");
oscP5.plug(this, "nostrilsReceived", "/gesture/nostrils");
oscP5.plug(this, "found", "/found");
oscP5.plug(this, "poseOrientation", "/pose/orientation");
oscP5.plug(this, "posePosition", "/pose/position");
oscP5.plug(this, "poseScale", "/pose/scale");
oscP5.plug(this, "loadMesh", "/raw");

void draw() {
for (int i=0; i 1) {
image(apple, 0, 0, 640, 480);
/* if (found) {
for (int i = 0; i 0) {
PVector prev = meshPoints[i-1];
line(prev.x, prev.y, p.x, p.y);

/translate(posePosition.x, posePosition.y);
// ellipse(0,0, 3,3);
ellipse(-20, eyeLeftHeight * -9, 20, 7);
ellipse(20, eyeRightHeight * -9, 20, 7);
ellipse(0, 20, mouthWidth
3, mouthHeight * 3);
ellipse(-5, nostrilHeight * -1, 7, 3);
ellipse(5, nostrilHeight * -1, 7, 3);
rect(-20, leftEyebrowHeight * -5, 25, 5);
rect(20, rightEyebrowHeight * -5, 25, 5);

public void mouthWidthReceived(float w) {
// println("mouth Width: " + w);
mouthWidth = w;

public void mouthHeightReceived(float h) {
println("mouth height: " + h);
mouthHeight = h;

public void eyebrowLeftReceived(float h) {
// println("eyebrow left: " + h);
leftEyebrowHeight = h;

public void eyebrowRightReceived(float h) {
// println("eyebrow right: " + h);
rightEyebrowHeight = h;

public void eyeLeftReceived(float h) {
// println("eye left: " + h);
eyeLeftHeight = h;

public void eyeRightReceived(float h) {
// println("eye right: " + h);
eyeRightHeight = h;

public void jawReceived(float h) {
// println("jaw: " + h);

public void nostrilsReceived(float h) {
// println("nostrils: " + h);
nostrilHeight = h;

public void found(int i) {
println("found: " + i); // 1 == found, 0 == not found
found = i == 1;

public void posePosition(float x, float y) {
//println("pose position\tX: " + x + " Y: " + y );
posePosition = new PVector(x, y);

public void poseScale(float s) {
//println("scale: " + s);
poseScale = s;

public void poseOrientation(float x, float y, float z) {
//println("pose orientation\tX: " + x + " Y: " + y + " Z: " + z);

public void loadMesh(float x0, float y0, float x1, float y1, float x2, float y2, float x3, float y3, float x4, float y4, float x5, float y5, float x6, float y6, float x7, float y7, float x8, float y8, float x9, float y9, float x10, float y10, float x11, float y11, float x12, float y12, float x13, float y13, float x14, float y14, float x15, float y15, float x16, float y16, float x17, float y17, float x18, float y18, float x19, float y19, float x20, float y20, float x21, float y21, float x22, float y22, float x23, float y23, float x24, float y24, float x25, float y25, float x26, float y26, float x27, float y27, float x28, float y28, float x29, float y29, float x30, float y30, float x31, float y31, float x32, float y32, float x33, float y33, float x34, float y34, float x35, float y35, float x36, float y36, float x37, float y37, float x38, float y38, float x39, float y39, float x40, float y40, float x41, float y41, float x42, float y42, float x43, float y43, float x44, float y44, float x45, float y45, float x46, float y46, float x47, float y47, float x48, float y48, float x49, float y49, float x50, float y50, float x51, float y51, float x52, float y52, float x53, float y53, float x54, float y54, float x55, float y55, float x56, float y56, float x57, float y57, float x58, float y58, float x59, float y59, float x60, float y60, float x61, float y61, float x62, float y62, float x63, float y63, float x64, float y64, float x65, float y65) {
println("loading mesh...");
meshPoints[0].x = x0;
meshPoints[0].y = y0;
meshPoints[1].x = x1;
meshPoints[1].y = y1;
meshPoints[2].x = x2;
meshPoints[2].y = y2;
meshPoints[3].x = x3;
meshPoints[3].y = y3;
meshPoints[4].x = x4;
meshPoints[4].y = y4;
meshPoints[5].x = x5;
meshPoints[5].y = y5;
meshPoints[6].x = x6;
meshPoints[6].y = y6;
meshPoints[7].x = x7;
meshPoints[7].y = y7;
meshPoints[8].x = x8;
meshPoints[8].y = y8;
meshPoints[9].x = x9;
meshPoints[9].y = y9;
meshPoints[10].x = x10;
meshPoints[10].y = y10;
meshPoints[11].x = x11;
meshPoints[11].y = y11;
meshPoints[12].x = x12;
meshPoints[12].y = y12;
meshPoints[13].x = x13;
meshPoints[13].y = y13;
meshPoints[14].x = x14;
meshPoints[14].y = y14;
meshPoints[15].x = x15;
meshPoints[15].y = y15;
meshPoints[16].x = x16;
meshPoints[16].y = y16;
meshPoints[17].x = x17;
meshPoints[17].y = y17;
meshPoints[18].x = x18;
meshPoints[18].y = y18;
meshPoints[19].x = x19;
meshPoints[19].y = y19;
meshPoints[20].x = x20;
meshPoints[20].y = y20;
meshPoints[21].x = x21;
meshPoints[21].y = y21;
meshPoints[22].x = x22;
meshPoints[22].y = y22;
meshPoints[23].x = x23;
meshPoints[23].y = y23;
meshPoints[24].x = x24;
meshPoints[24].y = y24;
meshPoints[25].x = x25;
meshPoints[25].y = y25;
meshPoints[26].x = x26;
meshPoints[26].y = y26;
meshPoints[27].x = x27;
meshPoints[27].y = y27;
meshPoints[28].x = x28;
meshPoints[28].y = y28;
meshPoints[29].x = x29;
meshPoints[29].y = y29;
meshPoints[30].x = x30;
meshPoints[30].y = y30;
meshPoints[31].x = x31;
meshPoints[31].y = y31;
meshPoints[32].x = x32;
meshPoints[32].y = y32;
meshPoints[33].x = x33;
meshPoints[33].y = y33;
meshPoints[34].x = x34;
meshPoints[34].y = y34;
meshPoints[35].x = x35;
meshPoints[35].y = y35;
meshPoints[36].x = x36;
meshPoints[36].y = y36;
meshPoints[37].x = x37;
meshPoints[37].y = y37;
meshPoints[38].x = x38;
meshPoints[38].y = y38;
meshPoints[39].x = x39;
meshPoints[39].y = y39;
meshPoints[40].x = x40;
meshPoints[40].y = y40;
meshPoints[41].x = x41;
meshPoints[41].y = y41;
meshPoints[42].x = x42;
meshPoints[42].y = y42;
meshPoints[43].x = x43;
meshPoints[43].y = y43;
meshPoints[44].x = x44;
meshPoints[44].y = y44;
meshPoints[45].x = x45;
meshPoints[45].y = y45;
meshPoints[46].x = x46;
meshPoints[46].y = y46;
meshPoints[47].x = x47;
meshPoints[47].y = y47;
meshPoints[48].x = x48;
meshPoints[48].y = y48;
meshPoints[49].x = x49;
meshPoints[49].y = y49;
meshPoints[50].x = x50;
meshPoints[50].y = y50;
meshPoints[51].x = x51;
meshPoints[51].y = y51;
meshPoints[52].x = x52;
meshPoints[52].y = y52;
meshPoints[53].x = x53;
meshPoints[53].y = y53;
meshPoints[54].x = x54;
meshPoints[54].y = y54;
meshPoints[55].x = x55;
meshPoints[55].y = y55;
meshPoints[56].x = x56;
meshPoints[56].y = y56;
meshPoints[57].x = x57;
meshPoints[57].y = y57;
meshPoints[58].x = x58;
meshPoints[58].y = y58;
meshPoints[59].x = x59;
meshPoints[59].y = y59;
meshPoints[60].x = x60;
meshPoints[60].y = y60;
meshPoints[61].x = x61;
meshPoints[61].y = y61;
meshPoints[62].x = x62;
meshPoints[62].y = y62;
meshPoints[63].x = x63;
meshPoints[63].y = y63;
meshPoints[64].x = x64;
meshPoints[64].y = y64;
meshPoints[65].x = x65;
meshPoints[65].y = y65;

void oscEvent(OscMessage theOscMessage) {
if (theOscMessage.isPlugged()==false) {
println("UNPLUGGED: " + theOscMessage);

Face tracking projects

  • In car face tracking – to detect driver fatigue and other behavior.
  • PS Eye head tracking to move the POV of the game according to the position of the head.
  • Example
  • Facial tracking in cars to control objects in the car and identify the driver.
  • Facial recognition for identification of people in photos as depicted in Apple’s iPhoto and Aperture
  • To target objects in real life and fire deadly missles like Apache Helicopter Pilots.

How to get from point A to point B

I’ve made documentaries before and it’s not something entirely new. We would basically tell a story and the one way of telling a story is just simply to move forward. “Connecting” seems to be just as complicated as it sounds.

For one, no longer do you have a captured audience for them to digest your work in one sitting. The audience can now jump from one part of your story to another. They can now view your documentary on their own time and only the parts that they are interested in. So is there a way for the filmmaker’s view to translate properly into the hyperlinked form?

I think it’s possible but I think the material requires two different approaches resulting in the same ending. There will be path as determined by the author to be the “true vision” for the work. This is exactly as the creator determined it and should be experienced as such. The other path would be the one created by the user as they go along the story on their time and their own interests.

With this in mind I think traditional documentary methods still hold true for two of my proposals. One is the train ride and the other is Woody Allen map of Manhattan but I think I can narrow it down to his film (Manhattan, 1979).,-95.677068&sspn=14.84512,90&t=w&ie=UTF8&hq=&hnear=New+York&ll=40.714353,-74.005973&spn=0.080645,0.175781&z=13&output=embed

View Larger Map

For “Manhattan”, there are various sources for the film such as the Internet Movie Database and Google Maps to determine these locations. For instance, the iconic scene with Woody Allen and Diane Keaton was shot exactly on this spot. As you can see today, there is no park bench. I wonder what this place looks like at sunrise, as depicted in the film.,+New+York,+NY&aq=0&oq=59&sll=40.761958,-73.973443&sspn=0.027565,0.175781&t=w&ie=UTF8&hq=&hnear=E+59th+St,+New+York&ll=40.761553,-73.966584&spn=79.695795,180&z=3&layer=c&panoid=69sIySJ7ZVcr3Yfe3RQIPQ&cbll=40.757793,-73.959552&cbp=13,107.65876970508172,,0,5.099324376664313&output=svembed,+New+York,+NY&aq=0&oq=59&sll=37.0625,-95.677068&sspn=42.987658,90&ie=UTF8&hq=&hnear=E+59th+St,+New+York&t=h&layer=c&cbll=40.757793,-73.959552&panoid=69sIySJ7ZVcr3Yfe3RQIPQ&cbp=13,109.18,,0,0.44&ll=40.751825,-73.96142&spn=0.020416,0.048237&z=14&output=svembed

View Larger Map

Woody Allen and Diane Keaton in Manhattan

Then embed this scene,

That should work. It would allow the film to be enjoyed in a different way. Something like that.

As for the train, I’ve downloaded maps and brochures from Amtrak that would get me to the west coast. Sadly I couldn’t find an API for Amtrak but Google Maps does the same anyway.

Based on the railyway system of the US, there are two ways to the west, either through the north or sounth, the tracks don’t exactly cross the US like the highways. Also, it would take two trains. The Lake Shore Limited and the California Zephyr. I think this is where the two paths I mentioned above can be created. One is my own journey on the train that would include, videos, pictures, and posts from along the train that would be the filmmaker’s point of view. The other is where the user follow the train as it went through the track in real time via Google Earth. The user can then see the stops that are marked along the way to get from point A to point B and any historical significance it would have. This would also work along the east coast, but I think riding across the country would be very exciting. Or maybe California Highway 1.

Technologies I think I see in integrating would be:

  • HTML5
  • Google Maps API
  • YouTube API
  • a bit of Javascript

That’s all I can think of for now.