Research ReviewKuo-Yen Lo羅国彦ロ コウエン2013.4.18Sato Laboratory, University of Tokyo
Short Curriculum Vitae• Personal Homepagehttp://www.hci.iis.u-tokyo.ac.jp/~kylo/• Period of past yearsUniversity (2006 – 2...
Short Curriculum Vitae• Language: English (TOEIC935), JLPT(N1), Chinese(Native)• Programming: C/C++, Matlab, Java(android)...
Visual CuesLow-levelMathematicsMachineLearningPsychologyComputer Vision
Overview2006NTU2007UPenn2008OpenCV2009RoboticsContestGenderRecognitionContest2012ResearchAssistantICPR2012ACCV_w20122010IS...
National Taiwan University2006NTU2007UPenn2008OpenCVPeople in Vision• Yi-Ping Hung(MM12, CVPR11, CHI11, UIST11)• Yung-Yu C...
Summer School in UPenn2006NTU2007UPenn2008OpenCVSummer Language ProgramUniversity of Pennsylvania
Join the Lab2006NTU2007UPenn2008OpenCVBiophotonics andBioimaging LaboratoryProf. Ta-Te LinOpenGLOpenCVBorlandC++ Builder
Robotics Contest @ ASABE 20092009RoboticsContestGenderRecognitionContest• Problem:Detecting and Positioning the circular o...
Robotics Contest @ ASABE 20092009RoboticsContestGenderRecognitionContestPlease Visit the following linkfor viewing the vid...
Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image...
Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image...
Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image...
Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image...
Fish Recognition2010ISMABNTUGraduateInternship@ TV corp.• Problem:Tuna species recognition for fisheryconservation and man...
Fish Recognition• Problem:Tuna species recognition for fisheryconservation and management• Task:Detection and Classificati...
Fish Recognition• Problem:Tuna species recognition for fisheryconservation and management• Task:Detection and Classificati...
Yeh, Graduation!2010ISMABNTUGraduateInternship@ TV corp.2011Navy
2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011Navy• Problem:Generating 3D videofrom 2D content.• Inspirat...
2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011NavyReality Comfort• Accurate depth map• Correct depth orde...
How people perceive depth?2010ISMABNTUGraduateInternship@ TV corp.2011Navy1. Low-level cue 2. Scene Recognition
2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011NavyVideo frame + motion estimation[ICCE 2009]Approaches1. ...
2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011Navy Video frame + Saliency map [SDA 2010]Approaches1. Dept...
Introduction to Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011NavyBilateral Filter [Tomasi, ICCV98]:f(x) h(x...
Introduction to Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011NavyBilateral Filter [Tomasi, ICCV98]:f(x) h(x...
Introduction to Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011NavyBilateral Filter [Tomasi, ICCV98]:f(x) h(x...
Application of Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011Navy“Bi” lateral = Spatial term + Range termSmo...
2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011NavyPrior fusion [Siggraph 2009]1. Decide Geometric perspec...
One-year in Navy2010ISMABNTUGraduateInternship@ TV corp.2011Navy
Academia Sinica2012ResearchAssistantICPR2012ACCV2012Institute of Information Science(Central Research Academy)People in Vi...
PHOTO AESTHETICS CLASSIFICATIONPredicting the visual appealing quality of photos
good?jcar@DPChallenge
good?Mnet @ DPChallenge
Which one is better?
Voted by online photo communityAverage: 5.088 votesAverage: 7.292 votes
Reason?
Reason?BoundaryAlternating repetition(Texture)Contrast Levels of scaleRoughnessStrong centersPositive spaceLocal symmetrie...
ApplicationImage Search & Management Photo evaluation systemEmbedded Camera system Media analysis
Photo Aesthetics2012ResearchAssistantICPR2012ACCV2012• Problem:Recognition the appealing quality ofphoto by computational ...
ICPR 20122012ResearchAssistantICPR2012ACCV2012As a Pattern Recognition Problem…Comparison of feature1. Edge distribution, ...
Extraction of Color InformationExtract NDominant colors(we set N=5)K-Nearest Neighbor(K=20)List of PalettesDictionaryHQ Pa...
Retrieved by FrequencyRetrieved by Kmeans(Cluster Center)Proposed(Weighted Kmeans)Finding the Dominant Colors
Video Demo2012ResearchAssistantICPR2012ACCV2012[ Intelligent Photographing Interface with On-Device Aesthetic Quality Asse...
Discussion 1Device : On-line assistive camera system• Contextual Information (Viewing angle)camera < human• Feedbackfrom a...
Discussion 2Algorithm: photo aesthetic value assessment• Definition of photo aestheticsExpert v.s. Volkswagen• Labeling pr...
Thanks for your attention!10 ratings5.00/7 average5 ratings4.90/7 average
of 46

Introduction to my Research

Personal Homepage http://www.hci.iis.u-tokyo.ac.jp/~kylo/
Published on: Mar 4, 2016
Source: www.slideshare.net


Transcripts - Introduction to my Research

  • 1. Research ReviewKuo-Yen Lo羅国彦ロ コウエン2013.4.18Sato Laboratory, University of Tokyo
  • 2. Short Curriculum Vitae• Personal Homepagehttp://www.hci.iis.u-tokyo.ac.jp/~kylo/• Period of past yearsUniversity (2006 – 2010)Internship (2010 – 2011)Research Assistant (2012 – 2013)• Two main research topic:1. 2D-to-3D conversion2. Photo Aesthetics.• Wrap up
  • 3. Short Curriculum Vitae• Language: English (TOEIC935), JLPT(N1), Chinese(Native)• Programming: C/C++, Matlab, Java(android)Technique: SIFT/SURF/HOG, K-means, GMM, kNN, SVM, PCA/LDA/ITML, bad-of-visual word, bilateral filter.• Have traveled to: USA, Korea.Want to travel to: China, Thailand, Spain• Why Japan-- historical and cultural connection-- camera companies and electronics maker here
  • 4. Visual CuesLow-levelMathematicsMachineLearningPsychologyComputer Vision
  • 5. Overview2006NTU2007UPenn2008OpenCV2009RoboticsContestGenderRecognitionContest2012ResearchAssistantICPR2012ACCV_w20122010ISMABNTUGraduateInternship@ TV corp.2011Navy
  • 6. National Taiwan University2006NTU2007UPenn2008OpenCVPeople in Vision• Yi-Ping Hung(MM12, CVPR11, CHI11, UIST11)• Yung-Yu Chuang(CVPR12*3)• C.J. Lin (Libsvm)• H.T. Lin(ICML12, NIPS12, CVPR11KDD12 Champion)• Winston H. Hsu(MM12*6)33,0001928 B.C.World rank 80
  • 7. Summer School in UPenn2006NTU2007UPenn2008OpenCVSummer Language ProgramUniversity of Pennsylvania
  • 8. Join the Lab2006NTU2007UPenn2008OpenCVBiophotonics andBioimaging LaboratoryProf. Ta-Te LinOpenGLOpenCVBorlandC++ Builder
  • 9. Robotics Contest @ ASABE 20092009RoboticsContestGenderRecognitionContest• Problem:Detecting and Positioning the circular obstacle• Technique:Graphical simulation(OpenGL), Sensor• Material:Boe-Bot Toolkit, IR sensor, Ultrasonicsensor, Zigbee wireless communication
  • 10. Robotics Contest @ ASABE 20092009RoboticsContestGenderRecognitionContestPlease Visit the following linkfor viewing the video:http://youtu.be/8EjON8Y2OJ0
  • 11. Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image• Technique:Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm checkRotate the image 35 degreeto detect all possible tilt face
  • 12. Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image• Technique:Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm checkUse Eye detectorto wrap the face tountilt view.
  • 13. Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image• Technique:Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm checkUtilize Skin color model, Eyeand Mouth detector to filterthe false-positive result fromthe V-J face detector.
  • 14. Gender Recognition Contest2009RoboticsContestGenderRecognitionContest• Problem:Recognize the gender with single face image• Technique:Viola-Jones Face detector (OpenCV)Feature-based alignmentFalse-alarm check• Performance1.2 second per 480*320 image• Result~85% face detection accuracy~75% gender recognition accuracyWin 3rd place among 20 teams (Taiwan andChina). Bonus 60,0000yen.
  • 15. Fish Recognition2010ISMABNTUGraduateInternship@ TV corp.• Problem:Tuna species recognition for fisheryconservation and management• Task:Detection and ClassificationBigeyeYellowfinAlbacore3 spices are considered2011Navy
  • 16. Fish Recognition• Problem:Tuna species recognition for fisheryconservation and management• Task:Detection and ClassificationFish Image are capturedin certain lighting conditionwith measurement plate.Body part is smooth,makes it reflect light well.72%2010ISMABNTUGraduateInternship@ TV corp.2011Navy
  • 17. Fish Recognition• Problem:Tuna species recognition for fisheryconservation and management• Task:Detection and ClassificationB Y AB 89 10 5Y 9 86 3A 10 9 8184%34% 72% 52% 58%Confusion Matrix(Head)(Abdomen)(Tail fin) (Tail)Discriminate part!2010ISMABNTUGraduateInternship@ TV corp.2011Navy
  • 18. Yeh, Graduation!2010ISMABNTUGraduateInternship@ TV corp.2011Navy
  • 19. 2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011Navy• Problem:Generating 3D videofrom 2D content.• Inspiration:3D information isrecovered by depthcuesCaptured View + Depth
  • 20. 2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011NavyReality Comfort• Accurate depth map• Correct depth order• Real-time processing• Clear boundary• Temporal smoothness• Visual impression
  • 21. How people perceive depth?2010ISMABNTUGraduateInternship@ TV corp.2011Navy1. Low-level cue 2. Scene Recognition
  • 22. 2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011NavyVideo frame + motion estimation[ICCE 2009]Approaches1. Depth map by motion2. Depth map by saliency3. Depth map by priorinformation fusion
  • 23. 2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011Navy Video frame + Saliency map [SDA 2010]Approaches1. Depth map by motion2. Depth map by saliency3. Depth map by priorinformation fusion
  • 24. Introduction to Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011NavyBilateral Filter [Tomasi, ICCV98]:f(x) h(x)“Bi” lateral = Spatial term + Range term
  • 25. Introduction to Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011NavyBilateral Filter [Tomasi, ICCV98]:f(x) h(x)“Bi” lateral = Spatial term + Range term
  • 26. Introduction to Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011NavyBilateral Filter [Tomasi, ICCV98]:f(x) h(x)“Bi” lateral = Spatial term + Range term
  • 27. Application of Bilateral Filter2010ISMABNTUGraduateInternship@ TV corp.2011Navy“Bi” lateral = Spatial term + Range termSmooth Target Edge-Preserving ResultAnd this one?
  • 28. 2D-to-3D conversion2010ISMABNTUGraduateInternship@ TV corp.2011NavyPrior fusion [Siggraph 2009]1. Decide Geometric perspective2. Integrate Image and Depth map by Bilateral filter
  • 29. One-year in Navy2010ISMABNTUGraduateInternship@ TV corp.2011Navy
  • 30. Academia Sinica2012ResearchAssistantICPR2012ACCV2012Institute of Information Science(Central Research Academy)People in Vision• Chu-Song Chen (CVPR12, CVPR11*2)• Mark H. Liao (MM12*2, MM11*2)• Y.-C. Frank Wang (ECCV12, CVPR12)• Yen-Yu Lin (CVPR13, MM12, TPAMI11)Prof. Chen
  • 31. PHOTO AESTHETICS CLASSIFICATIONPredicting the visual appealing quality of photos
  • 32. good?jcar@DPChallenge
  • 33. good?Mnet @ DPChallenge
  • 34. Which one is better?
  • 35. Voted by online photo communityAverage: 5.088 votesAverage: 7.292 votes
  • 36. Reason?
  • 37. Reason?BoundaryAlternating repetition(Texture)Contrast Levels of scaleRoughnessStrong centersPositive spaceLocal symmetriesThe VoidNot-separatenessGood shapeGradientsEchoesSimplicity and Inner CalmDeep interlock and ambiguityColorCompositionHarmoniumRichness
  • 38. ApplicationImage Search & Management Photo evaluation systemEmbedded Camera system Media analysis
  • 39. Photo Aesthetics2012ResearchAssistantICPR2012ACCV2012• Problem:Recognition the appealing quality ofphoto by computational approaches.• Technique:Image analysis, Pattern recognition,Crowdsourcing, Psychology, Photography• Application
  • 40. ICPR 20122012ResearchAssistantICPR2012ACCV2012As a Pattern Recognition Problem…Comparison of feature1. Edge distribution, Color histogram,Hue, Saturation.. [Ke, CVPR06]2. SIFT + BOV [Marchesotti , ICCV11]3. Composition layout (Edge + HSV),Color palette, contrast.. [Proposed]ResultItem Speed on PC AccuracyCVPR06 0.2s 81%ICCV11 4s 85%Proposed 0.16s 84%[ Photo aesthetics assessment with efficiency ]
  • 41. Extraction of Color InformationExtract NDominant colors(we set N=5)K-Nearest Neighbor(K=20)List of PalettesDictionaryHQ Palettes DictionaryLQ Palettes DictionaryPalettes of Photo
  • 42. Retrieved by FrequencyRetrieved by Kmeans(Cluster Center)Proposed(Weighted Kmeans)Finding the Dominant Colors
  • 43. Video Demo2012ResearchAssistantICPR2012ACCV2012[ Intelligent Photographing Interface with On-Device Aesthetic Quality Assessment ]Please Visit the following linkfor viewing the video:http://youtu.be/o8mKuTfO6ao
  • 44. Discussion 1Device : On-line assistive camera system• Contextual Information (Viewing angle)camera < human• Feedbackfrom analysis to advice• Human behaviorWhat do people take?How do people take?• ComputationServer-based v.s. Device• Market and Needs
  • 45. Discussion 2Algorithm: photo aesthetic value assessment• Definition of photo aestheticsExpert v.s. Volkswagen• Labeling processIndividual bias and variance.Absolute or Relative evaluationEffect of Labeling order• Quantify photo aestheticModeling, the Personalization
  • 46. Thanks for your attention!10 ratings5.00/7 average5 ratings4.90/7 average

Related Documents