4Dgen

Reinventing 4D-generation experience..

Comment

4Dgen

View demo video here: https://drive.google.com/open?id=1q_4JHCbZUtBG9qIqtWKfeQseYf7DNkxP (P.S. we dk how to upload)

Problem Statement

“A nation’s culture resides in the hearts and in the soul of its people.” - Mahatma Gandhi

In 2008, Singapore's government have spend S$122.07 million on preserving our heritage. It is apparent that there is intrinsic value in our unique Singaporean heritage that is worth preserving. However, preserving our heritage isn't a one way street where the authorities does everything. Our nations culture and heritage is part of our Singaporean indentity and it's OUR responsiblity to help preserve our heritage as well.

Our Solution

Preservation of heritage comes in many ways. What we indentify as the most relevant would be to use visuals and auditory materials to aid us in our goal, with this aim in mind we want to create an image and music app that can help our users acheive this goal.

Original App Features:

  • Image Styling
    • Singaporean landmarks as the target style where we transform any photo with state of the art machine learning models into the style we see from old Singapore photos
  • Background Music
    • Provide audio (Selected by pre-trained machine learning models) as background music to the new photo provided by our users. Thus allowing our users to get the full visual and auditory experience - similar to reliving the past!
  • Captioning
    • Captions of similar historical images will be generated. It may be a quote, fun-fact or even some random comments. Try our app to find out!

BUT

We realised that this is pointless to have a great app if no one were to use it. Thus we decided to include even more features that our users are able to use in their day to day lifes which would entice them to actually use the app.

Finalised App Features:

  • Image Fixing
    • This feature will allow users with damaged photos or pictures to upload a photo and our app will tap on to the latest in-painting model that NVIDIA developed just a year back.
    • They will also be able to erase unwanted portions of the photos away and the model will remake and patch up the photo just as though there isn't anything there in the first place!
    • Even unwanted photobombers can be removed.
  • Image Styling
    • Although it's a lot more effort to include this feature which multiple machine learning models to include and tweak. We managed to include custom image styling with ANY user provided image.
    • Any image styles can be used in tandem with the original image and a new image based on the original image would auto generated to mimic the style provided in the second image.
  • Captioning
    • Ability to auto caption ANY image have also been included
  • Audio Generation
    • Ability to generate the surrounding music based on the image uploaded to re-live the moment as if you are there

Technology Used

Implement a myriad of machine learning models in our bot. Technology such as RNN, LSTM, RNN-NADE, RL, VAE and generative model. Models from multiple machine learning model from multiple well known library such as pytorch, keras and tensorflow as well as models developed by NVIDIA and Google are also implemented.

  • Python
  • JS/JQuery
  • R
  • HTML
  • machine-learning
  • generative-adversarial-networks
  • recurrent-neural-network
  • auto-encoding
  • auto-decoding
  • music-generation
  • image-generation
  • text-generation
  • tensorflow
  • keras
  • pytorch

Difficulty faced

NONE! WE ARE THAT AMAZING! Just joking. We are not that good. Model outputs are chained to the input of other models. And since machine learning models are not perfect it is inevitable that there will be some accuracy loss. We had to tweak originals models to our requirement to improve the performance and with the limited time of 24 hours it had been a great challenge.

Share this project:

Updates