Virtual Nostalgia
Thesis
VR / Unity / AIGC / Python / Blender
Spring 2023, New York.
OVERVIEW
Virtual Nostalgia is an interactive virtual reality (VR) experience set in a virtual Cabinet of Curiosities. Players engage with objects in the cabinet to be transported into captivating stories and memories linked to each item, creating an engaging first-person encounter. All the objects, stories and interaction ways are “imagined” or generated by AI models.
It explores the possibilities and limitations of AI-generated content and workflows by using AI models to help create a comprehensive VR experience, as well as the role of AI in the creative process and its relationship to human creativity and artistic expression.
CONCEPT AND SCHEMATICS
Is creativity human superiority? Does AI possess creativity?
Answering these questions pose challenges due to the difficulty of capturing human perceptions of creativity in algorithms and the absence of a precise definition. However, subjectively speaking, AI has shown its ability to assist in music composition, replicate art styles, and generate intricate 3D models, among other achievements. AI has produced numerous novel and seemingly "creative" outputs, even winning art prizes.
What if we let AI explore more complex art forms? What if an entire project is guided by AI's decisions and advice? Would the result be genuinely creative, clichéd, or nonsensical? These questions drive my curiosity.
AI can be viewed as a reflection of human behaviors, experiences, and creations. It learns from vast amounts of human data to replicate human intelligence. The Cabinet of Curiosities metaphorically represents AI-generated output as a symbolic representation of the collective culture and history of humanity.
VR is chosen for its seamless integration of diverse AI-generated elements such as images, sounds, 3D models, and real-time interactions. It offers an immersive and interactive platform that effectively presents and explores AI-generated content.
RESEARCH AND PROCESS
The only prerequisite I provide is a “cabinet of curiosity”, with
ChatGPT selected four main items for the Cabinet of Curiosities: a quill pen, a kaleidoscope, a djembe, and a teddy bear. Each item is accompanied by its own story.
1. Quill Pen
The detailed description of the quill pen scene, as generated by chatGPT, is as follows:
“Scene Description: This quill pen once belonged to a young woman named Charlotte in the 19th century. Charlotte used this pen to write heartfelt letters to her same-gender lover, Elizabeth, with whom she could never be together. When players touch the quill pen within the Cabinet of Curiosities, they are transported to a dimly-lit Victorian-era bedroom adorned with antique furnishings, a crackling fireplace, and a writing desk. The desk holds a quill pen, inkwell, and a letter or paper. A handwritten diary with a cowhide cover reveals the romantic connection between Charlotte and Elizabeth through poetic entries. To further suggest their backstory, photographs of the couple can be included in the display.”
The interactions designed by chatGPT for this scene are as follows:
“Interactions: Players can explore the room and delve into their identities and backstories. As they read through the diary, they will hear voices and sounds, and the ambiance of the scene, including the lighting and fireplace, will change, evoking a sense of recreated memories. Using the quill pen, players can write letters to their loved ones and receive responses in return.”
Player writes and sends a letter, which is captured by a camera in Unity and sent to Google Vision AI as an image. Google Vision AI recognizes the letter's content and sends the text back to Unity. I fine-tuned a
2. Kaleidoscope
The chatGPT-generated description of the kaleidoscope scene is as follows:
“Scene Description: This interactive experience transports players into a fantastical and dreamlike world, reminiscent of a child's perspective. Within this captivating world, players will hear dreamy music and encounter an array of colorful objects such as crystals, prisms, mirrors, and stained glass windows. ”
The interactions described by chatGPT are:
“Interactions: As players pick up and rotates the kaleidoscope, the kaleidoscope's patterns shift and change, revealing the kaleidoscope scene. Within this mesmerizing environment, various objects reflect and refract light, creating a mesmerizing environment to freely explore and interact with.”
Here are some of the interactions actually developed in this scene:
3. Djembe (Ongoing)
“Scene Description: When the player taps the djembe, they are transported to a quaint African village characterized by traditional huts, a communal fire pit, and a gathering area where the community would congregate for celebrations and ceremonies. The immersive environment is teeming with the sounds of nature, such as birds chirping and leaves rustling in the wind. As the sun sets, painting the sky with shades of red and orange, a warm and inviting atmosphere envelops the surroundings.”
“Interactions: In the world of African Celebration, players can tap the djembe to hear drumming and traditional African music. People in traditional attire dance around a bonfire. The music and dancing rhythm would respond to the player's drumming.”
The next step is to integrate these animations into the character models.
4. Teddy Bear (Ongoing)
“Scene Description: The old tattered teddy bear belonged to a young child named Emily, who lived during World War II. Emily's father was a soldier fighting in the war, and the teddy bear was given to her as a comforting companion during his absence. The teddy bear became a symbol of love, hope, and resilience during a difficult time.”
“Interactions: When the player touches the teddy bear, they enter a dimly lit room with wartime photographs on the walls. Outside the window are ruins from the war. Next to the teddy bear is a fairy tale picture book. When the player reads the book while holding the teddy bear, the window scenery transforms into the book's illustrations.
The player can find a hidden letter from Emily's father in a drawer, filled with loving words. They also discover a makeshift toy hospital where they can stitch up the teddy bear. A small music box plays a nostalgic melody, triggering a memory of Emily and her father dancing together.
As the player explores, the room gradually transforms from dimly lit to bright and vibrant, symbolizing the return of peace and hope. ”
Tools Used:
- Meta Oculus2
Applications Used:
- Unity
-
ChatGPT
- Blender
- Adobe (Premiere, Illustrator, Photoshop)
- Stable Diffusion
- Midjourney
- Google Vision API
- Dreamfileds
- Dreamfusion
- CLIP-Mesh
- Mubert
- MDM: Human Motion Diffusion Model
- ...(various other AIGC models)
Skills Used:
- Machine learning
- Coding (C#, python)
- Moedeling & Rigging
REVIEW
Being an assistant to AI can be challenging, as the majority of the work still falls on me, and sometimes it's difficult to bring AI's ideas to life. AI has a tendency to generate random and unpredictable ideas, which can result in both unexpected and nonsensical outputs.
I wonder if ChatGPT solely dominates the idea composition process. During our ongoing conversations, I may unintentionally reject some of its proposals or unknowingly steer its output in a particular direction. It feels more like a collaborative effort, where we co-create the entire story through our interactions. AI has definitely inspired me with stories and interactions that I may not have considered on my own.
As my project progressed, I observed the emergence of integrated AI models like autoGPT and huggingGPT. These models have the capability to utilize various AI models and automate certain tasks. It made me contemplate the future, where the work I did here could potentially be accomplished with just a few prompts. Initially, I had concerns about these advancements. However, throughout this project, I realized that these tools are not meant to replace us but rather enhance efficiency and allow us to focus more on generating ideas.
It doesn't render what you have learned useless; on the contrary, it’s because what you have learned that you can extract maximum value from its capabilities.