SWITCH UP

Alright so. My previous idea wasn't working out so well. I couldn't think much of a final project for it, nor really what I wanted to do with the technology I'm experimenting with.

After some demonstrations of AI powered conversational digital characters, it got my mind racing. I love designing and writing characters--all of my illustration work is character based, I spend all of my free time drawing, writing, modeling, and animating my original characters, and love integrating them into everything I do. That led me to my conclusion, and a new goal to work toward. Why not make conversational AIs out of my characters? I'm already familiar enough with Blender for modeling and animating, and I know my way around Unity well enough to (hopefully) get something going.

I want to approach this by piecing a few different things together.

First, it's necessary to have the foundational conversational element in place. This can be done by writing pre-programmed responses to keywords, which, while useful in some contexts like help bots on websites, is not what I'm trying to achieve here.

I've been looking into ChatGPT and OpenAI and integration with Unity, and there seems to be a couple ways to integrate it as long as the project remains connected to their server. This is probably the path I will choose to take, as there is a lot of customization in the way prompts can be given and how the AI can respond.

Following that, I want to add dynamic text-to-speech generation. some APIs like Replica Studios exist for this purpose, and they offer dozens of pre-voice-acted voices. Whether or not this API will work with Unity is yet to be seen, haha. But that's the point of the experiment!

This is a fantastic demo of AI powered NPCs in a game environment. The developer uses OpenAI to give unique prompts to NPCs that inform their attitudes and personalities. The responses are then fed through Replica, an AI powered speech-to-text API to bring the OpenAI responses to life. It is then further combined with an animation system to give the NPCs some movement.

Right now, the most important steps that need to be taken are going to be:

• Get Open AI working with Unity to generate custom and dynamic conversation

• Send the generated text through an API like replica to produce dynamic, real time speech

• Rig a face/body model to account for all potential expressions and facial movements

• Use lip sync technology to combine speech and facial movements

There seems to be a few OpenAI to Unity open source projects in existence already, so I plan to dive into them and make sure I can get them working dynamically in the way I'm imagining.

I do plan to use Replica Studios' system as long as their API is still standing, and for that I need to learn how to make HTTP requests to their API through unity for real-time (or close to real time) speech.

So those will be my starting points. In the meantime, I'm already working on a model to be my guinea pig. He's in the texture painting stage at the moment, and after that he'll get all of his shape keys/blend shapes for facial animation.

woo!