Hello everyone.
WARNING The included video contains explicit violence.
I've been working on assets for Makayla's story and also writing the overall story for her and the other main characters. On my downtime (when my brain feels fried... which is happening a lot lately), I've been playing with video and voice generation just a little each day and I've managed to put together something I hope you enjoy.
It's a proof of concept of how a video with my characters would look like. We tried a short clip with Xiomara sleeping at the beach, in the Darwin Awards. That was an initial test to see if we could chain clips to form a little narrative. There were many things that came out cool but others that changed things too much from the original. If you compare the images from that set with the video you'll see quite a few things changed.This time the goal was to take a structured story and try to keep as close to the original with very little deviation. I chose Pandora's comic to see if I could do it and I think the result was quite nice. I would love to hear what you think.What's new here:
Voice generation. I installed VibeVoice in my local pc and used it to create specific voices for each character. The idea is to have an official voice for every named character. What do you think about the voice I chose for her?
Video and audio effects. Video editing is relatively new to me so it took me some time to study how to do certain things like the camera shake every time Pandora takes a step, and the directional audio to show when a sound comes from one side (like when Pandora is out of frame walking into view). I still have a lot to learn and hope each project like this allows me to integrate new things
What is missing / things to improve:
The main glaring issue with this animation is the lack of true lip synch of the character's mouths with the audio. The video generator i use (Grok) creates the speech with random voices, so the video and audio were created separately with different software. I tried to manually synch them in DaVinci Resolve (the video editor), and got it as best as I could. I created the video first from existing images (the original comic). and then created the voice. I know there are AI tools that does automatic lip synch. I'll try installing and testing them and see if I can find one that could do the trick.
A second issue is also with voice. There are some lines that sound fake or inconsistent with the video, and its due to the fact that the voice is generated from text, and the intentionality and "acting" is decided by the AI. I've learned how to "trick" it into giving some emotion from the context of the text. One solution I'll explore is looking for tools that clone voices from one audio to another, so I can record the lines myself with my voice, then run the AI to change my voice with Pandora's. That way I can control exactly the tone, speed, and even try to match the animation.
I think a combination of using my voice with auto lip synch to fine tune the lips with the audio might bring the best results.
For future projects I'll also look into local uncensored video generation. I'm currently using a public tool with lots of limitation regarding nudity and sexually suggestive material. The fact I can trick it to show gore is quite insane. I'm working on installing an uncensored video generator on my pc. It hasn't been easy since my pc is a bit outdated and the video card is a 1080TI, with regular video memory (11 gigs) which is good for many things but the modern video generators ask for a lot more. Still I think I should be able to get one of them working.
The video in question is in the attachment. as a Warning, the video contains very violent and gory sequences. If you don't like violence please don't look at it.
Finally, I would love to hear what you think. I'll continue delving into video stories in my down time between working on the comics. It's a fun way to clear my head. I think the next videos I make would be made with original content for that video. That is, generating images not for a comic but with the intent to be animated. Working with image to video workflows had made me understand that the approach to images for animation and images for comics are fundamentally different. Just as an example: In a comic if I want to show a girl punching a guy, I would render the image of the punch connecting, but if I want to make an animation I would have to render the build up to the punch and have the AI take it from there and create the punch.The fun thing is there is a lot of new territory to cover and new things to learn and play with. I'm happy you guys are here with me for that journey.
Cheers!
-Kurt