miniorchid - Tumblr blog

miniorchid · 8 months

Text

mysterious lotus casebook 🪷 as text posts pt.1

583 notes · View notes

miniorchid · 1 year

Text

If he departs from this world, I’ll bring grass and oil to cremate myself along with his body, so that even as we turn to ashes, we will still be together.

Based on a fic by @bluemorningsoup that yall should be reading for a good time.

464 notes · View notes

miniorchid · 2 years

Text

ZZH IG Video Deepfake Detection

I started researching deepfake detection method after IG video published, an approach that can objectively examine a clip without any bias. After researching published deepfake detection papers, I started to build the code based on one of the published deepfake detection models. It took me 2 days to have the first result generated without fine-tuning. The result was published on Weibo and banned 6 hours after posting. Based on the feedback from other experts, I refined the model to have the second version published here, with a detailed explanation of the method.

I retrained the model with raw/uncompressed video, and increased the training samples. The final output reduced false negative rate, leading to a more accurate reading.

The algorithm classifies each frame of video along with confidence values. The green box with 90% value, means the model classified this frame as real with 90% confidence. If the box shows red with 90% value, it means the model classified this frame as fake with 90% confidence.

I compared the IG video with zzh’s previous real-person interviews. The video showing my results are below and includes the ZZH IG video first and then two confirmed real interview clips with him. The confirmed real-person interviews were included to show what the model outputs when the video is real. After running the videos through the model, the results showed that where more facial expression was present, more fake frames were detected.

The below chart shows the detection result with confidence value within a video. When the algorithm detects the frame as being “real”, there will be a positive confidence value. When it detects the frame as being “fake”, there will be a negative confidence value.

This is the chart of the confidence values detected for the IG video. Note that the results fluctuate with high volatility between -100 to 100

This is the chart of the confidence values for the first confirmed real interview video used in the video clip. The results are between 90-92 with minor volatility

As comparison, I run the model through two known deep fake videos. Result is shows in a separate post.

Explanation of the Model Used

The model used in this simulation is called “Xception”. The paper describing the model can be found here: https://arxiv.org/abs/1901.08971

The model was trained via Faceforensics++ data which is collected by the Visual Computing Group. I downloaded 800 raw videos (200 real + 600 fake) covering 3 state-of-art video manipulation techniques such as deepfakes, faceswap and face2face.

Data are divided into training, testing, and cross-validation group. Then I took 101 face images from each video and pre-processed every image by applying a face tracker algorithm that detects face area pixels. Based on those classified videos, the AI model will learn how to distinguish between a manipulated video and a real video. It is similar to the process of a baby learning the difference between an apple and orange, the model will classify fake and real along with confidence values on each frame of the video.

At present, Deepfake and Deepfake detection technology is progressing simultaneously. A good model can detect a modified video made by a less trained/outdated deepfake model, but a better deepfake model can also fool a good trained detection model. No technology is perfect and it is possible for a very good modified video to escape detection given current Deepfake detection technology.

260 notes · View notes