[SHARING] DFL 2.0 ReadyToTrain model

[TMBDF]

6. GAN 0.1 power, patch size 28, gan_dims:32. Train until the src loss value has not increased in the last 12 hours.

7. finalize model by disabling masked training for 100-200 (not thousand) iterations

What is your mean (src loss value has not increased)?

He means what he wrote, you train until SRC loss value isn't changing anymore, usually it goes down but with GAN it may tempoarily go up a bit but eventually will stop moving up and as the previews clear up of the noise and you see detail you stop.

Do note that this is old workflow, iperov has suggested a different workflow for training rtm models, it's included in my guide (iperov's default new workflow, ignore my variations and his old workflow) in the RTM training step of the guide, it's also posted on his github page in one of the commits I think, I don't remember exactly where it was posted originally but I copied his 1:1 to the main guide: //deep.whitecatchel.ru/literotica/forums/thread-guide-deepfacelab-2-0-guide

 

[computera14]

for good result in RTM which one is better?
batch size=8
==            resolution: 320      ==
==            face_type: wf        ==
==    models_opt_on_gpu: True      ==
==                archi: liae-udt  ==
==              ae_dims: 512      ==
==                e_dims: 64        ==
==                d_dims: 64        ==
==          d_mask_dims: 32        ==

OR

batch size=4
==            resolution: 320      ==
==            face_type: wf        ==
==    models_opt_on_gpu: True      ==
==                archi: liae-udt  ==
==              ae_dims: 512      ==
==                e_dims: 90        ==
==                d_dims: 90        ==
==          d_mask_dims: 32        ==

 

[CheeseBiscuit]

for good result in RTM which one is better?
batch size=8
==            resolution: 320      ==
==            face_type: wf        ==
==    models_opt_on_gpu: True      ==
==                archi: liae-udt  ==
==              ae_dims: 512      ==
==                e_dims: 64        ==
==                d_dims: 64        ==
==          d_mask_dims: 32        ==

OR

batch size=4
==            resolution: 320      ==
==            face_type: wf        ==
==    models_opt_on_gpu: True      ==
==                archi: liae-udt  ==
==              ae_dims: 512      ==
==                e_dims: 90        ==
==                d_dims: 90        ==
==          d_mask_dims: 32        ==

The second one with e_dims: 90 and d_dims: 90 is better, it can handle more diverse variations in src and dst faces (variation in lighting, face detail etc.); however, it will be slower to train.

Would you be able to share your liae-udt model at 320 resolution for wf? I've been searching for one but can't seem to find one that's downloadable.

 

[computera14]

Would you be able to share your liae-udt model at 320 resolution for wf? I've been searching for one but can't seem to find one that's downloadable.

my model not trained I want start now

 

[DeliveryBoy]

File Renamed.

 

[TMBDF]

New RTM UDT 256 Female Face Completed


320 RTM Non UDT female Face


320 RTM UDT female Face Coming Soon .... i am up to phase 2

Running trainer.

Choose one of saved models, or enter a name to create a new model.
[r] : rename
[d] : delete

[0] : Jessica320RTM-UDT - latest
 :
0
Loading Jessica320RTM-UDT_SAEHD model...

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : NVIDIA GeForce RTX 3090

[0] Which GPU indexes to choose? :
0

Initializing models: 100%|###############################################################| 5/5 [00:04<00:00,  1.23it/s]
Loading samples: 100%|##########################################################| 12402/12402 [01:15<00:00, 163.57it/s]
Sort by yaw: 100%|##################################################################| 128/128 [00:00<00:00, 409.70it/s]
Loaded 63012 packed faces from D:\DeepFaceLab_NVIDIA_RTX3000_series\workspace\data_dst\aligned
Sort by yaw: 100%|###################################################################| 128/128 [00:01<00:00, 80.33it/s]
================== Model Summary ===================
==                                                ==
==            Model name: Jessica320RTM-UDT_SAEHD ==
==                                                ==
==     Current iteration: 1457464                 ==
==                                                ==
==---------------- Model Options -----------------==
==                                                ==
==            resolution: 320                     ==
==             face_type: wf                      ==
==     models_opt_on_gpu: True                    ==
==                 archi: liae-udt                ==
==               ae_dims: 256                     ==
==                e_dims: 72                      ==
==                d_dims: 72                      ==
==           d_mask_dims: 32                      ==
==       masked_training: True                    ==
==       eyes_mouth_prio: True                    ==
==           uniform_yaw: True                    ==
==         blur_out_mask: True                    ==
==             adabelief: True                    ==
==            lr_dropout: y                       ==
==           random_warp: True                    ==
==       true_face_power: 0.0                     ==
==      face_style_power: 0.0                     ==
==        bg_style_power: 0.0                     ==
==               ct_mode: lct                     ==
==              clipgrad: False                   ==
==              pretrain: False                   ==
==       autobackup_hour: 0                       ==
== write_preview_history: False                   ==
==           target_iter: 0                       ==
==       random_src_flip: False                   ==
==       random_dst_flip: True                    ==
==            batch_size: 8                       ==
==             gan_power: 0.0                     ==
==        gan_patch_size: 40                      ==
==              gan_dims: 16                      ==
==      random_hsv_power: 0.0                     ==
==                                                ==
==------------------ Running On ------------------==
==                                                ==
==          Device index: 0                       ==
==                  Name: NVIDIA GeForce RTX 3090 ==
==                  VRAM: 21.17GB                 ==
==                                                ==
====================================================
Starting. Press "Enter" to stop training and save model.
[04:13:30][#1460104][1390ms][0.2766][0.7349]
[04:38:48][#1462709][1484ms][0.2768][0.7258]
[05:03:24][#1465254][1202ms][0.2769][0.7269]
[05:28:24][#1467880][1343ms][0.2756][0.7317]
[05:53:24][#1470557][1374ms][0.2746][0.7272]
[06:18:25][#1473234][1484ms][0.2767][0.7347]
[06:43:24][#1475900][1374ms][0.2758][0.7275]
[07:08:25][#1478514][1391ms][0.2749][0.7260]
[07:33:25][#1481058][1429ms][0.2749][0.7222]
[07:58:24][#1483579][1000ms][0.2759][0.7321]
[08:23:24][#1486104][1530ms][0.2759][0.7269]
[08:48:25][#1488583][1485ms][0.2742][0.7303]
[09:13:24][#1491074][1421ms][0.2744][0.7277]
[09:38:24][#1493553][1484ms][0.2754][0.7248]
[10:03:25][#1496083][1645ms][0.2755][0.7297]
[10:28:26][#1498580][1459ms][0.2749][0.7223]
[10:53:24][#1501195][1408ms][0.2747][0.7252]
[11:18:24][#1503831][1459ms][0.2743][0.7262]
[11:43:24][#1506463][1407ms][0.2739][0.7251]
[12:08:24][#1509043][1176ms][0.2769][0.7260]
[12:15:55][#1509809][0569ms][0.2320][0.6612]

Jessica who? If these are RTM then provide full name of the celebrity.

If these are not specific celebs and are in fact heavily trained LIAE-UD/UDT then rename these to RTT or ReadyToTrain.

 

[DeliveryBoy]

OK i will rename it when i finished training.

 

[honor]

OK i will rename it when i finished training.

the links don't work any more ?

 

[MrMonkE]

OK i will rename it when i finished training.

the links don't work any more ?

They would work i think ..  IF DeliveryBoy would share his google account details with us.

"DeliveryBoy" , buddy, what's your login for gdrive?

(He's probably just disabled them untill his training is done? I think?)

 

[DeliveryBoy]

The new 320 RTM Liae UDT is almost finished and will be available within 2 weeks (i just want to do more 1 million iteration to make sure you are using the best heavily trained model).

Running trainer.

Choose one of saved models, or enter a name to create a new model.
[r] : rename
[d] : delete

[0] : Jessica320RTM-UDT - latest
 : r
r
Enter the name of the model you want to rename : Jessica320RTM-UDT
Jessica320RTM-UDT
Enter new name of the model : 320RTM-LIAE-UDT
320RTM-LIAE-UDT
Choose one of saved models, or enter a name to create a new model.
[r] : rename
[d] : delete

[0] : 320RTM-LIAE-UDT - latest
 :
0
Loading 320RTM-LIAE-UDT_SAEHD model...

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : NVIDIA GeForce RTX 3090

[0] Which GPU indexes to choose? :
0

Press enter in 2 seconds to override model settings.
[0] Autobackup every N hour ( 0..24 ?:help ) :
0
[n] Write preview history ( y/n ?:help ) :
n
[0] Target iteration :
0
[n] Flip SRC faces randomly ( y/n ?:help ) :
n
[y] Flip DST faces randomly ( y/n ?:help ) :
y
[8] Batch_size ( ?:help ) :
8
[y] Masked training ( y/n ?:help ) :
y
[y] Eyes and mouth priority ( y/n ?:help ) :
y
[n] Uniform yaw distribution of samples ( y/n ?:help ) :
n
[y] Blur out mask ( y/n ?:help ) :
y
[y] Place models and optimizer on GPU ( y/n ?:help ) :
y
[y] Use AdaBelief optimizer? ( y/n ?:help ) :
y
[y] Use learning rate dropout ( n/y/cpu ?:help ) :
y
[y] Enable random warp of samples ( y/n ?:help ) : n
[0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
0.0
[0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
0.0
[0.0] Face style power ( 0.0..100.0 ?:help ) :
0.0
[0.0] Background style power ( 0.0..100.0 ?:help ) :
0.0
[lct] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
lct
[n] Enable gradient clipping ( y/n ?:help ) :
n
[n] Enable pretraining mode ( y/n ?:help ) :
n
Initializing models: 100%|###############################################################| 5/5 [00:03<00:00,  1.50it/s]
Loading samples: 100%|##########################################################| 12402/12402 [01:24<00:00, 147.30it/s]
Loaded 63012 packed faces from D:\DeepFaceLab_NVIDIA_RTX3000_series\workspace\data_dst\aligned
================== Model Summary ===================
==                                                ==
==            Model name: 320RTM-LIAE-UDT_SAEHD   ==
==                                                ==
==     Current iteration: 2072499                 ==
==                                                ==
==---------------- Model Options -----------------==
==                                                ==
==            resolution: 320                     ==
==             face_type: wf                      ==
==     models_opt_on_gpu: True                    ==
==                 archi: liae-udt                ==
==               ae_dims: 256                     ==
==                e_dims: 72                      ==
==                d_dims: 72                      ==
==           d_mask_dims: 32                      ==
==       masked_training: True                    ==
==       eyes_mouth_prio: True                    ==
==           uniform_yaw: False                   ==
==         blur_out_mask: True                    ==
==             adabelief: True                    ==
==            lr_dropout: y                       ==
==           random_warp: False                   ==
==       true_face_power: 0.0                     ==
==      face_style_power: 0.0                     ==
==        bg_style_power: 0.0                     ==
==               ct_mode: lct                     ==
==              clipgrad: False                   ==
==              pretrain: False                   ==
==       autobackup_hour: 0                       ==
== write_preview_history: False                   ==
==           target_iter: 0                       ==
==       random_src_flip: False                   ==
==       random_dst_flip: True                    ==
==            batch_size: 8                       ==
==             gan_power: 0.0                     ==
==        gan_patch_size: 40                      ==
==              gan_dims: 16                      ==
==      random_hsv_power: 0.0                     ==
==                                                ==
==------------------ Running On ------------------==
==                                                ==
==          Device index: 0                       ==
==                  Name: NVIDIA GeForce RTX 3090 ==
==                  VRAM: 21.17GB                 ==
==                                                ==
====================================================
Starting. Press "Enter" to stop training and save model.
[00:16:45][#2312561][1919ms][0.2105][0.5539]
[00:41:40][#2315066][1934ms][0.2114][0.5538]
[01:06:38][#2317601][1449ms][0.2113][0.5528]
[01:31:39][#2320167][1909ms][0.2100][0.5540]
[01:56:39][#2322740][1469ms][0.2111][0.5544]
[02:21:39][#2325308][1919ms][0.2104][0.5482]
[02:46:38][#2327882][1569ms][0.2107][0.5499]
[03:11:38][#2330458][1449ms][0.2098][0.5539]
[03:36:39][#2333030][1949ms][0.2107][0.5536]
[04:01:38][#2335604][1954ms][0.2103][0.5500]
[04:26:39][#2338179][1945ms][0.2105][0.5499]
[04:51:39][#2340751][1994ms][0.2101][0.5522]
[05:16:38][#2343324][1510ms][0.2093][0.5500]
[05:41:39][#2345898][2000ms][0.2095][0.5513]
[06:06:39][#2348473][1969ms][0.2082][0.5510]
[06:31:38][#2351047][1934ms][0.2103][0.5452]
[06:56:39][#2353621][2079ms][0.2106][0.5503]
[07:21:39][#2356204][1970ms][0.2093][0.5516]
[07:46:40][#2358790][1972ms][0.2092][0.5497]
[08:11:39][#2361373][1939ms][0.2084][0.5516]
[08:36:39][#2363957][1739ms][0.2081][0.5453]
[09:01:39][#2366539][1979ms][0.2107][0.5497]
[09:26:39][#2369100][2109ms][0.2084][0.5478]
[09:51:38][#2371669][1560ms][0.2078][0.5490]
[10:16:39][#2374241][1960ms][0.2099][0.5504]
[10:41:39][#2376802][2210ms][0.2071][0.5483]
[11:06:38][#2379356][1489ms][0.2090][0.5466]
[11:31:39][#2381905][1504ms][0.2097][0.5463]
[11:56:39][#2384449][2010ms][0.2085][0.5468]
[12:21:40][#2386993][1988ms][0.2090][0.5465]
[12:32:51][#2388107][0560ms][0.2081][0.5495]
[12:46:39][#2389470][1935ms][0.2068][0.5462]
[13:01:36][#2390984][0523ms][0.2072][0.5424]
[13:11:38][#2391992][1801ms][0.2089][0.5431]
[13:15:31][#2392380][0580ms][0.1821][0.6273]

Link to my folder, within it are trained RTM models.

 

[honor]

The new 320 RTM Liae UDT is almost finished and will be available within 2 weeks (i just want to do more 1 million iteration to make sure you are using the best heavily trained model).


Link to my folder, within it are trained RTM models.

Thanks... when you say RTM... are you following iperov's RTM training instructions... or simply pretraining ( RTM dataset for both Src and Dst ) ?

 

[DeliveryBoy]

i trained the RTM as per guide line but sometimes i let each section reach over 1 million iterations instead of 500K, until the loss data value can no longer be lowered.

My src is over 12500 faces and the dst is over 23000, further more i flip the dst to get double faces to learn from. I also use open mouth (sexual orgasmic face and singing mouth for musical deepfake) and closed eyes as well as faces from all angles (specifically side profiles as well as angles from behind of the ear to the cheek bones (for extra src likeness) to make sure this RTM is not going to waste no one's time.


This is why it takes a lot longer to train this RTM.

You can then just replace the src and the dst and continue training without changing any perimeters or deleting any B or AB npy files. ... and wait until you've reached at least 40K iterations to see it the magic working.

Use once and throw away: dont keep training on other new src and or other new dst after you've already trained your deepfake.

If you use this model to train on new src and dst without changing the model options, and you've reached at least 200K iterations the magic is so real you'd think fake news is real !


* If you dont like the result then download this model again and delete the B npy file for your new dst or delete AB npy file to train on your own src and or delete both AB and B npy files and start training on new src and dst at the same time from scratch.


The 320 UDT RTM is at the 2nd last stage (Random Warp = NO), in this section i have done already 300K iterations.

Running trainer.

Choose one of saved models, or enter a name to create a new model.
[r] : rename
[d] : delete

[0] : 320RTM-LIAE-UDT - latest
 :
0
Loading 320RTM-LIAE-UDT_SAEHD model...

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : NVIDIA GeForce RTX 3090

[0] Which GPU indexes to choose? :
0

Press enter in 2 seconds to override model settings.
[0] Autobackup every N hour ( 0..24 ?:help ) :
0
[n] Write preview history ( y/n ?:help ) :
n
[0] Target iteration :
0
[n] Flip SRC faces randomly ( y/n ?:help ) :
n
[y] Flip DST faces randomly ( y/n ?:help ) :
y
[8] Batch_size ( ?:help ) :
8
[y] Masked training ( y/n ?:help ) :
y
[y] Eyes and mouth priority ( y/n ?:help ) :
y
[n] Uniform yaw distribution of samples ( y/n ?:help ) :
n
[y] Blur out mask ( y/n ?:help ) :
y
[y] Place models and optimizer on GPU ( y/n ?:help ) :
y
[y] Use AdaBelief optimizer? ( y/n ?:help ) :
y
[y] Use learning rate dropout ( n/y/cpu ?:help ) :
y
[n] Enable random warp of samples ( y/n ?:help ) :
n
[0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
0.0
[0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
0.0
[0.0] Face style power ( 0.0..100.0 ?:help ) :
0.0
[0.0] Background style power ( 0.0..100.0 ?:help ) :
0.0
[lct] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
lct
[n] Enable gradient clipping ( y/n ?:help ) :
n
[n] Enable pretraining mode ( y/n ?:help ) :
n
Initializing models: 100%|###############################################################| 5/5 [00:03<00:00,  1.49it/s]
Loading samples: 100%|##########################################################| 12402/12402 [01:34<00:00, 130.70it/s]
Loaded 63012 packed faces from D:\DeepFaceLab_NVIDIA_RTX3000_series\workspace\data_dst\aligned
================== Model Summary ===================
==                                                ==
==            Model name: 320RTM-LIAE-UDT_SAEHD   ==
==                                                ==
==     Current iteration: 2310005                 ==
==                                                ==
==---------------- Model Options -----------------==
==                                                ==
==            resolution: 320                     ==
==             face_type: wf                      ==
==     models_opt_on_gpu: True                    ==
==                 archi: liae-udt                ==
==               ae_dims: 256                     ==
==                e_dims: 72                      ==
==                d_dims: 72                      ==
==           d_mask_dims: 32                      ==
==       masked_training: True                    ==
==       eyes_mouth_prio: True                    ==
==           uniform_yaw: False                   ==
==         blur_out_mask: True                    ==
==             adabelief: True                    ==
==            lr_dropout: y                       ==
==           random_warp: False                   ==
==       true_face_power: 0.0                     ==
==      face_style_power: 0.0                     ==
==        bg_style_power: 0.0                     ==
==               ct_mode: lct                     ==
==              clipgrad: False                   ==
==              pretrain: False                   ==
==       autobackup_hour: 0                       ==
== write_preview_history: False                   ==
==           target_iter: 0                       ==
==       random_src_flip: False                   ==
==       random_dst_flip: True                    ==
==            batch_size: 8                       ==
==             gan_power: 0.0                     ==
==        gan_patch_size: 40                      ==
==              gan_dims: 16                      ==
==      random_hsv_power: 0.0                     ==
==                                                ==
==------------------ Running On ------------------==
==                                                ==
==          Device index: 0                       ==
==                  Name: NVIDIA GeForce RTX 3090 ==
==                  VRAM: 21.17GB                 ==
==                                                ==
====================================================
Starting. Press "Enter" to stop training and save model.
[00:16:45][#2312561][1919ms][0.2105][0.5539]
[00:41:40][#2315066][1934ms][0.2114][0.5538]
[01:06:38][#2317601][1449ms][0.2113][0.5528]
[01:31:39][#2320167][1909ms][0.2100][0.5540]
[01:56:39][#2322740][1469ms][0.2111][0.5544]
[02:21:39][#2325308][1919ms][0.2104][0.5482]
[02:46:38][#2327882][1569ms][0.2107][0.5499]
[03:11:38][#2330458][1449ms][0.2098][0.5539]
[03:36:39][#2333030][1949ms][0.2107][0.5536]
[04:01:38][#2335604][1954ms][0.2103][0.5500]
[04:26:39][#2338179][1945ms][0.2105][0.5499]
[04:51:39][#2340751][1994ms][0.2101][0.5522]
[05:16:38][#2343324][1510ms][0.2093][0.5500]
[05:41:39][#2345898][2000ms][0.2095][0.5513]
[06:06:39][#2348473][1969ms][0.2082][0.5510]
[06:31:38][#2351047][1934ms][0.2103][0.5452]
[06:56:39][#2353621][2079ms][0.2106][0.5503]
[07:21:39][#2356204][1970ms][0.2093][0.5516]
[07:46:40][#2358790][1972ms][0.2092][0.5497]
[08:11:39][#2361373][1939ms][0.2084][0.5516]
[08:36:39][#2363957][1739ms][0.2081][0.5453]
[09:01:39][#2366539][1979ms][0.2107][0.5497]
[09:26:39][#2369100][2109ms][0.2084][0.5478]
[09:51:38][#2371669][1560ms][0.2078][0.5490]
[10:16:39][#2374241][1960ms][0.2099][0.5504]
[10:41:39][#2376802][2210ms][0.2071][0.5483]
[11:06:38][#2379356][1489ms][0.2090][0.5466]
[11:31:39][#2381905][1504ms][0.2097][0.5463]
[11:56:39][#2384449][2010ms][0.2085][0.5468]
[12:21:40][#2386993][1988ms][0.2090][0.5465]
[12:32:51][#2388107][0560ms][0.2081][0.5495]
[12:46:39][#2389470][1935ms][0.2068][0.5462]
[13:01:36][#2390984][0523ms][0.2072][0.5424]
[13:11:38][#2391992][1801ms][0.2089][0.5431]
[13:36:39][#2394560][1740ms][0.2084][0.5498]
[14:01:39][#2397143][1763ms][0.2072][0.5448]
[14:26:39][#2399424][1998ms][0.2079][0.5447]
[14:51:39][#2401964][1979ms][0.2083][0.5477]
[15:16:39][#2404508][2039ms][0.2076][0.5481]
[15:41:38][#2407050][1879ms][0.2087][0.5429]
[16:06:38][#2409598][1585ms][0.2063][0.5433]
[16:16:03][#2410548][0582ms][0.2225][0.7480]

 

[honor]

appreciate the transparency...

seems like a lot of effort put in not only in training time, yet data prep and extending on to make it even better...

Good Going

 

[TMBDF]

I think it's time to tidy up this thread.

Since this thread is called "ready to train" and should only contain ready to train models and most people should know by now what RTT and RTM models are I'd like to ask everyone who has made actual RTT and RTM models to submit them to me via PM, I will prepare new separate threads for posting RTT and RTM models and only that, no discussion, only sharing.

I will delete this thread once the other threads are set up but I won't wait forever, I want these models by the end of the month or I'll just copy what is in this thread and post it as is.

In case someone still doesn't know the difference here is what is what:

- RTT - ready to train - a trained LIAE-UD or LIAE-UDT model by reusing it several times, preferably in the 4-6kk iterations range, meant to accelerate training of RTM models, this is very much like randomly trained models people would post (so not pretrained but actually trained on random data or reused with few different celebs over and over). Now to my knowledge usually trained models would be against actual DST and RTT I think is against the RTM DST WF dataset from the torrent, however as iperov never published official RTT workflow I'll just accept any pretrained/trained model of LIAE variant and put these in the RTT sharing thread but if you can tell me how you trained yours please send that along with links.

- RTM - ready to merge - that's actually fully trained model on RTM DST WF set and one specific celebrity - meant for use with DFLive, DFL to just merge straight without training or to do extra trainign on target DST. I'd prefer if people who post these also included the SRC set for those who may want to train them further when using DFL but if not then I'll just include them without a SRC set and call it a day, if you made any RTM models, also send these along with any relevant info.

I'm closing this thread for now, you've got time to submit your models until the end of the month, after that I take what's here and delete this thread. It's 20 pages of mostly asking questions and very little sharing so no point in cleaning it up.