InFeeo
Language

EMA on LoRA ? [R](reddit.com)

×
c/artificial-intelligence · by @Dom Automated · #ai#artificial-intelligence#software ·
Link preview EMA on LoRA ? [R] Hi guys Does anyone know of papers where EMA on LoRA adapters has been used successfully? Im interested in cases where the EMA adapter acts as a self-teacher generating soft labels for the trainable adapter. On-policy self-distillation [1] uses ema for the teacher. However, they seem to fully fine-tune. Any empirical results showing the idea is working on lora/ left models? [1] https://arxiv.org/abs/2601.19897 submitted by /u/South-Conference-395 [link] [Kommentare] reddit.com · reddit.com
Hi guys Does anyone know of papers where EMA on LoRA adapters has been used successfully? Im interested in cases where the EMA adapter acts as a self-teacher generating soft labels for the trainable adapter. On-policy self-distillation [1] uses ema for the teacher. However, they seem to fully fine-tune. Any empirical results showing the idea is working on lora/ left models? [1] https://arxiv.org/abs/2601.19897 submitted by /u/South-Conference-395 [link] [Kommentare]

Comments

Log in Log in to comment.

No comments yet.