Projects
Flow Map Language Models
Carnegie Mellon University (2025–2026)
The story behind this project is pretty epic. But first, let me summarize what we did. For technical details, please refer to the project page and the paper. High-level, we went against the prevailing wind of doing discrete diffusion for language and showed that a continuous formulation was competitive in the many-function-evaluations regime where discrete diffusion did best. Moreover, in the few-function-evaluations regime, our method was able to far outperform all baselines - we were able to generate coherent text in as few as 1 function evaluation!
Now, let me describe my journey to this project. I had been learning about how diffusion in images was going towards this new paradigm, called Flow Maps, that allowed ultra-fast generation of images (just a couple function evaluations were enough for pretty good results). I was also reading up on how people were very excited about doing discrete diffusion for language and trying to make it fast. So I thought that the million dollar question was: “Can we build a Flow Map for language?” Nick Boffi, my advisor, agreed! And so I kept reading papers and thinking about the problem, trying to understand the fundamental limits of being able to build Discrete Flow Maps. I landed on the conclusion that the vogue discrete formulation was, in fact, not the way to go.
Then, through a serendipitous conversation with Jaehoon at NeurIPS, where he was presenting his work on a related but different idea, things fit into place. I don’t know why he took me seriously, but Jaehoon asked me to meet his colleagues from KAIST. I spoke to them and realized that they were on the exact same page as me. In fact, they had the exact same chain of thought leading to the same conclusion. It was wonderful to hear that they had started working on the idea and had some promising initial results. It was even more wonderful to hear that they were happy to collaborate with our group at CMU. And things just clicked! I had a bunch of really nice discussions with Jinwoo and Nick, where we questioned several key ideas of our approach and the answers to them all fell into place. We rediscovered the right way to do flow training for discrete data and I also came up with new ways to do flow map training entirely on the simplex (where discrete data lives)! I am super happy with how the project has turned out; public response to the paper have been quite positive. I am excited to see where this goes!