[Question]: How to understand dense_decoding?

[lemyx]

Describe the issue

Dear author,

Thanks for your outstanding work!

There is a parameter named dende_decoding in minference/models_patch.py. And variables dense_k, dense_v in inf_llm.py don't appear in the original InfLLM implementation.

Could you please explain more details on the designs? Thanks a lot!

Best wishes

[iofu728]

Hi @lemyx, thanks for your interest in our work.

Since MInference focuses only on pre-filling speedup, we aligned all baselines to perform sparse pre-filling with dense decoding. Therefore, we re-implemented InfLLM's dense decoding ourselves to ensure a fair comparison, which improves upon the original implementation of InfLLM.