Support for Q4_0 and other formats.

[Azure-Tang]

Hi, I notice some dequant format, like Q4_0 Q5_0, are not supported yet. Will you support these formats?

[99991]

There usually are other formats with better perplexity and lower file size than Q4_0 and Q5_0. Do you really need them?

[99991]

I have not heard back from you, but I have now implemented Q4_0 and Q5_0 anyway. Could you test whether it works for your use case?

[Azure-Tang]

sorry i didn't check this message box, I would test it in a few days and give you feedback. Thanks!

[ExtReMLapin]

Should be fixed in #4

[99991]

Should be fixed in #4

What does that pull request have to do with this issue?

[ExtReMLapin]

At the end of the day (not literally), gguf package will handle dequantization, so all quantization type will be supported.