@Twu31 Thanks for the careful question. Mostly the typical response, not the slow tail.
On our laptop (RTX 3050 Ti, 4 GB) the median time to the first valid call is 2.1 s with Q4_K_M and 2.7 s with Q8_0, but the 95th percentile is about the same: 4.9 s against 4.7 s. In both builds the slowest turns are clarification questions, where the model has to write a question and the candidate ids. That is around 50 tokens, and at roughly 9 to 11 tokens per second it takes 4 to 5 seconds whichever file you use. So the tail follows how much the model has to write more than the size of the file.
Two caveats. This is one machine and 96 turns per build, so the 95th percentile rests on the slowest three or four turns. And the laptop warms up: Q4's median was 1.1 s in the first run and 2.6 s in the third, while Q8's did not drift the same way. How much you gain depends on how hot the GPU is.
For an interactive loop, I would try letting the app build the clarification question from the candidate ids instead of having the model write it. I expect that to help the tail more than the quantisation does, but we have not tested it yet.
