@aelaraji@aelaraji.com mutt has the $header_cache option, which might help a bit:

http://www.mutt.org/doc/manual/#header-cache

I disabled it over a year ago on my machine, because it didn’t make that much of a difference anymore. 🤔