Speed drops since b1442

[wro52]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [x ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• [x ] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering- and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
• [x ] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Speed was typical 6.3 tokens/sec for Win10, i5850, 32 GB both with supplied main.exe AVX2 up to build 1440
Test confirmed by GCC compiled under CodeBlocks.

Current Behavior

Beginning with Build b1442 (confirmed with b1443, b1444, b1445) speed drops to 2.84 tokens / second under same
test conditions.

Environment and Context

Test done at equal conditions with the following start:

main -t 6 -m models/META-2/7B/META_2_7B_f16_V4_q4_0.gguf --log-disable --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 --in-prefix-bos --in-prefix [INST] --in-suffix [/INST] -i -p "[INST] <> You are a helpful, respectful and honest assistant. <> Write Good Morning in German. [/INST]"

After the complete answer manually stopped by Ctrl C and speed token/sec taken.

• Physical (or virtual) hardware you are using, e.g. for Linux:

` i5 8350, 32 GB RAM, 1TB M.2 SSD

• Operating System, e.g. for Linux:

`WIN 10, up to date

• SDK version, e.g. for Linux:
  *Checked CPU only version AVX2 packages: e.g. llama-b1440-bin-win-avx2-x64.zip

Checked with provided b1440, b1442 ... b1445 AVX2 packages (main.exe and dll's from package)
Counter checked with GCC, IDE is CodeBlocks build1440 performs as expected, build 1445 same speed drop as referenced versions

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. step 1 Unpack build AVX2 dll's and main into clean test environment (root directory drive C:\LLaMa2\0_KI_test)
2. step 2 start main by batch file
3. step 3 Wait till system completes response.
4. step 4 Hit Ctrl C, confirm Yes and take note of tokens/sec

Notice speed drop after build 1440 (starting b1442) b1440:
llama-b1440-bin-win-avx2-x64.zip 6.3 tokens/sec,
llama-b1442-bin-win-avx2-x64.zip 2.84 tokens/sec
llama-b1443-bin-win-avx2-x64.zip 2.84 tokens/sec
llama-b1444-bin-win-avx2-x64.zip 2.84 tokens/sec
llama-b1445-bin-win-avx2-x64.zip 2.84 tokens/sec

[ggerganov]

I can confirm that d69d777 leads to drop in performance on macOS Arm as well when running on the CPU

[shibe2]

I would see if LTO helps. If it gets the speed up to previous level or higher, then the drop may be because of missed inlining.