Skip to content

Support multiple GPUs (split mode) on SYCL backend #5806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
NeoZhangJianyu merged 7 commits into from
Mar 2, 2024
Merged

Support multiple GPUs (split mode) on SYCL backend #5806

NeoZhangJianyu merged 7 commits into from
Mar 2, 2024

Conversation

@NeoZhangJianyu
Copy link
Collaborator

@NeoZhangJianyu NeoZhangJianyu commented Mar 1, 2024

  1. Support multiple GPUs (split mode) on SYCL backend.
    split mode: [none, layer] supported; [row] not supported, it's on developing.

  2. Unify the GPU setting as Cublas backend:

  • support set main gpu by: --main-gpu
  • support detecting all GPUs with level-zero and same top Max compute units.
  • remove use GGML_SYCL_DEVICE to set main gpu.
  1. format to show the device list, like:
found 6 SYCL devices:
|ID| Name                                        |compute capability|Max compute units|Max work group|Max sub group|Global mem size|
|--|---------------------------------------------|------------------|-----------------|--------------|-------------|---------------|
| 0|            Intel(R) Data Center GPU Flex 170|               1.3|              512|          1024|           32|    16225243136|
| 1|               Intel(R) FPGA Emulation Device|               1.2|               64|      67108864|           64|   540713414656|
| 2|     Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz|               3.0|               64|          8192|           64|   540713414656|
| 3|            Intel(R) Data Center GPU Flex 170|               3.0|              512|          1024|           32|    16225243136|
| 4|            Intel(R) Data Center GPU Flex 170|               3.0|              512|          1024|           32|    16225243136|
| 5|            Intel(R) Data Center GPU Flex 170|               1.3|              512|          1024|           32|    16225243136|
detect 2 SYCL GPUs: [0,5] with Max compute units:512
  1. Support OPs:
  • hardsigmoid
  • hardswish
  • pool2d
  1. Use device index to set/get GPU data internal data.
    same as cubals backend.
  2. Use device ID to set/get GPU device info.

@NeoZhangJianyu
Copy link
Collaborator Author

NeoZhangJianyu commented Mar 1, 2024

@airMeng , @luoyu-intel @abhilash1910 could help review this PR?

Thank you!

slaren
slaren reviewed Mar 1, 2024
View reviewed changes
examples/llama-bench/llama-bench.cpp Outdated
Comment on lines 126 to 127
int device_list[GGML_SYCL_MAX_DEVICES];
ggml_sycl_get_gpu_list(device_list, GGML_SYCL_MAX_DEVICES);
Copy link
Member

@slaren slaren Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be removed now, device_list does not seem to be used anymore.

Copy link
Collaborator Author

@NeoZhangJianyu NeoZhangJianyu Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, rm it.

examples/sycl/run-llama2.sh Outdated
#ZES_ENABLE_SYSMAN=1, Support to get free memory of GPU by sycl::aspect::ext_intel_free_memory. Recommended to use when --split-mode = layer.

#use all GPUs with same max compute units
ZES_ENABLE_SYSMAN=1 ./build/bin/main -m models/llama-2-7b.Q4_0.gguf -p "${INPUT2}" -n 400 -e -ngl 33 -s 0 -mg $GGML_SYCL_DEVICE
Copy link
Member

@slaren slaren Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that -mg is ignored with -sm layer, which is the default, so passing it here does nothing.

Copy link
Collaborator Author

@NeoZhangJianyu NeoZhangJianyu Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, rm -mg

ggerganov
ggerganov approved these changes Mar 1, 2024
View reviewed changes
common/common.cpp
Comment on lines 642 to 647
} else if (arg_next == "row") {
#ifdef GGML_USE_SYCL
fprintf(stderr, "warning: The split mode value:[row] is not supported by llama.cpp with SYCL. It's developing.\nExit!\n");
exit(1);
#endif // GGML_USE_SYCL
params.split_mode = LLAMA_SPLIT_MODE_ROW;
Copy link
Member

@ggerganov ggerganov Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
} else if (arg_next == "row") {
#ifdef GGML_USE_SYCL
fprintf(stderr, "warning: The split mode value:[row] is not supported by llama.cpp with SYCL. It's developing.\nExit!\n");
exit(1);
#endif // GGML_USE_SYCL
params.split_mode = LLAMA_SPLIT_MODE_ROW;
} else if (arg_next == "row") {
#ifdef GGML_USE_SYCL
fprintf(stderr, "warning: The split mode value:[row] is not supported by llama.cpp with SYCL. It's developing.\nExit!\n");
exit(1);
#endif // GGML_USE_SYCL
params.split_mode = LLAMA_SPLIT_MODE_ROW;

Copy link
Collaborator Author

@NeoZhangJianyu NeoZhangJianyu Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, accept it.

llama.cpp Outdated
Comment on lines 107 to 110
#if (defined(GGML_USE_CUBLAS) || defined(GGML_USE_SYCL))
#define GGML_USE_CUBLAS_SYCL
#endif

Copy link
Member

@ggerganov ggerganov Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GGML_USE_CUBLAS_SYCL appears to be unused

Copy link
Collaborator Author

@NeoZhangJianyu NeoZhangJianyu Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, rm it.

@NeoZhangJianyu NeoZhangJianyu merged commit 7156413 into ggml-org:master Mar 2, 2024
@NeoZhangJianyu NeoZhangJianyu mentioned this pull request Mar 5, 2024
Closed
5 tasks
hazelnutcloud pushed a commit to hazelnutcloud/llama.cpp that referenced this pull request Mar 10, 2024
@NeoZhangJianyu @hazelnutcloud
Support multiple GPUs (split mode) on SYCL backend (ggml-org#5806)
e789d25 
* suport multiple cards: split-mode - layer|row

* rm warning

* rebase with master, support tow new OPs, close feature for -sm=row, fix for unit test

* update news

* fix merge error

* update according to review comments
@AidanBeltonS AidanBeltonS mentioned this pull request Mar 12, 2024
Closed
jordankanter pushed a commit to jordankanter/llama.cpp that referenced this pull request Mar 13, 2024
@NeoZhangJianyu @jordankanter
Support multiple GPUs (split mode) on SYCL backend (ggml-org#5806)
ab86fba 
* suport multiple cards: split-mode - layer|row

* rm warning

* rebase with master, support tow new OPs, close feature for -sm=row, fix for unit test

* update news

* fix merge error

* update according to review comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes

@slaren slaren slaren approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NeoZhangJianyu @ggerganov @slaren