MPI partitioning issues for Turbomachinery runs

[auzbaig]

The number of cores I can run Turbomachinery cases always behaves unpredictably. Here are my observations:

1. Ran my case with 1 or 2 cores it runs fine.

2. Any higher than that, it gives a segmentation fault or the following error:

---------------------- Turbomachinery Preprocessing ---------------------
Initialize Turbo Vertex Structure.
Error in "virtual void CPhysicalGeometry::ComputeNSpan(CConfig*, short unsigned int, short unsigned int, bool)":

At the moment only turbomachinery with the same amount of span-wise section can be simulated
nSpan inflow 113, nSpan outflow 32510
------------------------------ Error Exit -------------------------------

3. I generate the mesh again, with the same NPOIN and NELEM but the mesh files are different in point numbering, and now the case runs with 3 cores as well and above that gives the same error.

Is this issue known? It may be a small fix in "CPhysicalGeometry::ComputeNSpan" to partition nSpan outflow correctly.

• OS: CentOS 7
• SU2 Version: v7.0.4

[economon]

I am not too familiar with that routine, but I do know that there are some checks related to periodic BCs inside. Are you using periodic BCs for your case? You might go through the routine to see that the counting of the spans is correct. Perhaps @salvovitale can comment if there's time.

Also, can you verify that the test cases in SU2/TestCases/turbomachinery/ are working for you on your system? Might check for additional options that appear in those examples.

[auzbaig]

Yes, I am using periodic boundaries. However, the points are matched without any issue during initialization so I don't think the issue is there. Turbomachinery cases are working but the issue is with MPI where after a certain number of cores a case may encounter segmentation fault. I cannot predict beforehand at what number of cores it will fail, or if it will even fail or not.

[stephansmit]

I had the same issue, its depended on the way the mesh is partitioned. There is an error in the calculation of NSpan (they should be the same for the inlet and outlet) when the boundary is on two different cores (points are counted twice). I resolved it by making an extra entry in the config that specifies the number of spanwise sections.

The testcases are meshed in such a way that the partitioning rarely lead to the outlet being partitioned across two cores, so the errors doesnt pop up there.

[auzbaig]

@stephansmit I think that should be able to resolve the issue temporarily. Can you let me know how you did it? Can you start a pull request for those changes to merge into turbo_output_opt?

[auzbaig]

@stephansmit Is it possible for you to share what you did? I looked at the code and don't want to mess with the MPI stuff as I don't know about it. Currently, I am limited to 3 cores which is greatly slowing down the optimizations. Thanks.

[stephansmit]

@auzbaig Sorry, I was a bit busy with other stuff, I thought you will figure it out. I'll send you a merge request for turbo_output_opt.

[auzbaig]

I didn't want to dive into hardcoding nSpan that is related to the MPI and have unintended downstream effects.

[auzbaig]

I was having several other issues with optimization in v7. I have downgraded to v6 where none of these issues are there. I will not need a resolution for this at the moment. Thanks! @stephansmit

[ashvinmahajan]

I am having a similar issue with V71 where in the turbo-machinery pre-processing part it incorrectly calculates the number of spanwise sections when I use more than 6 processors and then it gives a segmentation fault. I can run it for less than 6 processors, but when I increase the processor (> 6) count I notice the incorrect spanwise sections and then there is segmentation fault or MPI fatal error.
Is there any solution to this problem?