[SolidFire] Cannot attach volume which is detached from VM

[weizhouapache]

see title. The issue happens when use Solidfire as primary storage on vmware.

steps to reproduce the issue

1. create a volume and attach to vm
2. detach volume
3. attach volume again

Expected result: succeed
Actual result:

2023-01-27 14:40:19,380 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-489:ctx-057df38b xxx.yyy.zzz, job-44722/job-44723, cmd: AttachCommand) (logid:736275cc) Failed to attach volume!
com.vmware.vim25.DuplicateNameFaultMsg: The name '-iqn.2010-01.com.solidfire:3p53.data-vol-test-01.62-0' already exists.
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:117)
    at com.sun.xml.ws.client.sei.StubHandler.readResponse(StubHandler.java:223)
    at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:176)
    at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:263)
    at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
    at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:62)
    at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:131)
    at com.sun.proxy.$Proxy698.createVmfsDatastore(Unknown Source)
    at com.cloud.hypervisor.vmware.mo.HostDatastoreSystemMO.createVmfsDatastore(HostDatastoreSystemMO.java:211)
    at com.cloud.storage.resource.VmwareStorageProcessor.getVmfsDatastore(VmwareStorageProcessor.java:2825)
    at com.cloud.storage.resource.VmwareStorageProcessor.prepareManagedDatastore(VmwareStorageProcessor.java:2666)
    at com.cloud.storage.resource.VmwareStorageProcessor.prepareManagedStorage(VmwareStorageProcessor.java:3450)
    at com.cloud.storage.resource.VmwareStorageProcessor.attachVolume(VmwareStorageProcessor.java:2034)
    at com.cloud.storage.resource.VmwareStorageProcessor.attachVolume(VmwareStorageProcessor.java:1997)
    at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:163)
    at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:67)
    at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:586)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

As the datastore has been already created (before DetachVolume), cloudstack just needs to mount the datastore instead of creating a new datastore.

ISSUE TYPE

• Bug Report

COMPONENT NAME

vmware, storage

CLOUDSTACK VERSION

4.17.2.0 (and probably previous versions)

CONFIGURATION

vmware
solidfire as primary storage

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[weizhouapache]

as suggested by @rohityadavcloud and @DaanHoogland
keep this issue open until a solidfire user confirms it.

[DaanHoogland]

@weizhouapache @rohityadavcloud If no solidfire user confirms or invalidates this fix before 4.18.1 is out, let´ s close this one. Makes no sense to keep it open forever.

[rohityadavcloud]

@weizhouapache @shwstppr I've tested this fix backported to a 4.17 env with Solidfire and VMware; it fixes the issue of previously failured where it would attempt to create datastore on an already created datastore for a solidfire iSCSI target. However, it doesn't address the issue #7148 completely.

The issue now it with the fix from #7160, a solidfire data disk is detached successfully in CloudStack but we see failures to remove/unmount the iSCSI target on vCenter:

Furthermore, in vCenter we see the datastore gets inaccessible (umounted with errors):

Next time we try to attach the solidfire data disk in CloudStack, it fails as it somehow fails to mount the datastore on all hosts (error from cloudstack logs) and this is seen in vCenter:

However, surprisingly if I attach from CloudStack and manually mount the inaccessible datastore in vCenter, then attach operation succeeds.

[rohityadavcloud]

The previous PR doesn't fix the issue, I've tested the fixes and raised - #7761
cc @weizhouapache

[weizhouapache]

fixed by #7761