CLOUDSTACK-8609: [VMware] VM is not accessible after migration across clusters

[sureshanaparti]

[VMware] VM is not accessible after migration across clusters.

• Once a VM is successfully started, don't delete the files associated with the unregistered VM, if the files are in a storage that is being used by the new VM.
• Attempt to unregister a VM in another DC, only if there is a host associated with a VM.

This closes #556

[sureshanaparti]

@borisstoyanov Can you please run VMware trillian tests on this PR

[borisstoyanov]

@blueorangutan package

[blueorangutan]

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-704

[borisstoyanov]

@blueorangutan test

[blueorangutan]

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-1070)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38743 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2091-t1070-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermitten failure detected: /marvin/tests/smoke/test_iso.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Test completed. 48 look ok, 4 have error(s)

Test	Result	Time (s)	Test File
test_04_rvpc_privategw_static_routes	Failure	391.95	test_privategw_acl.py
ContextSuite context=TestInternalLb>:setup	Error	0.00	test_internal_lb.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	Error	0.00	test_deploy_virtio_scsi_vm.py
test_01_vpc_site2site_vpn	Success	185.72	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	Success	71.24	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	Success	286.26	test_vpc_vpn.py
test_02_VPC_default_routes	Success	270.83	test_vpc_router_nics.py
test_01_VPC_nics_after_destroy	Success	589.27	test_vpc_router_nics.py
test_05_rvpc_multi_tiers	Success	538.76	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	Success	1352.07	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Success	580.14	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	Success	775.69	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Success	1318.43	test_vpc_redundant.py
test_09_delete_detached_volume	Success	156.66	test_volumes.py
test_08_resize_volume	Success	156.46	test_volumes.py
test_07_resize_fail	Success	161.62	test_volumes.py
test_06_download_detached_volume	Success	151.42	test_volumes.py
test_05_detach_volume	Success	150.83	test_volumes.py
test_04_delete_attached_volume	Success	151.32	test_volumes.py
test_03_download_attached_volume	Success	156.48	test_volumes.py
test_02_attach_volume	Success	89.63	test_volumes.py
test_01_create_volume	Success	711.94	test_volumes.py
test_03_delete_vm_snapshots	Success	275.21	test_vm_snapshots.py
test_02_revert_vm_snapshots	Success	100.89	test_vm_snapshots.py
test_01_create_vm_snapshots	Success	168.96	test_vm_snapshots.py
test_deploy_vm_multiple	Success	293.09	test_vm_life_cycle.py
test_deploy_vm	Success	0.03	test_vm_life_cycle.py
test_advZoneVirtualRouter	Success	0.04	test_vm_life_cycle.py
test_10_attachAndDetach_iso	Success	26.80	test_vm_life_cycle.py
test_09_expunge_vm	Success	125.26	test_vm_life_cycle.py
test_08_migrate_vm	Success	30.98	test_vm_life_cycle.py
test_07_restore_vm	Success	0.13	test_vm_life_cycle.py
test_06_destroy_vm	Success	125.90	test_vm_life_cycle.py
test_03_reboot_vm	Success	125.92	test_vm_life_cycle.py
test_02_start_vm	Success	10.18	test_vm_life_cycle.py
test_01_stop_vm	Success	35.30	test_vm_life_cycle.py
test_CreateTemplateWithDuplicateName	Success	141.51	test_templates.py
test_08_list_system_templates	Success	0.04	test_templates.py
test_07_list_public_templates	Success	0.05	test_templates.py
test_05_template_permissions	Success	0.07	test_templates.py
test_04_extract_template	Success	5.20	test_templates.py
test_03_delete_template	Success	5.12	test_templates.py
test_02_edit_template	Success	90.22	test_templates.py
test_01_create_template	Success	85.76	test_templates.py
test_10_destroy_cpvm	Success	161.69	test_ssvm.py
test_09_destroy_ssvm	Success	193.79	test_ssvm.py
test_08_reboot_cpvm	Success	131.79	test_ssvm.py
test_07_reboot_ssvm	Success	133.64	test_ssvm.py
test_06_stop_cpvm	Success	131.86	test_ssvm.py
test_05_stop_ssvm	Success	138.81	test_ssvm.py
test_04_cpvm_internals	Success	1.24	test_ssvm.py
test_03_ssvm_internals	Success	3.67	test_ssvm.py
test_02_list_cpvm_vm	Success	0.16	test_ssvm.py
test_01_list_sec_storage_vm	Success	0.14	test_ssvm.py
test_02_list_snapshots_with_removed_data_store	Success	87.06	test_snapshots.py
test_01_snapshot_root_disk	Success	16.38	test_snapshots.py
test_04_change_offering_small	Success	239.69	test_service_offerings.py
test_03_delete_service_offering	Success	0.04	test_service_offerings.py
test_02_edit_service_offering	Success	0.06	test_service_offerings.py
test_01_create_service_offering	Success	0.13	test_service_offerings.py
test_02_sys_template_ready	Success	0.14	test_secondary_storage.py
test_01_sys_vm_start	Success	0.18	test_secondary_storage.py
test_09_reboot_router	Success	45.41	test_routers.py
test_08_start_router	Success	35.35	test_routers.py
test_07_stop_router	Success	10.17	test_routers.py
test_06_router_advanced	Success	0.06	test_routers.py
test_05_router_basic	Success	0.04	test_routers.py
test_04_restart_network_wo_cleanup	Success	5.77	test_routers.py
test_03_restart_network_cleanup	Success	70.71	test_routers.py
test_02_router_internal_adv	Success	1.13	test_routers.py
test_01_router_internal_basic	Success	0.67	test_routers.py
test_router_dns_guestipquery	Success	76.96	test_router_dns.py
test_router_dns_externalipquery	Success	0.07	test_router_dns.py
test_router_dhcphosts	Success	313.68	test_router_dhcphosts.py
test_router_dhcp_opts	Success	21.91	test_router_dhcphosts.py
test_01_updatevolumedetail	Success	0.12	test_resource_detail.py
test_01_reset_vm_on_reboot	Success	130.99	test_reset_vm_on_reboot.py
test_createRegion	Success	0.04	test_regions.py
test_create_pvlan_network	Success	5.23	test_pvlan.py
test_dedicatePublicIpRange	Success	0.53	test_public_ip_range.py
test_03_vpc_privategw_restart_vpc_cleanup	Success	541.09	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Success	416.39	test_privategw_acl.py
test_01_vpc_privategw_acl	Success	102.50	test_privategw_acl.py
test_03_migration_options_storage_tags	Success	161.96	test_primary_storage.py
test_02_edit_primary_storage_tags	Success	0.12	test_primary_storage.py
test_01_primary_storage_nfs	Success	35.80	test_primary_storage.py
test_01_deploy_vms_storage_tags	Success	31.22	test_primary_storage.py
test_createPortablePublicIPRange	Success	15.23	test_portable_publicip.py
test_createPortablePublicIPAcquire	Success	15.53	test_portable_publicip.py
test_isolate_network_password_server	Success	89.77	test_password_server.py
test_UpdateStorageOverProvisioningFactor	Success	0.16	test_over_provisioning.py
test_oobm_zchange_password	Success	25.80	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Success	16.46	test_outofbandmanagement.py
test_oobm_issue_power_status	Success	10.32	test_outofbandmanagement.py
test_oobm_issue_power_soft	Success	15.39	test_outofbandmanagement.py
test_oobm_issue_power_reset	Success	15.91	test_outofbandmanagement.py
test_oobm_issue_power_on	Success	10.45	test_outofbandmanagement.py
test_oobm_issue_power_off	Success	15.41	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Success	15.42	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Success	87.79	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Success	5.19	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Success	0.11	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Success	5.21	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Success	0.13	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Success	0.09	test_outofbandmanagement.py
test_oobm_configure_default_driver	Success	0.10	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Success	29.49	test_outofbandmanagement.py
test_extendPhysicalNetworkVlan	Success	15.47	test_non_contigiousvlan.py
test_01_nic	Success	434.59	test_nic.py
test_releaseIP	Success	293.30	test_network.py
test_reboot_router	Success	403.65	test_network.py
test_public_ip_user_account	Success	10.27	test_network.py
test_public_ip_admin_account	Success	40.32	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	Success	67.13	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	Success	62.04	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	Success	124.26	test_network.py
test_delete_account	Success	318.59	test_network.py
test_02_port_fwd_on_non_src_nat	Success	55.77	test_network.py
test_01_port_fwd_on_src_nat	Success	111.77	test_network.py
test_nic_secondaryip_add_remove	Success	248.65	test_multipleips_per_nic.py
test_list_zones_metrics	Success	0.29	test_metrics_api.py
test_list_volumes_metrics	Success	5.50	test_metrics_api.py
test_list_vms_metrics	Success	282.69	test_metrics_api.py
test_list_pstorage_metrics	Success	0.40	test_metrics_api.py
test_list_infrastructure_metrics	Success	0.47	test_metrics_api.py
test_list_hosts_metrics	Success	0.41	test_metrics_api.py
test_list_clusters_metrics	Success	0.35	test_metrics_api.py
login_test_saml_user	Success	19.58	test_login.py
test_assign_and_removal_lb	Success	151.07	test_loadbalance.py
test_02_create_lb_rule_non_nat	Success	187.59	test_loadbalance.py
test_01_create_lb_rule_src_nat	Success	242.99	test_loadbalance.py
test_03_list_snapshots	Success	0.10	test_list_ids_parameter.py
test_02_list_templates	Success	0.05	test_list_ids_parameter.py
test_01_list_volumes	Success	0.04	test_list_ids_parameter.py
test_07_list_default_iso	Success	0.07	test_iso.py
test_05_iso_permissions	Success	0.08	test_iso.py
test_04_extract_Iso	Success	18.58	test_iso.py
test_03_delete_iso	Success	95.17	test_iso.py
test_02_edit_iso	Success	0.07	test_iso.py
test_01_create_iso	Success	71.32	test_iso.py
test_dedicateGuestVlanRange	Success	10.35	test_guest_vlan_range.py
test_UpdateConfigParamWithScope	Success	0.14	test_global_settings.py
test_rolepermission_lifecycle_update	Success	6.28	test_dynamicroles.py
test_rolepermission_lifecycle_list	Success	6.02	test_dynamicroles.py
test_rolepermission_lifecycle_delete	Success	5.90	test_dynamicroles.py
test_rolepermission_lifecycle_create	Success	6.05	test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updates	Success	6.09	test_dynamicroles.py
test_role_lifecycle_update_role_inuse	Success	5.93	test_dynamicroles.py
test_role_lifecycle_update	Success	11.05	test_dynamicroles.py
test_role_lifecycle_list	Success	5.95	test_dynamicroles.py
test_role_lifecycle_delete	Success	11.02	test_dynamicroles.py
test_role_lifecycle_create	Success	5.96	test_dynamicroles.py
test_role_inuse_deletion	Success	5.92	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	Success	8.36	test_dynamicroles.py
test_role_account_acls	Success	8.58	test_dynamicroles.py
test_default_role_deletion	Success	6.04	test_dynamicroles.py
test_04_create_fat_type_disk_offering	Success	0.12	test_disk_offerings.py
test_03_delete_disk_offering	Success	0.05	test_disk_offerings.py
test_02_edit_disk_offering	Success	0.06	test_disk_offerings.py
test_02_create_sparse_type_disk_offering	Success	0.11	test_disk_offerings.py
test_01_create_disk_offering	Success	0.12	test_disk_offerings.py
test_deployvm_userdispersing	Success	25.89	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated	Success	20.77	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfit	Success	156.41	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_post	Success	25.51	test_deploy_vm_with_userdata.py
test_deployvm_userdata	Success	116.26	test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resize	Success	0.07	test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resize	Success	0.07	test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resize	Success	383.04	test_deploy_vm_root_resize.py
test_deploy_vm_from_iso	Success	288.24	test_deploy_vm_iso.py
test_DeployVmAntiAffinityGroup	Success	186.94	test_affinity_groups.py
test_change_service_offering_for_vm_with_snapshots	Skipped	0.00	test_vm_snapshots.py
test_06_copy_template	Skipped	0.00	test_templates.py
test_static_role_account_acls	Skipped	0.02	test_staticroles.py
test_11_ss_nfs_version_on_ssvm	Skipped	0.02	test_ssvm.py
test_01_scale_vm	Skipped	0.00	test_scale_vm.py
test_01_primary_storage_iscsi	Skipped	0.04	test_primary_storage.py
test_nested_virtualization_vmware	Skipped	0.00	test_nested_virtualization.py
test_06_copy_iso	Skipped	0.00	test_iso.py
test_deploy_vgpu_enabled_vm	Skipped	0.03	test_deploy_vgpu_enabled_vm.py
test_3d_gpu_support	Skipped	0.04	test_deploy_vgpu_enabled_vm.py

[ghost]

Fix verified. LGTM.

[cloudmonger]

ACS CI BVT Run

Sumarry:
Build Number 842
Hypervisor xenserver
NetworkType Advanced
Passed=109
Failed=4
Skipped=12

Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0

Failed tests:

• test_routers_network_ops.py

• test_01_isolate_network_FW_PF_default_routes_egress_true Failing since 30 runs

• test_02_isolate_network_FW_PF_default_routes_egress_false Failing since 30 runs

• test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failing since 30 runs

• test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failing since 30 runs

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

[rohityadavcloud]

Ping can you fix the conflicts @sureshanaparti

[resmo]

I would like to see this in 4.11.1 @rhtyd

[rohityadavcloud]

@resmo I'll inform that PR author, ultimately it will depend on how responsive the PR author is and if we're able to test,review,merge this PR before 4.11.1 RC1. You may also re-open the PR in case you would like to pursue with this PR yourself.
@sureshanaparti can you re-base the PR against 4.11, fix conflicts?

[rohityadavcloud]

@sureshanaparti can you rebase this against latest 4.11.
I'll in the interim move this to 4.11.2.0 milestone, if the author does not repond I may take over this PR in future.

[sureshanaparti]

@rhtyd I'll rebase and update. Thanks.

[rohityadavcloud]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2250

[DaanHoogland]

@blueorangutan help

[blueorangutan]

@DaanHoogland I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [additional tests]
Mgmt OS options: ['centos6', 'centos7', 'ubuntu']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-ubuntu', 'xenserver-71', 'xenserver-65sp1', 'xenserver-62sp1', 'vmware-65', 'vmware-60u2', 'vmware-55u3', 'vmware-51u1', 'vmware-50u1']
Additional tests: list of space separated tests with paths relative to the test/integration directory, for example: component/test_acl_listvm.py component/test_volumes.py
Note: when additional tests are passed, you need to specify mgmt server os and hypervisor or use the matrix command.

Blessed contributors for kicking Trillian test jobs: ['rhtyd', 'nvazquez', 'PaulAngus', 'borisstoyanov', 'DaanHoogland', 'ernjvr', 'dhlaluku', 'houthuis', 'charles-phiri']

[DaanHoogland]

@blueorangutan test centos6 vmware-55u3

[blueorangutan]

@DaanHoogland a Trillian-Jenkins test job (centos6 mgmt + vmware-55u3) has been kicked to run smoke tests

[rohityadavcloud]

@blueorangutan test centos7 vmware-65

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-65) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-2937)
Environment: vmware-65 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32660 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2091-t2937-vmware-65.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_usage.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:teardown	Error	0.00	test_deploy_virtio_scsi_vm.py
test_04_extract_template	Failure	134.42	test_templates.py
ContextSuite context=TestISOUsage>:setup	Error	0.00	test_usage.py
test_06_download_detached_volume	Failure	160.25	test_volumes.py

[rohityadavcloud]

LGTM

[rohityadavcloud]

Review request - @DaanHoogland @rafaelweingartner @resmo and others

[DaanHoogland]

system VMs did not get up ==>> timeout
@blueorangutan test centos6 vmware-55u3

[DaanHoogland]

@blueorangutan test centos6 vmware-55u3

[blueorangutan]

@DaanHoogland a Trillian-Jenkins test job (centos6 mgmt + vmware-55u3) has been kicked to run smoke tests

[rohityadavcloud]

@blueorangutan test centos7 vmware-55u3

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-55u3) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-2943)
Environment: vmware-55u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30840 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2091-t2943-vmware-55u3.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Smoke tests completed. 66 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:teardown	Error	0.00	test_deploy_virtio_scsi_vm.py

[DaanHoogland]

is scsi virtio supposed to work (in vmware)? @rhtyd @rafaelweingartner, if not, any other test needed?

[rohityadavcloud]

No @DaanHoogland, that test should skip for vmware. I would ignore the error for now, maybe fix that smoketest in a different PR.

[borisstoyanov]

Looks like it's properly skipping all non kvm hypervisors

[resmo]

LGTM

[rohityadavcloud]

@borisstoyanov looks like there was an issue in the tear down of that test

[rohityadavcloud]

Merging this based on test results and reviews