Fix Policy Based Routing for private gateway static routes #3604

DennisKonrad · 2019-09-23T15:19:41Z

Description

We noticed that VMs that are target of a static nat cannot use private gateways.

Investigation showed that these VMs behave differently than all other VMs in a VPC because their whole traffic gets marked and is only allowed to leave through the public ip provided by the static nat.

Could be related to: #3366
Maybe @richardlawley @rhtyd can contribute if this is an unwanted side-effect.

To not break anything we propose a solution that does only mark the nat portion of the traffic and allow the VM use the private gateway like normal.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

We built and manually tested this to check if function is restored and nothing else breaks.

richardlawley · 2019-09-23T15:31:52Z

I doubt this was introduced in #3366 - all I did there was change to the proper way of adding rules to the front. Before this change, rules were still added to the front, but this happened multiple times as the code couldn't detect it as a duplicate rule.

DennisKonrad · 2019-09-23T16:25:19Z

Thanks you for helping to find the source of this issue @richardlawley.
We noticed the behaviour of the nat'ed VM changed in the passed year somehow. Our systemvm where the private gatways were working (for the natted vm) was about a year old. Maybe something else changed that I did not find/see?

Or could the issue solved by #3366 have caused the packet mark not to work somehow?

richardlawley · 2019-09-23T16:52:14Z

I'm not sure. I made a few changes around the same time for static nat, especially when there are multiple public networks, but some of these were in code branches which don't get hit on a vpc.

Unfortunately I'm only running 4.11.2 in production, but have all of my fixes ported into our systemvm iso. If I get a chance I'll see if I can reproduce the problem.

DennisKonrad · 2019-09-23T17:00:31Z

That would be a great help. I think most cs users don't use the private gateways and maybe this nat issue was not noticed?

What do you think of the proposed changes? Do you think it breaks functionality?

DennisKonrad · 2019-10-07T14:14:54Z

@rhtyd I looked into the failed test and I think it failed because of:

deletenetworkoffering failed, due to: errorCode: 431, errorText:Can't delete network offering 75 as its used by 1 networks. To make the network offering unavaiable, disable it

It seems this is not caused by the changes in this PR. Can you confirm?

DaanHoogland · 2020-01-03T11:30:12Z

should this go on 4.13 as well, @andrijapanicsb @DennisKonrad ?

andrijapanicsb · 2020-01-16T22:12:33Z

Makes sense @DaanHoogland

andrijapanicsb · 2020-01-16T22:17:31Z

@DennisKonrad I would like to test this manually.
Can you please give EXACT explanation on the setup to reproduce the issue (create this, create that, etc....)

Thanks!

DennisKonrad · 2020-01-20T08:25:03Z

@andrijapanicsb I'll describe the setup to you so you can reproduce it. In short:

If you use VPC with private Gateways and static NAT, the one VM that the NAT points to isn't able to use the private gateway anymore.
The private gateway is used to connect to an outside network (with static routes).

This worked up to a change that I cannot pin down exactly and after we updated the NATed VM lost connection to the machines behind the private gateway. Therefor the fix

rohityadavcloud · 2020-01-28T05:17:45Z

@DennisKonrad can you change the PR base to 4.13 and rebase against origin/4.13?

DaanHoogland · 2020-01-28T08:14:37Z

@DennisKonrad can you change the PR base to 4.13 and rebase against origin/4.13?

and mark milestone 4.13.1, please?

DennisKonrad · 2020-01-29T10:34:41Z

@DaanHoogland @rhtyd After a lot of pain:
Rebased to origin/4.13
Changed PR base to 4.13
Changed Milestone to 4.13

DaanHoogland

changing the rule on source/internal to destination/public makes sense in view of the report. code looks good

DaanHoogland · 2020-01-29T12:19:08Z

@blueorangutan package

blueorangutan · 2020-01-29T12:19:21Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-29T13:25:09Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-691

DaanHoogland · 2020-01-29T14:15:39Z

@blueorangutan test

blueorangutan · 2020-01-29T14:16:20Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-01-29T23:35:02Z

Trillian test result (tid-844)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31585 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3604-t844-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2020-01-30T08:53:26Z

tests look good, second pair of eyes, @andrijapanicsb @wido @rhtyd @weizhouapache ?

weizhouapache

LGTM

tested manually
the issue can be reproduced, and fixed with this change.

wido

LGTM

DaanHoogland · 2020-01-30T10:31:43Z

tnx guys

* 4.13: Fix Policy Based Routing for private gateway static routes (#3604)

* Fix for routing table issue with NAT interfaces * Mark only packets with the public ip as destination

…pache#3604)" This reverts commit 82d94a8.

…3604)" This reverts commit 82d94a8.

* master: (25 commits) integration test: skip vlan of public ip range in get_free_vlan vpc vr: plugin nics by this order: public/private/guest vpc vr: fix Conflicting device id on private gw nic Adding zone name to physicalnetworkresponse (apache#4510) Disallowing udp for lb rules for haproxy (apache#4501) Make global setting non-dynamic (apache#4505) Adding cpuallocated percentage and value to host and hostsformigrationresponse (apache#4499) kvm: fix router.aggregation.command.each.timeout is reset to 600 when update other kvm configs (apache#4496) fix failures with test_multiple_nic_support.py (apache#4495) Fix hosts for migration count (apache#4500) sql: Fix Zones are returned in a random order (apache#3934) (apache#4494) integration test: update steps integration test: add private gateway in test integration test: verify public nics state bugfix apache#9 vpc vr: Add PREROUTING rule for vm with static nat to multiple private gateways bugfix apache#8 vpc: add rule for traffic between vm and private gateway bugfix apache#7 vpc vr: allow servers in private gateway to reach internet via the VPC VR if it is gateway bugfix apache#6 vpc vr: Add iptables rules for ACL of private gateway Revert "Fix Policy Based Routing for private gateway static routes (apache#3604)" Revert "Add private gateway IP to router initialization config" ...

DennisKonrad added type:bug component:virtual-router labels Sep 23, 2019

DennisKonrad added this to the 4.14.0.0 milestone Sep 23, 2019

DennisKonrad self-assigned this Sep 23, 2019

DennisKonrad requested review from rohityadavcloud and wido September 23, 2019 16:58

rohityadavcloud changed the base branch from master to 4.13 September 25, 2019 05:39

rohityadavcloud changed the base branch from 4.13 to master September 25, 2019 05:39

rohityadavcloud changed the base branch from master to 4.13 January 28, 2020 05:17

rohityadavcloud changed the base branch from 4.13 to master January 28, 2020 05:17

DennisKonrad changed the base branch from master to 4.13 January 28, 2020 09:18

DennisKonrad modified the milestones: 4.14.0.0, 4.13.1.0 Jan 28, 2020

Dennis Konrad added 2 commits January 29, 2020 11:31

Fix for routing table issue with NAT interfaces

4a8e384

Mark only packets with the public ip as destination

b1ee51f

apache deleted a comment from blueorangutan Jan 29, 2020

DaanHoogland approved these changes Jan 29, 2020

View reviewed changes

apache deleted a comment from rohityadavcloud Jan 29, 2020

apache deleted a comment from blueorangutan Jan 29, 2020

apache deleted a comment from rohityadavcloud Jan 29, 2020

apache deleted a comment from blueorangutan Jan 29, 2020

apache deleted a comment from andrijapanicsb Jan 29, 2020

apache deleted a comment from blueorangutan Jan 29, 2020

weizhouapache approved these changes Jan 30, 2020

View reviewed changes

wido approved these changes Jan 30, 2020

View reviewed changes

DaanHoogland merged commit 82d94a8 into apache:4.13 Jan 30, 2020

DaanHoogland added a commit that referenced this pull request Jan 30, 2020

Merge release branch 4.13 to master

99ec8a8

* 4.13: Fix Policy Based Routing for private gateway static routes (#3604)

ustcweizhou pushed a commit to ustcweizhou/cloudstack that referenced this pull request Feb 28, 2020

Fix Policy Based Routing for private gateway static routes (apache#3604)

959eb95

* Fix for routing table issue with NAT interfaces * Mark only packets with the public ip as destination

weizhouapache mentioned this pull request Aug 2, 2020

resolves issues Multiple Public Subnets static NAT #4236

Closed

5 tasks

ustcweizhou added a commit to ustcweizhou/cloudstack that referenced this pull request Nov 19, 2020

Revert "Fix Policy Based Routing for private gateway static routes (a…

f204d09

…pache#3604)" This reverts commit 82d94a8.

ustcweizhou mentioned this pull request Nov 19, 2020

VPC: fix some issues related to multiple public IP ranges and private gateway #4484

Merged

10 tasks

ustcweizhou added a commit to ustcweizhou/cloudstack that referenced this pull request Nov 23, 2020

Revert "Fix Policy Based Routing for private gateway static routes (a…

f12af22

…pache#3604)" This reverts commit 82d94a8.

DaanHoogland pushed a commit that referenced this pull request Nov 25, 2020

Revert "Fix Policy Based Routing for private gateway static routes (#…

7e6f484

…3604)" This reverts commit 82d94a8.

Fix Policy Based Routing for private gateway static routes #3604

Fix Policy Based Routing for private gateway static routes #3604

Uh oh!

Conversation

DennisKonrad commented Sep 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

How Has This Been Tested?

Uh oh!

richardlawley commented Sep 23, 2019

Uh oh!

DennisKonrad commented Sep 23, 2019

Uh oh!

richardlawley commented Sep 23, 2019

Uh oh!

DennisKonrad commented Sep 23, 2019

Uh oh!

DennisKonrad commented Oct 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DaanHoogland commented Jan 3, 2020

Uh oh!

andrijapanicsb commented Jan 16, 2020

Uh oh!

andrijapanicsb commented Jan 16, 2020

Uh oh!

DennisKonrad commented Jan 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rohityadavcloud commented Jan 28, 2020

Uh oh!

DaanHoogland commented Jan 28, 2020

Uh oh!

DennisKonrad commented Jan 29, 2020

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jan 29, 2020

Uh oh!

blueorangutan commented Jan 29, 2020

Uh oh!

blueorangutan commented Jan 29, 2020

Uh oh!

DaanHoogland commented Jan 29, 2020

Uh oh!

blueorangutan commented Jan 29, 2020

Uh oh!

blueorangutan commented Jan 29, 2020

Uh oh!

DaanHoogland commented Jan 30, 2020

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

wido left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jan 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

DennisKonrad commented Sep 23, 2019 •

edited

Loading

DennisKonrad commented Oct 7, 2019 •

edited

Loading

DennisKonrad commented Jan 20, 2020 •

edited

Loading