Clarify semantics and restrictions of user config for Peering and VpcExpose objects

[qmonnet]

When users define Expose objects to connect VPCs, we need to clearly define what is allowed and what is not, and what the meaning of some expressions is. Eventually, this should translate into making sure that the appropriate behaviour (matching the semantics) is implemented, and setting up the relevant validation steps to enforce restrictions.

Related:

• NAT: Clarify what happens when user passes blank prefix lists, non-empty exclusion prefix lists #779

As a reminder, we consider an Expose object as:

• A list of prefixes (and optional associated port ranges - this is coming soon) for internal prefixes exposed, ips
• A list of prefixes (and optional associated port ranges) for actually exposed addresses and ports, after NAT/PAT, as_range
• A list of exclusion prefixes (and optional associated port ranges) to apply to these exposed prefixes, not and not_as (for ips and as_range, respectively).

The lists may or may not be present from the CRD passed by the user.

Here's a non-exhaustive list of items to consider for an Expose object.

1. Can lists be present but empty? Does it change the behaviour compared to when they're not present at all?
2. What happens if ips list is not present?
3. In particular, what happens if ips is not present but if a not list is present?
4. What happens if ips list contains 0.0.0.0/0 as prefix (or ::/0)?
   • With a port range?
   • Without a port range?
5. What happens if as_range list is not present?
6. In particular, what happens if as_range is not present but if a not_as list is present?
7. What happens if as_range list contains 0.0.0.0/0 as prefix (or ::/0)?
   • With a port range?
   • Without a port range?
8. What happens if not or not_as contains 0.0.0.0/0 as prefix (or ::/0)?
   • With a port range?
   • Without a port range?
9. Can a list contain a mix of IPv4/IPv6 prefixes?
10. Can the different lists of an expose contain a mix of IPv4/IPv6 prefixes?
    • Today?
    • In the future for NAT46/NAT64?
11. Can prefixes in ips or as_range lists overlap? (IP addresses overlapping, or if both have port ranges, IP addresses + port ranges overlapping)
12. Can prefixes in not or not_as lists overlap?
13. Can prefixes in not or not_as contain item not in ips or as_range (respectively)?
    • If there's not overlap at all?
    • If there's some overlap? (Example: exclude 0.0.0.0/0, ports 5001-6000 from some ips)
14. Can prefixes in ips for different Expose objects of a peering, once their exclusion prefixes applied, overlap?
15. Can prefixes in as_range for different Expose objects of a peering, once their exclusion prefixes applied, overlap?
16. Can we have multiple peerings between any pair of VPCs?
17. Can prefixes in ips for Expose objects from different peerings between a pair of VPC overlap with prefixes from ips in the Expose of another peering for the same VPCs?
18. Can prefixes in as_range for Expose objects from different peerings between a pair of VPC overlap with prefixes from as_range in the Expose of another peering for the same VPCs?
19. Do we need a particular treatment for some ranges of IPs (0.0.0.0, 255.255.255.255, etc.)?
20. Do we need a particular treatment for port 0 (mostly invalid for TCP/UDP)?
21. When port ranges are in use, what are the expectations regarding ICMP?
22. ...

We should settle on the expectations for each of these items (some of them we already know) and formalise it, so we can document it for the users and implement the relevant behaviour/validation. We should also double-check the existing validation steps.

[qmonnet]

22. Same again with port forwarding

[qmonnet]

Here are the results from today's discussion:

---

1. Can lists be present but empty? Does it change the behaviour compared to when they're not present at all?

   • illegal

2. What happens if ips list is not present?

   • illegal

3. In particular, what happens if ips is not present but if a not list is present?
   a. In the generic case?
   b. For "default" + nots?

   • illegal in generic case
   • "default" + nots is illegal too for now, but might make sense in the future

4. What happens if ips list contains 0.0.0.0/0 as prefix (or ::/0)?
   a. With a port range?
   b. Without a port range?

   • should be allowed (but DIFFERENT from default) -> TODO: change validation
   • port ranges doesn't change a thing

5. What happens if as_range list is not present?

   • With NAT set up: illegal
   • Otherwise, normal

6. In particular, what happens if as_range is not present but if a not_as list is present?

   • illegal

7. What happens if as_range list contains 0.0.0.0/0 as prefix (or ::/0)?
   a. With a port range?
   b. Without a port range?

   • no special case, it's OK -> TODO: check if validation supports this

8. What happens if not or not_as contains 0.0.0.0/0 as prefix (or ::/0)?
   a. With a port range?
   b. Without a port range?

   • With port range: fine if prefixes (with uncovered ports) remain
   • No port range: illegal (but because no prefixes left, nothing to do with 0s)

9. Can a list contain a mix of IPv4/IPv6 prefixes?

   • no, doesn't make sense within one expose

10. Can the different lists of an expose contain a mix of IPv4/IPv6 prefixes?
    a. Today?
    b. In the future for NAT46/NAT64?

    • not today
    • in future, likely

11. Can prefixes in ips or as_range lists overlap? (IP addresses overlapping, or if both have port ranges, IP addresses + port ranges overlapping) within same expose

    • currently not supported but should be allowed (need merging prefixes) -> TODO: Create an Issue

12. Can prefixes in not or not_as lists overlap? within 1 expose

    • same as 11

13. Can prefixes in not or not_as contain item not in ips or as_range (respectively)?
    a. If there's not overlap at all?
    b. If there's some overlap? (Example: exclude 0.0.0.0/0, ports 5001-6000 from some ips)

    • some overlap is OK (with warning)
    • no overlap is OK (with warning)
    • -> We reject it at the moment, we should just ignore (and warn on) these non-relevant exclusion prefixes

14. Can prefixes in ips for different Expose objects of a peering, once their exclusion prefixes applied, overlap?

    • No in generic case
    • Yes between port forwarding + masquerade only

15. Can prefixes in as_range for different Expose objects of a peering, once their exclusion prefixes applied, overlap?

    • No in generic case
    • Yes between port forwarding + masquerade only

16. Can we have multiple peerings between any pair of VPCs?

    • illegal

17. Can prefixes in ips for Expose objects from different peerings between a pair of VPCs overlap with prefixes from ips in the Expose of another peering for the same VPCs?

    • n/a, see 16

18. Can prefixes in as_range for Expose objects from different peerings between a pair of VPCs overlap with prefixes from as_range in the Expose of another peering for the same VPCs?

    • n/a, see 16

---

Due to time constraints, we didn't have time to go through the whole list. Remaining items include:

---

19. Can prefixes in ips for Expose objects from a peering between VPC A and VPC B overlap with prefixes from ips from a peering between VPC A and VPC C?
    a. Full overlap?
    b. Partial overlap?

20. Can prefixes in as_range for Expose objects from a peering between VPC A and VPC B overlap with prefixes from as_range from a peering between VPC A and VPC C?
    a. Full overlap?
    b. Partial overlap?

21. Do we need a particular treatment for some ranges of IPs (0.0.0.0, 255.255.255.255, multicast, etc.)?

22. Do we need a particular treatment for port 0 (mostly invalid for TCP/UDP)?

23. When port ranges are in use, what are the expectations regarding ICMP?

24. Can we have a "default" connect to a "default"?

25. Can a peering contain multiple "default" Expose on the same side?

26. Can we have a "default" with NAT on one side of a peering?

27. Can we have multiple "default" exposed to a VPC?

28. NAT combinations ON SAME PEERING END (distinct Expose blocks): all allowed?

29. NAT combinations ON TWO ENDS of a peering (one on each side)
    a. no NAT + no NAT
    b. no NAT + stateless NAT
    c. no NAT + stateful NAT
    d. no NAT + port forwarding
    e. stateless NAT + stateless NAT
    f. stateless NAT + stateful NAT
    g. stateless NAT + port forwarding
    h. stateful NAT + stateful NAT
    i. stateful NAT + port forwarding
    j. port forwarding + port forwarding

30. Stateful NAT + port forwarding combination (same peering end):
    a. Can we have stateful NAT prefixes superset of port forwarding prefixes?
    b. Can we have port forwarding prefixes superset of stateful NAT prefixes?
    c. Can we have partial overlap, with non-null XOR set?
    d. Can we have overlap on ips, but not on as_range?
    e. Can we have overlap on as_range, but not on ips?

[edipascale]

a few requests for clarification:

4. What happens if ips list contains 0.0.0.0/0 as prefix (or ::/0)?
   a. With a port range?
   b. Without a port range?

should be allowed (but DIFFERENT from default) -> TODO: change validation
port ranges doesn't change a thing

Can we explicitly state what the difference is? I presume that other exposes are not allowed to overlap with it, so essentially it becomes the only legal expose for that VPC?

7. What happens if as_range list contains 0.0.0.0/0 as prefix (or ::/0)?
   a. With a port range?
   b. Without a port range?

no special case, it's OK -> TODO: check if validation supports this

What is the expected behavior in this case? are addresses in the IP block going to be mapped to any other address, or to themselves? Not sure I understand the use case.

Additional questions:

• is there a use case for wanting port forwarding to the same IP, i.e. only remapping the ports within a range? I understand the current API requires an as_range for NAT, hence the question
• are we making any promises on how we are going to map NATed addresses in the various cases?
• why do we have static NAT as the default, given that in practice (imho) it's the least likely to be useful for the average user? would masquerade be a better default?

And finally a comment that will rightly be ignored given how close we are to release, but I'll make it anyway for the record: I think we should have dropped the not ranges, they are going to make our API and validation way more complex and the same result could be achieved without them.

[qmonnet]

4. What happens if ips list contains 0.0.0.0/0 as prefix (or ::/0)?
   a. With a port range?
   b. Without a port range?

should be allowed (but DIFFERENT from default) -> TODO: change validation
port ranges doesn't change a thing

Can we explicitly state what the difference is? I presume that other exposes are not allowed to overlap with it, so essentially it becomes the only legal expose for that VPC?

Correct, no overlap so no other possible expose (in the context of that peering - VPC can have other peerings with other VPCs of course)

---

7. What happens if as_range list contains 0.0.0.0/0 as prefix (or ::/0)?
   a. With a port range?
   b. Without a port range?

no special case, it's OK -> TODO: check if validation supports this

What is the expected behavior in this case? are addresses in the IP block going to be mapped to any other address, or to themselves? Not sure I understand the use case.

There's no special case, so IPs are mapped to any other address, just like for other prefixes. For stateful NAT, this means we can use the whole IP space in the pool for the addresses we can allocate (although in practice we might run out of memory before that). For stateless NAT, given that we need a 1:1 mapping, it means the ips list must also be 0.0.0.0/0 (unless we use port ranges, in which cases other combinations are possible too).

---

• is there a use case for wanting port forwarding to the same IP, i.e. only remapping the ports within a range? I understand the current API requires an as_range for NAT, hence the question

You can specify the same IP CIDR in ips and in as_range if you want to, they don't have to be different, so yes it's possible

• are we making any promises on how we are going to map NATed addresses in the various cases?

At the moment we don't. We could for some specific cases (for example, only one range on each side), but the generic case is hard to codify and it could change based on the implementation.

• why do we have static NAT as the default, given that in practice (imho) it's the least likely to be useful for the average user? would masquerade be a better default?

Good question, it might be time to revisit this (or make the NAT config block mandatory)

I think we should have dropped the not ranges, they are going to make our API and validation way more complex and the same result could be achieved without them.

You're not the only one to think that 🙂 We discussed it several times and the conclusion was that we needed to keep it.

[edipascale]

Thanks @qmonnet, that's helpful! just one more comment on this:

• is there a use case for wanting port forwarding to the same IP, i.e. only remapping the ports within a range? I understand the current API requires an as_range for NAT, hence the question

You can specify the same IP CIDR in ips and in as_range if you want to, they don't have to be different, so yes it's possible

In this case I propose we allow not specifying the as_range if port forwarding is selected, and assuming that the IPs are going to be mapped to themselves. I don't see the point in forcing the user to write the same ranges twice. Not a big deal either way but I think it makes for a better UX and it costs nothing (I think?)

[qmonnet]

The rest of the discussion, from today - thank you all!

---

19. Can prefixes in as_range for Expose objects from a peering between VPC A and VPC B overlap with prefixes from as_range from a peering between VPC A and VPC C?

    1. Full overlap?
    2. Partial overlap?

    • Yes:

      • if B and C both have masquerade
      • if B has port forwarding and C has masquerade
      • if B has masquerade and C has port forwarding
      • if B and C both have port forwarding, with no ports overlap (so no overlap from dataplane's point of view)

    • No: all other cases

20. Can prefixes in ips for Expose objects from a peering between VPC A and VPC B overlap with prefixes from ips from a peering between VPC A and VPC C?

    1. Full overlap?
    2. Partial overlap?
    • Yes (provided question 19. is addressed so we can differentiate based on destination address or flow table)

21. Do we need a particular treatment for some ranges of IPs (0.0.0.0, 255.255.255.255, multicast, etc.)?

    • 0.0.0.0/32 as a CIDR must be forbidden
    • 10.0.0.0/24 -> 0.0.0.0/24: error
    • 10.0.0.0/24 -> (0.0.0.0/24 + 1.1.1.1/32) -> ???
    • => No target prefix can include 0.0.0.0/32
    • What about 0.0.0.0/8 => Forbid as well
    • Range with 255.255.255.255 must be forbidden
    • Above means 0.0.0.0/0 must be forbidden too, this overwrites questions 4. and 7.
    • 224.0.0.0/4 is reserved: should be restricted too. (multicast)
    • IPv6 multicast too
    • 127.0.0.0... same
    • Better to forbid them now and allow them later if necessary.
    • IPv6: must look into the equivalents (ff00/8, ban address-link as well?: fead/10)
    • Forbid from both ips and as_ranges
    • TODO: Look into the list of prefixes to forbid, implement rejection

22. Do we need a particular treatment for port 0 (mostly invalid for TCP/UDP)?

    • Block 0 from port ranges (both ips and as_range/port forwarding ports)

23. When port ranges are in use, what are the expectations regarding ICMP?

    • Drop ICMP, for now. Longer term: needs to be configurable

24. Can we have a "default" connect to a "default"?

    • Let's not support this for now, intuition is it looks bad
    • VRF: both routes would be installed and we'd have issues

25. Can a peering contain multiple "default" Expose on the same side?

    • Not for now

26. Can we have a "default" with NAT on one side of a peering?

    • Not for now; maybe static later

27. Can we have multiple "default" exposed to a VPC?

    • Not for now; maybe via ECMP in the future

28. NAT combinations ON SAME PEERING END (distinct Expose blocks): all allowed?

    • no overlap, then all good
    • overlap: if masquerade + port forwarding only

29. NAT combinations ON TWO ENDS of a peering (one on each side)

    1. no NAT + no NAT
       • yes
    2. no NAT + stateless NAT
       • yes
    3. no NAT + stateful NAT
       • yes
    4. no NAT + port forwarding
       • yes
    5. stateless NAT + stateless NAT
       • yes
    6. stateless NAT + stateful NAT
       • not for now (Fredi and Quentin started to look into it)
    7. stateless NAT + port forwarding
       • not for now (same as vi.)
    8. stateful NAT + stateful NAT
       • not for now
    9. stateful NAT + port forwarding
       • not for now, same as vi.
    10. port forwarding + port forwarding
        • just PF + PF makes no sense (no side can initiate connection)
        • OK if we also have masquerading in addition, on both sides (or some other mechanism to initiate connection)
        • so not for now because we don't support masquerading on both sides; to be allowed later

30. Stateful NAT + port forwarding combination (same peering end):

    1. Can we have stateful NAT prefixes superset of port forwarding prefixes?
    2. Can we have port forwarding prefixes superset of stateful NAT prefixes?
    3. Can we have partial overlap, with non-null XOR set?
    4. Can we have overlap on ips, but not on as_range?
    5. Can we have overlap on as_range (+ ports), but not on ips?
    • Yes to all

31. Default NAT mode?

    • TODO: Make NAT spec mandatory for now

---

Further questions / corner cases remain welcome to discuss, even after the issue is closed.