Enforce xn rate limit per ENI

[KrisNey-MSFT]

Related information

Scope - P4 DASH pipeline.

[BsRamaKrishna]

How this can be tested with the software switch?

[KrisNey-MSFT]

Good question and possibly difficult to achieve? Does bmv2 support this? Meter on the add or miss; need to figure out how to implement. Andy Fingerhut knows of a sample program to show this. Discuss in PNA meetings or other, may need a separate meeting to discuss. (Condition within an action is allowed, however not required).

Public PNA meeting is Monday at 1pm Pacific.

[BsRamaKrishna]

Andy has a comment on this.

Note: If you are asking specifically about tables with the ability to add new entries in the data plane, without the control plane software having to add them, and rate-limiting how quickly those additions may occur per ENI in DASH, or something similar to that, then I have not thought in detail yet whether P4 meters are a good fit for that use case, but it seems like they might be. Happy to discuss it more if that is your use case.

[KrisNey-MSFT]

@marian-pritsak - question for this Issue:

We have 2 types of meters

1. Direct
2. indirect/index based
   Which one needs to be supported?

[mariobaldi]

I think we need first of all to implement connection rate limiting and then we’ll have the answer. Right now, at least to me, it is not completely clear how we’ll do it and which type of meters we’ll need.

…

--- Mario From: KrisNey-MSFT ***@***.*** Sent: Wednesday, April 13, 2022 1:07 PM To: Azure/DASH ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [Azure/DASH] Enforce xn rate limit per ENI (Issue #90) @marian-pritsak <https://github.com/marian-pritsak> - question for this Issue: We have 2 types of meters 1. Direct 2. indirect/index based Which one needs to be supported? — Reply to this email directly, view it on GitHub <#90 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AICNJE6IQ2WG7TJZS2A4JWLVE35MJANCNFSM5SCP4XKQ> . You are receiving this because you are subscribed to this thread. <https://github.com/notifications/beacon/AICNJE3PIXGI5QZM42SQC43VE35MJA5CNFSM5SCP4XK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIF3ISPQ.gif> Message ID: ***@***.*** ***@***.***> >

[KrisNey-MSFT]

@keshugupta1 is working w/ @sureshrghub
@marian-pritsak to help answer connection tracking code - bmv2 and p4 have some support for metering. Can we implement rate limiting in p4? Per Andy, a red/green/yellow the p4 can decide what to do with the packet. Common use in on byte-length of packet, however Andy to check w/respect to rate. You could have an index by integer meter (calculated by the ENI). Andy and P4.org know a lot about packet metering. It's possible meters were not intended for rate limiting. @mariobaldi is working on connection tracking, he may be helpful as well.

Further discussion re: how to handle external to P4.

Short term plan to have a stub, long-term plan to support in Simulator

[keshugupta1]

'How many CPS per ENI' is the functional requirement. We could be very literal in the behavioral model (@marianp)

@jfingerh Do you see any issue of implementing connection rate-limiting by meter?
No solution for this: need to build into the add-on-miss table?
If we do a meter before the add-on-miss, there may be inaccuracies in rate limiting. Only update token counts if you add an entry successfully.

I have few question regarding implementing connection rate-limiting by doing metering incase of connection-table miss
@marian-pritsak @mariobaldi ,

1. Is the flow addition is inline with each packet. I means two consecutive packet of same new flow will always result in miss for first packet and hit for second packet.
2. Can we assume the target implementation will have sufficient meters i.e one metering index per ENI
   which I think will have bearing on selecting direct meter vs indirect meter.
   Is there a possibility of pointing rate-limiting of two ENI interface to same meter index?

[KrisNey-MSFT]

If we use P4 meters, the problem is - a meter update in P4 (token bucket that is replenished at a rate - burst size minus # of tokens for example). Tokens are always taken away, but later in the add-on-miss they key may be in the table, no reason to create the entry in the table? Jai indicates that we do not usually have meters on the Ingress side of the pipeline. Placing a meter on a learning policy needs to be thought through.

Suppose 1 of every packet is one of 64 ENIs, do a stateless lookup for ENI, somewhere in dataplane there is a configured maximum rate of addition to the table (ex: 1000 per second, etc...). Each of these packets might already have an entry in the table. We would just want a 'hit', not perform an 'add'.

Hari indicates there are so many other parameters that can be used. Is there a known use case? This is the same as related to connection-tracking for VMs. We want fairness (upon maximum space). An upper boundary? An add-on-miss will need to carry another field in the table, that is set by a different/properties table. (rate, allocated memory, replacement policy for example). How would we link these tables together? You could argue that profiles are not part of the P4 pipeline, but they kind of are.

Alan Lo had mentioned LRU. Do we want this treated like a cache? Will we have an eviction policy?

@jfingerh may be able to summarize the output of the P4 meeting where this was discussed.

[keshugupta1]

I didn't quite get the metering limitation here? In which way the replenishing mechanism in P4 metering would be an issue for XN rate limiting? Not sure if it was discussed in DASH weekly meeting as I was on leave almost this whole week so might have missed the details here.

I have done the initial implementation of XN rate-limiting using a direct meter and raised the below pull request. Please see, though it may not be so relevant If P4 meters is not suitable for connection-rate-limiting implementation.
#111

[KrisNey-MSFT]

@marian-pritsak will need a bit more time to review. In the P4 WG the requirements were defined. Can existing mechanisms like meters be used (see above)? Kristina to check for P4 meeting minutes and share them here.