Improved index resolution and revised index authorization

[nibix]

Description

This introduces all the necessary changes for the improved index resolution and revised index authorization described in #5814.

This has several advantages:

• The old index authorization and index resolution system had many inconsistencies and oddities. These are fixed by the new approach.
• The responsibility for index resolution is moved from the security plugin to the individual transport actions. This makes the whole authorization process much more robust and the code less messy and fragile.
• As each transport action knows its indices it is operating on very well, it can do the resolution in a more efficient manner (often actually no resolution takes place at all, as all indices are already determined as constant values). This reduces the performance overhead induced by the security plugin.
• The security plugin no longer needs to resolve data streams to its individual backing indices to check privileges; this brings further performance improvements.

• Category: Enhancement
• Why these changes are required?
  • Fundamental improvements for index authorization were due
• What is the old behavior before changes and new behavior after changes?
  • By default, no behavioral changes are introduced; if the new index authorization system is enabled in config.yml, quite fundamental behavioral changes take place. These are described in [RFC] Revised index action authorization scheme #5814.

Issues Resolved

#5814

Testing

• extensive integration tests

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

❌ Patch coverage is 86.94539% with 153 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.41%. Comparing base (db9efb6) to head (f0ab722).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...s/actionlevel/nextgen/PrivilegesEvaluatorImpl.java	86.71%	13 Missing and 23 partials ⚠️
...tgen/DashboardsMultitenancySystemIndexHandler.java	88.41%	8 Missing and 11 partials ⚠️
...nsearch/security/privileges/DocumentAllowList.java	52.17%	10 Missing and 1 partial ⚠️
...curity/privileges/PrivilegesEvaluatorResponse.java	73.17%	6 Missing and 5 partials ⚠️
.../actionlevel/RuntimeOptimizedActionPrivileges.java	90.83%	3 Missing and 8 partials ⚠️
...eges/actionlevel/SubjectBasedActionPrivileges.java	85.29%	5 Missing and 5 partials ⚠️
...earch/security/privileges/PrivilegesEvaluator.java	76.31%	5 Missing and 4 partials ⚠️
...vileges/actionlevel/RoleBasedActionPrivileges.java	85.93%	4 Missing and 5 partials ⚠️
...leges/actionlevel/nextgen/ActionConfiguration.java	86.20%	5 Missing and 3 partials ⚠️
...search/security/configuration/DlsFlsValveImpl.java	75.86%	4 Missing and 3 partials ⚠️
... and 11 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5827      +/-   ##
==========================================
+ Coverage   73.98%   74.41%   +0.43%     
==========================================
  Files         441      446       +5     
  Lines       27413    28387     +974     
  Branches     4110     4328     +218     
==========================================
+ Hits        20281    21125     +844     
- Misses       5193     5251      +58     
- Partials     1939     2011      +72     

Files with missing lines	Coverage Δ
.../opensearch/security/OpenSearchSecurityPlugin.java	84.91% <100.00%> (+0.03%)	⬆️
...rity/configuration/SystemIndexSearcherWrapper.java	94.82% <100.00%> (ø)
...ensearch/security/privileges/ActionPrivileges.java	100.00% <100.00%> (ø)
...leges/ClusterStateMetadataDependentPrivileges.java	86.66% <100.00%> (ø)
.../opensearch/security/privileges/CompiledRoles.java	80.00% <100.00%> (+1.42%)	⬆️
...g/opensearch/security/privileges/IndexPattern.java	94.82% <100.00%> (+0.41%)	⬆️
...ch/security/privileges/IndicesRequestResolver.java	100.00% <100.00%> (ø)
...curity/privileges/PrivilegesEvaluationContext.java	97.43% <100.00%> (+4.75%)	⬆️
...ges/actionlevel/legacy/PitPrivilegesEvaluator.java	96.29% <100.00%> (ø)
...es/actionlevel/legacy/PrivilegesEvaluatorImpl.java	85.66% <100.00%> (ø)
... and 29 more

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cwperks]

@nibix apologies for a delay in comments. I plan to look at this in earnest over today and the coming days.

[cwperks]

TY for this change @nibix and introducing a dynamic feature flag through the security configuration.

I think this will greatly enhance the user experience, particularly for clusters where users are given fine-grained permissions over a subset of indices.

[cwperks]

@nibix When a securityconfig update is made will it add an entry for privileges_evaluation_type with the v3 default?

When preparing for OpenSearch v4, what code change would need to take place in the security repo?

[cwperks]

on second thought, what do you think about making this a dynamic cluster setting instead of securityconfig?

I'm a bit concerned about the persistence part of the securityconfig and generally prefer to make things dynamic cluster settings where its easier to change the default from one version to the next without concern about whether its been persisted with the default from previous major version

[nibix]

When a securityconfig update is made will it add an entry for privileges_evaluation_type with the v3 default?

Good question. Need to check that as well, will update later.

When preparing for OpenSearch v4, what code change would need to take place in the security repo?

It depends on the question whether we just want to flip the feature flag default for v4 or whether the old logic shall be completely retired and deleted then.

If just the feature flag default value is flipped, the changes in the main code are limited to flipping the default value. Test code would need to make sure to use explicit feature flag values where it is targeting the old behavior.

If the old logic shall be completely deleted, of course, we would need to delete quite a lot of main code. However, that should be fairly well separated from the new code. Regarding tests, we'd need to delete quite a lot of legacy tests which only cover the old behavior. But we should already have new tests which cover the new behavior, so it should be fine to delete such tests.

[nibix]

I have rebased this and fixed a conflict. With the merged test fixes, this is now IMHO ready for merging. @cwperks @shikharj05 wdyt?

[nibix]

@shikharj05 @DarshitChanpura @derek-ho @RyanL1997 @reta @willyborankin

I kind this is a big PR; is there something I can facilitate the approval? It is already sitting around for a while.

[cwperks]

@nibix can you fix the conflict?

[nibix]

will resolve conflicts as soon as #6037 is merged .. this will introduce further conflicts

[nibix]

@cwperks

Conflicts have been resolved now. That was actually quite a lot of work, it shows the age of the PR.

@shikharj05 @DarshitChanpura @derek-ho @RyanL1997 @reta @willyborankin @finnegancarroll

As said before, if I can do anything to facilitate the review, please let me know!

[cwperks]

Thank you @nibix ! I focused on reviewing the merge conflict resolutions as that is what changed from last approval. The changes LGTM. Its great to see this introduced as the current default behavior can be very confusing for users without all_access. Definitely a step in the direct for where I'd like to get to with security.

[willyborankin]

@nibix yet another conflict