Skip to content

ATLAS-4988: BusinessMetadata with attribute of data type Array takes time to Delete. #490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
UmeshPatil-1 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

ATLAS-4988: BusinessMetadata with attribute of data type Array takes time to Delete. #490

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fbf8033
ATLAS_4988: Resolve slow deletion issue of Business Metadata with arr…
UmeshPatil-1 Dec 27, 2025
5bf649f
ATLAS-4988 : Replaced manual iteration with graph query, added error …
UmeshPatil-1 Dec 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 70 additions & 17 deletions ...main/java/org/apache/atlas/repository/store/graph/v2/AtlasBusinessMetadataDefStoreV2.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,13 @@
import org.apache.atlas.model.TypeCategory;
import org.apache.atlas.model.discovery.AtlasSearchResult;
import org.apache.atlas.model.discovery.SearchParameters;
import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.model.typedef.AtlasBaseTypeDef;
import org.apache.atlas.model.typedef.AtlasBusinessMetadataDef;
import org.apache.atlas.model.typedef.AtlasStructDef;
import org.apache.atlas.repository.Constants;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.repository.graphdb.AtlasGraphQuery;
import org.apache.atlas.repository.graphdb.AtlasVertex;
import org.apache.atlas.type.AtlasBusinessMetadataType;
import org.apache.atlas.type.AtlasStructType;
Expand Down Expand Up @@ -58,6 +61,9 @@ public class AtlasBusinessMetadataDefStoreV2 extends AtlasAbstractDefStoreV2<Atl

private final EntityDiscoveryService entityDiscoveryService;

@Inject
private AtlasGraph graph;

@Inject
public AtlasBusinessMetadataDefStoreV2(AtlasTypeDefGraphStoreV2 typeDefStore, AtlasTypeRegistry typeRegistry, EntityDiscoveryService entityDiscoveryService) {
super(typeDefStore, typeRegistry);
Expand Down Expand Up @@ -363,44 +369,91 @@ private AtlasBusinessMetadataDef toBusinessMetadataDef(AtlasVertex vertex) throw
private void checkBusinessMetadataRef(String typeName) throws AtlasBaseException {
AtlasBusinessMetadataDef businessMetadataDef = typeRegistry.getBusinessMetadataDefByName(typeName);

if (businessMetadataDef != null) {
List<AtlasStructDef.AtlasAttributeDef> attributeDefs = businessMetadataDef.getAttributeDefs();
if (businessMetadataDef == null || CollectionUtils.isEmpty(businessMetadataDef.getAttributeDefs())) {
return;
}

for (AtlasStructDef.AtlasAttributeDef attributeDef : attributeDefs) {
String qualifiedName = AtlasStructType.AtlasAttribute.getQualifiedAttributeName(businessMetadataDef, attributeDef.getName());
String vertexPropertyName = AtlasStructType.AtlasAttribute.generateVertexPropertyName(businessMetadataDef, attributeDef, qualifiedName);
Set<String> applicableTypes = AtlasJson.fromJson(attributeDef.getOption(AtlasBusinessMetadataDef.ATTR_OPTION_APPLICABLE_ENTITY_TYPES), Set.class);
for (AtlasStructDef.AtlasAttributeDef attributeDef : businessMetadataDef.getAttributeDefs()) {
Copy link
Contributor

@sheetalshah1007 sheetalshah1007 Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extract the attribute iteration logic in checkBusinessMetadataRef to a private method (e.g., validateAttributeReferences) for better readability.

Copy link
Author

@UmeshPatil-1 UmeshPatil-1 Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

validateAttributeReferences(businessMetadataDef, attributeDef);
}
}

if (CollectionUtils.isNotEmpty(applicableTypes) && isBusinessAttributePresent(vertexPropertyName, applicableTypes)) {
throw new AtlasBaseException(AtlasErrorCode.TYPE_HAS_REFERENCES, typeName);
}
}
private void validateAttributeReferences(AtlasBusinessMetadataDef bmDef, AtlasStructDef.AtlasAttributeDef attributeDef) throws AtlasBaseException {
String applicableTypesStr = attributeDef.getOption(ATTR_OPTION_APPLICABLE_ENTITY_TYPES);
Set<String> applicableTypes = StringUtils.isBlank(applicableTypesStr) ? null : AtlasJson.fromJson(applicableTypesStr, Set.class);

if (CollectionUtils.isEmpty(applicableTypes)) {
return;
}

String qualifiedName = AtlasStructType.AtlasAttribute.getQualifiedAttributeName(bmDef, attributeDef.getName());
String vertexPropertyName = AtlasStructType.AtlasAttribute.generateVertexPropertyName(bmDef, attributeDef, qualifiedName);

boolean isPresent;
long startTime = System.currentTimeMillis();

if (Boolean.TRUE.equals(attributeDef.getIsIndexable())) {
isPresent = isBusinessAttributePresent(vertexPropertyName, applicableTypes);
} else {
isPresent = isBusinessAttributePresentInGraph(vertexPropertyName, applicableTypes);
}

if (LOG.isDebugEnabled()) {
LOG.info("Reference check for attribute {} took {} ms. Found: {}",
attributeDef.getName(), (System.currentTimeMillis() - startTime), isPresent);
}

if (isPresent) {
throw new AtlasBaseException(AtlasErrorCode.TYPE_HAS_REFERENCES, bmDef.getName());
}
}

private boolean isBusinessAttributePresent(String attrName, Set<String> applicableTypes) throws AtlasBaseException {
SearchParameters.FilterCriteria criteria = new SearchParameters.FilterCriteria();

criteria.setAttributeName(attrName);
criteria.setOperator(SearchParameters.Operator.NOT_EMPTY);
criteria.setOperator(SearchParameters.Operator.NOT_NULL);

SearchParameters.FilterCriteria entityFilters = new SearchParameters.FilterCriteria();

entityFilters.setCondition(SearchParameters.FilterCriteria.Condition.OR);
entityFilters.setCriterion(Collections.singletonList(criteria));

SearchParameters searchParameters = new SearchParameters();

searchParameters.setTypeName(String.join(SearchContext.TYPENAME_DELIMITER, applicableTypes));
searchParameters.setExcludeDeletedEntities(true);
searchParameters.setIncludeSubClassifications(false);
searchParameters.setEntityFilters(entityFilters);
searchParameters.setAttributes(Collections.singleton(attrName));
searchParameters.setOffset(0);
searchParameters.setAttributes(Collections.emptySet());
searchParameters.setLimit(1);

AtlasSearchResult atlasSearchResult = entityDiscoveryService.searchWithParameters(searchParameters);
return atlasSearchResult != null && CollectionUtils.isNotEmpty(atlasSearchResult.getEntities());
}

private boolean isBusinessAttributePresentInGraph(String vertexPropertyName, Set<String> applicableTypes) {
Copy link
Contributor

@sheetalshah1007 sheetalshah1007 Dec 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1. Replace manual iteration with Graph Query for better performance:
The method isBusinessAttributePresentInGraph manually iterates over all vertices of each applicable type using graph.getVertices(Constants.ENTITY_TYPE_PROPERTY_KEY, typeName), then checks each vertex for the property and state. This can be slow for large graphs, as it loads and inspects every vertex in memory. This needs to be improved

2. Add error handling and logging:
No exception handling in the graph operations.
If the graph is unavailable or queries fail, it could lead to unhandled exceptions. Wrap the query in a try-catch and log warnings/errors.
Also, add optional performance logging to monitor query times in production.

3. Add JUnit tests for the new method covering:
-Non-indexable attributes with/without assignments.
-Inactive entities (should not count).

if (graph == null || CollectionUtils.isEmpty(applicableTypes)) {
return false;
}

try {
for (String typeName : applicableTypes) {

return CollectionUtils.isNotEmpty(atlasSearchResult.getEntities());
Iterable<AtlasVertex> vertices = graph.query()
.has(Constants.ENTITY_TYPE_PROPERTY_KEY, typeName)
Copy link
Contributor

@mneethiraj mneethiraj Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary to add ENTITY_TYPE_PROPERTY_KEY filter here? If yes, note that all subTypes of types inapplicableTypes should be checked as well; consider a business attribute associated with a type like DataSet.

Copy link
Author

@UmeshPatil-1 UmeshPatil-1 Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct. The ENTITY_TYPE_PROPERTY_KEY filter is not necessary and will remove these.

.has(vertexPropertyName, AtlasGraphQuery.ComparisionOperator.NOT_EQUAL, (Object) null)
.has(Constants.STATE_PROPERTY_KEY, AtlasEntity.Status.ACTIVE.name())
Copy link
Contributor

@mneethiraj mneethiraj Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary to check only for ACTIVE entities? The business metadata can't be deleted even when a deleted entity has reference to it, right?

Copy link
Author

@UmeshPatil-1 UmeshPatil-1 Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank's for pointing these out, To ensure full data integrity, we should block the deletion of Business Metadata if any entity—Active or Deleted—still holds a reference to it.

.vertices();

if (vertices != null && vertices.iterator().hasNext()) {
if (LOG.isDebugEnabled()) {
LOG.info("Found at least one reference for {} on type {}", vertexPropertyName, typeName);
}
return true;
}
}
} catch (Exception e) {
LOG.error("Error occurred while querying graph for references of property: {}", vertexPropertyName, e);
return true;
}
return false;
}
}