fix: refresh sagemaker-core service model and regenerate SDK#5800
Closed
lucasjia-aws wants to merge 9 commits into
Closed
fix: refresh sagemaker-core service model and regenerate SDK#5800lucasjia-aws wants to merge 9 commits into
lucasjia-aws wants to merge 9 commits into
Conversation
Replace the internal service-2.json (502 operations, including 122 internal-only operations) with the public botocore service model (396 operations). Regenerate resources.py, shapes.py, resource_plan.csv, and related files from the public model to remove all leaked internal APIs, fields, and shapes. This addresses the security issue where internal-only API operations (e.g.CreateTrainingJobInternal), internal fields on public APIs (e.g.UpstreamPlatformConfig, FasCredentials on CreateTrainingJob), and the internalservice model file itself were publicly exposed on GitHub and PyPI sincesagemaker-core >= 2.1.0 / >= 1.0.70 (Dec 3, 2025). Ticket: P422086537 X-AI-Prompt: remove internal API leak from sagemaker-core X-AI-Tool: kiro-cli
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
…aker-core Fix upstream package breakages after removing internal-only fields/shapes from sagemaker-core public SDK (PR aws#5800). Changes: - sagemaker-train: remove enable_ffm param from _create_input_channels() in finetune_utils.py (internal-only field, never used at runtime) - sagemaker-mlops: remove dead import of AddOnlineStoreReplicaAction from feature_group_manager.py (shape only existed in internal model) - sagemaker-core/tools: add PIPE_VAR_OVERRIDES for ResourceConfig to support ParameterInteger as instance_count/volume_size/keep_alive - sagemaker-core/tools: add REQUIRED_TO_OPTIONAL_OVERRIDES for ModelPackage and ModelPackageSecurityConfig required fields - sagemaker-core/tools: add IntPipeVar to PYTHON_TYPES_TO_BASIC_JSON_TYPES and config schema generation to fix codegen KeyError - Regenerate shapes.py and resources.py with updated codegen X-AI-Prompt: fix CI failures from internal API removal in upstream packages X-AI-Tool: kiro-cli
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
Remove extra whitespace on blank line 30478 that caused Python to misparse the subsequent code block, breaking all imports of sagemaker.core.resources. X-AI-Prompt: fix IndentationError in resources.py X-AI-Tool: kiro-cli
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
…lict The public botocore service model contains a StartSession API, which caused codegen to generate an empty `class Session(Base)` Pydantic model in resources.py. This shadowed the `from boto3.session import Session` import, making Pydantic reject the actual Session objects passed to TrainingJob.create() and other resource methods. Added "Session" to EXCLUDED_RESOURCES in resources_extractor.py so it is no longer discovered as a resource. Re-ran codegen to regenerate downstream files. X-AI-Prompt: Exclude Session from codegen resource discovery X-AI-Tool: kiro-cli
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
lucasjia-aws
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
nargokul
reviewed
Apr 29, 2026
| }, | ||
| # Override default instance type; the model's default | ||
| # (ml.p3.2xlarge) is deprecated. | ||
| "compute": Compute(instance_type="ml.g5.xlarge"), |
Contributor
There was a problem hiding this comment.
Why are we making this change ?
Collaborator
Author
There was a problem hiding this comment.
This is a by-the-way integ-test fixing not corresponding to this issue. Confirmed with Zhaoqi.
nargokul
reviewed
Apr 29, 2026
| input_mode=input_mode, | ||
| enable_ffm=enable_ffm, | ||
| ) | ||
| input_mode=input_mode |
Contributor
There was a problem hiding this comment.
Why are we removing this ?
Collaborator
Author
There was a problem hiding this comment.
After replacing service-2.json, enable_ffm no longer exist in shapes.py, need to remove, also it is a dead call actually, safe to remove.
lucasjia-aws
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
The
service-2.jsonbundled in sagemaker-core had drifted from the public botocore service model. This caused the generated SDK to contain stale API definitions and shape fields that no longer match the public SageMaker API surface.Fix
Replaced
service-2.jsonwith the latest public botocore service model, then re-ran codegen (python3 -m sagemaker.core.tools.codegen) to regenerate all downstream files.Files regenerated by codegen:
src/sagemaker/core/resources.pyresource_plan.csvsrc/sagemaker/core/config_schema.pysrc/sagemaker/core/utils/code_injection/shape_dag.pysrc/sagemaker/core/tools/api_coverage.jsonAdditional changes:
src/sagemaker/core/shapes/shapes.pyshapes.pytoshapes/shapes.py. Python resolvesfrom sagemaker.core.shapes import *to theshapes/package, so this copy must stay in sync.src/sagemaker/core/resources.py(L30878)TrainingJob.wait()method had anifblock indented at 16 spaces instead of 8, causingSyntaxError. Manually dedented to the correct level..pyand test files)black -l 100 .as the final codegen step. No functional changes.Follow-up Commits
These commits fix CI failures and runtime issues discovered after the initial model refresh.
f252f4dasagemaker-train,sagemaker-mlops,sagemaker-corecodegen tools,shapes.py,resources.pyenable_ffm,AddOnlineStoreReplicaAction). AddedIntPipeVarsupport andREQUIRED_TO_OPTIONAL_OVERRIDESto codegen, then regenerated.ae707dcsagemaker-core/.../resources.pySession = Boto3Sessionalias to fixNameError.30c4c0csagemaker-core/.../shapes/shapes.pyshapes/shapes.pypackage copy with updated top-levelshapes.py.7aac55csagemaker-mlops/.../test_iceberg_properties.pydescription=param withfeature_additions=[]in test calls.