Add concept of global resources which can be shared across applications#1
Add concept of global resources which can be shared across applications#1lceleban-dd wants to merge 8 commits intomasterfrom
Conversation
| @@ -71,9 +82,7 @@ private[akkeeper] final class YarnLocalResourceManager(conf: Configuration, | |||
| } | |||
|
|
|||
| def getExistingLocalResource(dstPath: Path): LocalResource = { | |||
There was a problem hiding this comment.
Both getExistingLocalResource and getExistingResource returns LocalResource. Could you please rename
getExistingLocalResource -> getLocalResourceFromStagingDir and
getExistingResource -> getLocalResource
?
| val updatedUserConfig = args.userConfig.map(addStagingDirToUserConfig(_, baseStagingDir)) | ||
| val updatedUserConfig = args.userConfig | ||
| .map(addStagingDirToUserConfig(_, baseStagingDir)) | ||
| .map(addGlobalResourcesToUserConfig(_, args.globalResources)) |
There was a problem hiding this comment.
I'm not sure why the staging directory configuration is not passed when a user config is missing. It was introduced fairly recently (10 months ago) in a PR named: "Fix usage of the default staging directory in case when a user that submitted application is different from a user that runs the application". I've asked Iaroslav this question.
Since MasterRunner fallbacks to ConfigFactory.load() anyway (https://github.com/ramencloud/akkeeper/blob/master/akkeeper/src/main/scala/akkeeper/master/MasterRunner.scala#L124), I'd suggest always sending the config:
val updatedUserConfig = args.userConfig.orElse(Some(ConfigFactory.empty()))
.map(addStagingDirToUserConfig(_, baseStagingDir))
.map(addGlobalResourcesToUserConfig(_, args.globalResources))
(something along theses lines) or putting global resources into a separate file (launcher config). I'm about to prepare a commit with a suggestion.
There was a problem hiding this comment.
I'd like to propose something along these lines: global_resources...ramencloud:global_resources_proposal
| LocalResource.newInstance( | ||
| ConverterUtils.getYarnUrlFromURI(fs.makeQualified(status.getPath).toUri), | ||
| LocalResourceType.FILE, LocalResourceVisibility.APPLICATION, | ||
| ConverterUtils.getYarnUrlFromURI(fs. makeQualified(status.getPath).toUri), |
2 participants
No description provided.