.git/modules/<modulename>/info/refs not always present

[ottomata]

I believe that for dumb transports (http?), this causes submodule checkouts to fail on targets. This can be fixed by running

cd .git/modules/<modulename> && git update-server-info

whenever git-deploy start is called.

I see that the trigger-submodule-update might be an attempt to do this, but I'm not sure how open('.git', 'r') will ever succeed, since .git is a directory. Doing so for me gives IOError: [Errno 21] Is a directory: '.git'.

Should this util be fixed, and should it also always be called for every submodule upon git-deploy start?

[ottomata]

Ah, I see that this command is intended to read /.git, which contains a path to the module inside of the parent .git directory. Ignore the part about 'fixing' trigger-submodule-update then.

[bd808]

My git fu for submodules is weak. Would git submodule foreach update-server-info be the right thing to add in general?

[ryan-lane]

That doesn't work, because when it runs it's not actually in the git directory of the submodule. The actual directory is pointed at with the .git file, so you need to follow the link and run the command in that location. That's what the utility is doing. It's pretty annoying, overall.

This would be easier if we didn't need to use non-bare repos, but for Wikimedia's use-case, that's a whole other can of worms.

[bd808]

This would be easier if we didn't need to use non-bare repos, but for Wikimedia's use-case, that's a whole other can of worms.

I'm not exactly sure why we couldn't have bare repos on the deploy servers. If it's because we need to have a staging area to control the current HEAD it seems like we could add a bare mirror that is managed with triggers to be served up by apache. More complexity, yes, but if it helps mitigate these bootstrapping issues it might be worthwhile.

[ottomata]

Can we just run:

git submodule foreach --recursive 'cd $(sed "s/gitdir: //" .git) && git update-server-info'

whenever the deployment server inits its repos?

I've submitted a change for WMF's version here:

https://gerrit.wikimedia.org/r/#/c/126846

[ryan-lane]

If you've verified that works, I'm totally happy with it. It needs to be run every time before a deployment occurs and after any changes to the repo, btw.

[ottomata]

Yeah was thinking that too. I had only tested with checking out a previous submodule revision and deploying. You are right, when a new SHA comes in that the target doesn't have, this has to be run.

Where do we run it? git deploy start? (If so, where is that?). Perhaps at the start of sync_all?

[ottomata]

Ah no, sorry, sync_all is a minion command. This needs to be run on the deploy server. Hm.

[ottomata]

Should this be done in a hook? Either on the parent repo with a git submodule foreach command, or in each submodule gitdir's hook directory, e.g. .git/modules/<module>/hooks? If so, which hook? post-checkout might work, unless someone patches a submodule locally on the deploy server. In that case, maybe both post-checkout and post-commit? Really not sure.

[ottomata]

Ok, I updated the change in WMF operations/puppet. It now installs post-checkout and post-commit hooks to run this command. It works! But perhaps it is not the best way to do this? Not sure.

[ryan-lane]

I was actually looking for a way to do this via hooks (it's definitely better). The only difficult thing about the hooks (#14) is that we'd need an automated way of getting those hooks in place. A trigger command that will do so reliably would be nice, maybe via setup (#2).

[ottomata]

This is sort of what I'm doing then in my operations/puppet commit, ja?

https://gerrit.wikimedia.org/r/#/c/126846/

[ryan-lane]

Hm. Yeah, I do wonder if it's best to have that in trebuchet, or in trigger. It seems like the kind of thing that trigger should be doing. Either way, that looks like it solves the issue.