-
Notifications
You must be signed in to change notification settings - Fork 86
Masterslave algorithm #234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 2 commits
07c5c32
bbc8731
5bc5038
d3bf69e
1ff8606
027ce30
be724b2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| //Master slave algorithm | ||
| //Each process initially assumes it's the master process | ||
| //Master process is determined by oldest process start time | ||
| //Connect to other slave processes and retrieve details to determine which process is master | ||
|
|
||
| //run lines | ||
| //q torq.q -load masterslave.q -proctype lb -procname lb1 -p 6100 -debug -parentproctype wdb | ||
| //q torq.q -load masterslave.q -proctype lb -procname lb2 -p 6101 -debug -parentproctype wdb | ||
|
|
||
|
|
||
| // the list of processes to connect to | ||
| .servers.CONNECTIONS:`lb | ||
|
|
||
| // custom function, this is invoked when a new outbound connection is created | ||
| // to be customised to invoke negotiation of processes | ||
| .servers.connectcustom:{.lg.o[`connect;"found new connection"]; show x} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok so need to implement this. When it receives a new connection from the discovery service or on a retry, it needs to see if there are any procs of the same proctype that it needs to communicate with to find out who is master. This definition also needs to include any prior definitions (i.e. to not override previous definitions) .servers.connectcustom:{[f;x] ... your code in here ...; f@x }@[value;`.servers.connectcustom;{{}}] This should also only be done in the init function Use this same approach for .z.pc (capture the previous definition, add your own) |
||
|
|
||
| // create connections | ||
| .servers.startup[] | ||
|
|
||
| \d .masterslave | ||
|
|
||
| //table scehma of connected lb processes with ismaster status | ||
| statustable:([handle:`int$()] procname:`symbol$();starttimeUTC:`timestamp$();ismaster:`boolean$()) | ||
|
|
||
| //store own process start timestamp | ||
| start:.z.p | ||
|
|
||
| //set details dict with own details | ||
| details:{`procname`starttimeUTC`ismaster!(.proc.procname;.masterslave.start;1b)} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think probably for now we can drop the ismaster value |
||
|
|
||
| //get details of procname provided | ||
| getdetails:{[processname] (first exec w from .servers.SERVERS where procname like processname) ".masterslave.details[]"} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When doing IPC can you avoid strings here, i.e. use I think this should do something on the remote side too i.e. a getdetails call from A->B causes both sides of the negotation to know about each other. So when B receives this call, it should use the details passed in from A to update it's own status |
||
|
|
||
| //update .masterslave.statustable with other proc details and update ismaster col to determine which process is master | ||
| addmember:{[processname] | ||
| update ismaster:0b from (`.masterslave.statustable upsert .masterslave.getdetails[processname],(enlist `handle)!(enlist first exec w from .servers.SERVERS where procname like processname)) where starttimeUTC<>min starttimeUTC} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you split this out across multilines / separate statements. Kinda hard to read |
||
|
|
||
| //is the process itself the master | ||
| ammaster:{exec ismaster from .masterslave.statustable where handle=0} | ||
|
|
||
| //find which process is the master | ||
| findmaster:{first exec handle from .masterslave.statustable where ismaster=1b} | ||
|
|
||
| //wrapper func for finding master | ||
| checkifmaster:{ | ||
| .servers.startup[]; | ||
| .masterslave.addmember each string exec procname from .servers.SERVERS where proctype like "lb", not null w | ||
| } | ||
|
|
||
| //add pc override here | ||
| //remove dropped connection from statustable | ||
| //tell other alive processes to renegotiate who the master is | ||
|
|
||
|
|
||
| \d . | ||
|
|
||
| //populate statutable and find which process is master | ||
| .masterslave.checkifmaster[] | ||
|
|
||
|
|
||
| //.z.pc call | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should just be appended to the list of connections, and do it within an init[] function. Also just use .proc.proctype rather than `lb