Skip to content
Open
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions masterslave.q
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
//Master slave algorithm
//Each process initially assumes it's the master process
//Master process is determined by oldest process start time
//Connect to other slave processes and retrieve details to determine which process is master

//run lines
//q torq.q -load masterslave.q -proctype lb -procname lb1 -p 6100 -debug -parentproctype wdb
//q torq.q -load masterslave.q -proctype lb -procname lb2 -p 6101 -debug -parentproctype wdb


// the list of processes to connect to
.servers.CONNECTIONS:`lb
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should just be appended to the list of connections, and do it within an init[] function. Also just use .proc.proctype rather than `lb


// custom function, this is invoked when a new outbound connection is created
// to be customised to invoke negotiation of processes
.servers.connectcustom:{.lg.o[`connect;"found new connection"]; show x}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok so need to implement this. When it receives a new connection from the discovery service or on a retry, it needs to see if there are any procs of the same proctype that it needs to communicate with to find out who is master.

This definition also needs to include any prior definitions (i.e. to not override previous definitions)

.servers.connectcustom:{[f;x] ... your code in here ...; f@x }@[value;`.servers.connectcustom;{{}}]

This should also only be done in the init function

Use this same approach for .z.pc (capture the previous definition, add your own)


// create connections
.servers.startup[]

\d .masterslave

//table scehma of connected lb processes with ismaster status
statustable:([handle:`int$()] procname:`symbol$();starttimeUTC:`timestamp$();ismaster:`boolean$())

//store own process start timestamp
start:.z.p

//set details dict with own details
details:{`procname`starttimeUTC`ismaster!(.proc.procname;.masterslave.start;1b)}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think probably for now we can drop the ismaster value
We can just pass the name and the start time, and then both sides of the negotiation can work out which is the master (which might be neither of them)


//get details of procname provided
getdetails:{[processname] (first exec w from .servers.SERVERS where procname like processname) ".masterslave.details[]"}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When doing IPC can you avoid strings here, i.e. use
(.masterslave.details;)
This should be an async request, but we'll do that later.

I think this should do something on the remote side too i.e. a getdetails call from A->B causes both sides of the negotation to know about each other. So when B receives this call, it should use the details passed in from A to update it's own status


//update .masterslave.statustable with other proc details and update ismaster col to determine which process is master
addmember:{[processname]
update ismaster:0b from (`.masterslave.statustable upsert .masterslave.getdetails[processname],(enlist `handle)!(enlist first exec w from .servers.SERVERS where procname like processname)) where starttimeUTC<>min starttimeUTC}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you split this out across multilines / separate statements. Kinda hard to read


//is the process itself the master
ammaster:{exec ismaster from .masterslave.statustable where handle=0}

//find which process is the master
findmaster:{first exec handle from .masterslave.statustable where ismaster=1b}

//wrapper func for finding master
checkifmaster:{
.servers.startup[];
.masterslave.addmember each string exec procname from .servers.SERVERS where proctype like "lb", not null w
}

//add pc override here
//remove dropped connection from statustable
//tell other alive processes to renegotiate who the master is


\d .

//populate statutable and find which process is master
.masterslave.checkifmaster[]


//.z.pc call