Skip to content

spawn io worker for drive subsystem#4567

Open
invisig0th wants to merge 17 commits intomasterfrom
visi-io-spawn
Open

spawn io worker for drive subsystem#4567
invisig0th wants to merge 17 commits intomasterfrom
visi-io-spawn

Conversation

@invisig0th
Copy link
Contributor

No description provided.

@codecov
Copy link

codecov bot commented Nov 6, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.73%. Comparing base (f3f571f) to head (d3abcd0).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4567      +/-   ##
==========================================
+ Coverage   97.72%   97.73%   +0.01%     
==========================================
  Files         298      299       +1     
  Lines       62352    62453     +101     
==========================================
+ Hits        60934    61040     +106     
+ Misses       1418     1413       -5     
Flag Coverage Δ
linux 97.67% <100.00%> (+0.01%) ⬆️
linux_replay 93.40% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@invisig0th invisig0th changed the base branch from master to synapse-3xx January 16, 2026 12:41
@OCBender OCBender changed the base branch from synapse-3xx to master February 19, 2026 15:37
@OCBender OCBender changed the base branch from master to synapse-3xx February 19, 2026 15:38
@OCBender OCBender changed the base branch from synapse-3xx to master February 19, 2026 15:55
@OCBender OCBender changed the title WIP: prototype of spawn io worker spawn io worker for drive subsystem Feb 26, 2026
@OCBender OCBender marked this pull request as ready for review February 26, 2026 15:37
dmon.share('item', item)

# bind last so we're ready to go...
await dmon.listen(f'unix://{sockpath}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the event of a failure to open this listening socket, how does the application respond?

In the base cell and the default listening socket, we warn about a failure to open the default listener, but that isn't quite so fatal since regular telepath / http api listeners are going to work. In this case, it feels much more fatal.

class FileDrive(Drive, s_spawner.SpawnerMixin):

async def __anit__(self, path):
slab = await s_lmdbslab.Slab.anit(path)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no fini handler registered for this Slab that I can find?


async def __anit__(self, path):
slab = await s_lmdbslab.Slab.anit(path)
return await Drive.__anit__(self, slab, 'celldrive')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think anit would normally have anything to return?


base.schedCoro(s_process.spawn((_ioWorkProc, (todo, sockpath), {})))

await s_link.unixwait(sockpath)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that if the schedCoro has an error + infinite loop of unixwait means that it is possible to have a situation where we have blindly blocked execution of the application?

path = s_common.gendir(self.dirn, 'slabs', 'drive.lmdb')
sockpath = s_common.genpath(self.sockdirn, 'drive')

if len(sockpath) > s_const.UNIX_PATH_MAX:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it feels like this check should be in the spawner.py to prevent the multiprocess target from getting bad input?

MAX_FIELD_SIZE = kibibyte * 64

# Socket constants
UNIX_PATH_MAX = 107
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BSD derivatives ( which I believe there may be at least one Cortex deployment on in the wild ) have a maxlenth of 104 characters, vs the linux 108. We probably need to be conservative with this value and set it to 103.

info = {'path': path, 'unix': True}
return await Link.anit(reader, writer, info=info)

async def unixwait(path):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be able to write a test for this in test_lib_link that does the following:

  1. create a base
  2. create the path we're going to listen on
  3. spawn a schedcoro waiting for the listening path which blocks on unixwait
  4. startup the unix listener
  5. waits for the schedCoro task a limited amount of time.

# The scope data set in the task is not present outside of it.
self.none(s_scope.get('hehe'))

async def test_base_spawner_fini(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be moved to test_lib_spawner.py ? Its no longer a part of the Base object.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants