Synchronizing NetworkX and Joblib configurations in nx-parallel

[Schefflera-Arboricola]

Background:

PR #75 adds implementation to make nx-parallel compatible with the networkx's configuration system, offered for networkx backends, and documents how to configure nx-parallel using joblib’s parallel_config.

PR #68 was an attempt to create a unified configuration system in nx-parallel by wrapping the joblib.Parallel()(inside nx-parallel algorithms) within a with joblib.parallel_config(configs) context, here configs are extracted from nx.config.backends.parallel. This approach made NetworkX’s config closely mirror joblib’s, giving the appearance of synchronization between the two systems. However, in the last meeting, Dan mentioned that this approach complicates things.

Requirement:

We need a robust layer in nx-parallel which updates NetworkX’s config as soon as any of joblib’s config parameters are updated, and vice versa. The goal is to ensure that when a user updates a configuration in either of the systems, the other system is instantaneously updated by this layer/interface in nx-parallel to maintain consistency.

Challenges:

1. Order of Updates:

   • As an nx-parallel developer, how can I determine which config system (NetworkX or joblib) the user has updated and in what order?
   • For example, consider the following:

     joblib.parallel_config(n_jobs=5)
     nx.config.backends.parallel.n_jobs = 8
     
     nx.config.backends.parallel.n_jobs = 8
     joblib.parallel_config(n_jobs=5)

     • It’s simple to extract config stored in networkx and in joblib’s parallel_config. So, in both cases, we will extract n_jobs = 8 from NetworkX and n_jobs = 5 from joblib. However, the expected behavior is different: in the first case, n_jobs should be 8 at the end, and in the second case, it should be 5. How can I detect the order of updates to apply the correct configuration?

2. Keeping Context Managers in Sync:

   • Is there a way in Python to determine the context manager we are currently in and the order in which multiple context managers are applied?
   • Consider the following:

     nx.config.backends.parallel.n_jobs = 6 
     joblib.parallel_config(verbose=50) 
     G = nx.complete_graph(5) 
     nx.square_clustering(G, backend="parallel")  # n_jobs = 6, verbose = 50 
     
     with nx.config.backends.parallel(n_jobs=8): 
         nx.square_clustering(G, backend="parallel")  # n_jobs = 8, verbose = 50 
         with joblib.parallel_config(verbose=10): 
             nx.square_clustering(G, backend="parallel")  # n_jobs = 8, verbose = 10

   • How can we ensure that the configurations within nested context managers are correctly synchronized and applied in the expected order?

3. Updating Configurations Without Modifying External Libraries?:

   • Is it even possible to update both NetworkX’s and joblib’s configurations to keep them in sync without requiring any code changes in either of the libraries?

Thank you :)

[ANJAN672]

I just finished reproducing this! Basically, if I tell Joblib to use 4 workers, but the NetworkX config happens to be set to 2, nx-parallel ignores my manual request and uses 2 instead.

The Problem: Inside the code, there is a 'Gatekeeper' (the _configure_if_nx_active decorator). Right now, it is too bossy. It always forces the NetworkX settings on top of whatever the user already set in Joblib. Because of this, it 'blinds' the system to what I actually want.

How we can fix this: We can fix this permanently by making that 'Gatekeeper' smarter. Instead of just blindly forcing the NetworkX config, we should make it check: 'Has the user already set a custom Joblib config?'

If they have, we should respect it. If they haven't, only then should we use the NetworkX settings. Even better, we can link the two systems so that if I change one, the other updates automatically. This way, '4 means 4' no matter which system I use!

@Schefflera-Arboricola , Will Raise a PR if this sounds good!?

Its fixed now !!

FIX: Synchronize NetworkX and Joblib configs by respecting active contexts.

Problem:
_configure_if_nx_active blindly overrode any existing Joblib context with nx.config settings.

Solution: Updated the decorator and get_n_jobs() to peek at get_active_backend(). If an outer Joblib context is already setting n_jobs, we now respect it instead of overriding it.

Verified by:
repro_issue_76.py ( Test Script made and verified this fix )

[ANJAN672]

@Schefflera-Arboricola , Please can u review the PR #154

Note :
Modified decorators.py to peek at Joblib's active context before overriding
Modified chunk.py to prioritize Joblib's active value over NX config

!! Known Limitation !!
When NX context is nested inside Joblib context, Joblib still wins. Full sync requires upstream changes to both libraries.
To fix this permanently we need the timestamp logs from upstream.

[Schefflera-Arboricola]

Upon revisiting this after a while-- I think synchronizing the 2 config systems/context manager (i.e. nx's and joblib's) won't make a very nice user API-- For example:

with joblib.parallel_config(n_jobs=5):
    with nx.config.backends.parallel(n_jobs=8): 
        G = nx.complete_graph(5) 
        nx.square_clustering(G, backend="parallel") # will use networkx's n_jobs --> makes sense!
        joblib.Parallel()(delayed(sqrt)(i ** 2) for i in range(5)) # --> should this use n_jobs from joblib's context manager or networkx's context manager?

... above in the last line it's not obvious if n_jobs from nx.config.backends.parallel would be used or n_jobs from joblib.parallel_config would be used

• Why it would make sense that n_jobs from nx.config.backends.parallel might be used? : because both context managers are in sync therefore the inner context's n_jobs should be considered
• Why it would make sense that n_jobs from joblib.parallel_config might be used? : because joblib.Parallel() is a joblib command, not a networkx or nx-parallel command -- so networkx's config manager should not influence this.

If I remove the networkx code from inside the context, then it is even more confusing from a user API standpoint (bad code readability) :

with joblib.parallel_config(n_jobs=5):
    with nx.config.backends.parallel(n_jobs=8): 
        joblib.Parallel()(delayed(sqrt)(i ** 2) for i in range(5)) # n_jobs=8 --> if both context managers were in sync

And if I don't specify n_jobs inside nx.config.backends.parallel, it might even seem like as if the code is behaving unexpectedly:

with joblib.parallel_config(n_jobs=5):
    with nx.config.backends.parallel(): 
        joblib.Parallel()(delayed(sqrt)(i ** 2) for i in range(5)) # n_jobs=8 --> default for nx-parallel ; but by reading the code it gives a sense that n_jobs is 5

Also implementing syncing the 2 context manager classes (nx's and joblib's) would make the current codebase quite complex as they are not structurally same. Also from Zen of Python -- "Explicit is better than implicit" and "Simple is better than complex". So, I think the current active flag variable approach is clear and simple -- although the confusion mentioned in the above code snippet still exists-- but that would go away if we use the joblib's context manager by default-- but then n_jobs would not be -1 by default-- and we changed the default n_jobs to -1 (and the default context manager to networkx) so that a nx-parallel user could just add backend="parallel" and utilise all their CPU cores without needing to set any configs and stuff.. which i think is a reasonable trade-off..

So, I think we can close this as not planned -- LMKWYT @ANJAN672 @dschult

EDIT:

-- although the confusion mentioned in the above code snippet still exists ... a reasonable trade-off..

I take this back-- i don't think changing the default config system would avoid the confusion of "if the n_jobs is 8 or 5" but it would be worth mentioning somewhere in the docs that nx.config only controls the configs for networkx or nx-parallel function calls whereas joblib's context manager controls both joblib's commands and (when active=False i.e. joblib's context is active) nx and nx-parallel commands.

[ANJAN672]

Thanks for the detailed writeup @Schefflera-Arboricola! Your explanation makes sense - implicit syncing would create more confusion than it solves. The active=False approach is explicit and predictable.

Appreciate the learning opportunity! 🙌🙌