Schedule onto multiple cores in round-robin fashion #91

arihant2math · 2025-12-25T06:24:33Z

Exposes better APIs to collect all tasks/querying tasks across CPUs. Schedules in a round-robin ordering for now. Eventually we'd want work stealing from other cores and a quick way to get the current queue lengths for each core. Also this forces timer interrupts and extends them to non-main cores.

Currently fixes the bug where background tasks force a hang.

Includes improvements to the EEVDF implementation which fix many bugs.

hexagonal-sun · 2025-12-25T09:45:02Z

src/sched/mod.rs

+}
+
+// TODO: arbitrary cap.
+pub static SCHED_STATES: [SyncUnsafeCell<Option<SchedState>>; 128] =


I presume you’ve ditched the per_cpu! macro here since it precludes access to state held by other cores?

I’ve been thinking about this for a while and wondering what the best approach is. I can see two options:

What you’ve done here: i.e., a static array which allows arbitrary access from any other core. The main downside I see is that everything must be locked, since it can be accessed by another core. I see that you’ve used SyncUnsafeCell; in that case, I think functions such as get_sched_state_by_id and with_cpu_sched_state are inherently unsafe, as they permit lock-free access to SchedState across CPUs.

Accessing/modifying CPU state via the CPUMessenger. In this approach, we fire off an IPI to the target CPU, which performs the read/modify locally (lock-free) and returns the result.

I think the second option is a good compromise. Updating or accessing the state of another CPU isn’t done nearly as often as accessing the current CPU’s state. With the first option, we’d need locking for every scheduler state access or update (even something as frequent as retrieving the current task). Given that, my vote would be to try the CPUMessenger approach. I’m happy to give that a go if you’d like.

For an example of this, I've just pushed this branch which pushes tasks onto other CPUs via IPIs with the CPUMessenger. When I run usertest, I'm getting deadlock at the moment:

bash-5.3# ./bin/usertest Running userspace tests ... [ 35.35031303] interrupts::cpu_messenger: Putting task on CPU 1 Testing sync syscall ... OK

I suspect this is the child process that's finished on CPU1, the waker seting the task in CPU0 to be runnable again, but it's stuck in the idle task. I need to implement something such that a Waker, when signalling to another CPU causes the rechedule IPI message.

Yes. I've intentionally left out cross-cpu access beyond task insertion. To my knowledge get_sched_state_by_id et. al. should be safe since I'm never overwriting any item in the array, and the interior mutability of other threads is limited to run_queue, which is protected by a SpinLock.

A better approach could be to Arc the runqueue and just share that.

On the topic of using messengers, how fast are they?

I think the optimal cross-CPU task balancing method is work-stealing rather than work-giving, how would that work with messengers? It seems like you'd need to query for the number of tasks first, then request a steal.

Yes. I've intentionally left out cross-cpu access beyond task insertion. To my knowledge get_sched_state_by_id et. al. should be safe since I'm never overwriting any item in the array, and the interior mutability of other threads is limited to run_queue, which is protected by a SpinLock.

I disagree. I see a code path where the procfs module can call find_task_by_descriptor. That call can originate from any task on any CPU, which means it will very likely result in cross-CPU data access (i.e. accessing the run_queue of another CPU). That code iterates through all active slots and walks the run_queues until it finds the task.

As far as I can see, this is highly unsafe and will lead to data races. SchedState itself is not protected by a mutex, nor are any of its fields.

If we can guarantee that the only core accessing a given piece of data is the CPU that owns it, then we can safely access it lock-free. That is precisely the guarantee provided by per_cpu!: there is no way to access data that does not belong to the current CPU. In this case, however, that guarantee does not hold. You need a different mechanism here—this is exactly the problem the CPUMessenger is trying to solve.

A better approach could be to Arc the runqueue and just share that.

Possibly. I did consider wrapping all of this state in a large Arc, but that effectively means locking everything. Any on-CPU operation would then require taking a lock, which I think introduces too much overhead for the scheduler fast path.

On the topic of using messengers, how fast are they?

IPIs are fast. You can hit additional latency if interrupts are disabled on the target CPU, but Linux uses them extensively in practice. For example, on my machine:

$ cat /proc/interrupts [...] IWI: 103089 111648 51802 50290 3480843 50846 37640 101369 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 53258 64925 39159 37986 143873 61857 38363 49915 Rescheduling interrupts CAL: 1434334 1127025 974039 838055 844695 993721 932218 876858 Function call interrupts TLB: 684503 742799 728184 629270 646185 829325 783279 722283 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 66 65 65 65 65 65 65 65 Machine check polls ERR: 0

These are the various IPIs Linux uses, and as you can see, they are heavily exercised.

I've just pushed some additional code to the branch I mentioned earlier, this resolves the deadlock that I had and allows me to get through nearly all of usertest without any kind of preemption. I'm just using IPIs to wake-up other CPUs when needed.

I think the optimal cross-CPU task balancing method is work-stealing rather than work-giving, how would that work with messengers? It seems like you'd need to query for the number of tasks first, then request a steal.

That’s a fair point, and I agree that work-stealing is generally the better default for schedulers.

With messengers, the model shifts slightly compared to shared-memory designs, but it still maps cleanly. The key idea is that ownership never changes implicitly: the stealing CPU never directly inspects or mutates another CPU’s run queue. Instead, it asks the owning CPU to do the inspection and transfer on its behalf. Concretely, the stealing CPU raises a message to a target CPU with the necessary parameters, and the target CPU decides what (if any) tasks it can push back.

Alright, I think we can worry about the work stealing implementation later, I'll switch what I have currently to use messengers.

src/drivers/interrupts/arm_gic_v3.rs

src/drivers/timer/mod.rs

hexagonal-sun · 2025-12-25T21:50:50Z

I've just tried to run the usertest and it looks like you're getting the same problem as I am on my cpu-messenger-task-migration branch:

Testing rust threads ... OK
[    5.5169154] moss: Kernel panicked at src/arch/arm64/memory/fault.rs:121:17: Page fault defered error, SIGBUS on process

I Need to investigate what's going on there.

hexagonal-sun

Looks good, some minor comments - nothing major. Other than that, happy to merge.

It might be work creating an issue for per-cpu driver initialisation logic for per-CPU devices (the armv8 timer being the primary one). That decouples that driver from the gic code.

hexagonal-sun · 2025-12-26T06:20:22Z

src/sched/mod.rs

+    }
+}
+
+// TODO: arbitrary cap.


Redundant comment now.

hexagonal-sun · 2025-12-26T06:26:56Z

src/sched/mod.rs

+    sched_state.advance_vclock(now_inst);
+
    let next_task = sched_state.find_next_runnable_task();
+    // if previous_task.tid != next_task.tid {


I presume this is useful debug code?

It'd be a shame to remove such code when it's useful for debugging. Perhaps we should make use of log's release_max_level_info feature flags or similar, so that useful code isn't removed.

I hope with a proper procfs we can remove this and rely on top instead.

src/sched/mod.rs

hexagonal-sun · 2025-12-26T06:32:52Z

src/sched/mod.rs

        .activate();

    *init_task.state.lock_save_irq() = TaskState::Running;
+    SCHED_STATE.borrow_mut().running_task = Some(idle_task.clone());


Don't we want init to be the first task executed?

arihant2math · 2025-12-26T09:13:39Z

FYI still debugging some weird crashes with usertest. Also there's a lot of intermittent hanging.

hexagonal-sun · 2025-12-26T09:31:39Z

I found 50e5b0b helped a lot with intermittent hanging. Note that you need the prior commit to the interrupt manager to help prevent a deadlock in the interrupt manager.

hexagonal-sun · 2025-12-26T11:50:06Z

I’m seeing some lock-ups due to the interrupt manager not being able to handle interrupts reentrantly. That should be fixed if you cherry-pick 6ae2375.

I'm also seeing a lot of lock-ups in the allocator:

#0  spinning_top::spinlock::{impl#1}::lock (self=0xffff800001213170 <moss::arch::arm64::memory::HEAP_ALLOCATOR>) at src/spinlock.rs:57
#1  0xffff800000164f94 in lock_api::mutex::Mutex<spinning_top::spinlock::RawSpinlock, linked_list_allocator::Heap>::lock<spinning_top::spinlock::RawSpinlock, linked_list_allocator::Heap> (self=0xffff800001213170 <moss::arch::arm64::memory::HEAP_ALLOCATOR>)
    at /home/matthew/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lock_api-0.4.14/src/mutex.rs:213
#2  0xffff800000166144 in linked_list_allocator::{impl#4}::alloc (self=0xffff800001213170 <moss::arch::arm64::memory::HEAP_ALLOCATOR>, layout=...)
    at src/lib.rs:336
#3  0xffff800000151088 in moss::arch::arm64::memory::_::__rust_alloc (size=24, align=8) at src/arch/arm64/memory/mod.rs:21
#4  0xffff800000119dc8 in alloc::alloc::alloc (layout=...)
    at /home/matthew/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:95
#5  alloc::alloc::Global::alloc_impl (self=0x1, layout=..., zeroed=false)
    at /home/matthew/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:190
#6  0xffff800000119f04 in alloc::alloc::{impl#1}::allocate (layout=...)
    at /home/matthew/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:251
#7  alloc::alloc::exchange_malloc (size=24, align=8)
    at /home/matthew/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:352
#8  0xffff8000000e4790 in alloc::boxed::{impl#0}::new<moss::drivers::interrupts::arm_gic_v3::ArmGicV3InterruptContext> (x=...)
    at /home/matthew/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:265
#9  moss::drivers::interrupts::arm_gic_v3::{impl#5}::read_active_interrupt (self=0xffffb00000000ce0) at src/drivers/interrupts/arm_gic_v3.rs:395
#10 0xffff800000066360 in moss::interrupts::InterruptManager::get_active_handler (self=0xffffb00000000d08) at src/interrupts/mod.rs:152
#11 0xffff800000066140 in moss::interrupts::InterruptManager::handle_interrupt (self=0xffffb00000000d08) at src/interrupts/mod.rs:161
#12 0xffff8000000e0a2c in moss::arch::arm64::exceptions::el1_irq_spx (state=0xffffb8000001a8d0) at src/arch/arm64/exceptions/mod.rs:125
#13 0xffffe000000002f4 in ?? ()

This makes total sense: the spinlock used in the allocator does not disable interrupts. Therefore, when an ISR is running, the heap spinlock may already be held. If an allocation occurs during the ISR, this will deadlock.

hexagonal-sun · 2025-12-26T16:32:15Z

Okay, I've just pushed fixes for the heap allocator and the interrupt manager to this PR which should help. I now get a successful boot to bash pretty much every time which means I think we've fixed interrupt locking issues. I'm now hitting:

[    0.717370] drivers::interrupts::arm_gic_v3: GICv3: CPU interface for core 2 enabled.
[    0.718020] arch::arm64::boot::secondary: CPU 2 online.
[    0.719852] drivers::interrupts::arm_gic_v3: GICv3: Redistributor for core 3 (MPIDR=0x80000003) is awake.
[    0.720558] drivers::interrupts::arm_gic_v3: GICv3: CPU interface for core 3 enabled.
[    0.721818] arch::arm64::boot::secondary: CPU 3 online.
[    0.725946] interrupts::cpu_messenger: CPU 1 recieved ping from CPU 0
bash-5.3# [    1.1196409] moss: Kernel panicked at src/sched/mod.rs:335:13: assertion `left == right` failed
  left: Sleeping
 right: Runnable

~~Which looks like it's a scheduler related issue.~~ This was an incorrect assertion check which I added to the sched code. I've pushed a commit to remove it.

arihant2math · 2025-12-26T22:14:30Z

Seems like I can spawn 3 cat /dev/zero &s, then it starts to slow down. This is fine for now.

However usertest still doesn't run to completion. I think there might be a bug with respect to cross-cpu task completion.

arihant2math · 2025-12-27T02:16:22Z

Also interrupts being scheduled on top of times when the borrow is acquired is turning out to be a problem. (i.e. insert_task is running and then the timer interrupt strikes. This seems to cause hangups a lot.

src/sched/mod.rs

hexagonal-sun · 2025-12-27T23:05:37Z

Fixed a bunch of issues. The thread exit logic was utterly wrong - we were sending a SIGCHLD to the parent for every thread exit; I'm surprised threading code worked prior to this PR!

Now we've introduced proper SMP, the fault handling code didn't handle page-fault collisions on two different CPUs. It treated an AlreadyMapped condition as a hard error, see 709845c for details.

I think we're getting closer to this being workable now.

arihant2math · 2025-12-28T02:43:26Z

Still being very non-deterministic from my end. Usertest does pass sometimes. I suspect there might be some sort of race condition somewhere. From a black-box perspective it seems the algorithm is fine, so I suspect it's something else.

arihant2math · 2025-12-28T02:48:08Z

Ah ok, I see that preempts are being scheduled but not executed (sometimes).

arihant2math · 2025-12-28T03:04:24Z

~~Ok I see the problem, we should disable interrupts when modifying the wait queue because it might be locked during a timer interrupt.~~

Or something is just holding the wakeup_q and never dropping it.

Nevermind, I see the problem:
Essentially SYS_TIMER seems to be running on a random CPU. However sched_yield yields the current CPU's scheduler. So, if the yield was meant for a different CPU it'll never go through.

Indeed, modifying WakeupEvent::preempt to include the callers CPU and adding the following patch:

                    WakeupKind::Preempt(cpu_id) => {
                        if cpu_id == ArchImpl::id() {
                            crate::sched::sched_yield();
                        } else {
                            log::warn!(
                                "SysTimer: received preemption event for CPU {}, but running on CPU {}",
                                cpu_id,
                                ArchImpl::id()
                            );
                        }
                    }
                }

Shows:

[    2.2638996] drivers::timer: SysTimer: received preemption event for CPU 1, but running on CPU 0

right before a hang.

Ideally we'd have some sort of per-cpu handling (either the wakeup-queue being per-cpu or the timer interrupts being handled per-cpu). But for now I'll just send an IPI.

Still running into the timer interrupt not firing (I think it's still being deadlocked somehow).

Ah, it's a cyclic lock acquire (sched_yield -> schedule -> schedule_preempt).

arihant2math · 2025-12-28T05:10:52Z

Ok, fixed the hanging bug with 8a09fe0. Essentially I move to interrupt the idle task every 4 ms.

However I'm seeing a speed regression of ~10x when running usertests. I'd like to probably resolve that before merging.

hexagonal-sun · 2025-12-28T07:13:46Z

Do you think we need a commit like 50e5b0b to force other CPUs out of the idle task if a waker changes the state of the task? I realise that the preempt code would eventually do this with a tick interrupt but I perhaps it may reduce latency?

arihant2math · 2025-12-29T02:03:27Z

src/sched/mod.rs

+    if let Some(task) = SCHED_STATE.borrow().run_queue.get(descriptor) {
+        return Some(task.clone());
+    }
+    // TODO: Ping other CPUs to find the task.


Should probably be handled by an IPI in a future PR

Agreed, althogh I've got a patch in the works which makes TASK_LIST a global list of all tasks, not just task states which should help with this.

That works as well.

src/interrupts/cpu_messenger.rs

src/drivers/fs/proc.rs

arihant2math · 2025-12-29T02:12:33Z

src/drivers/timer/mod.rs

                // After pushing, we must update the hardware timer in case our
                // new event is the earliest one.
-                if let Some(next_event) = wake_q.peek() {
+                if let Some(next_event) = wakeup_q.peek() {


This might stop being accurate now that schedule() is called after this.

Although it doesn't matter because we'll probably remove this in the near-future anyhow.

arihant2math · 2025-12-29T02:13:00Z

src/drivers/timer/mod.rs

+    }
+}
+
+pub fn schedule_force_preempt() {


We should find a way to do without this later.

Agreed, this feels like a bit of a hack.

src/process/threading/futex/mod.rs

hexagonal-sun · 2025-12-29T06:46:09Z

I've split out independent fixes into sepereate PRs which has reduced the size of this PR. Ideally they need merging before this does. They are #108 #107 #106 #105 #104.

Use EEVDF concepts like virtual deadline correctly and actually calculate the necessary deadline and use it to schedule. Also dynamically preempts based on the deadline.

# Conflicts: # libkernel/src/sync/per_cpu.rs

When switching tasks, we may well be swithing away from a task which is going to `Sleep`. Therefore the check ```rust debug_assert_eq!(*prev_task.state.lock_save_irq(), TaskState::Runnable); ``` Is incorrect.

This prevents a bug where `sys_exit` calls `exit_group` for the thread's process, even when there are still active threads.

Currently, a global wakeup queue is used for all CPUs on the system. This leads to inefficient behavior regarding preemption. When the scheduler requests a preemption event, it is inserted into a global list alongside events from all other CPUs. When processing IRQs, there is no guarantee which CPU will handle the timer interrupt. If the current CPU processes a preemption event intended for a different CPU, it must signal the target CPU via an IPI. This causes a severe bottleneck, as one CPU may end up distributing preemption events for the entire system. Fix this by implementing a per-cpu wakeup queue. Preemption events are now strictly scheduled for the current CPU, ensuring they are handled locally by the core that scheduled them. This significantly simplifies the preemption logic and eliminates the need for IPIs to signal preemption events.

Fix formatting in the sched module to keep CI happy.

arihant2math · 2025-12-29T07:55:41Z

I think the sub-PRs all look reasonable; I've merged them all. I also cleaned up my commit logs a little bit.
I think that merging this after your review, given that there are no issues, looks reasonable

hexagonal-sun

One minor comment, but happy to merge. I don't fully understand the EEVDF scheduling aspects but it seems to work fine. I should set some time aside and study the paper really.

src/process/mod.rs

Use the new `CpuId` type for the `last_cpu` field in `Task`.

arihant2math force-pushed the multicore-sched branch 2 times, most recently from b620a40 to a63ce3b Compare December 25, 2025 08:39

arihant2math changed the title ~~schedule in round-robin fashion~~ Schedule onto multiple cores in round-robin fashion Dec 25, 2025

arihant2math marked this pull request as draft December 25, 2025 09:00

hexagonal-sun reviewed Dec 25, 2025

View reviewed changes

arihant2math force-pushed the multicore-sched branch 2 times, most recently from 5c890be to a25bb55 Compare December 25, 2025 21:04

arihant2math marked this pull request as ready for review December 25, 2025 21:04

arihant2math force-pushed the multicore-sched branch 3 times, most recently from bc729dc to 427aae4 Compare December 26, 2025 05:52

hexagonal-sun approved these changes Dec 26, 2025

View reviewed changes

hexagonal-sun force-pushed the multicore-sched branch from 70ea99c to c9d0ef4 Compare December 26, 2025 16:29

arihant2math commented Dec 27, 2025

View reviewed changes

src/sched/mod.rs Show resolved Hide resolved

hexagonal-sun force-pushed the multicore-sched branch 2 times, most recently from c6b44cb to c52b786 Compare December 27, 2025 21:13

arihant2math force-pushed the multicore-sched branch from 9697eea to e1dadca Compare December 28, 2025 04:46

arihant2math commented Dec 29, 2025

View reviewed changes

arihant2math mentioned this pull request Dec 29, 2025

Support cwd symlink in procfs #101

Merged

hexagonal-sun force-pushed the multicore-sched branch from e268746 to f6d4feb Compare December 29, 2025 06:44

arihant2math force-pushed the multicore-sched branch from f6d4feb to 70f0753 Compare December 29, 2025 07:46

arihant2math added 2 commits December 28, 2025 23:49

schedule in round-robin fashion

b75f298

try fix scheduling on non-main cores

5e55309

arihant2math force-pushed the multicore-sched branch 2 times, most recently from 3b5c835 to 2818775 Compare December 29, 2025 07:50

arihant2math and others added 15 commits December 28, 2025 23:51

EEVDF improvements

39d0bba

Use EEVDF concepts like virtual deadline correctly and actually calculate the necessary deadline and use it to schedule. Also dynamically preempts based on the deadline.

include last run CPU

0a3d085

use messengers

ca85552

# Conflicts: # libkernel/src/sync/per_cpu.rs

sched: remove incorrect assertion check

0f566f3

When switching tasks, we may well be swithing away from a task which is going to `Sleep`. Therefore the check ```rust debug_assert_eq!(*prev_task.state.lock_save_irq(), TaskState::Runnable); ``` Is incorrect.

clone: add all tasks to process task list

116a1ad

This prevents a bug where `sys_exit` calls `exit_group` for the thread's process, even when there are still active threads.

Preempt IPI

0227e0d

fix deadline guard to minimize overhead

4513531

update debug statements

b0d214d

fix deadlock in interrupt handler

b818047

fix preemption not being scheduled

018c1d9

Force preemption of idle task

0f1a486

Optimize current-cpu case in task insertion

4fedf19

sched: fix formatting

108b580

Fix formatting in the sched module to keep CI happy.

sched: remove unused function sched_yield

bbde9f0

arihant2math force-pushed the multicore-sched branch from 2818775 to bbde9f0 Compare December 29, 2025 07:51

hexagonal-sun approved these changes Dec 29, 2025

View reviewed changes

src/process/mod.rs Outdated Show resolved Hide resolved

process: mod: last_cpu: use CpuId

9f0bf1f

Use the new `CpuId` type for the `last_cpu` field in `Task`.

hexagonal-sun merged commit 0258645 into hexagonal-sun:master Dec 29, 2025
1 check passed

arihant2math deleted the multicore-sched branch December 29, 2025 19:56

Schedule onto multiple cores in round-robin fashion #91

Schedule onto multiple cores in round-robin fashion #91

Uh oh!

Conversation

arihant2math commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hexagonal-sun commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hexagonal-sun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arihant2math commented Dec 26, 2025

Uh oh!

hexagonal-sun commented Dec 26, 2025

Uh oh!

hexagonal-sun commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hexagonal-sun commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arihant2math commented Dec 26, 2025

Uh oh!

arihant2math commented Dec 27, 2025

Uh oh!

Uh oh!

hexagonal-sun commented Dec 27, 2025

Uh oh!

arihant2math commented Dec 28, 2025

Uh oh!

arihant2math commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arihant2math commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arihant2math commented Dec 28, 2025

Uh oh!

hexagonal-sun commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arihant2math commented Dec 25, 2025 •

edited

Loading

hexagonal-sun commented Dec 25, 2025 •

edited

Loading

hexagonal-sun commented Dec 26, 2025 •

edited

Loading

hexagonal-sun commented Dec 26, 2025 •

edited

Loading

arihant2math commented Dec 28, 2025 •

edited

Loading

arihant2math commented Dec 28, 2025 •

edited

Loading

hexagonal-sun commented Dec 28, 2025 •

edited

Loading

arihant2math commented Dec 29, 2025 •

edited

Loading