Skip to content

Commit 3d36375

Browse files
lockshawlockshaw
authored
Generalize index space constructs and add data dependence info to op-attrs (#1569)
* Add utils/orthotope targeting #1528 * Pass test for deconflict_overlapping_dims * Pass initial orthotope projection tests * Start moving some machine view types over to use orthotope * Add OneToMany, ManyToOne, and DimProjection to fix projection equality issue * Improved OneToMany and ManyToOne, and some bidict refactoring * Fri Dec 27 03:20:32 PM PST 2024 * Additional changes. * Move over to new projection infra * Fix build errors * Test fixes * Add more testing and implement more projection functions * Format * Start creating local-pcg-execution * Checkpoint version with dim projections * Add MinimalDimDomain and MinimalOrthotope * Fix test build * Pre-cleanup * Implement new communication modeling approach * Fix a bunch of build errors in compiler tests * Pass pcg and utils tests * Save before tackling the minimal domain issue * Fix failing subcase of get_abstracted_tensor_set_movement_across_split * Fix pcg_get_mapping_along_edge test by composing minimal dim domain mappings instead of dim domain mappings * Refactor abstract communication modeling to get get_tensor_set_movement_across_split tests passing * More test case fixing (this time for get_optimial_machine_mapping_with_memory) * Fix machine view offset issues in optimal machine mapping algorithms * Fix get_optimal_machine_mapping test * Fix misc other tests * Updated MappedParallelComputationGraph with computed task dependency information * More refactoring, add plan for how we get from here to realm execution * Move training logic into task-spec and define AtomicTaskInvocation and associated types * Rearchitect local-execution and local-pcg-execution * Wire up shard bindings into local pcg execution * Reimplement local-execution's ModelTrainingInstance * Start on dynamic graph attempt * Update proj, layout fixes * Cleanup * task-spec reorganization * Update proj * Pass simple pass expansion test * Remove task binding and more * kwarg graph impl * Pass KwargDataflowGraph tests * More kwarg graph code to support pcg * Remove variadic args, more kwarg graph algorithms * Finish kwarg fixes for ComputationGraph * Pass pcg tests * Extend kwarg dataflow graph changes into substitutions * Update a bunch more of substitutions * Fix substitutions build * Get substitutions passing tests * Pass compiler tests * Pre-remove OpTaskInvocation * Remove OpTaskInvocation * Remove symbolic in task-spec * Pass basic pass_expansion and shard_expansion tests * Remove runtime_task_invocation dir in task-spec * PR todo cleanup * Pass tests * Format * Format again
1 parent 5abb225 commit 3d36375

2,130 files changed

Lines changed: 53274 additions & 30913 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.flake/pkgs/ffdb/ffdb.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
from proj.config_file import get_config_root
1+
from proj import get_repo_root
22
from pathlib import Path
33
import gdb
44

5-
gdb.execute(f'directory {get_config_root(Path.cwd())}')
5+
gdb.execute(f'directory {get_repo_root(Path.cwd())}')
66
gdb.prompt_hook = lambda x: '(ffdb) '
77
gdb.execute('set history save on')
88
gdb.execute('catch throw')

‎.proj.toml‎

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,21 @@ cuda_launch_cmd = [
66
"nixGL",
77
"--",
88
]
9+
layout_ignore_paths = [
10+
"lib/runtime",
11+
"lib/kernels",
12+
"lib/utils/test/common",
13+
"lib/compiler/ffi",
14+
"lib/op-attrs/ffi",
15+
"lib/pcg/ffi",
16+
"lib/substitutions/ffi",
17+
"lib/utils/ffi",
18+
"lib/ffi",
19+
"lib/utils/include/utils/graph/docs",
20+
"lib/compiler/test/src/internal",
21+
"lib/local-execution/test/src/internal",
22+
"bin/protobuf-to-json",
23+
]
924

1025
[targets.utils]
1126
type = "lib"
@@ -63,12 +78,19 @@ has-cpu-only-benchmarks = false
6378
has-cuda-tests = false
6479
has-cuda-benchmarks = false
6580

66-
[targets.local-execution]
67-
type = "lib"
68-
has-cpu-only-tests = true
69-
has-cpu-only-benchmarks = false
70-
has-cuda-tests = true
71-
has-cuda-benchmarks = false
81+
# [targets.local-execution]
82+
# type = "lib"
83+
# has-cpu-only-tests = true
84+
# has-cpu-only-benchmarks = false
85+
# has-cuda-tests = true
86+
# has-cuda-benchmarks = false
87+
88+
# [targets.local-pcg-execution]
89+
# type = "lib"
90+
# has-cpu-only-tests = true
91+
# has-cpu-only-benchmarks = false
92+
# has-cuda-tests = false
93+
# has-cuda-benchmarks = false
7294

7395
[targets.models]
7496
type = "lib"

‎CMakeLists.txt‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,6 @@ include(spdlog)
102102
include(doctestlib) # named doctestlib to avoid a name collision with doctest.cmake in rapidcheck
103103
include(gbenchmark)
104104
include(libassert)
105-
include(visit_struct)
106105
include(CTest)
107106
include(fmt)
108107
include(legion)

‎bin/export-model-arch/include/export-model-arch/json_sp_model_export.dtg.toml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
namespace = "FlexFlow"
2+
name = "JsonSPModelExport"
3+
type = "struct"
4+
features = [
5+
"eq",
6+
"hash",
7+
"fmt",
8+
"json",
9+
]
10+
11+
includes = [
12+
"pcg/file_format/v1/v1_computation_graph.dtg.h",
13+
"pcg/file_format/v1/v1_binary_sp_decomposition/v1_binary_sp_decomposition.dtg.h",
14+
]
15+
16+
src_includes = [
17+
"pcg/file_format/v1/v1_binary_sp_decomposition/json.h",
18+
]
19+
20+
[[fields]]
21+
name = "sp_decomposition"
22+
type = "::FlexFlow::V1BinarySPDecomposition"
23+
24+
[[fields]]
25+
name = "computation_graph"
26+
type = "::FlexFlow::V1ComputationGraph"

‎bin/export-model-arch/include/export_model_arch/json_sp_model_export.struct.toml‎

Lines changed: 0 additions & 25 deletions
This file was deleted.

bin/export-model-arch/src/export_model_arch.cc renamed to bin/export-model-arch/src/export-model-arch/main.cc

File renamed without changes.

‎bin/substitution-to-dot/CMakeLists.txt‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,9 @@ ff_add_executable(
22
NAME
33
substitution-to-dot
44
SRC_PATTERNS
5-
*.cc
5+
src/*.cc
6+
PRIVATE_INCLUDE
7+
include/
68
DEPS
79
substitution-generator
810
)

bin/substitution-to-dot/substitution_to_dot.cc renamed to bin/substitution-to-dot/src/substitution-to-dot/main.cc

File renamed without changes.

‎cmake/visit_struct.cmake‎

Lines changed: 0 additions & 16 deletions
This file was deleted.

‎flake.lock‎

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)