Skip to content

Commit 2dfb306

Browse files
milanmmilanm
committed
Add tests, CI, and project polish
- Add 25 unit tests with numerical gradient checking (xUnit) - Add GitHub Actions CI workflow (build + test) - Fix double.Parse to use InvariantCulture (locale bug) - Add test project to solution file - Add .editorconfig, global.json, nuget.config - Add badges, architecture diagram, and test section to README - Add TestResults/ to .gitignore
1 parent 00b0e07 commit 2dfb306

10 files changed

Lines changed: 545 additions & 3 deletions

File tree

‎.editorconfig‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
root = true
2+
3+
[*]
4+
indent_style = space
5+
indent_size = 4
6+
end_of_line = lf
7+
charset = utf-8
8+
trim_trailing_whitespace = true
9+
insert_final_newline = true
10+
11+
[*.cs]
12+
dotnet_sort_system_directives_first = true
13+
csharp_new_line_before_open_brace = all
14+
csharp_style_var_for_built_in_types = false:suggestion
15+
csharp_style_var_when_type_is_apparent = true:suggestion
16+
17+
[*.{json,yml,yaml}]
18+
indent_size = 2
19+
20+
[*.md]
21+
trim_trailing_whitespace = false

‎.github/workflows/build.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: Build & Test
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
jobs:
10+
build:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- name: Setup .NET
17+
uses: actions/setup-dotnet@v4
18+
with:
19+
dotnet-version: '10.0.x'
20+
21+
- name: Restore
22+
run: dotnet restore
23+
24+
- name: Build
25+
run: dotnet build --no-restore --configuration Release
26+
27+
- name: Test
28+
run: dotnet test --no-build --configuration Release --verbosity normal

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,8 @@ Thumbs.db
1818
desktop.ini
1919
.DS_Store
2020

21+
## Test
22+
TestResults/
23+
2124
## Data
2225
input.txt

‎AutogradEngine.slnx‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,7 @@
22
<Folder Name="/src/">
33
<Project Path="src/AutogradEngine/AutogradEngine.csproj" />
44
</Folder>
5+
<Folder Name="/tests/">
6+
<Project Path="tests/AutogradEngine.Tests/AutogradEngine.Tests.csproj" />
7+
</Folder>
58
</Solution>

‎README.md‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# MicroGPT.cs
22

3+
[![Build & Test](https://github.com/milanm/AutoGrad-Engine/actions/workflows/build.yml/badge.svg)](https://github.com/milanm/AutoGrad-Engine/actions/workflows/build.yml)
4+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
5+
[![.NET](https://img.shields.io/badge/.NET-10.0-512bd4)](https://dotnet.microsoft.com/)
6+
37
A complete [GPT](https://en.wikipedia.org/wiki/Generative_pre-trained_transformer) language model — training and inference — in pure C# with zero dependencies.
48

59
Faithful port of [Andrej Karpathy's microgpt.py](https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95).
@@ -22,6 +26,53 @@ It trains a tiny GPT model on a list of human names, then generates new ones tha
2226
| `Tokenizer.cs` | Character-level tokenizer with `Encode()`/`Decode()` |
2327
| `NeuralOps.cs` | Stateless neural-net building blocks: `Linear`, `Softmax`, `RMSNorm` |
2428
| `Program.cs` | GPT model, training loop (`Train`), and generation (`Generate`) |
29+
| `ValueTests.cs` | 25 tests — numerical gradient checking, ops correctness, roundtrips |
30+
31+
## Architecture
32+
33+
```
34+
┌──────────────────────────────────────────────────────────────────────┐
35+
│ TRAINING LOOP │
36+
│ │
37+
│ "emma" ──► Tokenizer ──► [BOS, e, m, m, a, EOS] │
38+
│ │ │
39+
│ ┌─────────────────────▼──────────────────────┐ │
40+
│ │ GPT MODEL │ │
41+
│ │ │ │
42+
│ │ Token Embedding ──┐ │ │
43+
│ │ ├──► x │ │
44+
│ │ Position Embedding┘ │ │
45+
│ │ │ │
46+
│ │ ┌─────── Transformer Layer ──────────┐ │ │
47+
│ │ │ │ │ │
48+
│ │ │ RMSNorm ──► Multi-Head Attention │ │ │
49+
│ │ │ (Q·K/√d → softmax → V)│ │ │
50+
│ │ │ + Residual Connection │ │ │
51+
│ │ │ │ │ │
52+
│ │ │ RMSNorm ──► MLP (expand → ReLU² │ │ │
53+
│ │ │ → compress) │ │ │
54+
│ │ │ + Residual Connection │ │ │
55+
│ │ │ │ │ │
56+
│ │ └──────────── × n_layer ────────────┘ │ │
57+
│ │ │ │
58+
│ │ Linear (weight-tied with Token Embedding) │ │
59+
│ │ │ │ │
60+
│ └─────────┼──────────────────────────────────┘ │
61+
│ ▼ │
62+
│ Softmax ──► Probabilities │
63+
│ │ │
64+
│ ▼ │
65+
│ Cross-Entropy Loss │
66+
│ │ │
67+
│ ▼ │
68+
│ Backward() ◄── Value autograd engine │
69+
│ │ (chain rule through │
70+
│ ▼ computation graph) │
71+
│ Adam Optimizer │
72+
│ (update all parameters) │
73+
│ │
74+
└─────────────────────── × num_steps ─────────────────────────────────┘
75+
```
2576

2677
## Quick Start
2778

@@ -36,6 +87,14 @@ Or with custom settings:
3687
dotnet run -- --n_embd 32 --n_layer 2 --num_steps 2000
3788
```
3889

90+
### Running Tests
91+
92+
The autograd engine is verified with numerical gradient checking — the same technique PyTorch uses in `torch.autograd.gradcheck`:
93+
94+
```bash
95+
dotnet test
96+
```
97+
3998
### CLI Arguments
4099

41100
| Argument | Default | What it does |
@@ -243,4 +302,4 @@ This implementation uses more modern design choices (closer to [LLaMA](https://e
243302

244303
## License
245304

246-
MIT — learn from it, play with it, share it. See [LICENSE](LICENSE).
305+
MIT — learn from it, play with it, share it. See [LICENSE](LICENSE).

‎global.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"sdk": {
3+
"version": "10.0.100",
4+
"rollForward": "latestFeature"
5+
}
6+
}

‎nuget.config‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<configuration>
3+
<packageSources>
4+
<clear />
5+
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
6+
</packageSources>
7+
</configuration>

‎src/AutogradEngine/Program.cs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121

2222
using System;
2323
using System.Collections.Generic;
24+
using System.Globalization;
2425
using System.IO;
2526
using System.Linq;
2627
using System.Net.Http;
@@ -86,7 +87,7 @@ static int ParseArg(string[] args, string name, int defaultVal)
8687
static double ParseArg(string[] args, string name, double defaultVal)
8788
{
8889
for (int i = 0; i < args.Length - 1; i++)
89-
if (args[i] == $"--{name}") return double.Parse(args[i + 1]);
90+
if (args[i] == $"--{name}") return double.Parse(args[i + 1], CultureInfo.InvariantCulture);
9091
return defaultVal;
9192
}
9293

@@ -529,4 +530,4 @@ static void Generate(int numSamples)
529530
Console.WriteLine($"sample {sampleIdx}: {string.Join("", generated)}");
530531
}
531532
}
532-
}
533+
}

‎tests/AutogradEngine.Tests/AutogradEngine.Tests.csproj‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
3+
<PropertyGroup>
4+
<TargetFramework>net10.0</TargetFramework>
5+
<ImplicitUsings>enable</ImplicitUsings>
6+
<Nullable>enable</Nullable>
7+
<IsPackable>false</IsPackable>
8+
</PropertyGroup>
9+
10+
<ItemGroup>
11+
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.*" />
12+
<PackageReference Include="xunit" Version="2.*" />
13+
<PackageReference Include="xunit.runner.visualstudio" Version="2.*" />
14+
</ItemGroup>
15+
16+
<ItemGroup>
17+
<ProjectReference Include="..\..\src\AutogradEngine\AutogradEngine.csproj" />
18+
</ItemGroup>
19+
20+
</Project>

0 commit comments

Comments
 (0)