Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Editor and IDE specific files
.idea/
.vs/
.vscode/
*.sln.DotSettings.user

# C# build directories
Expand All @@ -20,4 +21,4 @@ __pycache__/

# Artifacts
artifacts/
packages/
packages/
41 changes: 13 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# LibStored.Net
[![.NET](https://github.com/DEMCON/libstored.net/actions/workflows/dotnet.yml/badge.svg)](https://github.com/DEMCON/libstored.net/actions/workflows/dotnet.yml)
[![NuGet](https://img.shields.io/nuget/dt/LibStored.Net.svg)](https://www.nuget.org/packages/LibStored.Net)
[![NuGet](https://img.shields.io/nuget/vpre/LibStored.Net.svg)](https://www.nuget.org/packages/LibStored.Net)

A 100% native C# implementation of the [libstored](https://github.com/DEMCON/libstored) library, which is a library for storing and retrieving data in a structured way, using a store definition file. The library provides a set of layers to build applications that can communicate with a Debugger or Synchroniser.

Expand Down Expand Up @@ -71,26 +74,16 @@ flowchart TD
- SyncZeroMQLayer: to connect to the Synchronizer (connection)

### Store generation
C# stores can be generated from the `<name>Meta.py`. Use the [libstored](https://demcon.github.io/libstored) generator to parse the `<name>.st` and output the `<name>Meta.py`.
The `generator.py` script in this repository under `/python`

Make sure a virtual environment is created, activated and the requirements.txt is installed in it.
```bash
cd python
python -m venv .venv
./.venv/Scripts/activate
pip install -r requirements.txt
```

Call the generator script to create a C# store that implements the `Store` abstract class, which can then be mapped to the `Synchronizer` and `Debugger` in LibStored.Net.
This project uses a C# Source Generator to create strongly-typed store classes from store metadata YAML (.yml) files at compile time.
This YAML file is automatically created when generating code for a store (.st) using [libstored](https://demcon.github.io/libstored) (from version v2.1.0).

```bash
python generator.py -m <name>Meta.py -o output_dir
```
How to use the Source Generator:
- Add the store metadata files (*.yml) to your project as AdditionalFiles so the source generator can read them during compilation.
- Reference the LibStored.Net either by adding the NuGet package. This package include the Source Generator.
- The generator emits C# store classes implementing the `Store` base class. Use them directly from your code after a build.

#### Source Generator

The C# code can be automatically generated from a metadata`*.yml` file generated by the `generator.py` script (or in the future - see [libstored issue#76](https://github.com/DEMCON/libstored/issues/76)). Reference the store(s) metadata as AddinionalFiles:
Example csproj snippet:
```xml
<ItemGroup>
<AdditionalFiles Include="TestStore.yml" />
Expand All @@ -116,15 +109,6 @@ Another option would be to create a store per thread, and synchronize them using
- StdioLayer (use `libstored.Stdio2Zmq` as alternative for local debugging, not production ready)
- SerialLayer (use `libstored.Serial2Zmq` as alternative for local debugging, not production ready)

#### Store source generation
Currently, C# stores are created manually by running a Python script using the Meta.py output of [libstored](demcon.github.io/libstored).
A C# Source Generator could automate this process of converting the store data structure to a C# class, using the hash, names, types, offset and
sizes of all objects in the store.

A C# Source Generator could generate the stores automatically during compile-time of the C# application using this library. The input of the source generated is the `.rtf` / `.csv` / `Meta.py`, or parse the actual store definition in `.st` files. Maybe [libstored](demcon.github.io/libstored) can be extended to generate a language agnostic `.json` or `.yml` file with the store definition, which can be used by the C# Source Generator to generate the store class. The `.json` file would be the easiest to implement since its easy to parse and can contain all relevant store data. The Source Generator that transform a `*.yml` has been implemented, as described in the [README.md](src/LibStored.Net.Generator/README.md).

Another option is to create a template file for the C# version of the store in [libstored](demcon.github.io/libstored).

#### Unsupported features

Types: `Pointer`, `Pointer32` and `Pointer64` are not supported, as they are not needed in C# applications. The `ptr32` and `ptr64` types are used to store pointers to other objects in the store, which is not needed in C# applications as the objects are stored in managed memory. These types will be mapped to unsigned integers (`uint` and `ulong`) in the C# implementation.
Expand All @@ -139,7 +123,7 @@ using LibStored.Net.ZeroMQ;
using NetMQ;
using NetMQ.Sockets;

// Make sure this ExampleStore is created by the source generator or python script.
// Make sure this ExampleStore is created by the source generator from ExampleStore.yml.
ExampleStore store = new();

// Attach the store to the debugger
Expand Down Expand Up @@ -173,6 +157,7 @@ python -m libstored.gui -p 5555
## Compatibility

Tested with libstored versions:
- [v2.1.0](https://github.com/DEMCON/libstored/releases/tag/v2.1.0)
- [v2.0.0](https://github.com/DEMCON/libstored/releases/tag/v2.0.0)
- [v1.8.0](https://github.com/DEMCON/libstored/releases/tag/v1.8.0)
- [v1.7.1](https://github.com/DEMCON/libstored/releases/tag/v1.7.1) (see note below)
Expand All @@ -186,7 +171,7 @@ The ZeroMQ socket type changed from `PAIR` to `DEALER` for the SyncZeroMQLayer.

## Build

Install the [.NET SDK](https://dotnet.microsoft.com/download), version 9.0 or higher, and run the following command in the root directory of the project:
Install the [.NET SDK](https://dotnet.microsoft.com/download), version 10.0 or higher, and run the following command in the root directory of the project:

Build the library and examples:
```bash
Expand Down
10 changes: 4 additions & 6 deletions examples/Arq/LibStored.Arq/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
# Add ARG for libstored version
ARG LIBSTORED_VERSION=2.0.0

FROM ubuntu:22.04 as builder

# Use the ARG after FROM
ARG LIBSTORED_VERSION

# Set noninteractive frontend for apt
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections

RUN apt-get update && \
Expand Down Expand Up @@ -36,6 +31,9 @@ RUN apt-get update && \

WORKDIR /src/extern

# Add ARG for libstored version
ARG LIBSTORED_VERSION=2.0.0

RUN wget https://github.com/DEMCON/libstored/archive/refs/tags/v${LIBSTORED_VERSION}.tar.gz && \
tar -xzf v${LIBSTORED_VERSION}.tar.gz --transform="s/libstored-${LIBSTORED_VERSION}/libstored/" && \
rm v${LIBSTORED_VERSION}.tar.gz
Expand Down
2 changes: 1 addition & 1 deletion examples/Arq/LibStored.Net.Arq.AppHost/AppHost.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

var builder = DistributedApplication.CreateBuilder(args);

const string libstoredVersion = "2.0.0";
const string libstoredVersion = "2.1.0";
var arq = builder.AddDockerfile("libstored-arq", "../LibStored.Arq/", "Dockerfile")
.WithImage("demcon/libstored-arq-example")
.WithImageTag(libstoredVersion)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">

<Sdk Name="Aspire.AppHost.Sdk" Version="13.0.0" />
<Project Sdk="Aspire.AppHost.Sdk/13.0.1">

<PropertyGroup>
<OutputType>Exe</OutputType>
Expand All @@ -10,7 +8,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Aspire.Hosting.AppHost" Version="13.0.0" />
<PackageReference Include="Aspire.Hosting.AppHost" Version="13.0.1" />
</ItemGroup>

<ItemGroup>
Expand Down
2 changes: 1 addition & 1 deletion examples/Arq/LibStored.Net.Arq.Console/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
bool connected = false;
arqLayer.EventOccurred += (_, e) =>
{
_logger.LogInformation("Arc Event: {Event}", e.Event);
_logger.LogInformation("Arq Event: {Event}", e.Event);
if (e.Event is ArqEvent.Connected or ArqEvent.Reconnect)
{
connected = true;
Expand Down
23 changes: 23 additions & 0 deletions examples/Arq/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Arq Example

This example demonstrates cross-language synchronization over a lossy channel. The Arq layer will retransmist lost messages between C++ and C# components. The C++ component runs inside a container while the C# implementation runs natively. The setup is adapted from the libstored project's synchronization examples and uses the Arq transport/adapter to route changes between the stores.

In this example, the C++ code runs inside a container, while the C# implementation runs natively.

## Prerequisites

- [.NET 10 SDK](https://dotnet.microsoft.com/download)
- [Aspire CLI](https://aspire.dev/docs/cli/)
- [Podman](https://podman.io/) or [Docker](https://www.docker.com/products/docker-desktop)

## Running the Example

Start the example environment with Aspire:

```sh
aspire run
```

## Learn More

- [Aspire Documentation](https://aspire.dev/docs/)
10 changes: 4 additions & 6 deletions examples/Sync/LibStored.Net.Example.AppHost/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
# Add ARG for libstored version
ARG LIBSTORED_VERSION=2.0.0

FROM ubuntu:22.04 as builder

# Use the ARG after FROM
ARG LIBSTORED_VERSION

# Set noninteractive frontend for apt
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections

RUN apt-get update && \
Expand Down Expand Up @@ -36,6 +31,9 @@ RUN apt-get update && \

WORKDIR /tmp

# Add ARG for libstored version
ARG LIBSTORED_VERSION=2.0.0

RUN wget https://github.com/DEMCON/libstored/archive/refs/tags/v${LIBSTORED_VERSION}.tar.gz && \
tar -xzf v${LIBSTORED_VERSION}.tar.gz --transform="s/libstored-${LIBSTORED_VERSION}/libstored/" && \
rm v${LIBSTORED_VERSION}.tar.gz && \
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
<Project Sdk="Microsoft.NET.Sdk">

<Sdk Name="Aspire.AppHost.Sdk" Version="13.0.1" />
<Project Sdk="Aspire.AppHost.Sdk/13.0.1">

<PropertyGroup>
<OutputType>Exe</OutputType>
<NoWarn>CS1591</NoWarn>
<IsAspireHost>true</IsAspireHost>
<UserSecretsId>d11b339a-7d36-4112-8969-2ebfad7419e0</UserSecretsId>
</PropertyGroup>

Expand Down
2 changes: 1 addition & 1 deletion examples/Sync/LibStored.Net.Example.AppHost/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

builder.AddDockerComposeEnvironment("docker-compose");

const string libstoredVersion = "2.0.0";
const string libstoredVersion = "2.1.0";
var sync = builder.AddDockerfile("libstored", ".", "Dockerfile")
.WithImage("demcon/libstored")
.WithImageTag(libstoredVersion)
Expand Down
31 changes: 0 additions & 31 deletions python/ExampleMeta.g.cs.json

This file was deleted.

22 changes: 0 additions & 22 deletions python/ExampleMeta.g.cs.yml

This file was deleted.

7 changes: 0 additions & 7 deletions python/ExampleMeta.st

This file was deleted.

46 changes: 0 additions & 46 deletions python/ExampleMetaMeta.py

This file was deleted.

Loading
Loading