- added new macro
$gensym, to generate a unique symbol identifier to use in macros append,concat, andpopcan be use as values
- all paths inside
ifshould return a value, when used as an expression. If anelsebranch is missing,nilwill be returned - new compile time error when trying to use
append!,concat!,pop!,@=and@@=as values
- in function calls, the function to call is now always evaluated first
- in function calls, the arguments are now evaluated from left to right
- the bytecode reader can print the argument of a
PUSH_RETURN_ADDRESSinstruction as a hex number - new super instruction
CALL_SYMBOL_BY_INDEXto optimise aLOAD_FAST_BY_INDEXfollowed by aCALL
- instruction counter in the bytecode reader are displayed in hex, and count each instruction instead of each byte
let/mut/setpush a copy of their value when used as expression (instead of an internal reference)
- removed a nearly never emitted
GET_CURRENT_PAGE_ADDRinstruction, since it's now always optimised withCALLinto aCALL_CURRENT_PAGEinstruction - removed
list:size,dict:size,dict:contains,math:evenandmath:oddsince they were deprecated since ArkScript 4.2.0
list:permutationsis deprecated in favor oflist:combinationslist:permutationsWithReplacementis deprecated in favor oflist:combinationsWithReplacement
- new debugger commands:
stack <n>andlocals <n>to print the values on the stack and in the current locals scope - custom format specifiers for lists:
:nto remove surrounding brackets,:c/:ncto use,as a separator instead of,:l/:nlto use\nas a separator,:?sto format as an escaped quoted string,:sto format as a quoted string
formatcan use format specifiers for integers:b,#b,B,#B,c,d,o,x,#x,X, and#Xif the argument is an integer- display a warning to
stderrwhen using a deprecated function/value (checks for@deprecatedinside the attached comment of functions / values)
pop!can return the removed value@=and@@=return the inserted valueappend!andconcat!return the modified listlet,mutandsetcan return the assigned value- fix formatter: when the condition of a
whileloop is on multiple lines, add the right amount of indentation before it
- runtime type checking errors are on stderr instead of stdout
- runtime exceptions are on stderr instead of stdout
- VM error outputs are on stderr instead of stdout
- new
TAIL_CALL_SELFinstruction to take care of tail calls in functions: jumps to address 0 in the current page, and reset the scope
- error outputs are on stderr instead of stdout
- in macros,
len,empty?,head,tail,@have been renamed to$len,$empty?,$head,$tailand$at. Those versions only work inside macros too, inside of having a weird dichotomy where they sometimes got applied and sometimes not
applyfunction:(apply func [args...]), to call a function with a set of arguments stored in a list. Works with functions, closures and builtins+,-,*,/and many other operators can now be passed around, like builtins. This now works:(list:reduce [1 2 3] +), where before we would get a compile time error about a "freestanding operator '+'"builtin__slicebuiltin, for strings and lists:(builtin__slice data start end [step=1]); this is an experimentation and may be removed in future versions- arguments of builtin macros are properly type-checked and will now raise runtime errors if the type is incorrect
-fno-cachecli option to disable the creation of the bytecode cache folder__arkscript__- in the CLI,
filecan be-to read code from stdin
- when using the cli flag
-fdump-ir, the IR is dumped in the cache folder
assertis no longer an instruction but a builtin- when comparing values of different types using
<,>,<=,>=and=, the result will always befalse(it used to rely on the type index)
- added
BREAKPOINTinstruction - breakpoints can be placed using
(breakpoint)and(breakpoint condition) - added a debugger that can be triggered on error or on breakpoint by passing
-fdebuggerto the CLI (see the docs for the debugger) - diagnostics can now be generated when using
State.doString, usingDiagnostics::generateWithCode(as the original code must be passed to the diagnostics generator) empty?can now take a dict, and returnstrueif it has no key/value pairslencan now work on dictionaries, counting the number of keys
- changed the runpath of
arkscriptto look forlibArkReactorunder its (arkscript's) directory, {arkscript}/bin, {arkscript}/lib, and {arkscript}/../lib andandorrequire valid expressions, so(or 1 (mut x 3))is no longer valid code, as(mut x 3)doesn't return a value(not (dict "a" 2))now returnsfalse, asnotchecks if the dict is empty
- the repl prints the output of the last expression it ran
- new super instructions:
MUL_BY,MUL_BY_INDEX,MUL_SET_VALthat can do multiplications (and optional storing in vars) in place - new super instruction:
FUSED_MATH, which can fuse 2 to 3 math operations in one go (ADD, SUB, MUL, DIV) - new
LOAD_SYMBOLinstruction that avoids creating a reference
- the REPL doesn't color
importin two colors (red forimp__tand blue for___or_), it keeps the first color that matched (red for import here) - page numbers are correctly counted when using the bytecode reader with '--only-names', instead of displaying
0every time
- quotes are added around strings in type errors
disassemblecan show a file bytecodeempty?now acceptsniland returnstruefor this value- the REPL adds
(repl:history)and(repl:save filename)as builtins - the REPL attempts to load a file from
ARKSCRIPT_REPL_STARTUPenvironment variable, to preload code - rename LOAD_SYMBOL and LOAD_SYMBOL_BY_INDEX to LOAD_FAST and LOAD_FAST_BY_INDEX to emphasize they load refs
- the formatter was breaking functions' arguments list containing argument attributes on multiple lines for no reason
- the formatter was formatting begin nodes inside conditions badly, putting the
{on the same line as the condition, making it hard to know if the condition hadthenandelsenodes or a single multi nodesthennode
- long function calls are split on multiple lines
- Function arguments are now immutable by default and an argument attribute
mutmust be added:(fun (a b c) (set b 5))->(fun (a (mut b) c) (set b 5))
dict:contains, usedict:contains?math:even, usemath:even?math:odd, usemath:odd?
- new builtin
disassembleto print the bytecode of a function - new builtin
io:readFileLinesto read lines from a file as a list of strings
- the formatter properly formats dictionaries (key-value pairs on their own line, always)
- renamed
dict:containstodict:contains?so that all functions returning booleans have?suffix ; added temporary aliasdict:contains - renamed
math:eventomath:even?, andmath:oddtomath:odd? string:removeAtcan work with negative indexes
- more tests for the io builtins
- added lines and code coloration in the error context
- new dependency:
fmtlib - added the padding/instruction/argumentation values when displaying instructions in the bytecode reader
$reprmacro to get a string representation of a given node- added boost-ext/ut to write unit tests in C++
- basic ArkScript code formatter, available through the CLI:
arkscript -f|--format - comments are now tracked in the AST and attached to the nearest node below them
VM::forceReloadPlugins, to be used by the REPL to force reload the plugins and be sure that their symbols are all define- added
help,save(save history to disk),history(print history),reset(reset vm and code) commands to the REPL - REPL can now show when a code block isn't terminated (prompt changes from
>to:) - more controls available inside the REPL
- fuzzing step in the CI
- better error reporting on unknown import
- check on number of arguments passed to
type - warning when the formatter deletes comment(s) by mistake
- check on arguments passed to
list,concat,appendand friends to only push valid nodes (that produces a value) - introduced
Ark::internal::Passto describe compiler passes: they all output an AST (parser, import solver, macro processor, and optimizer for now) - add
-f(no-)importsolver,-f(no-)macroprocessorand-f(no-)optimizerto toggle on and off those compiler passes - added resolving
empty?as a macro when possible - added short-circuiting to
andandorimplementation - added
--checkto the formatter as an option: returns 0 if the code is correctly formatted, 1 otherwise - the name & scope resolution pass now checks for mutability errors
- compile time checks for mutability errors with
append!,concat!andpop! - new
MAKE_CLOSURE <page addr>instruction, generated in place of aLOAD_CONSTwhen a closure is made - added
-fdump-irto dump the IR entities to a file named{file}.ark.ir - added 11 super instructions and their implementation to the VM
- support for the glob import syntax and symbol import syntax
- modify list and return a copy
(string:setAt string index char)(bound checked) - added in place list mutation:
(@= list|string index new_value),(@@= list|list<string> index1 index2 new_value|char)(bound checked) - compile time argument count check for
andandor - basic dead code elimination in the AST optimizer
- new operator
@@to get elements in list of lists / list of strings - new builtin
random, returning a random number between INT_MIN and INT_MAX, or in a custom range $as-isto paste a node inside a maro without evaluating it further ; useful to stop recursive evaluation of nodes inside function macrosLOAD_SYMBOL_BY_INDEXinstruction, loading a local from the current scope by an index (0 being the last element added to the scope)STORE_FROM_INDEXandSET_VAL_FROM_INDEXinstructions for parity with the super instructions not using load by indexINCREMENT_BY_INDEXandDECREMENT_BY_INDEXinstructions for parity with the super instructions not using load by indexSTORE_TAIL_BY_INDEX,STORE_HEAD_BY_INDEX,SET_VAL_TAIL_BY_INDEX,SET_VAL_HEAD_BY_INDEXsuper instructions added for parity with the super instructions not using load by indexRESET_SCOPE_JUMPinstruction emitted at the end of a while loop to reset a scope so that we can create multiple variables and useLOAD_SYMBOL_BY_INDEX- instruction source location ; two new bytecode tables were added: one for filenames, another for (page pointer, instruction pointer, file id, line), allowing the VM to display better error messages when the source is available
- show source location when a runtime error is thrown in the VM
LT_CONST_JUMP_IF_FALSEandLT_SYM_JUMP_IF_FALSEto compare a symbol to a const and a symbol to a symbol (respectively), then jump to an address if false (useful for while loops that check a simple(< x n)condition)LT_CONST_JUMP_IF_TRUE, counterpart ofLT_CONST_JUMP_IF_FALSEGT_CONST_JUMP_IF_TRUE, counterpart ofLT_CONST_JUMP_IF_TRUEGT_CONST_JUMP_IF_FALSE, counterpart ofLT_CONST_JUMP_IF_FALSEGT_SYM_JUMP_IF_FALSE, counterpart ofLT_SYM_JUMP_IF_FALSECALL_SYMBOLsuper instruction to load and call a symbol in a single instructionGET_FIELD_FROM_SYMBOLandGET_FIELD_FROM_SYMBOL_INDEXsuper instructions to get a field from a closure and push it to the stackEQ_CONST_JUMP_IF_TRUEandEQ_SYM_INDEX_JUMP_IF_TRUEto compare a symbol to a const and a symbol to a symbol (respectively), then jump to an address if true (useful for conditions that check a simple(= x n)condition)NEQ_CONST_JUMP_IF_TRUEas a super instruction counterpart toEQ_CONST_JUMP_IF_TRUENEQ_SYM_JUMP_IF_FALSE, counterpart ofLT_SYM_JUMP_IF_FALSEfor inequalityAT_SYM_SYMandAT_SYM_INDEX_SYM_INDEXsuper instructions, to get an element from a list in a single instruction, avoiding 2 push and 2 popCHECK_TYPE_OFandCHECK_TYPE_OF_BY_INDEXsuper instructions, to check the type of variable against a constant in a single instructionINCREMENT_STOREandDECREMENT_STOREsuper instructions, to update a value in place when incrementing/decrementing it by a set amountAPPEND_IN_PLACE_SYMandAPPEND_IN_PLACE_SYM_INDEXsuper instructionsPUSH_RETURN_ADDRESSinstruction now replaces the VM auto push of IP/PP- remove the stack swapping by pushing arguments in the reverse order by which they are loaded
- wasm export: we can now run ArkScript code on the web!
GET_CURRENT_PAGE_ADDRESSinstruction to push the current page address to the stackCALL_CURRENT_PAGEsuper instruction, calling the current page with a given number of arguments (avoid loading a page address on the stack, then popping it to perform the call)- new data type
Dict, which can be created with(dict "key" "value" ...), and manipulated withdict:get,dict:add,dict:contains,dict:remove,dict:keysanddict:size - added program name under `builtin__sys:programName
STORE_LENsuper instruction, to load a symbol by index and store its length (if it's a string or list) in a new variableAT_SYM_INDEX_CONSTsuper instruction, to load a value from a container using a constant as the index
- instructions are on 4 bytes: 1 byte for the instruction, 1 byte of padding, 2 bytes for an immediate argument
- enhanced the bytecode reader and its command line interface
- added the padding/instruction/argumentation values when displaying instructions in the bytecode reader
- fixed underline bug in the error context
- the str:format functions now expects strings following this syntax: https://fmt.dev/12.0/syntax/
- more documentation about the compiler implementation
- more documentation about the virtual machine
- closures can be now be compared field per field:
(= closure1 closure2)will work only if they have the same fields (name) and if the values match - macros are now defined like
(macro name value)/(macro name (args args args) body)/($if cond then else) - upgraded from C++17 to C++20
- new parser, new syntax for imports:
(import package.sub.file) - allow nodes to be empty when dumping the AST to JSON
- macros can be declared inside a
beginblock within a cond macro and used in the scope surrounding the cond macro arkscript --versionandarkscript --helpnow output ArkScript version with the commit hashvoid Value::toString(std::ostream&, VM&)now becomesstd::string Value::toString(VM&)- removed
Node::operator<<to replace it withNode::debugPrint - fixed a bug in the compiler where one could pass a non symbol to
let,mutorset, resulting in a compiler crash - fixed a bug in the macro processor where one could pass an unknown symbol to
argcountand crash the processor - fixed a bug in the compiler where one could pass something other than a list to
(fun)as the argument block, resulting in a crash - fixed a bug in the compiler generating not callable functions
- fixed a bug in the macro processor generating invalid
let/mut/setnodes - fixed a bug in the macro processor allowing out of bounds access with
(macro test (@ [1 2 3] -5)) - fixed a bug in the vm which wrongfully allowed self concat in place:
(concat! lst lst) - fixed a bug in the compiler where one could "use" operators without calling them:
(print nil?) - fixed a bug in the compiler allowing the use of operators without any argument:
(+) - fixed a bug in the vm during error reporting when a non-function was used as a function
- refactored code inside the bytecode reader to promote code reuse
- fixed a bug in the compiler generating invalid
whilenodes - fixed a bug when passing the wrong number of arguments to a function inside an async call was crashing the VM because the function couldn't be named
- fixed a bug in the compiler generating invalid
funnodes - fixed a bug when generating
let,mutorsetnodes inside macros with an invalid node type - fixed a bug when reading invalid UTF8 codepoints in the parser caused out of bounds reads
- fixed a bug with recursive macro, exhausting the stack space due to recursive evaluation
- futures can be awaited again, they will return nil on all the tries
- checking for reused argument name in macros during parsing
- enhanced comment after node handling in macros
- adding a hard limit on package names length (255 characters, to comply with posix limits)
- disallow passing invalid nodes as arguments to functions and operators
- checking for unevaluated spread inside macros
- checking for invalid symbols when defining a function through a macro
- added a max macro unification depth (256)
- added a max macro evaluation depth (256)
- introduced
internal::listInstructionswith the different instructions, to be used by the compiler and name resolution pass - checking for forbidden variable/constant name in the name & scope resolution pass, to give errors to the user before compiling some weird code
- repl completion and colors are now generated automatically from the builtins, keywords & operators
- fixed formating of comments inside function declarations
- renamed the macros
symcatandargcountto$symcatand$argcountfor uniformity - the
Ark::VMclass is nowfinal - the
STOREinstruction has been renamedSET_VAL - the
STOREinstruction is emitted in place of theLETandMUTinstructions, without any mutability checking now io:writeFileno longer takes a mode and has been split intoio:writeFileandio:appendToFile- instructions are now positioned like this:
inst byte1 byte2 byte3- byte1 is 0 if the instruction takes a single argument on 16 bits, split on byte2 and byte3
- if the instruction takes two arguments, they each have 12 bits ; the second one is on byte1 and upper half of byte2, the first on lower half of byte2 and then byte3
- ast-to-json dump now supports macros
- the parser can detect ill-formed macros (that are seen as function macros while being value macros)
- adding a
CALL_BUILTIN <builtin> <arg count>super instruction - fixed formatting of comments after the last symbol in an import node
- renamed
str:xyzbuiltins tostring:xyzfor uniformity with the standard library string:findtakes an optional third argument, startIndex (where to start the lookup from, default 0list:setAtcan work with negative indexes, and is now bound checked- re-enabled the AST optimizer, only used for the main
arkscriptexecutable (not enabled when embedding arkscript, so that one can grab variables from the VM) - loops have their own scope: variables created inside a loop won't leak outside it
- upgraded
fmtlibto 11.1.3-13 - allow capture in nested scope (before it was targeting only the current scope)
-bcroption can be given a source file, it will then be compiled before its bytecode is shown- magic numbers for tables start in bytecode files have been changed from 0x01, 0x02, 0x03 to 0xA1, 0xA2, 0xA3 (symbols, values, code) to make them stand out in hex editors
- magic numbers for value types in bytecode files have been changed from 0x01, 0x02, 0x03 to 0xF1, 0xF2, 0xF3 (number, string, function)
- numbers in the values table in bytecode files are no longer turned to string, but their IEEE754 representation is now encoded on 12 bytes (4 for the exponent, 8 for the mantissa)
- changed how scopes are stored inside the VM to enhance performances. All scope data are now contiguous!
- when possible, accessing variables from the current scope is compiled to a new instruction
LOAD_SYMBOL_BY_INDEX, to avoid the sometimes expansive lookup by id- this works inside normal scopes (introduced by while loops) and functions scopes, but not for closures
- VM stack size is now 4096 instead of 8192
Ark::CodeErrornow takes aCodeErrorContextto store the source (filename, line, column, expression) of an error- renamed
string:formattoformat io:removeFilesis nowio:removeFileand works on a single file/path- renamed almost all builtins to prefix them with
builtin__, to have them proxied in the standard library (to be able to import and scope them properly) - new super instruction
CALL_BUILTIN_WITHOUT_RETURN_ADDRESSto optimize the proxied builtins, skipping the return address deletion - the VM no longer store a reference to the current function being called in the newly created scope
- execution contexts can be reused for async calls if they are not active, to avoid constantly requesting memory and creating (heavy) contexts
- if there is more than 5 contexts, the 6th one will be destroyed once it completes
- execution contexts are now marked as free to be reused (or deleted) once a value has been computed, without waiting for a call to
await - captures are not renamed anymore by the NameResolutionPass (which used to fully qualify captured names when possible, which isn't desirable: when you capture
&foo, you expect to be able to use.foonot.module:foo) - when loading a module, its mappings are loaded in the current scope instead of the global scope
- argument order in the CLI changed: the file to run (and its optional script arguments) are now last, to be more consistent with all the other existing tooling (Python, Docker...)
- VM stack size has been upped to 4096 + 256, to have a buffer to be able to catch stack overflows without hindering performances too much
- we can not create a variable in a function, shadowing said function, to prevent weird bugs when trying to do recursion for example
- removed unused
NodeType::Closure - removing the custom string, replacing it with std::string (the format engine of the custom string had a lot of memory leaks)
Utils::digPlacesandUtils::decPlacesgot removed as they were no longer needed- removed deprecated code (
list:removeAt,arkexecutable now replaced byarkscript) - removed
VM::getUserPointerandVM::setUserPointer - removed
ARK_PROFILER_COUNTdefine - removed useless
\0escape in strings - removed
termcolordependency to rely onfmtfor coloring outputs - removed
andandorinstructions in favor of a better implementation to support short-circuiting - removed
LETandMUTinstructions in favor of a single newSTOREinstruction - removed
SAVE_ENVinstruction - removed
Value VM::resolve(const Value* val, Args&&... args), which has been deprecated in ArkScript v3.4.0
- added fuzzing tools and corpus for AFL
- added some tests for errors
- added recursion limit reached detection
- plugins can be constructed from outside ArkScript lib/modules folder, easing the development process
- plugins loading now works as intended: look alongside the given file/bytecode file, then in the std lib folder
- new way to create modules, easier to use
- calling a non-callable anonymous object do not result in a segfault
- macro processor function registering now handles empty nodes
- added a fix to avoid crashes when capturing unbound variables
- checking if the given operator takes one or more arguments at compile time
- adding bound checking on operator @
- adding bound checking on operator @ when used in macros
- better arity check for macros
- fixed a bug in the macro processor where macros were deleted when they shouldn't
- fixed a bug where macro functions with no argument would crash the macro processor
- added new
asyncandawaitbuiltins- they can access the outer scope
- added methods to create and destroy an execution context and a future in the VM
- added new CLI option
--astto generate JSON from the generated abstract syntax tree - added an AST to JSON compiler
- added warnings on unused functions/quotes and statements without any effect
- printing a closure will now print its fields instead of
Closure<1432> - macros are always evaluated, even when they aren't any user defined macro
argcountworks on symbols and anonymous functions
- deprecating
Value VM::resolve(const Value* val, Args&&... args)
- removed the
std::ostream& operator<<of the Value, now using the.toString(stream, vm reference) - removed the global VM lock
- removed coz and ARK_PROFILER
- running the modules tests in the CI
- new bytecode instruction
POP, removing the last value from the stack - more documentation about ArkScript and its modules
- more tests for the io builtins
- added lines and code coloration in the error context
- added documentation about the compiler implementation
- added documentation about the virtual machine
- ArkScript now supports LTO if the compiler can do it
- this is disabled in GCC 8 as this causes a runtime crash due to an ABI breakage
- fixed underline bug in the error context
- moved the frame counter of the VM to the ExecutionContext as this should be local to the context, not to the VM
- changing the way we count received arguments in arity / type errors for failed function call
- the CLI can now take a list of paths to the standard library, separated by ';'
- running the modules tests in the CI
- new bytecode instruction
POP, removing the last value from the stack - the compiler can finally optimize tail calls
- suggesting symbols to the user when the compiler encounters an unbound symbol
- the compiler can now remove unused values from the stack
- enhancing the compiler code
- removing bloat in the parser methods argument's lists
- adding an ExecutionContext to host the pointers (instruction, page, stack) and execution related structures (stack, locals, scopes), to ease the transition to a parallelized VM
- the VM can have multiple independent context running on the same bytecode
- the VM now takes a reference to an
Ark::Stateinstead of a raw non-owning pointer - adding
ARK_PROFILER_MIPSto toggle instruction per second calculation - adding new way to typecheck in builtins
- new CI build step now running valgrind to check for memory leaks
- new type checker (to be used by builtins)
- better type errors generation (with the list of arguments, if they are matching or not, and more)
- splitting Utils.hpp into multiple files for easier maintenance and contextualisation
- reserving a default scope size of 3, which yields good performance results compared to nothing being reserved
- upgrading the builtins error handling to use the
BetterTypeError - the VM now displays the debug info (ip, pp, sp) at the end of the backtrace instead of the beginning
BetterTypeErrorhas been removed in favor of a type checker using templates and an error generator
- deprecating
VM(State*)in favor ofVM(State&)
- adding support for append_in_place, concat_in_place, pop_list and pop_list_in_place in the bytecode reader
- added
page_ptr(int)in the compiler to replace&page(int) - added literals
_u8and_u16 - added table overflow detection in the compiler, to avoid creating unusable bytecode (checks if the symbols/values table is full or not)
- new Installer.iss (inno setup script) to generate a Windows installer
- new exceptions for type errors
- using
doc_formatting.first_columninstead ofdoc_formatting.start_columnwhen displaying the CLI help - brand new cmake build system
- renaming
Ark/Config.hpptoArk/Platform.hpp - refactored compiler handling of keywords
- removed
using Inst_t = uint8_tin the compiler - moved everything related to the AST in
Ark/Compiler/AST/ - moved everything related to the macros in
Ark/Compiler/Macros/ - renamed unclear file
CValuetoValTableElem - the parser is now an internal class
- the AST Optimizer was moved to
Compiler/AST - changed the ARKSCRIPT_PATH to be a collection of paths to look into, separated by
; - updating
replxxto avoid a bug when compiling with clang
- removed
ARK_SCOPE_DICHOTOMYflag so that scopes don't use dichotomy search but a linear one, since it proved to be faster on small sets of values. This goes toward prioritizing small functions, and code being cut in multiple smaller scopes - removing
download-arkscript.shfrom the repo - removed
isFraction,isInteger,isFloatfrom Ark/Utils.hpp (worked on strings and used regex) - removed
mpark::variantto use standard variant Ark::FeatureFunctionArityCheckwas removed, making arity checks mandatory
- ArkDoc documentation for the builtins
- Now using clang-format to ensure the code is correctly formatted
- the macro processor can now handle multiple macro definitions in an if-macro:
!{if true { !{a 1} !{b 2} }}is finally working
arkcommand is now marked as deprecated, in favor ofarkscript
- lists are mutated in place if they are mutable, through
append!andconcat! - instructions for
popandpop!were added, to replacelist:removeAt
list:removeAtwas deprecated
- adding of new string function for manipulation of utf8 string (str:ord and str:chr)
- utf8 support for lexer
UserType::del(), used only by the virtual machine to free memory- a new unique stack based on a
std::array<Value, ARK_STACK_SIZE>, the default stack size being 8192 - more profiling tests
- more options on the
display()method of the bytecode reader, allowing us to selecto segment of bytecode instead of displaying everything - added a new token type:
Spread; it handles...identifier, needed in macros - the parser can now handle macros definitions
- macros are being handled right after the parsing, before the AST optimizer can run
- if macros:
!{if compile-time-value then [optional else]} - values macros:
!{name value} - functions macros:
!{name (a b c ...args) body}
- if macros:
sys:platform, containing the current platform name- updated the CLI so that we can slice the bytecode when displaying it
- the bytecode reader can now display
- all the segments
- only the values segment
- only the symbols segment
- only the code segment (all of them or a given one)
- only the segments' titles and length
- verifying that we give enough arguments
- we can now import macros from other files
- undefined macros is now possible by using
!{undef macro_name} str:joinadded in the standard librarystr:splitcan now take longer separators- added
symcatin macros to concatenate a symbol and a number/string/symbol to create a new one - added
argcountin macros to count (at compile time) the number of arguments of a function - fixed a bug where
(bloc)and(print bloc), given a!{bloc value}macro, didn't give the same result (one was applied, the other was partial) - new module to manipulate bits:
bitwise - enhanced standard library
- updating
doxyfileand some docstrings - updating the download script
- enhancing examples
- creating a Scope allocates 4 pairs instead of 2, reducing the number of reallocations needed
tailOfrenamed totailandheadOftohead; no need to keep the relics of the pastheadOf(nowhead) returns the real head of a container (List or String), the first element (nil if the list is empty, "" if the string is empty)- the http module was updated to add
http:params:toListand fix thehttp:server:[method]when passing a function - fixing the compiler when we encounter get fields in lists
- updating the parser to support usually invalid constructions when they are in macros, to allow things like
!{defun (name args body) (let name (fun args body))} - updated the lexer to add UTF8 support and allow unconventional identifiers as long as they aren't keyword nor operators, so things like
->now works - fixing the code optimizer to avoid removing unused variables which are defined on function calls
- fixed the traceback generation on errors, it should now display the correct function names
- reorganizing the compiler code
- reorganizing the parser code to make it more maintainable
- adding
make_node<T>andmake_node_listinternally to avoid repetitive code - enhancing the parser
atommethod - enhancing the way we choose the subparser to use in the parser
- avoid using
std::endlif it's not useful - CI was split into multiple files to ease maintenance
- moving ArkScript tests from
tests/*.arktotests/arkscript/*.ark - fixed macros adding useless begin blocks, sometimes breaking code generation from macros
- moving std lib related tests into std/tests/
- lists are mutated in place if they are mutable, through
appendandconcat - fixed macro chaining
- fixed lexer, which wasn't adding the last token it read under some specific conditions
~UserType, since we are doing manual memory management nowFramewere removed because they were giving bad performancesfirstOfwas removed because it's basically a(@ list 0)and it was doing the job ofheadArk::Utils::toString, our internal version ofstd::to_string- use of static in the MacroProcessor and in the NodeType to string conversion function
Ark::Loggerwas removed in favor ofstd::cout/cerr+termcolor
- new submodule,
plasma-umass/coz(a profiler) - macros for profiling, enabled only if
ARK_PROFILEis defined - cmake flags using -D to turn on/off sys:exec and the coz profiler
mpark::variantis now the default used instead of the default STL variant (faster, better, stronger, and its creator is quite a god)- new cmake flag, -DARK_SCOPE_DICHOTOMY=On|Off (default Off)
- using internal only references to constants and symbols to reduce the number of useless copies of the value type
- updated standard library
- updated modules, adding hash
- updated the error handlers to avoid errors (sigsegv) when handling errors (lexing, parsing, optimization and compilation error)
- better error message at runtime when a plugin can not be found
- fixes issue #203 (imports are ill-formed when given an absolute path)
- fixes issue #205 (search for the standard library folder in more common places)
- transitioning from C++ streams to printf
- replaced the
thirdparty/folder with a git submodule inthirdparties/ - now checking that a scope doesn't have our symbol before doing a
mutoperation (in dichotomy mode it was automatically handled, but not in linear mode) - enhancing the cmake defines (
-DARK_XYZ) and the code using them - lighter Frame (from 40B to 32B), moved some unrelated logic from the frame to the virtual machine
(sys:exec)now returns the stdout output of the given command
- the parser can handle
(let|mut a b.c)(bug fix) f[ruv|no-ruv]CLI switch to control the optimizer (ruv stands for remove unused variables)- error message when we have too many parenthesis (at parse time)
- error message when using an operator not right after a
( - error message when we're capturing an unbound variable
- added
(sys:exit code)as a builtin - bytecode integrity checking through a sha256 in the header
- tests for
math:fiboandmath:divs - added the ability to give scripts arguments, through
sys:args
- the parser checks if set is given a dot expression as an identifier (which is an error)
- the parser should take in account captured variables as well, otherwise some variables are optimized while they are captured, resulting in runtime errors
- better unbound variable error message
- (implementation) every constructor with a single argument is now marked as explicit
- REPL does not need to add extra surrounding {}
- the Ark::State (re)compiles a file even if there is a bytecode version available
- the parser is now stricter and gives better error messages when we give too many/not enough arguments to a keyword
- better handling of the code given to the REPL (adds new line)
- renamed the executable from
Arktoark - now using GitHub Actions instead of Travis
- the parser can now detect when let/mut/set are fed too many arguments, and generate an error
- the compiler now handles
(set a b.c.d) - using a new plugin interface, more C-like
- class
Ark::internal::Instwhich was used as a wrapper betweenuint8_tandInstruction - worthless examples were removed
- removing
f[no-]aitapsince it wasn't used anymore in the code
- string tests
- list tests
- range tests
- unbound variable checker at compile time (won't break on plugin symbols)
list:findreturns -1 to stay consistent withstr:find- hot fix
(mut a 10) (let b 12) (set a b) (set a 11), the immutability was transferred from b to a - converting
list,appendandconcatto instructions - instructions
LISTCONCATandAPPENDadded to replace the corresponding builtins
- using a macro to define the default filename (when none is given, e.g. when loading bytecode files or from the REPL)
PLUGIN <const id>instruction to load plugin dynamically and not when the VM boots up- updated search paths for
(import "lib.ark"), looking in ./, lib/std/ and lib/ - added a case to display NOT instructions in the bytecode reader
T& as<T>()in usertype- enhanced error message when calling a non-function object
- eliminating unused global scope variables in the compiler
- adding a new feature enabled by default:
FeatureRemoveUnusedVars(not enabled for the REPL for obvious reasons) - added
replxxas a submodule - added custom destructor to the user type, called when a scope is destroyed and when we use
(del obj) - added a GVL (global virtual machine lock) to be able to use the VM in a multithreaded context
- dockerfile + specific GitHub action to build and push stable and nightly docker images, thanks to @yardenshoham
- added guards to the bytecode reader to stop reading if we're missing an entry point; now telling the user about it
- updated the string module to benefit from the new
formatmember function - updated the logger to remove
fmt/format - changed the argument order for
Ark::State - renamed the cache directory
__arkscript__ - operator
@can now handle negative indexes to get elements from the end of the given container - the standard library is now in another repository
- moved the modules to lib/ext
- the value of
CODE_SEGMENT_STARTis again 0x03 (because we removed the plugin table) - renamed
isDir?todir?for consistency - the lexer is no longer using regexes but a char after char method
- an ArkScript program is no longer a single bloc, but can be composed of multiple bloc, thus we don't need to use a single big {} or (begin) bloc for all the program
- enhancing lexer and parser error messages
- else clause in if constructions is now optional
- updating error messages in the VM
- updated the repl to add auto-completion, coloration and persistence by @PierrePharel
- moving the parser, lexer and AST node to Compiler/ from Parser/
- better import error messages at parsing
- format can now handle any value type
- updated the tests to use the new standard library, and testing every VM instruction and builtins (we didn't test everything before, this way we can be sure we don't break anything in the VM after each update)
- renaming builtins to add a namespace to them (math:, sys:, str:, list: and such)
- firstOf, tailOf and headOf now returns [] or "" instead of nil when they have nothing to do
- adding a brand new scoping system, lighter, more powerful
str:findnow returns the index where the substring was foundstr:removeAtwas fixed to throw an error when the index is strictly equal to the length of the string (can not work since accessing elements in string is 0-index based)
- removed
fmt/formatfrom our dependencies PLUGIN_TABLEwas removed to use thePLUGINinstructionnot_()from usertype- removed Parser/Utf8Converter
- member function
resolve(Args&& args...)to Value, callable by plugins to resolve the value of a function called with specific arguments given by the plugin (fill qu value)create a list ofquvalues(setListAt list at new-value)modify a list in place and return the new list value- adding UTF-8 support in programs (experimental)
- more benchmarks
- on error, the VM now display the value of each variable in the current scope
- added
thirdparty/madureira/String, to replace std::string in Ark::internal::Value which was heavy and slower than the new implementation - minimizing the size of the usertype
- UserType does not need to be given a manually defined type id but relies on
typeid(T) - performance boost of the VM by using pointers to avoid unnecessary copies
- renaming
isNaNtoNaN?,isInftoInf?for uniformization (seeempty?,nil?) - renaming CLI feature options:
-ffunction-arity-checkbecomes-ffac, same for the-fno-version-fauthorize-invalid-token-after-parenbecomes-faitap, some for the-fno-version
- improving compiler performances by using const ref when passing nodes around
- renaming the FFI "builtins" because it's not an FFI but a set of functions using the VM API
- the VM should display a backtrace even if an unknown error occurred
- transforming inline code from the vm into not inline code when possible to speed compilation, using macros instead of inline functions
- smaller value class
- smaller vm frames
- forked
madureira/Stringand modified it for the needs of the project, added it as a submodule - removed the VM pointer from the value class to make it lighter, now the VM is sending a pointer of itself to the C procedures
- removed const and type from value, now using an uint8_t to store this information
- removed NFT from the internal API to rely only on the value type
- adding
sortto sort a list - added
\t,\n,\vand\rescape codes (available in strings only) - adding
listFileswhich returns a list of paths (strings) - adding
(makeDir path)and(removeFiles ...) - added
(filter func list)inlib/Functional/Functional.ark - adding
puts. Does the same thing as print but doesn't print a newline character - added a msgpack module by @PierrePharel
- added a user type (to be defined in C++)
- adding a
notoperator - adding an http module by @SuperFola
- updated output of tests with number of passed tests
- updated REPL so that it doesn't try to compile blank lines or comments, by @rstefanic
- the option
-d|--debugis now repeatable, to set a debug level. Can go from 1 (compilation information) to 3 (a lot of information). - better precision when using
(time) - better tests presentation
- moved the modules to https://github.com/ArkScript-lang/modules
- fixing segfault when the VM receives an empty code page (generated from an empty block)
(print (fun () ()))will now printFunction @ 1instead of just its page addr,1(if true () ())now returns nil (can be generalized to() -> nil)- anonymous functions are now available ; eg:
((fun () (print "a"))) - added
forEachin library - added
-c|--compileoption to the CLI so that we can only compile an ArkScript file instead of compiling and running it, by @DontBelieveMe - added
minandmaxinlib/Math/Arithmetic.ark, by @FrenchMasterSword - added
reduceinlib/Functional/Reduce.ark, by @FrenchMasterSword - added
productinlib/List/Product.ark, by @FrenchMasterSword
- a quoted code (deferred evaluation) isn't capturing anymore surrounding variables' values, thus increasing greatly performances
- lists are printed like
["string" 1 true]now, instead of( string 1 true ) - updated
zipso that it can work with lists of different sizes, by @FrenchMasterSword - better cyclic includes detection
- better VM error message when redefining a variable through
let
- it's now possible to compare Values using
operator< reverseList(added to the FFI) by @rinz13r- a warning will now pop up during compilation if code appears to be ill-formed (e.g. number/string/list right after a
() - option
-f(allow-invalid-token-after-paren|no-invalid-token-after-paren)was added (default: OFF)
- the internal API used to compare values has been updated to be much shorter
- the REPL can take into account the lib directory, by @rstefanic
isNaNandisInfshould work on any type but return false if they aren't numbers- replacing Ark with ArkScript in source code and files (Ark being the shortname for ArkScript, as JS is the shortname for Javascript)
findInListnow returnsnilwhen the object can not be found, otherwise it returns its index in the list
cos,arccos,sin,arcsin,tan,arctanE(exp 1),Pi,Tau(2 * Pi),NaN,InfisNaN,isInfexp,ln(standard logarithm),pow,sqrtceil,round,floorzip,map,sum- REPL, made by @rstefanic
- moved the lib files in subfolders to be more organized
- re-updating the import rules on the parser side to be able to import files in subfolders from the standard library
- updating naming convention of the modules
- adding
lib/Functional.arkto store(compose f g)
- the VM now deletes all scopes except the global one in case of failure, when the persisting flag is set
- fixing plugin importation
- function arity handling in the VM, can be disabled with the option
-fno-function-arity-check sliceStrinlib/Slice.ark, taking a string, a starting index (can't be less than 0), and the length (can't be less than 1), returning a portion of the given stringreverseStrinlib/Reverse.arktaking a string and returning it in reverse
- updated the include module, which was randomly adding
/before the path of the files - dispatched the unit tests in multiple files
- now using a separated class to hold the description values for the VM
- updated assertions in
splitinlib/Split.ark(works only with single character separators) - fixing import bug
- the parser can now recognize expressions like
((f f) x) - we can now create
Ark::Valuewith floats
- the
init()internal method of the VM shouldn't stop when a bound function isn't used in the code, just ignore it - the compiler shouldn't crash on a bad code block
- better line counter in lexer
- the VM shouldn't try to run a non-existing bytecode file if the compilation fails (when calling
doFile()) VM.callshould returnnilif the stack is empty, otherwise it results in avector subscript out of rangeand that's bad- the SFML plugin was updated to run on Windows
- with the option
-L|--libwe can set the path to the ArkScript standard library - we can now load C++ lambdas into the ArkVM, as well as C++ functions
- adding function
sleep, taking a duration in milliseconds - adding function
system, taking a command from a string ; can be deleted ifARK_ENABLE_SYSTEMis set to 0 in the CMakeLists
- updating CMakeLists.txt to avoid building unuseful stuff from Google benchmark
toNumberdoesn't raise an unrecoverable error but returnsnilif the conversion failedprintno longer add a space between each element- updating the way the builtins are handled (in VM/FFI) to make it easier to add functions
docfolder, now everything is on the wiki
- should be able to compare lists
- chained operators:
(+ 1 2 3)is automatically expanded (at compile time) into(+ (+ 1 2) 3)by the compiler
- some functions playing with list should also be able to play with Strings:
headof,tailof,firstof,len,empty?,@ firstofshould segfault when the list/String is empty- fixing type of
nilto be"Nil"instead ofnilwhen using(type nil) - using uniformed names of builtins: pascal case (impacted functions are
firstOf,headOfandtailOf, as well ashasField) - fixing bug with
writeFilewhen sending a mode: the mode was also the content of the file, it no longer is
- cmake options
ARK_BUILD_EXEandARK_BUILD_BENCHMARKto choose what to build - when the VM crash, displaying stack trace
- added function
timeto the FFI (time in seconds since epoch) - adding VM.doFile
- updated the VM to be able to call functions defined in ArkScript from C++
del symsetsymtoundefined(internal value only, not theundefinedof Javascript) instead ofnil- fixed imports
- fixed automatic compilation of not-up-to-date files
- flag
-c|--compileto force compilation was not useful
- we can now call functions captured by closures, inside the scope of the closure, using the dot notation
- the CLI is checking the timestamp of the file to know if it should recompile it or not
- the CLI knows if it should recompile the given file or not
- adding
delandmutkeywords. Nowletis for settings constants andmutfor variables. Also, it isn't possible to useletto define the same constant twice google/benchmarklibrary for the benchmarks- ArkScript version section in bytecode
- timestamp (build date)
- major versions of the compiler and the virtual machine used must match, a compatibility across versions will be kept if they have the same major number
- many opcodes to handle the operators
- persist flag for the VM (if persist is false (default value), each time we call vm.run(), the frames will be reset)
- adding captures through functions arguments:
(fun (&captured std-argument) (...)) - adding closure fields reading (readonly)
- moved everything from the "folder namespaces" to a single
Ark::internalnamespace - using
#instead of'for the comments, using'to quote instead of` - the lexer is now detecting the type of the tokens it's playing with
- using
std::runtime_errors instead ofexit(1)when an error occurred - the VM should throw an error if we try to use
seton a constant - we can avoid passing all the arguments to a function, they will just be undefined
- the CLI is now able to determine if it should compile & run, run from the arkscript cache or run the file as a bytecode file
- Lexer::check, we should see if the program is correct when building the AST
- removed from the bytecode
NEW_ENV
- option in the CMakeLists.txt to use
MPIRor not (defaults to noMPIR) - information about the compilation options used for ArkScript in the CLI
- we can now use
`to quote
- using a vector instead of a map in the
Frameto speed up things - using double or
MPIRdepending on the compilation options - moving
modin the FFI - renamed methods in
Node
Defer.arkfrom the standard library- supporting both BigNum and double is a bad idea, using only double now
- removed the interpreter
- adding
switchanddefer1in the standard library (defer1shall be rewritten usingquote) - keyword
quote, macro version is` - added VM::get(name) to retrieve values from the top stack of the Virtual Machine. Types currently supported are
Ark::BigNum,std::stringandbool - adding
modin the standard library - module
sfml, need the SFML 2.5 - adding
@,and,orandheadofin the FFI - adding a guard in the VM to be sure the builtins are all present in the VM FFI
- the frames stack is handled differently, using shared pointers to avoid unnecessary copies of frame's environments, it improves execution speed by a lot
- new CLI
- handling floating pointer numbers and rational numbers
- configure.py script, to download, build and install
mpir3.0.0 - builtins functions: input, toNumber, toString
- breaking change adding
PLUGIN_TABLE_STARTwith a value of 3 in the compiler/VM - adding plugins management
- split
lib/Exceptions.arkintolib/Exceptions.arkandlib/Either.ark - renamed FindGMP FindMPIR, and we're now searching for
MPIRand linking with it - proper exception handling
- the VM shouldn't throw a runtime error if it can't link a function name and a function address
- breaking change the
CODE_SEGMENT_STARTis now equal to 4 - fixing a bug in the bytecode reader: it didn't handle
NOP importshould be able to load plugins, alsoimporttakes only one argument now- breaking change
POP_JUMP_IF_FALSEis now an absolute jump as well - upgrading CMakeLists to add
-rpathoption to the linker (with GCC), so that it still finds the lib after being installed
hastypekeyword because I never had to implement compile time typechecking, so it's not useful
- adding
importkeyword (handled by parser), throwing an error if a cyclic included is detected
- CMakeLists.txt to add
installrules: installing ArkScript inbin/and the ArkScript standard library inshare/.Ark/lib/ - updated documentation
- runtime typechecking
- exceptions (in the C++ ArkScript API)
- updated the FFI to add the runtime typechecking
- micro optimization: using numbers as variable names internally, instead of strings
- unnecessary destructors removed to let the compiler auto generate
T(T&&)(to avoid implicitly usingT(const T&))
- syntactic sugar handling in the parser
- GMP lib to handle very large number
- REPL (can be launched from the CLI)
- tests
- changed syntax: using
{...}as a(begin ...)and[...]as a(list ...) - updated documentation according the new syntax
- the lexer is now using a Token structure to store the line and column as well as the token itself
- generating the FFI using include/Ark/MakeFFI.hpp, everything defined in one file to avoid having 2 files to update
- tests
dozerg::HugeNumber, it was too slow
- test.cpp to try to embed ArkScript into a C++ project
- updated the documentation
- the compiler can now return a read only version of the bytecode being executed
- the VM can take a bytecode or a filename
- OOP test with ArkScript using closures
- closures support
- Types.hpp (for the VM) to store the definitions of the NFT (Nil/True/False enum class) and the PageAddr_t
- Function.hpp to get a lambda from the interpreter and call it from C++ code
- CMakeLists.txt, adding an option to chose between compiling main.cpp or test.cpp
- moved the VM FFI into include/Ark/VM
- beginning of the documentation
- compiler (ark code to ark bytecode)
- bytecode reader (human readable format)
dozerg::HugeNumberto handle big numbers- simple VM handling all the instructions, able to run an ark bytecode
- interpreter and VM FFI
- logger
- Node (to represent an AST node and a Node in the language)
- Environment to map variables and values
- Program executing ArkScript code from the AST
- standard library (builtin functions)
- Lexer and parser
- default CLI can handle the interpreter
- tests
- utils to play with files
- utils to play with strings and numbers
- default CLI (using clipp)
- CMakeLists to compile the project
ryjen::formatto format strings