Performs is designed to visualize and synthesize humanoid animations for customized avatars. It supports two types of animations: keyframe animations (glTF, BVH) and script animations (SiGML, BML). The second mode integrates a variety of BML (Behavior Markup Language) instructions for Manual Features (MF) and Non-Manual Features (NMF) to create cohesive animations, performed in real-time. The project has made significant progress in synthesizing NMF, which is essential for enhancing the realism of sign language animations. This initiative began as part of the SignON project, a user-centric and community-driven effort aimed at facilitating communication among Deaf, hard of hearing, and hearing individuals across Europe.
The application allows users to customize the following features:
- Avatar:
- Character
- Cloth color
- Background:
- Space: Open space, Studio, Photocall
- Color
- Photocall image
- Illumination:
- Light position
- Light color
It can be easily integrated into other applications by inserting an HTML iframe or by including the js libary.
Insert Performs inside your application using the iframe HTML element.
<iframe src='https://performs.gti.upf.edu'>Important
You can customize the default settings combining the available options with multiple parameters by concatenating them with &.
So for example showing a custom avatar in a blue chroma would look like that:
<iframe src='https://performs.gti.upf.edu?avatar=https://models.readyplayer.me/67162be7608ab3c0a85ceb2d.glb&background=studio&color=rgb(54,54,190)'>By default, Performs generates the code for creating a fully functional web application with its built-in GUI. But there are two versions of performs modules:
performs.module.js: Performs with a default GUI. Requires importing lexgui.js, treejs and include style.css.
performs.nogui.module.js: Performs without GUI. It only requires importing treejs. If you want to develop your own GUI, see Performs API for useful functions.
index.html
<!DOCTYPE html>
<html>
<head>
<title>My app using Performs</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="icon" type="image/x-icon" href="./data/imgs/favicon.png">
<link rel="stylesheet" href="https://cdn.skypack.dev/lexgui@0.7.6/build/lexgui.css">
<link rel="stylesheet" href="./style.css">
<script async src="https://unpkg.com/es-module-shims@0.10.7/dist/es-module-shims.js"></script>
<script type="importmap">
{
"imports": {
"three": "../external/three/build/three.module.js",
"three/addons/": "../external/three/jsm/",
"lexgui": "https://cdn.skypack.dev/lexgui@0.7.10/build/lexgui.module.js",
"lexgui/extensions/": "https://cdn.skypack.dev/lexgui@0.7.10/build/extensions/"
}
}
</script>
</head>
<body>
<script type="module" src="main.js"></script>
</body>
</html>main.js
import { Performs } from './performs.module.js';
const performs = new Performs();
const customOptions = {restrictView: false};
performs.init( customOptions );You can load the customization features through the GUI, by loading a JSON file or by appending URL parameters. It can also be passed as a JSON object, calling functions like init(options) and setConfiguration(options). For a complete list of available options, please refer to Performs Settings.
Important
To load the default avatar, an internet connection is required. If you prefer to work offline, you can change the modelToLoad in the init() method of Performs.js. You can use the resources from the /data folder or add your own.
The following sections provide examples of how to initialize the application without a GUI or configure it to initialize with a specific mode as the default. You can also check the demos: Demo with default GUI or Demo with custom GUI.
This mode visualizes clip animations across different avatars. To ensure effective retargeting, certain options need to be adjusted for the source animation and the target avatar:
-
Embedded transforms: Retargeting only takes into account the transforms from the actual bone objects (local transforms). If set to true, external (parent) transforms are computed and embedded into the root joint (only once, on construction). Afterwards, parent transforms/matrices can be safely modified and will not affect in retargeting. Useful when it is easier to modify the container of the skeleton rather than the actual skeleton in order to align source and target poses.
trueorfalse
-
Reference pose mode: Determines which pose will be used as the retargeting bind pose.
DEFAULT: Uses skeleton's actual bind poseCURRENT: Uses skeleton's current poseTPOSE: Forces the bind pose to be a T-pose and sets CURRENT mode.
A detailed explanation of these options can be found in the retargeting repository.
Important
Currently only .gltf, .glb, .bvh and .bvhe (bvh extended to support facial intensities) are supported. If you happen to use another format, please convert it to the mentioned formats.
main.js
By default, Keyframe mode is enabled. However, if a configuration file for Script mode is detected, the mode will automatically switch. When multiple animations are loaded, a crossfade blending is applied during transitions between animations. The blend time can be adjusted using the blendTime attribute of keyframeApp.
import { PERFORMS } from 'performs.module.js'
const performs = new PERFORMS.Performs();
// Set customization options
const options = {
srcEmbeddedTransforms: true,
trgEmbeddedTransforms: true,
srcReferencePose: 0, // DEFAULT
trgReferencePose: 2, // TPOSE
autoplay: true,
animations = [{name: './data/animations/myAnimation.glb', data: null}], // Set keyframe animation files' URL. 'data' can be an already parsed file
onReady = () => { // Function called after loading the application
// Change to Keyframe mode
performs.changeMode(PERFORMS.Modes.KEYFRAME);
// Play the animation after 1s
setTimeout(() => performs.changePlayState(true), 1000);
}
}
// Init performs with the options
performs.init(options);]() This mode allows for the generation of procedural animations based on instructions. The system interprets the instructions based in SiGML (Signing Gesture Markup Language), and translates them into an extended version of BML (Behavior Markup Language), or directly into BML in JSON format. While the results may not match the quality of keyframe animations, this approach requires neither expensive equipment nor extensive manual labor, making it highly suitable for scalable sign synthesis. These instructions can be written manually or generated and exported using Animics' script mode, which enables the creation and editing of this type of animation through a visual timeline-clip representation.
The current supported instructions are explained in detail in BML Instructions. An example:
{
type: "faceLexeme",
start: 0.1,
attackPeak: 0.6,
relax: 1.5,
end: 1.8,
amount: 0.1,
lexeme: "ARCH"
}main.js
Include ./data/dictionaries/ folder in your project to use the Idle animation and the NGT gloss dictionary.
import { PERFORMS } from 'performs.module.js'
const performs = new PERFORMS.Performs();
const options = {
config = './data/configFile.json', // Set required config file for Script mode
scripts = [ // Set script instruction files' URL and/or parsed data
{
type: 'sigml',
name: './data/animations/scriptAnimation.sigml'
},
{
type: 'bml',
data: {
behaviours: [{
type: "faceLexeme",
start: 4,
attackPeak: 0.6,
relax: 1.5,
end: 1.8,
amount: 1,
lexeme: "ARCH"
}]
}
},
],
applyIdle: true, // Enable Idle animation to blend with recieved instructions
onReady = () => { // Function called after loading the application
setTimeout(() => {
// Play the animation after 1s
performs.changePlayState(true);
}, 1000)
}
};
// Init performs with the options
performs.init(options);Clone the repository:
git clone https://github.com/upf-gti/performs.gitTo run locally, host a server from the main project folder.
To modify the code, you should change the source files and then build it to get the module updated.
To build you must run the following command:
npm run buildImportant
You may need to install the rollup package.
Some examples on simple NGT (Dutch Sign Language)
Kind lezen boek (child reads a book):
Man rijden fiets (Man rides a bicycle) :
Important
Currently only .gltf and .glb are supported. If you happen to use another format, please convert it to the mentioned formats.
To add a new avatar to Performs, you must follow this steps:
- Make sure the avatar is rigged (if it is not rigged, we recommend using Mixamo)
- Check that your avatar has the correct scale and orientation.
- Use the performs-atelier tool to configure all the parameters needed for the application (only required for Script mode), this will generate a configuration .json file containing all the needed information.
- Jaume Pozo @japopra
- Eva Valls @evallsg
- Carolina del Corral @carolinadcf
- Alex Rodríguez @jxarco
- Víctor Ubieto @victorubieto
- Pablo García @PZerua
- Three.js - An open-source JavaScript library for creating interactive 3D and 2D graphics in web browsers.
- Lexgui.js - A simple and lightweight GUI library for creating graphical user interfaces for web applications.
We would like to extend our gratitude to the creators and maintainers of these libraries for their invaluable contributions to the open-source community.
This project is being developed with partial financial support of:
| EMERALD Project (2023-2026) | SignON Project (2021-2023) |
|---|---|
 |
 |