Introduction

Beaming Scene Service

A key step towards software/hardware integration in Beaming is the data communication between modules. One of the requirements is the interfacing of heterogeneous systems (sensors on Linux/iOS/Windows/OSX). Interfaces range from external services (e.g. SecondLife), embedded systems (e.g. Arduino) to specific hardware (e.g. robots). The Beaming Scene Service aims to create a framework for managing data in a loosely-coupled heterogeneous environment.

A client-server replicated shared object database approach is adopted, supported using a centralised-server model. Every process in a Beaming session (clients) a copy of those shared objects when they connect to the server. Every process can edit a shared object, changes are transparently copied to all other processes. Each process can publish information simply by creating a shared object. All slow, or limited volume data is published in this database (providing single transport, easy logging) while all time-critical, high volume data use native streams (but configuration of those streams is stored in the database).

How to connect to the Beaming Scene Service

All client processes are expected to load the netClient library. The following describes the function calls and transmission of data types that occur during a typical session. See client.h for a more detailed description of the parameters. SampleClient VC++ project and SampleXVRClient XVR projects are thin clients you can use as a guide.

1. Start netServer on a fast machine with a fast Internet connection.

2. Using startclient(...) function, client connects to server IP address, server port and specifies a client name, client type(e.g. VISITOR, LOCAL, SPECTATOR, ROBOT, HAPTIC_DEVICE, etc), configuration file, a flag to denote whether it is a unidirectional viewer (read access) or wants to create and update objects (read\write access). You are also able to change the packet priority, reliability and serialization frequency.

3. The check() function was implemented to continually receive incoming packets. Put in a continuous loop (e.g. onTimer for XVR, Idle for glut). The server regularly pings all clients to ensure they are still live. The function also checks for new connections to the server and gets the packet data.

4. If you've started your client with write access, then you would be able to create objects, relating for example to avatar joints using addRocketBoxAvatar(...) function. Clients can add multiple avatars. Clients should also setAvatarName.

5. A client (avatar type) updates each joint with a starting value containing x, y, z position and rx, ry, rz, and rw rotation using updateRocketBoxAvatar(...) function. The RocketBox avatars have 123 joints.

NOTE: Each client type would update nodes using the appropriate function. Client (type emotion) uses updateEmotionNodes.

IMPORTANT: All AVATARS adhere to the Rocketbox standard as agreed during the development of the initial theatre application.

6. Other clients connect in the same manner as described above and are able to share data with your client. (i.e. they are engaged in a BEAMING session).

7. To read data from other remotely connected clients(see client.h and the sampleclient C++ and XVR projects for further info):

Using getPeersID/getPeersInfo, getNodesInfo and GetAvatarData/GetAvatarSpecificData functions(preferred method)

NOTE: Each client type would read data using the appropriate function. To read emotion data from a client of emotion type, call GetEmotionData and GetPeerSpecificEmotionData.

8. Your client disconnects by removing all owned objects and publishing the event to the server:

void removeAllNodes();

void stop();

9. Other clients are notified of this disconnection, and destroy any locally-stored data structures pertaining to your client. Additionally, if a client process does not respond in a timely manner, eventually the server process will delete the objects that the client created.

Data Types

Version 1.0 of the netManager application [1] catered for avatar joints data (i.e. transmission of 3D positions and orientations). This worked well for the theatre demo. Version 2.0 supports the Beam-enabling process through a variety of replicated objects and multiple data types. Hence 8 more data types have been created.

Based on the visitor-destination paradigm, the data types fall into two categories: (i) user/visitor (typically represented by an avatar, robot, etc) and a room/destination (typically represented by video, audio, point cloud, etc).

Beaming User

Avatar Skeleton (BeamingAvatarJointReplica)

- Represents a 'Visitor' or 'Local'
- Based on Rocketbox avatars: 123 joints
- Pelvis joint is relative to world-frame, others are relative to the pelvis/parent joint
- Each joint has a unique node in Raknet
- 7 floats per node (3 translational, 4 quaternions).

ID	X	Y	Z	AX	AY	AZ	AW
1	0.87	0.98	1.2	0.6	0.44	0.32	0.88
...	...	...	...	...	...	...	...
123	3.12	-1.38	9.7	6.2	-2.24	5.30	9.38

Facial Expression (BeamingFacialReplica)

- Requires different data depending on avatar or robot representation.
- Therefore needs higher-level abstraction than the joint rotations used for driving the skeleton
- Float/boolean attributes, each corresponding to a particular expression or mouth shape

ID	Blink	Smile	Frown	'o'	'e'	'p'
avatar_face	true	0.5	0.1	0.25	0.0	0.75

Affective State (BeamingEmotionReplica)

Real time affective state recognition from EEG, GSR & HRV:

- Valence: [-1.0 - 1.0]
- Arousal: [-1.0 - 1.0]
- Three double data types

ID	Valence	Arousal	misc
affect_state	0.5	-0.5	...

Tactile Data (BeamingTactileReplica)

Technion implementation:

- Raknet shared object for vibration:
o Duration of vibration: time in milliseconds
o Intensity of vibration: [0(low) - 1(high)]
- Raknet shared object for temperature:
o Temperature [0 (low) - 1(high)]
- 3 doubles

ID	Duration	Intensity	Temperature
tactile	125	0.5	0.8

Kali-Type Robot (BeamingRobotReplica)

Data provided by intention recognition module:

- Type of interaction (human-human(=0), human-object (=1))
- Details of interaction, e.g. object type

o Details contained in database (eventually coded by numbers)

- Free space/contact
- Location of interaction in global coordinate system
- Expected remaining time till interaction in seconds
- Type of interaction in contact: handshaking, touching, stroking, lifting, etc.

ID	Type	Details	Free space/contact	X	Y	Z	AX	AY	AZ	AW	Remaining time	Contact interaction
kali1	1	35	1	0.87	0.98	1.2	0.6	0.44	0.32	0.88	5.3	3

Generic Data (BeamingGenericReplica)

As the needs of different partners might change over time, these data type allows sending of generic data (e.g. pointer to a struct in memory - see sampleClient):

- Any data: [sent as void pointer]
- size: [1024 Bytes]

ID	Any data	size (in bytes)
generic_id	e.g. struct, class, etc.	1024 bytes (limit)

Beaming Room

All replicas in the Room category have a host and port for dynamic scenes and a file_url for static scenes (in addition to the data types specified below).

3D Objects (BeamingObjectReplica)

- Represents dynamic objects in the scenes such as meshes, point cloud data, etc.
- 7 floats and one string

ID	X	Y	Z	AX	AY	AZ	host	port	file_url
object1	0.87	0.98	1.2	0.6	0.44	0.32	127.0.0.1	80	http://.../

Video Source (BeamingVideoReplica)

- Although video data is sent through separate network channels, information about the video source is published on the Beaming Scene Service.
- This includes a file_url for static scenes
- 3 integers, 1 double and 2 strings

ID	width	height	bandwidth	host	port	file_url
video1	640	480	100	127.0.0.1	80	http://../

Audio Source (BeamingAudioReplica)

- Although audio data is also sent through separate network channels, information about the audio server is published on the Beaming Scene Service.
- 1 integer and 3 strings

ID	host	port	file_url	config
audio1	127.0.0.1	80	http://.../	...

Point Cloud (BeamingPointCloudReplica)

- Although point cloud data is sent through separate network channels, information about the source of the point cloud is published on the Beaming Scene Service.
- This includes a file_url for static scenes
- Size represents the number of points
- 2 integers, 1 double, 1 boolean and 2 strings

ID	size	bandwidth	RGB_flag	host	port	file_url
video1	10	100	true/false	127.0.0.1	80	http://../

[1] See netManager directory on the Beaming svn (http://svn.beaming-eu.org)

See sampleClient.cpp for an example client

NetServer

Project: Beaming Server Application

Description: This application act as server in the Client-Server architecture. It handles link to RakNet. 
Logfile recording is also handled on the server side.

Related projects: ReplicaManager3 is used for object creation, destruction, and automatic serialization

NetClient

Project: Network Client Library 

Description: This DLL/LIB connects any client application to RakNet (see sampleClient and sampleXVRclient for how to connect). 
Comments in client.h provides more indepth explanations. The client (sampleClient) acts as a thin client, in that 
it simply connects to the server and simply sends and recieves data from server.

Dependencies: RakNet Static Library project included in the same visual studio solution file. See external_libs directoy on the svn.

Installation

INTRODUCTION
------------
This application deals with the network transmission layer of WP2 using RakNet middleware. 
Object sharing between clients is managed by RakNet's ReplicaManager3.


Linux Instructions
------------------
Follow the build instructions under ../external_libs/raknet to compile the raknet libraries using "cmake ."

When raknet is finished, run linux_build script ("./linux_build") which builds the client libraries, sampleclient 
and server applications.

To test client-server, run server by typing "netServer/server" on a terminal and run one or more of "netClient/sampleclient" 
on a seperate terminal.


Windows Instructions:
--------------------
Open netManager.sln in VC2005 and Build solution

To test, run server.exe and one or more of sampleClient.exe

Mac OS X Instructions
------------------
How to install Raknet, libclient on OsX.

-RakNet

NOTE: Use preferably GNU gcc to compile. to do so, set an environmental var before each CMAKE call:
CC=gcc CXX=g++ cmake (...)

NOTE2: There's no __int32 on mac, on $RAKNET/Source/NativeTypes.h add
#if defined(__APPLE__)
typedef int __int32;
#endif

1) cd into the raknet root directory (external_libs/rakent). cd into the Source dir, 
    and change NativeTypes.h with NativeTypes_OSX.h (by physically renaming the files).
2) if you've cmake GUI, run it and disable the dependencies. Otherwise type
	cmake .
3) cmake will tell you that the dependencies can't be found. Please do run again cmake 
    with the flag to disable them (cmake will tell you the flag)
	cmake -FLAG_CMAKE_SUGGESTS .
4) type make
5) make will fail on about 75%. We don't care, we already have the static library. 
    Is in Lib/LibStatic/libRakNetStatic.a
6) copy it in /usr/lib/
	sudo cp Lib/LibStatic/libRakNetStatic.a /usr/lib/
7) you are done.

-LibClient

1) cd into netManager
2) run osx_build
	./osx_build
3) you should get everything working fine. If not, check the paths in osx_build.
4) copy libclient (netClent/libclient.a) into /usr/lib/
	sudo cp netClent/libclient.a /usr/lib/
5) you are done.

Now you can build the app. Make sure you've Qt installed!