Skip to main content

Boomy Comprehensive Usage Document

I. About the Boomy​

(I) Component Information​

info

The Boomy machine has 21 degrees of freedom. One arm has 4 degrees of freedom, including (left-handed coordinate system):

  1. Shoulder joint pitch
  2. Shoulder joint roll
  3. Upper arm joint yaw
  4. Elbow joint pitch

One leg has 6 degrees of freedom, including:

  1. Hip joint pitch
  2. Hip joint roll
  3. Thigh joint yaw
  4. Knee joint pitch
  5. Ankle joint pitch
  6. Ankle joint roll

The lumbar joint has one degree of freedom, including:

  1. Lumbar joint yaw

Having multiple joint motor degrees of freedom enables the Boomy robot to achieve precise motion and attitude control.

The detailed information is as follows:

ParameterBoomyBoomy-Education Edition
Height, width, and thickness (standing)97cm*35cm*20cm
Weight including batteryAbout 20.5kg
Total degrees of freedom21
Single leg degree of freedom6
Single arm degree of freedom4
Waist degree of freedom1
Computing power module/NVIDIA Jetson Orin NX
Joint communication methodEtherCAT & CAN FD
Maximum joint torque70 N·m
Lower-Leg and Thigh LengthAbout 513mm
Arm SpanAbout 379mm
Power supply methodLithium battery
Emergency stop functionSoftware emergency stop & hardware emergency stop
Basic computing power6 TOPS6 + 117 TOPS
Camera modelDC_V93Realsense D435I
Perception sensorsIMU, camera
Microphone arrayStandard configuration
SpeakerStandard configuration
Voice interaction modelChatGPT
Voice controlSupport
Intelligent batteryQuick-release battery
Image transmissionWithin the app

During development, hardware can be directly accessed on the computing board

ChargerYes
Mobile APPYes
OTA updateYes
Battery life2~3H
Secondary development (HighController)Yes
Secondary development (LowController)/Yes
Development languagepython C++ Graphical Programming
Component Introduction DiagramComponent Introduction DiagramComponent Introduction Diagram - Side

(II) Appearance Dimensions​

Appearance Dimensions

Battery and charger dock
Dock Charger
Power adapter
Battery Charger

(III) Material Composition​

Material Composition

(IV) Electrical Interface​

On the back of Boomy, the following electrical interfaces are provided:

Electrical Interface

Serial numberTypeExplanationQuantity
1RJ45Switch 1000 BASE-T1
2USBMotion Control Board/Computing Board USB 3.02
3HDMIHigh definition display cable for Motion Control Board/Compute Board1
tip

The back interface of Boomy with Computing Board is the interface of the Computing Board, while Boomy without Computing Board has the back interface of the Motion Control Board

(V) Airborne Computer​

TypeModelUserPasswordIPComputing powerFrequencyMemory
Motion Control BoardRockchip RK3576The Motion Control Board is not open to users192.168.55.1026 TOPS2.4GHz8+64GB
Secondary development Computing BoardNVIDIA Jetson Orin NXnoetixBoomy192.168.55.101117 TOPS2.0GHz16+128GB
danger

The Motion Control Board (Rockchip RK3576) is not open to users. If users access the Motion Control Board without authorization, causing the motion control program to lose control and resulting in losses, the user shall bear the losses themselves

(VI) Camera Information​

Boomy-EDU: Realsense D435i

Intel® RealSense™ Depth Camera D435i - Product Specifications | Intel

Boomy Standard: DC_V93

Boomy Standard: DC_V93 parameters

(VII) Battery Information​

ParameterInformation
BrandGrepow
ModelGRP6752097-13S1P
Capacity5100mAh
Voltage48V
Weight1360g

(VIII) Charger Specifications and Parameters​

ParameterInformation
Product ModelTA300 Robot
Input100-240V~50/60Hz 4.5A
Battery TypeLiPo/LiFe/LiHV 13S-16S
Size172*97*46mm
Product NameSmart charger
Output35-72V ⎓ 7.5A 350W Max
Work ModeCharge
Weight510g±10g

II. Instructions for Use​

(I) Startup Equipment and Precautions​

info
  1. Equipment removal and placement

  2. Remove Boomy from the storage position.

  3. Hold it steady and make it sit on the ground.

  4. Battery installation

  5. Insert the battery into the Boomy back battery compartment.

  6. Confirm that the battery is installed in place (snap lock).

  7. Equipment startup

  8. Press the battery switch briefly once to confirm that the battery indicator light is on.

  9. Then press and hold the battery switch until the flowing light effect appears.

  10. Wait for the robot to complete startup.

  11. Initialize attitude adjustment

  12. Place Boomy face-up flat on the ground, make sure the ground is flat and non-slip, there are no foreign objects under the Boomy soles or around it, so as to avoid the Boomy automatic stand-up being affected by surrounding objects and failing

  13. System initialization instructions

  14. After turning on, Boomy will automatically enter full-body enable mode:

    • All joints will automatically move to their zero positions
    • Do not interfere with or obstruct joint movement during the process

⚠️ Precautions

  • Avoid touching joints during startup to prevent pinching or interference with initialization
  • Ensure the ground is level to avoid abnormal posture
  • Do not power on the robot unless the battery is installed correctly
  • When powering on, ensure that the emergency stop button is in the released (popped-up) position. During use, do not press the emergency stop button except in an emergency.

(II) Boomy Connection Method​

Boomy connection method

danger

After connecting to Boomy, do not select 'Forget Device' in the system settings, otherwise the Bluetooth record will be cleared, causing the device to be unable to connect again. If there is any misoperation, please contact customer support.

(III) Connecting to and Controlling Boomy with the Mobile App​

NOETIX Run App Download:

Android:

Android link

iOS: NOETIX Run App - App Store

NOETIX Run App usage instructions: NOETIX Run App usage instructions

First use app connect Boomy: NOETIX Run App User Guide (Boomy)

(IV) Emergency Stop Function​

danger

This product supports two emergency-stop methods: the emergency-stop button on the Boomy device body and the emergency-stop control in the App:

  1. Device body emergency stop button After pressing the button, the device will be immediately powered off and the whole machine will collapse to the ground. Do not press it accidentally during normal operation to prevent the device from falling and being damaged; in case of an accident, please press it immediately to avoid further losses.

  2. App emergency stop (global floating button) The button is hidden by default in the sidebar; tap the sidebar to display it. After pressing, the robot will not be powered off, it will only go limp all over. Do not touch it by mistake during normal use to avoid the device falling and being damaged; in case of an accident, please press it immediately to avoid further losses.

App emergency stop process

App emergency stop process 1 App emergency stop process 2 App emergency stop process 3

Emergency stop button

Emergency stop button

(V) Device Shutdown and Battery Charging​

info
  1. Equipment storage preparation

a. Perform the one-click storage operation in the App. b. After the storage is completed, hold Boomy steady and make it sit on the ground.

  1. Device shutdown

c. Press and hold the battery switch until Boomy is completely turned off.

  1. Battery disassembly

d. Press and hold the battery clip. e. Remove the battery** from the Boomy back battery compartment.

  1. Battery charging

f. Place the battery on the battery base and ensure it is securely connected. g. Connect the charging adapter to the battery base. h. Insert the adapter into a 220V power supply.

  1. Charging status confirmation

i. Observe the battery indicator light:

  • Indicator light flashing: Charging in progress
  • No indication: The battery is fully charged

(VI) Tutorial Videos​

1. Boomy Unboxing Video​

III. Boomy Development Guide​

(I) Boomy Supports the Following Development Interfaces​

  • DDS: An SDK encapsulated from the DDS interface, supporting python and C++
    • More lightweight, integrated and encapsulated into simple and easy-to-understand function interfaces
  • ROS2: Topics encapsulated and integrated from ROS2, exposed for external access
    • More suitable for research laboratory scenarios, offering hardcore, professional-grade ROS2 topic access

noetix_ros2_boomy: https://github.com/Noetix-Robotics/noetix_ros2_boomy

(II)Boomy URDF File​

BoomyURDF.zip

(III) Boomy-DDS-SDK​

github link:

GitHub - Noetix-Robotics/noetix_sdk_boomy: This is SDK for Noetix robot Boomy with DDS interface.

tip

SDK Introduction

This document introduces Boomy's communication development based on the DDS message middleware. This SDK adopts a C/S architecture. Under an isolation mechanism, it enables the DDS client program deployed on the secondary-development Compute Board of the Boomy robot and the motion control program on the Motion Control Board, which acts as the server, to issue control commands at a frequency of 500Hz and continuously obtain the current robot status. The DDS used is CYCLONEDDS 0.11.0.

1. System Architecture Diagram​

System Architecture Diagram

Under the isolation mechanism between the Motion Control Board and the Compute Board, the DDS interface is exposed for the Compute Board to access and develop against.

2. Application Development: HighController​

tip

Introduction to Application Development

HighController performs upper-layer scheduling development based on the OEM's motion control strategy. Without changing the underlying control logic, it enables the rapid construction of robot automation applications and supports integration with Foundation Models to create customized intelligent behavior systems.

Typical applications include:

  • Combining vision algorithms with the OEM motion control system to achieve autonomous perception and autonomous movement
  • Integrating a voice interaction module to enable natural language control
  • Integrating Foundation Models to dynamically generate and invoke OEM control actions based on input (this entails certain risks and requires strict safety policies)
  • Working in conjunction with an externally mounted perception radar to implement a positioning, navigation, and inspection system

In short, through high-level scheduling and encapsulation of the OEM's control capabilities, HighController enables automated and programmatic invocation of behaviors controllable via a remote controller or App, and on this basis extends certain autonomous and intelligent capabilities.

(1) HighController Architecture Diagram​

HighController Architecture Diagram

a. State Reporting Mechanism

Once the client SDK has established a connection with the robot's motion control program, the motion control program will continuously push real-time robot status information to the client, including:

  • Full-body motor parameter data (pos: position, vel: velocity, tau: torque, kp, kd, temperature, error: error information)
  • IMU status data
  • Current working mode (workmode, indicating the control strategy currently in operation)
  • AOLION controller data (used to synchronize manual input or auxiliary control)
  • Battery level and related system status information

b. Client Control Command Issuance

Based on its own application scenarios, the client sends control commands to the motion control program, mainly including:

  • (x, y, z) movement control data

    Valid only in specific modes, such as walking mode, handshake, greeting, cheering, teaching mode, etc.

  • action command

    Corresponds to workmode and is used to switch or trigger the algorithm strategy currently running on the robot.

  • data: additional DDS data

    Usually used in PLAYTEACH (playback teaching mode) to specify additional parameters such as the teaching file index.


c. Internal Processing Flow of the Motion Control Program

After the motion control program receives a command from the client:

  • Pass the control parameters to the internal control algorithm
  • The algorithm performs policy computation based on a reinforcement learning model
  • Generate the motor target setpoints (target position, velocity, or torque) and send them to the motor control interface
  • Send the computation results down to the hardware control layer

d. Hardware Execution Layer

After the hardware layer receives the target control data:

  • Execute the motor control commands
  • Complete the corresponding desired action

Throughout the entire process, users only need to issue some high-level control parameters.

The specific motion planning, dynamic balancing, and motor-level control computations are all handled by the black-box algorithms in the OEM motion control program.

Therefore, under HighController mode, the range of capabilities that users can invoke and automate does not, in principle, exceed the functional boundary originally achievable via the App or controller, but underlying motor-level control permissions are not exposed.

(2) HighController Operation Flowchart​

HighController Operation Flowchart

a. Initialization Phase (init)

Upon system startup, the following initialization procedure is performed:

  1. Establish a communication connection between the client and the robot's motion control program
  2. Register a callback function for receiving robot status data

b. The DDS callback thread continuously receives the returned status

The motion control program continuously pushes robot status data to the client.

After receiving the data, the callback function encapsulates it and stores it separately in the buffer and in the global state variables:

  • motor_state_buffer_: motor state buffer
  • imu_buffer_: IMU state buffer
  • joy_buffer_: controller data buffer
  • curmode: current motion control strategy mode
  • BmsState: current battery information
(3) HighController Common Interfaces and Parameters Table​
HighController Interface
Interface NameInterface DescriptionReturn Type
instanceGet the HighController singleton pointerObject Pointer
initHighController Object InitializationNone
publish_cmdPublish DDS control command dataNone
subscribe_robot_hardware_statusSubscribe to robot hardware status informationNone

Get the HighController singleton object

Function Nameinstance
Function Prototypestatic HighController *Instance()
Function OverviewGet the HighController singleton
ParametersNone
Return ValueHighController singleton object pointer
Remarks

C++ Example:

HighController *ctrl = HighController::Instance();

Python Example:

ctrl = HighController.instance()

HighController Object Initialization

Function Nameinit
Function Prototypebool init()
Function OverviewHighController Object Initialization
ParametersNone
Return Valuebool
Remarks

C++ Example:

ctrl->init();

Python Example:

ctrl.init()

Publish robot control commands

Function Namepublish_cmd
Function Prototype

void publish_cmd(double x, double y, double z, ControlCmd action, uint16_t index)

Function OverviewUse HighController to send DDS control commands
Parameters
  • x range: [-1,1]; robot longitudinal movement value (>0 means forward)

  • y range: [-1,1]; robot lateral movement value (>0 means move left)

  • z range: [-1,1]; robot turning value (>0 means turn left)
  • action: robot mode-switching enum
  • index: index specified for teaching-related modes
Return ValueNone
Remarks

This interface must not be called continuously; otherwise, the robot may receive commands too quickly to respond. A delay of at least 2 ms must be added after each command is sent

C++ Example:

ctrl->publish_cmd(0, 0, 0, ControlCmd.CHEER, 0);

Python Example:

ctrl.publish_cmd(0, 0, 0, ControlCmd.CHEER, 0)

The action parameter indicates switching the robot to the corresponding mode

Parameter NameDefine PrototypeSerial No.Example ValueDescription
ControlCmd
enum class ControlCmd : uint32_t {
WALK,
SWING,
SHAKE,
CHEER,
FULLWALK,
START,
SWITCH,
STARTTEACH,
SAVETEACH,
ENDTEACH,
PLAYTEACH,
DANCE0,
FALLTOSTAND,
STANDTOFALL,
DANCE1,
DANCE2,
ZHOUJI,
DEFAULT,
};
0WALKWalking Mode (Half-Body Walking)
1SWINGFactory-Preset Action: Wave
2SHAKEFactory-Preset Action: Handshake
3CHEERFactory-Preset Action: Cheer
4FULLWALKFull-Body Walking
5STARTEnable/Disable
6SWITCHPreparation Mode
7STARTTEACHStart Teaching
8SAVETEACHSave Teaching
9ENDTEACHEnd Teaching
10PLAYTEACHPlay Demonstration
11DANCE0Dance 1 Mode (what is love)
12FALLTOSTANDStand Up
13STANDTOFALLStow
14DANCE1Dance 2 Mode (jueshi)
15DANCE2Dance 3 Mode (gee)
16ZHOUJIElbow Strike
17DEFAULTNull Mode
tip

Purpose of Using DEFAULT (Empty Mode)

The entire robot thread executes the process control function in a periodic loop. The control function is executed periodically in a loop. If teaching-related commands such as

action = ControlCmd::PLAYTEACH; are continuously issued within this loop,

the robot may repeatedly interrupt the teaching playback action before it has finished, causing the robot to appear "stuck and unresponsive."

Correct Approach

PLAYTEACH is a one-shot trigger command (event) and should be sent only once; in subsequent loop iterations, send DEFAULT (an empty command) to avoid repeated triggering.

Example Code (Corrected)

if (is_first) {
action = ControlCmd::PLAYTEACH; // Trigger only once
is_first = false;
} else {
action = ControlCmd::DEFAULT;
}

Effect

  • Upon receiving PLAYTEACH → the teaching action is triggered and executed to completion
  • Subsequent DEFAULT → no longer interrupts the action
  • The robot behaves normally and predictably

Action-type commands should be "edge-triggered" and must not be continuously issued within the control loop.

tip

START Enable/Disable

workmode == 30, send START to put the robot into enabled mode

workmode != 30, send START to immediately put the robot into disabled mode

For switching between modes, please follow the state machine below

Boomy State Machine

Boomy State Machine

index is used to specify the teaching file index in teaching-related modes

ctrl.publish_cmd(0, 0, 0, ControlCmd.STARTTEACH, 0) # Start teaching
sleep(10) # Record the teaching action
ctrl.publish_cmd(0, 0, 0, ControlCmd.SAVETEACH, 1) # End and save the teaching action, specifying index 1
sleep(2) # Allow the motion control program to finish the save process to prevent it from being interrupted by the next command
ctrl.publish_cmd(0, 0, 0, ControlCmd.PLAYTEACH, 1) # Play teaching file 1
tip

The parameters x and yaw take effect only when action is one of the following modes: PLAYTEACH/WALK/DEFAULT

and are used to control robot movement; in other modes, x and yaw have no effect.

Subscribe to robot hardware status information

Function Namesubscribe_robot_hardware_status
Function Prototypevoid subscribe_robot_hardware_status(RobotHardwareStatusCallback callback);
Function OverviewSubscribe to robot hardware status information
Parameters- callback: Callback function for processing robot status data
Return ValueNone

| Remarks | |

C++ Example:

ctrl->subscribe_robot_hardware_status(
[](const RobotHardwareStatus &status) {
latest_hw_status = status;
});

Python Example:

def _on_hw_status(status):
global latest_hw_status
latest_hw_status = status

ctrl.subscribe_robot_hardware_status(_on_hw_status)
Parameter NameDefine PrototypeParametersDescription
RobotHardwareStatus
struct RobotHardwareStatus {
NingImuData imu_data;
joydata remote_data;
std::array<MotorState, 21> motor_data;
RobotBmsData bms_data;
int workmode;
};
imu_dataIMU data
remote_dataController data
motor_dataMotor data
bms_dataBattery data
workmodeCurrent working mode

workmode parameter mapping table

The workmode value is valid only in HighController mode

ValueMode Description
0Enable Mode
1Preparation Mode
2Half-Body Walking Mode
3Full-Body Walking Mode
4Run Mode
5Dance Mode 0
8Factory Preset Action: Greeting
9Factory Default Action: Handshake
10Factory Default Action: Cheer
11Start Teaching
12End Teaching
13Play Demonstration
14Save Teaching
23Play Demonstration
26Protection Mode
27Get up from the ground
28Get up and fall to the ground
29Save Demonstrated Motion
30Disabled Mode
31Dance Mode 1
32Dance Mode 2
33Elbow Strike Motion
tip

Save Teaching
After the SDK sends SAVETEACH to the robot, the robot will successively return three statuses: Exit Teaching, Save Teaching 1, and Save Teaching 2

caution

Protection Mode

When the robot is under standing control strategies (walking, teaching, dancing, etc.), the motion control program continuously monitors IMU attitude data.

If the attitude is detected to deviate from the stable standing range and exceed the preset threshold, the system will immediately trigger the protection mechanism and forcibly switch to safe mode.

In protection mode:

  • All joints enter the disabled state (the motors stop active output)
  • The current motion control strategy is terminated
  • Further action commands are blocked from execution

The purpose of this mechanism is to:

  • Prevent the robot from continuously outputting control torque during a fall or loss of stability
  • Avoid secondary damage caused by uncontrolled swinging of the limbs
  • Reduce potential damage to the device itself and the surrounding environment

This protection logic is an underlying safety protection mechanism with the highest priority and is not affected by upper-layer control commands.

Controller Data Structure

Parameter NameDefine PrototypeExample ValueDescription
joydata
struct joydata {
double axes[2];
int button[14];
};
axesRemote controller joystick linear value
buttonRemote controller button press mapping values: 1 = pressed, 0 = normal
Controller Joystick MappingMapping DescriptionRange
axes[0]Right Joystick Horizontal Value[-1, 1]
axes[1]Left Joystick Vertical Value[-1, 1]
tip

For a Boomy equipped with a Compute Board, the rear interface is the Compute Board interface. The Motion Control Board cannot accept a controller receiver, so DDS cannot obtain controller data; therefore, controller data must be obtained directly on the Compute Board. For details, please see the compute board directly obtains controller data

IMU State Data Structure

Parameter NameDefine PrototypeExample ValueDescription
NingImuData
struct NingImuData {
double ori[4];
double ori_cov[9];
double angular_vel[3];
double angular_vel_cov[9];
double linear_acc[3];
double linear_acc_cov[9];
};
oriAttitude Quaternion
ori_covAttitude Covariance
angular_velAngular Velocity
angular_vel_covAngular Velocity Covariance
linear_accLinear Acceleration
linear_acc_covLinear Acceleration Covariance

Motor State Data Structure

Parameter NameDefine PrototypeExample ValueDescription
MotorState
struct MotorState {
double pos;
double vel;
double tau;
uint16_t motor_id;
uint8_t error;
int temperature;
};
posMotor Position
velMotor Speed
tauMotor Torque
motor_idMotor ID
error

Motor Error Codes:


02H: Motor overcurrent


03H: Motor undervoltage


04H: Motor encoder error


06H: Motor brake overvoltage


07H: DRV driver error


08H: Overvoltage


09H: Undervoltage


0AH: Overcurrent


0BH: MOS overtemperature


0CH: Motor coil overtemperature


0DH: Communication loss


0EH: Overload

temperatureTemperature

Battery State Data Structure

Parameter NameDefine PrototypeExample ValueDescription
RobotBmsData
struct RobotBmsData {
uint16_t battery_temp_ = 0;
uint16_t battery_alarm_ = 0;
uint16_t battery_soc_ = 0;
uint16_t battery_soh_ = 0;
};
battery_temp_Temperature
battery_alarm_Alarm Information
battery_soc_Battery Level
battery_soh_Battery Health
(4) Motion Control State Transition Flowchart​

Motion Control State Transition Flowchart

danger

Note: During development with HighController, please follow the above flowchart for mode switching to ensure proper and safe operation

(5) Demo Program Introduction​

The examples/cpp/example_high.cpp and examples/py/test_high.py files in the SDK folder provide a sample demo for parsing AOLION controller input data and generating corresponding control commands based on controller operations to send to the robot.

3. Motion Control Development: LowController​

tip

Motion Control Development Introduction

LowController exposes the motor control interfaces in the motion control program to the SDK, allowing users to directly adjust low-level motor parameters at a control frequency of 500 Hz. Combined with the deployment of reinforcement learning algorithms, this enables deep customization of Boomy's motion capabilities, allowing it to perform diverse new actions and complex behaviors such as dances.

caution

Note: Only the EDU version of Boomy supports LowController development

(1) Boomy Training Framework​

Coming soon!

(2) LowController Framework Diagram​

LowController Framework Diagram

a. Status Data Reporting

Once the client SDK has established a connection with the robot's motion control program, the motion control program will continuously push real-time robot status information to the client, including:

  • Full-body motor parameter data (pos: position, vel: velocity, tau: torque, kp, kd, temperature, error: error information)
  • IMU status data
  • AOLION controller data

b. Client-side control computation

The client uses its own algorithms (typically a reinforcement learning model) to perform real-time policy computation, generate target setpoints for each motor (such as target position), and write the motor control command array into the buffer before sending it to the motion control program.


c. Motion Control Program Forwarding Mechanism

After receiving the motor command array sent by the client, the motion control program does not perform high-level policy computation; instead, it directly forwards the control parameters to the hardware control layer.


d. Hardware Execution Layer

After receiving the motor target parameters, the hardware layer executes the control commands and drives the motors to complete the desired actions.


In LowController mode:

  • The reinforcement learning model is deployed on the client side
  • Users can directly participate in motor-level control
  • The motion control program mainly handles communication and safe forwarding
  • The control frequency can reach 500Hz

This mode provides higher control freedom and flexibility, but it also means:

  • Higher requirements for algorithm stability
  • Stricter requirements for real-time performance and system safety
  • If the control strategy is unstable, it may directly affect robot motion safety
(3) LowController Operation Flowchart​

LowController Operation Flowchart

a. Initialization Phase (init)

Upon system startup, the following initialization steps are performed:

  1. Establish a communication connection between the client and the robot's motion control program
  2. Register a callback function for receiving robot status data
  3. Create the send_thread data sending thread

b. Status Data Callback Mechanism

The motion control program continuously pushes robot status data to the client.

After receiving the data, the callback function encapsulates it and stores it in the corresponding buffer or global variable:

  • motor_state_buffer_: motor state buffer
  • imu_buffer_: IMU state buffer
  • joy_buffer_: controller data buffer

This phase is only responsible for data reception and caching, and does not participate in control policy computation.


c. send_thread (Data Sending Thread)

The data sending thread also runs at a 2 ms period:

  • If motor_cmd_buffer_ is not empty
  • Read the latest motor control command
  • Send the motor target array to the motion control program

The motion control program is only responsible for forwarding data to the hardware layer and does not participate in high-level control decisions.


d. Architecture Features

The core features of LowController mode are:

  • The client directly participates in motor-level control
  • Reinforcement learning or custom control algorithms run on the client
  • The motion control program no longer handles high-level policy computation and only handles communication and safe forwarding
  • The control frequency can reach 500Hz

This mode provides higher control freedom and algorithm plasticity, but at the same time places higher demands on:

  • Real-time performance
  • Algorithm stability
  • Thread safety and scheduling accuracy
(4) Joint Motor Information​
IDNameJointAngle RangeRadian Range
0arm_l1_jointLeft arm shoulder joint pitch-135135-2.362.36
1arm_l2_jointLeft arm shoulder joint roll-8111-0.141.94
2arm_l3_jointLeft arm joint yaw-9090-1.571.57
3arm_l4_jointLeft arm elbow joint pitch-1290-2.260.00
4leg_l1_jointLeft leg hip joint pitch-120120-2.092.09
5leg_l2_jointLeft leg hip joint roll-3890-0.661.57
6leg_l3_jointLeft thigh joint yaw-145145-2.532.53
7leg_l4_jointLeft leg knee joint pitch01280.002.24
8leg_l5_jointLeft leg ankle joint pitch-5525-0.960.44
9leg_l6_jointLeft leg ankle joint roll-1010-0.170.17
10arm_r1_jointRight arm shoulder joint pitch-135135-2.362.36
11arm_r2_jointRight arm shoulder joint roll-1118-1.940.14
12arm_r3_jointRight arm joint yaw-9090-1.571.57
13arm_r4_jointRight arm elbow joint pitch-1290-2.260.00
14leg_r1_jointRight leg hip joint pitch-120120-2.092.09
15leg_r2_jointRight leg hip joint roll-9038-1.570.66
16leg_r3_jointRight thigh joint yaw-145145-2.532.53
17leg_r4_jointRight leg knee joint pitch01280.002.24
18leg_r5_jointRight leg ankle joint pitch-5525-0.960.44
19leg_r6_jointRight leg ankle joint roll-1010-0.170.17
20waist_1_jointWaist joint yaw-9090-1.571.57

MotorCmd.pos corresponds to radian information

(5) AOLION Controller Data Mapping​

AOLION controller. The button[] and button mapping table is as follows:

Button DescriptionButton index
Right shoulder button1
Left shoulder button2
X5
Y6
B7
A8
+9
-10
None11
Left joystick press12
Controller Joystick MappingMapping Description
axes[0]Right joystick Y-axis value
axes[1]Left joystick X-axis value

Controller Joystick Data Range

MinimumMaximum
-11
(6) Commonly Used LowController Interfaces and Usage Tutorial​
LowController Interface
Interface NameInterface DescriptionReturn Type
instanceGet the LowController singleton pointerObject pointer
initInitialize the LowController objectNone
set_jointWrite motor control commands into the bufferNone
getJointsIndexGet the corresponding ID by joint nameint
subscribe_robot_hardware_statusSubscribe to robot hardware status informationNone

Get the LowController singleton object

Function Nameinstance
Function Prototypestatic LowController *Instance()
Function OverviewGet the LowController singleton
ParametersNone
Return ValueLowController singleton object pointer
Remarks

C++ Example:

LowController *ctrl = LowController::Instance();

Python Example:

ctrl = LowController.instance()

LowController Object Initialization

Function Nameinit
Function Prototypebool init()
Function OverviewInitialize the LowController object
ParametersNone
Return Valuebool
Remarks

C++ Example:

ctrl->init();

Python Example:

ctrl.init()

Store motor control command array into buffer

After being stored in the buffer, it will be automatically sent by the send_thread thread.

Function Nameset_joint
Function Prototypevoid set_joint(std::array<MotorCmd, 21> motorcmd)
Function OverviewSet joint data
Parametersmotorcmd: motor command structure
Return ValueNone
Remarks
Parameter NameDefinition PrototypeExample ValueDescription
MotorCmd
struct MotorCmd {
double pos;
double vel;
double tau;
double kp;
double kd;
uint16_t motor_id;
};
posMotor position
velMotor speed
tauMotor torque
kpPID parameter
kdPID parameter
motor_idMotor ID
danger

When issuing motor control commands, strictly refer to the specified parameter ranges in.

C++ Example:

std::array<MotorCmd, 21> motorcmd;
// Algorithm logic omitted ...
ctrl->set_joint(motorcmd);

Python Example:

cmds = [MotorCmd() for _ in range(21)]
# Algorithm logic omitted ...
ctrl.set_joint(cmds)

Get the corresponding joint motor ID by joint motor name

Function NamegetJointsIndex
Function Prototypeint getJointsIndex(std::string jointname)
Function OverviewGet the corresponding joint motor ID by joint motor name
Parametersjointname: joint motor name
Return Valueint: corresponding joint motor ID
Remarks

C++ Example:

int id = ctrl->getJointsIndex("arm_l1_joint");

Python Example:

id = ctrl.getJointsIndex("arm_l1_joint")

Subscribe to robot hardware status information

Function Namesubscribe_robot_hardware_status
Function Prototypevoid subscribe_robot_hardware_status(RobotHardwareStatusCallback callback);
Function OverviewSubscribe to robot hardware status information
Parameters- callback: callback function for processing robot status data
Return ValueNone
RemarksUsage is the same as HighController subscribe_robot_hardware_status
Obtain AOLION controller data (obtained directly on the compute board for Boomy with a compute board)​

AoLionDriver class

AoLionDriver Class Interface
initInitialize
getremotedataObtain controller data
Function Nameinit
Function Prototypebool init(std::string port, int baudrate)
Function OverviewInitialize the controller device port
Parameters
  • port: device port number, default /dev/input/js0
  • baudrate: baud rate, default 115200
Return Value

true: initialization succeeded


false: initialization failed

Remarks
Function Namegetremotedata
Function Prototypeconst joydata getremotedata()
Function OverviewObtain controller data
ParametersNone
Return Valuejoydata: controller data structure
Remarks

Usage Example

C++

AoLionDriver aoliondriver;
joydata remote_data;
aoliondriver.init("/dev/input/js0", 115200);
const joydata jdata=aoliondriver.getremotedata();
for (int i = 0; i < 14; i++)
std::cout << jdata.button[i] << std::endl;
std::cout << jdata.axes[0] << std::endl;
std::cout << jdata.axes[1] << std::endl;

Python

al = AoLionDriver()
al.init("/dev/input/js0", 115200)
jdata = al.getremotedata()
for i in range(14):
print(jdata.button[i])
print(jdata.axes[0])
print(jdata.axes[1])

4. Audio and Video Development​

tip

Introduction to Audio Development

MediaController establishes a communication link with the voice module on the Motion Control Board via DDS topics, enabling control of Boomy audio peripherals (such as speakers and microphones), and providing capabilities for voice interaction agent configuration and data stream management.

caution

Note Only the EDU version of Boomy supports MediaController development
The open-source interfaces are as follows:

Status control and acquisition interface: supports wakeup control, sleep control, module restart, operating status acquisition, and error information acquisition;

General configuration interface: an interface for configuring and retrieving general parameters, including: non-dialogue sleep timeout duration, wake-up word, wake-up reply word, sleep reply word, as well as audio-visual data stream control;

Audio interface: audio stream push and retrieval, audio control commands (pause/resume, volume adjustment, etc.)

Video interface: video stream push and retrieval, video control commands (pause/resume, etc.)

(1) MediaController Architecture Diagram​

MediaController Architecture Diagram

MediaController Architecture Diagram 2

a. Media service status and data reporting

After the client SDK establishes a connection with the robot media service program, the media service program will continuously push real-time robot media status information to the client, including:

  1. Continuous acquisition of internal audio and video data streams (only Boomy models without a Compute Board can acquire internal video streams)
  2. Partial agent configuration reporting (actively obtained for the first time, and reported after subsequent changes)
  3. Acquisition of desensitized processed video stream (the version equipped with a Compute Board needs to continuously publish external video streams to the audio-visual interaction system)

b. Delivery of media data streams and configuration data

  1. Publish external audio stream data to the agent as input, or send it to the internal speaker for output
  2. Publish external video stream data for agent recognition
  3. Modify some agent configuration settings, including: volume, timeout, wake word, wake-up response, sleep response, and the data stream routing switch, etc.

The input and output data for audio stream capture currently only support: channels=8, sample_rate=16000, format=2 (16-bit little-endian)

Audio stream capture info 1

The input and output data for audio stream playback currently only support: channels=2, sample_rate=16000, format=2 (16-bit little-endian)

Audio stream capture info 2

The captured data and playback data are interleaved

(2) MediaController Startup Flowchart​

MediaController Startup Flowchart

(3) Common Parameters of MediaController​

DDS Packet Header Structure

Structure NameDefine PrototypeMemberDescription
Header
struct Header {
uint64_t message_id = 0;
uint64_t timestamp_us = 0;
std::string sn;
};
message_idDDS Message ID
timestamp_usus timestamp
snThe current SN number of the Boomy device

System Status Control

SystemControlType
EnumerationDescription
TO_WAKEUPWake up the audio-visual interaction system agent
TO_SLEEPPut the audio-visual interaction system agent to sleep
TO_RESETRestart the audio-visual interaction system agent

System Status

Parameter NameDefine PrototypeMemberDescription
SystemStatus
struct SystemStatus {
Header header;
WorkStatus value = WorkStatus::READY;
StatusChangeReason reason = StatusChangeReason::SYSTEM_LAUNCH;
};
headerDDS Header
valueagent status
reasonStatus Reason
WorkStatus
EnumerationDescription
READYReady State
SLEEPEDSleep State
WAKEUPEDAwake State
EXITExit Status
StatusChangeReason
EnumerationDescription
SYSTEM_LAUNCHSystem Startup
CMD_RESETCommand Reboot
AUDIO_WAKEUPEDVoice Wake-up
CMD_WAKEUPEDCommand Wake-up
AUDIO_SLEEPEDVoice Sleep
CMD_SLEEPEDCommand Sleep
TIMEOUT_SLEEPEDTimeout Hibernation
ERROR_SLEEPEDError Sleep

Voice wake-up refers to triggering through actual human-computer voice interaction; command wake-up refers to triggering through DDS or other forms of commands;

System Error

Parameter NameDefine PrototypeMemberDescription
SystemError
struct SystemError {
Header header;
int32_t code = 0;
std::string message;
};
headerDDS Header
codeError Code
messageError Message

Audio stream data

Parameter NameDefine PrototypeExample ValueDescription
AudioStream
struct AudioStream {
Header header;
uint64_t timestamp_us = 0;
uint32_t channels = 0;
uint32_t sample_rate = 0;
uint32_t format = 0;
uint32_t duration_ms = 0;
std::vector<int16_t> audio_data;
};
headerDDS Header
timestamp_usTimestamp (us)
channelsNumber of channels
sample_rateSampling rate
formatData format
duration_msTime (ms)
audio_dataAudio data

Video stream data

Parameter NameDefine PrototypeExample ValueDescription
VideoStream
struct VideoStream {
Header header;
uint64_t timestamp_us = 0;
uint32_t format = 0;
uint32_t width = 0;
uint32_t height = 0;
uint32_t fps = 0;
std::vector<uint8_t> video_data;
};
headerDDS Header
timestamp_usTimestamp (us)
format

Format:


1: YUYV


Currently, only the YUYV format is supported

widthHorizontal pixels per frame
heightVertical pixels per frame
fpsFrame rate
video_dataFrame image data
(4) Common Interfaces of MediaController​
MediaController Interface
InstanceObtain the singleton pointer of MediaControllerObject pointer
initMediaController Object Initializationbool
set_system_controlSet the control status of the voice interaction systemNone
subscribe_system_statusSubscribe to status information of the audio-visual interaction systemNone
subscribe_system_errorSubscribe to error status information of the audio-visual interaction systemNone
get_volumeGet the current volume of the audio-visual interaction system Range: 0~200int
set_volumeSet the volume of the current audio-visual interaction system Range: 0~200None
get_timeoutGet the inactivity timeout before entering sleep modeint
set_timeoutSet the inactivity timeout before entering sleep modeNone
get_audio_cue_enableGet the enabled status of the prompt tone of the audio-visual interaction systembool
set_audio_cue_enableSet whether the prompt sound of the audio-visual interaction system is enabledNone
get_internal_capture_audio_data_to_agent_enableGet the enabled status of internal microphone audio stream input to the agent
bool
set_internal_capture_audio_data_to_agent_enableSet the enabled status of internal microphone audio stream input to the agent
None
get_external_custom_audio_data_to_agent_enableGet the enabled status of external audio stream input for the agentbool
set_external_custom_audio_data_to_agent_enableSet the enabled status of external audio stream input for the agentNone
get_internal_agent_audio_data_to_playback_enableGet the enabled status of agent audio stream output to the internal speakerbool
set_internal_agent_audio_data_to_playback_enableSet the enabled status of agent audio stream output to the internal speakerNone
get_external_custom_audio_data_to_playback_enableGet the enabled status of external audio stream output to the internal speakerbool
set_external_custom_audio_data_to_playback_enableSet the enabled status of external audio stream output to the internal speakerNone
get_internal_capture_video_data_to_agent_enableGet the enabled status of internal video stream input to the agentbool
set_internal_capture_video_data_to_agent_enableSet the enabled status of internal video stream input to the agentNone
get_external_custom_video_data_to_agent_enableGet the enabled status of external video stream input to the agentbool
set_external_custom_video_data_to_agent_enableSet the enabled status of external video stream input to the agentNone
get_external_custom_audio_data_to_agent_use_internal_3aGet the enabled status of using the internal 3A algorithm for external audio stream input to the agentbool
set_external_custom_audio_data_to_agent_use_internal_3aSet the enabled status of using the internal 3A algorithm for external audio stream input to the agentNone
get_wakeup_response_wordsGet wake-up response wordsstring
set_wakeup_response_wordsSet wake-up response wordsNone
get_sleep_response_wordsGet dormant response wordsstring
set_sleep_response_wordsSet sleep response wordsNone
get_wakeup_wordsGet all wake wordsstring
add_wakeup_wordsAdd wake-up wordsbool
reset_wakeup_wordsReset factory default wake-up wordsNone
subscribe_internal_audio_captureSubscribe to the audio stream captured by the internal microphoneNone
subscribe_internal_audio_playbackSubscribe to the audio stream played by the robot's internal speakerNone
subscribe_internal_video_captureSubscribe to the video stream collected by the camera inside the robot (only available for Boomy without Compute Board)VideoStream struct
subscribe_internal_video_desensedSubscribe to the desensitized video stream processed by the audio-visual interaction system (the Boomy equipped with a Compute Board needs to continuously publish the external video stream)VideoStream struct
pause_video_capturePause the internal camera data acquisition of the robot (only available for Boomy without Compute Board)None
resume_video_captureResume the internal camera data acquisition of the robot (only available for Boomy without Compute Board)None
publish_external_video_streamPublish external video streams to the audio-visual interaction systemNone
publish_external_audio_streamPublish external audio stream input to the agentNone
publish_external_audio_playback_streamPublish external audio stream output to the internal speakerNone
pause_audio_capturePause internal microphone audio stream captureNone
resume_audio_captureResume internal microphone audio stream captureNone
pause_audio_playbackPause the audio stream output of the internal speakerNone
resume_audio_playbackResume the audio stream output of the internal speakerNone
danger

The call interval of all set agent configuration interfaces (such as changing the volume, changing data switches, etc.) must not be less than 500 ms; otherwise, it may cause the voice interaction system to crash. If this problem occurs, please contact Songyan after-sales support

Get the singleton of the MediaController object

Function Nameinstance
Function Prototypestatic MediaController *Instance()
Function OverviewGet the MediaController singleton
ParametersNone
Return ValueSingleton pointer to the MediaController object
Remarks

C++ Example:

MediaController *ctrl = MediaController::Instance();

Python Example:

ctrl = MediaController.instance()

MediaController Object Initialization

Function Nameinit
Function Prototypebool init()
Function OverviewMediaController Object Initialization
ParametersNone
Return Valuebool
Remarks

C++ Example:

ctrl->init();

Python Example:

ctrl.init()

Set the status of the audio-visual interaction system

Function Nameset_system_control
Function Prototypevoid set_system_control(media::SystemControlType type, bool is_need_reply);
Function OverviewSet the status of the audio-visual interaction system (wake-up, sleep, restart)
Parameters
- type: system status
- is_need_reply: whether to trigger the wake-up/sleep reply words
Return ValueNone
Remarks

C++ Example:

ctrl->set_system_control(media::SystemControlType::TO_WAKEUP, true);

Python Example:

ctrl.set_system_control(media::SystemControlType::TO_WAKEUP, true)

Subscribe to the status information of the audio-visual interaction system

Function Namesubscribe_system_status
Function Prototypevoid subscribe_system_status(SystemstatusCallback callback);
Function OverviewSubscribe to status information of the audio-visual interaction system
Parameters- callback: the callback function that needs to be triggered when the state changes
Return ValueNone
Remarks

C++ Example:

ctrl->subscribe_system_status([&](const media::SystemStatus &status){
printf("status is %d \r\n" , static_cast<int>(status.value));
});

Python Example:

def on_status_switch(status):
print(status.value)

ctrl.subscribe_system_status(on_status_switch)

Subscribe to system error messages of the audio-visual interaction system

Function Namesubscribe_system_error
Function Prototypevoid subscribe_system_error(SystemerrorCallback callback);
Function OverviewSubscribe to system error messages of the audio-visual interaction system
Parameters- callback: the callback triggered by a DDS message containing error information
Return ValueNone
Remarks

C++ Example:

ctrl->subscribe_system_error([&](const media::SystemError &error){
printf("status is %s \r\n", error.message.c_str());
});

Python Example:

def on_error(error):
print(error.message)

ctrl.subscribe_system_error(on_error)

Get the current volume

Function Nameget_volume
Function Prototypeint get_volume()
Function OverviewGet the current volume of the audio-visual interaction system
ParametersNone
Return Valueint, volume value, range: 0~200
Remarks

C++ Example:

int volume = ctrl->get_volume();
std::cout << volume << std::endl;

Python Example:

volume = ctrl.get_volume()
print(volume)

Set the current volume

Function Nameset_volume
Function Prototypevoid set_volume(int volume)
Function OverviewSet the volume of the current audio-visual interaction system
Parametersvolume: volume value, range: 0~200
Return ValueNone
Remarks

C++ Example:

ctrl->set_volume(100);

Python Example:

ctrl.set_volume(100)

Get the inactivity timeout before entering sleep

Function Nameget_timeout
Function Prototypeint get_timeout()
Function OverviewGet the inactivity timeout before entering sleep
ParametersNone
Return Valueint, timeout (ms)
Remarks

C++ Example:

int timeout = ctrl->get_timeout();
std::cout << timeout << std::endl;

Python Example:

timeout = ctrl.get_timeout()
print(timeout)

Set the inactivity timeout before entering sleep

Function Nameset_timeout
Function Prototypevoid set_timeout(int timeout)
Function OverviewSet the inactivity timeout before entering sleep
Parameterstimeout: timeout duration (ms)
Return ValueNone
Remarks

C++ Example:

ctrl->set_timeout(3000);

Python Example:

ctrl.set_timeout(3000)

Check whether the prompt tone is enabled

Function Nameget_audio_cue_enable
Function Prototypebool get_audio_cue_enable()
Function OverviewGet the enabled status of the prompt tone of the audio-visual interaction system
ParametersNone
Return Valuebool
Remarks

C++ Example:

bool enabled = ctrl->get_audio_cue_enable();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_audio_cue_enable()
print(enabled)

Set whether the prompt tone is enabled

Function Nameset_audio_cue_enable
Function Prototypevoid set_audio_cue_enable(bool enable)
Function OverviewSet whether the prompt sound of the audio-visual interaction system is enabled
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_audio_cue_enable(true);

Python Example:

ctrl.set_audio_cue_enable(True)

Get the enabled status of internal microphone audio stream input to the agent

Function Nameget_internal_capture_audio_data_to_agent_enable
Function Prototypebool get_internal_capture_audio_data_to_agent_enable()
Function OverviewGet the enabled status of internal microphone audio stream input to the agent
ParametersNone
Return Valuebool
Remarks

C++ Example:

bool enabled = ctrl->get_internal_capture_audio_data_to_agent_enable();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_internal_capture_audio_data_to_agent_enable()
print(enabled)

Set the enabled status of internal microphone audio stream input to the agent

Function Nameset_internal_capture_audio_data_to_agent_enable
Function Prototypevoid set_internal_capture_audio_data_to_agent_enable(bool enable)
Function OverviewSet whether the internal microphone is enabled to collect audio stream and input it to the agent
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_internal_capture_audio_data_to_agent_enable(true);

Python Example:

ctrl.set_internal_capture_audio_data_to_agent_enable(True)

Get the enabled status of external audio stream input to the agent

Function Nameget_external_custom_audio_data_to_agent_enable
Function Prototypebool get_external_custom_audio_data_to_agent_enable()
Function OverviewGet the enabled status of external audio stream input for the agent
ParametersNone
Return Valuebool: true indicates enabled, false indicates disabled
Remarks

C++ Example:

bool enabled = ctrl->get_external_custom_audio_data_to_agent_enable();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_external_custom_audio_data_to_agent_enable()
print(enabled)

Set the enabled status of external audio stream input to the agent

Function Nameset_external_custom_audio_data_to_agent_enable
Function Prototypevoid set_external_custom_audio_data_to_agent_enable(bool enable)
Function OverviewSet whether the external audio stream input to the agent is enabled
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_external_custom_audio_data_to_agent_enable(true);

Python Example:

ctrl.set_external_custom_audio_data_to_agent_enable(True)

Get the enabled status of agent audio stream output to the internal speaker

Function Nameget_internal_agent_audio_data_to_playback_enable
Function Prototypebool get_internal_agent_audio_data_to_playback_enable()
Function OverviewGet whether the agent audio stream output to the internal speaker is enabled
ParametersNone
Return Valuebool: true indicates enabled, false indicates disabled
Remarks

C++ Example:

bool enabled = ctrl->get_internal_agent_audio_data_to_playback_enable();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_internal_agent_audio_data_to_playback_enable()
print(enabled)

Set the enabled status of agent audio stream output to the internal speaker

Function Nameset_internal_agent_audio_data_to_playback_enable
Function Prototypevoid set_internal_agent_audio_data_to_playback_enable(bool enable)
Function OverviewSet whether the agent audio stream output to the internal speaker is enabled
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_internal_agent_audio_data_to_playback_enable(true);

Python Example:

ctrl.set_internal_agent_audio_data_to_playback_enable(True)

Get the enabled status of external audio stream output to the internal speaker

Function Nameget_external_custom_audio_data_to_playback_enable
Function Prototypebool get_external_custom_audio_data_to_playback_enable()
Function OverviewGet the enabled status of outputting external audio stream to the internal speaker
ParametersNone
Return Valuebool: true to enable, false to disable
Remarks

C++ Example:

bool enabled = ctrl->get_external_custom_audio_data_to_playback_enable();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_external_custom_audio_data_to_playback_enable()
print(enabled)

Set the enabled status of external audio stream output to the internal speaker

Function Nameset_external_custom_audio_data_to_playback_enable
Function Prototypevoid set_external_custom_audio_data_to_playback_enable(bool enable)
Function OverviewSet whether the external audio stream is output to the internal speaker
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_external_custom_audio_data_to_playback_enable(true);

Python Example:

ctrl.set_external_custom_audio_data_to_playback_enable(True)

Get the enabled status of internal video stream input to the agent

Function Nameget_internal_capture_video_data_to_agent_enable
Function Prototypebool get_internal_capture_video_data_to_agent_enable()
Function OverviewGet the enabled status of internal video stream input to the agent
ParametersNone
Return Valuebool: true indicates enabled, false indicates disabled
Remarks

C++ Example:

bool enabled = ctrl->get_internal_capture_video_data_to_agent_enable();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_internal_capture_video_data_to_agent_enable()
print(enabled)

Set the enabled status of internal video stream input to the agent

Function Nameset_internal_capture_video_data_to_agent_enable
Function Prototypevoid set_internal_capture_video_data_to_agent_enable(bool enable)
Function OverviewSet whether to enable the internal video stream input for the agent
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_internal_capture_video_data_to_agent_enable(true);

Python Example:

ctrl.set_internal_capture_video_data_to_agent_enable(True)

Get the enabled status of external video stream input to the agent

Function Nameget_external_custom_video_data_to_agent_enable
Function Prototypebool get_external_custom_video_data_to_agent_enable()
Function OverviewGet the enabled status of external video stream input to the agent
ParametersNone
Return Valuebool: true indicates enabled, false indicates disabled
Remarks

C++ Example:

bool enabled = ctrl->get_external_custom_video_data_to_agent_enable();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_external_custom_video_data_to_agent_enable()
print(enabled)

Set the enabled status of external video stream input to the agent

Function Nameset_external_custom_video_data_to_agent_enable
Function Prototypevoid set_external_custom_video_data_to_agent_enable(bool enable)
Function OverviewSet whether the external video stream input to the agent is enabled
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_external_custom_video_data_to_agent_enable(true);

Python Example:

ctrl.set_external_custom_video_data_to_agent_enable(True)

Get the enabled status of using the internal 3A algorithm for external audio stream input to the agent

Function Nameget_external_custom_audio_data_to_agent_use_internal_3a
Function Prototypebool get_external_custom_audio_data_to_agent_use_internal_3a()
Function OverviewGet the status of whether the agent enables the internal 3A algorithm when receiving external audio stream input
ParametersNone
Return Valuebool: true indicates enabled, false indicates disabled
Remarks

C++ Example:

bool enabled = ctrl->get_external_custom_audio_data_to_agent_use_internal_3a();
std::cout << (enabled ? "true" : "false") << std::endl;

Python Example:

enabled = ctrl.get_external_custom_audio_data_to_agent_use_internal_3a()
print(enabled)

Set the enabled status of using the internal 3A algorithm for external audio stream input to the agent

Function Nameset_external_custom_audio_data_to_agent_use_internal_3a
Function Prototypevoid set_external_custom_audio_data_to_agent_use_internal_3a(bool enable)
Function OverviewSet whether the external audio stream input to the agent uses the internal 3A algorithm
Parametersenable: true on, false off
Return ValueNone
Remarks

C++ Example:

ctrl->set_external_custom_audio_data_to_agent_use_internal_3a(true);

Python Example:

ctrl.set_external_custom_audio_data_to_agent_use_internal_3a(True)

Get wake-up response words

Function Nameget_wakeup_response_words
Function Prototypestring get_wakeup_response_words()
Function OverviewGet wake-up response words
ParametersNone
Return Valuestring, the current wake-up reply word
Remarks

C++ Example:

std::string words = ctrl->get_wakeup_response_words();
std::cout << words << std::endl;

Python Example:

words = ctrl.get_wakeup_response_words()
print(words)

Set wake-up response words

Function Nameset_wakeup_response_words
Function Prototypevoid set_wakeup_response_words(string words)
Function OverviewSet wake-up response words
Parameterswords: content of the wake-up reply word
Return ValueNone
Remarks

C++ Example:

ctrl->set_wakeup_response_words("I'am");

Python Example:

ctrl.set_wakeup_response_words("I'am")

Get dormant response words

Function Nameget_sleep_response_words
Function Prototypestring get_sleep_response_words()
Function OverviewGet dormant response words
ParametersNone
Return Valuestring, the current dormant response word
Remarks

C++ Example:

std::string words = ctrl->get_sleep_response_words();
std::cout << words << std::endl;

Python Example:

words = ctrl.get_sleep_response_words()
print(words)

Set sleep response words

Function Nameset_sleep_response_words
Function Prototypevoid set_sleep_response_words(string words)
Function OverviewSet up auto-reply messages for idle status
Parameterswords: content of the auto-reply for inactivity
Return ValueNone
Remarks

C++ Example:

ctrl->set_sleep_response_words("ok");

Python Example:

ctrl.set_sleep_response_words("ok")

Get all wake-up words

Function Nameget_wakeup_words
Function Prototypestring get_wakeup_words()
Function OverviewGet all wake-up words
ParametersNone
Return ValueString, all current wake words
Remarks

C++ Example:

std::string words = ctrl->get_wakeup_words();
std::cout << words << std::endl;

Python Example:

words = ctrl.get_wakeup_words()
print(words)

Add wake-up words

Function Nameadd_wakeup_words
Function Prototypebool add_wakeup_words(const std::string &words);
Function OverviewAdd a new wake word
Parameters- words: new wake word string
Return Valuebool: whether the addition via internal Regular Expression succeeded
RemarksRegular Expression: (^(((([a-z]+) )?([^\s]+) ))+@(.+)$)
Example: B UW1 M IY1 @boomy

C++ Example:

if (ctrl->add_wakeup_words("B UW1 M IY1 @boomy"))
std::cout << "add succeed" << std::endl;
else
std::cout << "add fail" << std::endl;

Python Example:

if ctrl.add_wakeup_words("B UW1 M IY1 @boomy"):
print("add succeed")
else:
print("add fail")

Reset wake-up words

Function Namereset_wakeup_words
Function Prototypevoid reset_wakeup_words();
Function OverviewReset default factory wake-up words
ParametersNone
Return ValueNone
Remarks

C++ Example:

ctrl->reset_wakeup_words();

Python Example:

ctrl.reset_wakeup_words()

Subscribe to the audio stream collected by the robot's internal microphone

Function Namesubscribe_internal_audio_capture
Function Prototypevoid subscribe_internal_audio_capture(AudiostreamCallback callback);
Function OverviewSubscribe to the audio stream collected by the robot's internal microphone
Parameters- callback: callback function for processing audio stream data
Return ValueNone
RemarksThe collected data is 8-channel data: the first four channels correspond to the four microphones mounted on the top of Boomy, the fifth and sixth channels are reserved channels, and the seventh and eighth channels are for loudspeaker echo acquisition

C++ Example:

ctrl->subscribe_internal_audio_capture(
[&](const media::AudioStream &stream) {
std::lock_guard<std::mutex> lk(mtx);
if (channels == 0) {
channels = stream.channels;
sample_rate = stream.sample_rate;
}
pcm.insert(pcm.end(), stream.audio_data.begin(),
stream.audio_data.end());
});

Python Example:

def on_audio(stream):
with lock:
if channels_ref[0] == 0:
channels_ref[0] = stream.channels
sample_rate_ref[0] = stream.sample_rate
pcm_all.extend(stream.audio_data)

ctrl.subscribe_internal_audio_capture(on_audio)

Subscribe to the audio stream played by the robot's internal speaker

Function Namesubscribe_internal_audio_playback
Function Prototypevoid subscribe_internal_audio_playback(AudiostreamCallback callback);
Function OverviewSubscribe to the audio stream played by the robot's internal speaker
Parameters- callback: callback function for processing audio stream data
Return ValueNone
RemarksThe collected data is 2-channel data from the Boomy ontology speaker

C++ Example:

ctrl->subscribe_internal_audio_playback(
[&](const media::AudioStream &stream) {
std::lock_guard<std::mutex> lk(mtx);
if (channels == 0) {
channels = stream.channels;
sample_rate = stream.sample_rate;
}
pcm.insert(pcm.end(), stream.audio_data.begin(),
stream.audio_data.end());
});

Python Example:

def on_audio(stream):
with lock:
if channels_ref[0] == 0:
channels_ref[0] = stream.channels
sample_rate_ref[0] = stream.sample_rate
pcm_all.extend(stream.audio_data)

ctrl.subscribe_internal_audio_playback(on_audio)

Subscribe to the video stream collected by the camera inside the robot (this interface is only available for Boomy without a Compute Board)

Function Namesubscribe_internal_video_capture
Function Prototypevoid subscribe_internal_video_capture(VideostreamCallback callback);
Function OverviewSubscribe to the video stream collected by the camera in the robot
Parameters- callback: callback function for processing video stream data
Return ValueNone
RemarksOnly available for Boomy models without a Compute Board

C++ Example:

ctrl->subscribe_internal_video_capture(
[&](const media::VideoStream &stream) {
printf(
"[DEBUG] video frame: %ux%u format=%u data=%zu bytes\n",
stream.width, stream.height, stream.format,
stream.video_data.size());
if (got.exchange(true))
return;
save_video_frame("internal_video_capture", stream);
});

Python Example:

def on_video(stream):
if got.is_set():
return
result[0] = stream
got.set()

ctrl.subscribe_internal_video_capture(on_video)

Subscribe to the desensitized video stream processed by the audio-visual interaction system

Function Namesubscribe_internal_video_desensed
Function Prototypevoid subscribe_internal_video_desensed(VideostreamCallback callback);
Function OverviewSubscribe to the desensitized video stream processed by the audio-visual interaction system (the Boomy equipped with a Compute Board needs to continuously publish the external video stream)
Parameters- callback: callback function for processing video stream data
Return ValueNone
RemarksBoomy equipped with a Compute Board needs to continuously stream external video first before it can be used

C++ Example:

ctrl->subscribe_internal_video_desensed(
[&](const media::VideoStream &stream) {
printf(
"[DEBUG] video frame: %ux%u format=%u data=%zu bytes\n",
stream.width, stream.height, stream.format,
stream.video_data.size());
if (got.exchange(true))
return;
save_video_frame("internal_video_desensed", stream);
});

Python Example:

def on_video(stream):
if got.is_set():
return
result[0] = stream
got.set()

ctrl.subscribe_internal_video_desensed(on_video)

Suspend the internal camera data acquisition of the robot

Function Namepause_video_capture
Function Prototypevoid pause_video_capture()
Function OverviewPause the internal camera data acquisition of the robot (only applicable to Boomy models without a Compute Board)
ParametersNone
Return ValueNone
RemarksOnly available for Boomy models without a Compute Board

C++ Example:

ctrl->pause_video_capture();

Python Example:

ctrl.pause_video_capture()

Resume the internal camera data acquisition of the robot

Function Nameresume_video_capture
Function Prototypevoid resume_video_capture()
Function OverviewResume the internal camera data acquisition of the robot (only applicable to Boomy without a Compute Board)
ParametersNone
Return ValueNone
RemarksOnly available for Boomy models without a Compute Board

C++ Example:

ctrl->resume_video_capture();

Python Example:

ctrl.resume_video_capture()

Publish external video streams to the audio-visual interaction system

Function Namepublish_external_video_stream
Function Prototypevoid publish_external_video_stream(VideoStream stream)
Function OverviewPublish external video streams to the audio-visual interaction system
Parametersstream: VideoStream struct, containing video frame data
Return ValueNone
Remarks

C++ Example:

VideoStream stream;
// Populate stream data ...
ctrl->publish_external_video_stream(stream);

Python Example:

stream = VideoStream()
# Populate stream data ...
ctrl.publish_external_video_stream(stream)

Publish external audio stream input to the agent

Function Namepublish_external_audio_stream
Function Prototypevoid publish_external_audio_stream(AudioStream stream)
Function OverviewPublish external audio stream input to the agent
Parametersstream: AudioStream structure, which contains audio data
Return ValueNone
Remarks

C++ Example:

AudioStream stream;
// Populate stream data ...
ctrl->publish_external_audio_stream(stream);

Python Example:

stream = AudioStream()
# Populate stream data ...
ctrl.publish_external_audio_stream(stream)

Publish external audio stream output to the internal speaker

Function Namepublish_external_audio_playback_stream
Function Prototypevoid publish_external_audio_playback_stream(AudioStream stream)
Function OverviewPublish external audio stream output to the internal speaker
Parametersstream: AudioStream structure, which contains audio data
Return ValueNone
Remarks

C++ Example:

AudioStream stream;
// Populate stream data ...
ctrl->publish_external_audio_playback_stream(stream);

Python Example:

stream = AudioStream()
# Populate stream data ...
ctrl.publish_external_audio_playback_stream(stream)

Pause internal microphone audio stream capture

Function Namepause_audio_capture
Function Prototypevoid pause_audio_capture()
Function OverviewPause internal microphone audio stream capture
ParametersNone
Return ValueNone
Remarks

C++ Example:

ctrl->pause_audio_capture();

Python Example:

ctrl.pause_audio_capture()

Resume internal microphone audio stream capture

Function Nameresume_audio_capture
Function Prototypevoid resume_audio_capture()
Function OverviewResume internal microphone audio stream capture
ParametersNone
Return ValueNone
Remarks

C++ Example:

ctrl->resume_audio_capture();

Python Example:

ctrl.resume_audio_capture()

Pause the audio stream output of the internal speaker

Function Namepause_audio_playback
Function Prototypevoid pause_audio_playback()
Function OverviewPause the audio stream output of the internal speaker
ParametersNone
Return ValueNone
Remarks

C++ Example:

ctrl->pause_audio_playback();

Python Example:

ctrl.pause_audio_playback()

Resume the audio stream output of the internal speaker

Function Nameresume_audio_playback
Function Prototypevoid resume_audio_playback()
Function OverviewResume the audio stream output of the internal speaker
ParametersNone
Return ValueNone
Remarks

C++ Example:

ctrl->resume_audio_playback();

Python Example:

ctrl.resume_audio_playback()
(5) Demo Test Cases​

The SDK provides two complete sample programs with identical functions, implemented in C++ and Python respectively:

  • C++: examples/cpp/example_media.cpp compile and run
./build/example_media <command> [args]
  • Python: examples/py/test_media.py run
python3 test_media.py <command> [args]

All output files are saved by default in the out/ folder of the SDK root directory.

Test Case List

  • Obtain all common_config configuration information
python3 examples/py/test_media.py
./build/example_media
  • Subscribe to the internal microphone to capture a 10-second audio stream
python3 examples/py/test_media.py internal_audio_capture
./build/example_media internal_audio_capture

Subscribe to the internal microphone, capture audio for 10 seconds, and save it to out/internal_audio_capture.wav

  • Record the audio stream played by the internal speaker
python3 examples/py/test_media.py internal_audio_playback
./build/example_media internal_audio_playback

Subscribe to the internal speaker, collect data for 10 seconds, and save it to out/internal_audio_playback.wav

  • Input the wav audio file to the audio-visual interaction system agent
python3 examples/py/test_media.py external_audio_agent <wav>
./build/example_media external_audio_agent <wav>

Read the wav file, and based on the status of the switches set_internal_capture_audio_data_to_agent_enable and set_external_custom_audio_data_to_agent_enable, then use TO_WAKEUP to wake up the agent, then send the audio in 20ms chunks via publish_external_audio_stream, and resume the switch status after completion

  • Publish external audio stream output to the internal speaker
python3 examples/py/test_media.py external_audio_playback <wav>
./build/example_media external_audio_playback <wav>

Read the wav file, convert it to 2-channel interleaved format, split it into 20ms chunks, and transmit them via publish_external_audio_playback_stream for playback

  • Subscribe to the internal video stream to capture a frame of image (for models without a Compute Board)
python3 examples/py/test_media.py internal_video_capture
./build/example_media internal_video_capture

Subscribe to the internal video stream and capture a frame to save it to out/internal_video_capture.png

  • Subscribe to the internal desensitized video stream to capture a frame of image
python3 examples/py/test_media.py internal_video_desensed
./build/example_media internal_video_desensed

Subscribe to the internal desensitized video stream and capture one frame to save it to out/internal_video_desensed.png

Models equipped with Compute Board need to first publish the external video stream to the agent, as shown in the example below

  • Publish external video stream input to the audio-visual interaction program agent
python3 examples/py/test_media.py external_video_publish
./build/example_media external_video_publish

ffmpeg collects /dev/video4 (640x480 yuyv422 @30fps) and publishes it via publish_external_video_stream

5. Camera Example Program​

info

The camera hardware of the Boomy EDU version is directly connected to the Compute Board, which can directly access the hardware interfaces on the Compute Board to obtain data

Please refer to the following link for the Intel D435i camera example programs: https://github.com/IntelRealSense/librealsense

For models with a Compute Board, if you want to use the camera, please stop the official camera image transmission program first

sudo systemctl stop noetix-video-capture.service # **Stop the image transmission service**
sudo systemctl disable noetix-video-capture.service # **Disable auto-start**

sudo systemctl start noetix-video-capture.service # **Start the image transmission service**
sudo systemctl enable noetix-video-capture.service # **Enable auto-start**

6. Rapid Development​

All the following operations are performed on the Boomy Compute Board. The Compute Board comes pre-installed with the required compilation environment. You may also deploy the compilation environment via Docker.

For the external development environment setup, refer to: SDK Compilation Environment Deployment Process

info

Introduction to External Plugin Development

Since the Motion Control Board of humanoid products is not open to the public, user programs cannot be directly deployed and run on the Motion Control Board, so an additional computing carrier is required to execute user programs.

If the model purchased by the user is not equipped with a Compute Board, the "external plugin development" method shall be adopted: the user shall prepare the Compute Board or host by themselves, and connect to the Boomy rear interface via a network cable to communicate with the Motion Control Board.

Since the Motion Control Board only allows communication with the fixed IP of the original Compute Board, and the communication is based on the wired connection of the internal switch, the external plugin device needs to be configured with the same IP as the original Compute Board to simulate the original Compute Board and complete the access.

Wired SSH Login to the Computing Board

ssh noetix@192.168.55.101

Wireless SSH Login to the Computing Board (The wireless IP can be checked via the mobile App's WiFi connection page. The Compute Board uses the WiFi module of the Motion Control Board)

ssh -p 2222 noetix@[ROBOT_WIFI-IP]
(1) SDK Directory Structure​
.
├── build.sh # Build script
├── CMakeLists.txt
├── config # Configuration files
├── dds # DDS library files
├── examples # Example codes
├── include
├── lib # Dynamic library files
├── LICENSE
├── policy # Model files
├── pybind # Pybind files
└── README.md
(2) Docker Development Environment Image Deployment​

noetix_dds.tar.gz

Note: This image package is for the aarch64 architecture and can only be deployed on computers with the aarch64 architecture

a. Extract the compressed package

tar -zxvf noetix_dds.tar.gz

After extraction, the compressed package contains the noetix_dds.tar development image file and the noetix_dds.yaml configuration deployment file

version: "3.8"

services:
noetix_dds:
image: noetix_dds:arm64
container_name: noetix_dds
network_mode: host
volumes:
- /home/noetix/work:/home/noetix/work:rw # Mount working directory
# /home/noetix/work The part before the colon should be changed to the user's working directory; create this directory in advance
stdin_open: true
tty: true

b. Deployment

docker load -i noetix_dds.tar
docker images # Confirm that the noetix_dds arm64 image exists
mkdir -p /home/noetix/work # Create a working directory; the user can choose it themselves
docker compose -f noetix_dds.yaml up -d # Deploy
docker ps # Check if noetix_dds exists

c. Enter the Docker container

docker exec -it noetix_dds /bin/bash

You can start development by simply placing the DDS SDK in the working directory afterwards.

(3) Compile the SDK​
cd noetix_sdk_Boomy
./build.sh # One-click build script

After a successful compile, the executable files example_low, example_high, and example_policy_walk will be generated in the root directory of the SDK, and meanwhile the pybind libraries highcontrol_py and lowcontrol_py will be generated in the build directory for Python to call.

(4) Launch the Program​

C++

cd noetix_sdk_Boomy
sudo ./build/example_high # or sudo ./build/example_low

Python

cd noetix_sdk_Boomy
python3 examples/py/test_low.py # python3 examples/py/test_high.py
caution

Safety Tips

During the debugging process, please use the robot hanger throughout the whole process to avoid losses caused by out-of-control faulty programs

7. Common Issues and Notes​

  • DDS client demo: runtime error (1)

ARM architecture:

error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

libssl1.1_1.1.1f-1ubuntu2.16_arm64.deb

sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2.16_arm64.deb

X86 architecture:

error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

libssl1.1_1.1.1f-1ubuntu2_amd64.deb

sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2_amd64.deb
©Beijing Noetix Robotics Technology Group Co., Ltd.
Jing ICP Bei No. 2024092119-1 | Jing Gong Wang An Bei No. 11010802045221
Internet Filing No. 110114067446901260013
Beijing-SongYanDongLiFangShengJiQiRen-20260901S0213
Beijing-SongYanDongLiRenXingJiQiRen-20260901S0214