Visual Recognition Sorting Robot Arm

Report International Exposure Program PDEU (Pandit Deendayal Energy University) × AIT (Asian Institute Of Technology) By Anshul chawla Dharmik Parekh Saheb Bansal Nationality: Indian Pursuing Degree: Bachelor of Technology in Mechanical Engineering, Automobile Engineering At Pandit Deendayal Energy University(PDEU), Gandhinagar, Gujarat, India. Under the guidance of: Dr. Poom Konghuayrob At King Mongkut's Institute of Technology Ladkrabang School of Robotics and AI, Thailand.

ACKNOWLEDGMENTS We are immensely indebted to Pandit Deendayal Energy University (PDEU) for organizing the International Exposure Program at Asian Institute of Technology (AIT), Thailand. We would like to thank the Office of international relations and Mr. Maulik Shah for having good collaborative links with the Asian Institute of Technology and being all-time interactive to provide us with the best outcomes from the IEP. It allowed us to get exposed to the latest technology and helped us learn the research perspectives in our domain. It gave us great pleasure to have completed the entire month as a student of AIT, gaining knowledge in the field of wherein the world is progressing rapidly. We thank Dr. Nitin Tripathi for managing the entire IEP and organising the domain-specific programs for students of all branches. Our sincere thanks to Mr. Ranadheer Reddy, project coordinator- Special Degree Programs at AIT, for mentoring us in the entire program for a month and being all time available to us for any guidance needed whether it be in the academic-specific aspect or anything related to non-academic purpose. Furthermore, thanking Arthur Lance Gonzales, Program Officer at AIT, for giving us the best possible hospitality over the entire program duration. Most importantly we thank our professor and supervisor at KMITL, Dr. Poom Konghuayrob(HOD of Robotics and Artificial Intelligence) and our mentors Mr. Supun Dissanayaka, Mickey and Dio under whose guidance we were trained for the overall program and gained the knowledge in the field of Robotics and Artificial intelligence with a research perspective. Lastly, we thank our student coordinators May Hnin, Sai, and Jirawat D. for being all time available to us, providing us with our needs, and being the best guides to us. 2

ABSTRACT The \"Visual Recognition Sorting Robot Arm\" project presents an innovative solution to automate and optimise the sorting process in industrial environments. This study presents the development and implementation of an intelligent robotic arm system with shape recognition capabilities using deep learning techniques. The robotic arm is constructed using an Arduino Mega microcontroller board, a CNC shield, and three stepper motors, enabling precise control over its movements. CNN is trained to achieve accurate and real-time shape recognition for various objects within the robotic arm's workspace. The integration of the shape recognition model enables the robotic arm to autonomously identify and differentiate different shapes, allowing for dynamic and efficient pick-and-place tasks in unstructured environments. The CNN model is trained on a diverse dataset of shapes, achieving impressive recognition accuracy and processing speed. The performance evaluation demonstrates the robotic arm's adaptability and usability in handling objects based on their recognized shapes. The findings reveal that the shape recognition-enabled robotic arm outperforms traditional pre-programmed approaches, offering enhanced versatility and responsiveness. The study's contributions lie in advancing the field of intelligent robotics and automation by providing a practical and efficient solution for robotic perception capabilities. The developed system holds promising applications across industries, including manufacturing, logistics, and warehouse automation, where accurate and efficient object manipulation is critical. In conclusion, this research establishes a significant step towards enhancing robotic arm capabilities by incorporating shape recognition using deep learning, empowering the robotic arm to interact seamlessly with its surroundings and opening new possibilities for automation in real-world scenarios. 3

TABLE OF CONTENTS SR NO. CONTENTS PAGE NO. 02 1. ACKNOWLEDGMENT 03 05 2. ABSTRACT 06 3. LIST OF FIGURES 09 4. CHAPTER 1: INTRODUCTION 018 1.1 Background of the study 019 1.2 Statement of the problem 1.3 Research questions 1.4 Objective of the study 5. CHAPTER 2: DESCRIPTION 2.1 Hardware and control 2.2 Shape Recognition and Autonomous Operation 6. FUTURE WORK 7. REFERENCES 4

LIST OF FIGURES 2.1.1: Image of robot arm, with labeled components. 2.1.2: Image of A4988 motor driver. 2.1.3: Image of CNC Shield for Arduino. 2.1.4: Image of Jetson Nano. 2.1.5: Image of Arduino Mega. 2.2.1: Image depicting Feature Extraction in humans. 2.2.2: Image with Distinguishing features for identification. 2.2.3: Images showcasing differences in features through different perspectives. 2.2.4: Image showcasing pixel coordinates. 5

CHAPTER 1: INTRODUCTION This report outlines the design, development, and implementation of the Visual Recognition Sorting Robot Arm, starting with an in-depth analysis of existing sorting methods and their limitations. The project's key focus is to create a system that can visually recognize different shapes, handle diverse object types, and maintain high accuracy rates. The first phase of this project focuses on the selection and integration of visual recognition technology for training the A.I. bot to visually recognize various objects and their environment to be further integrated into the interface for the robotic arm for it to operate in simulated conditions imitating an assembly line. In the second phase, the report details the hardware and mechanical design of the robot arm. development of a robust control system is also explored, enabling precise and efficient object manipulation. Next, the implementation process includes the integration of computer vision algorithms with the robotic arm's control software. The report outlines the challenges encountered during integration and the strategies incorporated to overcome them. 1.1 Background of the study: Robotic arms have emerged as essential tools in various industries, offering precise and automated solutions for repetitive tasks. The integration of intelligent perception capabilities into robotic systems has become a focal point of research, enhancing their adaptability and usability in dynamic environments. Among these perceptual capabilities, shape recognition plays a fundamental role in enabling robotic arms to interact effectively with objects and environments. 6

The integration of shape recognition with robotic arms represents a significant step towards more versatile and intelligent automation solutions. This study explores the potential of deep learning techniques to empower robotic arms with the ability to recognize shapes, making them more efficient, adaptable, and capable of real-world interactions. The findings of this research can have far-reaching implications, opening new avenues for advancements in robotics and automation in diverse industries. 1.2 Statement of the problem: Despite the advancements in robotics and automation, the efficient interaction of robotic arms with objects in unstructured environments remains a challenging task. ● - Robotic arms lack efficient interaction with objects in unstructured environments, limiting their adaptability. ● - Traditional approaches rely on pre-programmed trajectories, making them time-consuming and labor-intensive for each specific task and object. ● - The absence of intelligent perception capabilities, especially shape recognition, hinders the robotic arm's ability to autonomously identify and differentiate shapes and objects. ● - Existing computer vision techniques for shape recognition in robotics may have accuracy and real-time performance limitations, especially in complex and cluttered environments. ● - This study aims to develop an intelligent robotic arm system with shape recognition using deep learning techniques, particularly Convolutional Neural Networks (CNNs). 7

1.3 Research Questions: The study will address the following research questions: ● How can we design and implement an intelligent robotic arm system with shape recognition using deep learning techniques? ● How can a Convolutional Neural Network be trained to achieve accurate and efficient shape recognition for various objects in dynamic environments? ● How does the integration of shape recognition impact the adaptability and usability of the robotic arm in performing pick-and-place tasks? ● What are the performance characteristics and limitations of the shape recognition-enabled robotic arm concerning recognition accuracy, processing speed, and generalisation to different objects and environments? 1.4 Objective of the Study: The Objectives of the study are as follows: ● Design and implement an intelligent robotic arm system with shape recognition using deep learning techniques. ● Train a Convolutional Neural Network (CNN) model for accurate and real-time shape recognition. ● Integrate the shape recognition model with the robotic arm to enable autonomous identification and differentiation of shapes. ● Evaluate the robotic arm's performance in pick-and-place tasks based on recognized shapes in unstructured environments. ● Contribute to the advancement of intelligent robotics and automation through practical shape recognition solutions. ● Explore potential applications of the shape recognition-enabled robotic arm in various industries. 8

CHAPTER 2: DESCRIPTION To Achieve our target for this project we divided the Major tasks into two important segments: Hardware and Control; Shape Recognition and Autonomous Control. This was done to maximise each individual’s focus on the task at hand. This research’s main objective was to find and train a suitable AI model which could be fine-tuned as per the requirements to perform the pick and place function with Visual recognition of the objects presented before it. 2.1 Hardware and Control: Fig. 2.1.1 2.1.1 Robot Arm As shown in the image above we can observe the robot arm has 5 major components: 9

● The base, upon which the entire structure of the arm is resting. Its purpose is to control the direction of the arm ● The shoulder, which has a hinge-type joint to control the orientation of the elbow, assists in raising and lowering the arm The robot arm consisted of three stepper motors 1. The first stepper motor was used for pivoting the base of the arm, to change the direction it is facing. 2. The second stepper motor was used to move the shoulder of the arm to extend the reach of the arm. 3. The third motor was used to pivot the elbow joint of the arm for further extension. 2.1.2 Stepper Motor Control: Fig 2.1.2 We have enabled communication between the stepper motors with the Arduino Mega microcontroller using the CNC shield. Smooth and controlled motion can be achieved using Move It (ROS) in Linux. 10

Fig. 2.1.3 2.1.3 End-Effector: The use of suction cups as end effectors in robotic arms offers a flexible and gentle gripping solution for a wide range of objects with smooth and non-porous surfaces. Their adaptability and ability to handle delicate items make them valuable tools in various industrial and automation applications. Proper integration of control mechanisms and sensor systems ensures the efficient and reliable performance of suction cups, making them an essential component in modern robotic arm systems. 2.1.4 Jetson Nano and Arduino Configuration: Jetson Nano is a SBC(single-board computer) that has been developed by NVIDIA. The computer mainly runs on Linux due to Hardware limitations. With the help of RoboFlow and YOLO(Deep Learning Framework) among other Python packages, we are able to detect the location and class of a shape in an image, and provide the optimal trajectory for the robot arm. 11

Fig. 2.1.4 Arduino is a microcontroller which is mainly used to control the motors in the limited scope of the project. It runs on C and uses many input methods including Serial (method of data transmission where bits of data are sent sequentially) to receive and send data. Fig 2.1.5 2.2 Shape Recognition and Autonomous Operation: Shape recognition may be understood as a subset of object detection, which has a wide range of applications ranging from shape detection, to more advanced profiling tools which were only fictional a few years ago. 12

2.2.1 Working of object detection: Object detection may be generally understood to work on the concept of feature extraction, which we as humans subconsciously perform. Let us consider the following two photos: Fig. 2.2.1 We know that the image on the left is a rabbit and on the right a cat, but why is that the case? What exactly has made us label them as ‘cat’ or ‘rabbit’? The reason is quite simple, they all have certain characteristics that stand out, and make it the easiest for us to identify them. Fig. 2.2.2 13

While this may seem quite simple to understand, it is quite hard to implement mathematically. THis is because not every photo is the same, i.e. some images may contain the object, but rotated, or at a different distance from the camera. While for us it is easy to understand, it is not the same for the computer. For example: Fig 2.2.3 The main feature for the above image might look as follows But if we were to zoom in, the same features would now look like The computer will not detect these as a part of the same object as the features do not match 14

2.2.2 Conversion of image coordinates to world coordinates Using Image detection, we can get the location of the object in the image itself, that is which row and column in terms of pixels it is located at. However, this location bears no meaning to the robot arm. There are two ways of converting image coordinates to real world coordinates. 1) Manually define the location of the base of the robot to be the world origin and determine the location of any object relative to the robot in terms of Cartesian coordinates. 2) Train another AI model with image position as input and series of movements as output, allowing us to bypass the maths for the most part. While the second option seems more attractive, it will involve a lot more work in the beginning stages and less work later on. For our case, the first option is much more feasible. We do this by converting pixels of the image to a set measurement in the real world, that is, if a picture of resolution 1000x1000 covers 100x100 mm² that would mean each pixel covers an area of 0.01 mm². This implies that one pixel of the image corresponds to 0.1mm, so if the centre of the object is at (200,500) in the image, it would be at (20,50) in terms of real world 15

coordinates. Fig 2.2.4 We can use this to move the robot arm and pick up and drop the object. 2.2.3 Communication between Jetson and Arduino Once the position of the object has been calculated, the Jetson nano, along with the help of MoveIt model provides the optimal trajectory to reach the position of the object, and thereon to the drop location. 16

However this data may be in gcode format or just a series of strings that are generally not recognisable by arduino, hence we will need to convert this data into a series of instructions that arduino can follow. There are multiple ways to go about this, however the two most promising ones that we found were 1) Using GRBL: GRBL is a software that is used for controlling multiple motors at the same time, while it is generally used to make customised 3-d printers, it may also be used in our case to move the motors in tandem and reach the end position. 2) Writing Base level code for all motors: We can write simple code that takes serial data in and in response to the data, moves the motors one at a time with the ability to increase or reduce motor speed, and distance travelled. While using GRBL may seem like the obvious choice, the software is actually quite buggy and the working is very obfuscated, hence opting for an easier to understand and control software seemed like a much better option. 2.2.4 The end effector For the end effector we have decided to use a cup that creates a suction force with the help of a force sensing switch and a compressor. When the robot arm reaches the location of the object, It gently presses on it, the triggers the force sensor and causes the compressor to start, creating a partial vacuum and picking up the object in the process. Then upon reaching the final position, the arm once again gently pushes down, reactivating the force sensor and causing the object to be dropped. 17

2.3 Future Work: This project can optimised by doing the following ● Improve shape recognition model accuracy through larger and diverse datasets. ● Extend shape recognition to handle multiple shapes simultaneously. ● Implement real-time object detection for dynamic environments. ● Integrate force feedback for adaptive and compliant gripping. ● Explore dual-arm collaboration for complex tasks. ● Design adaptive suction cups for a wider range of objects. ● Test the robotic arm in real-world applications and optimise performance. ● Implement autonomous navigation and path planning. ● Incorporate human-robot interaction features for intuitive control. ● Introduce safety measures like collision detection and avoidance. ● Optimise energy efficiency for longer operation periods. These future work points aim to enhance the robotic arm's capabilities, efficiency, and safety, enabling it to handle various tasks in practical scenarios. 18

REFERENCES Intisar, M., Khan, M. M., Islam, M. R., & Masud, M. (2021). Computer Vision Based Robotic Arm Controlled Using Interactive GUI. Intelligent Automation & Soft Computing, 27(2). Cabre, T. P., Cairol, M. T., Calafell, D. F., Ribes, M. T., & Roca, J. P. (2013). Project-based learning example: controlling an educational robotic arm with computer vision. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 8(3), 135-142. 19