A steel melting shop has many activities, which include the different activities in the setup and the arching and blowing, tapping, End Brick Removal, HM launder pouring and positioning, and others. It starts with the hot metal from the blast furnace and ends after the first heat is completed with the tapping process. The existing condition before the use of computer vision-based machine learning for detecting the delays and reducing the setup time in the SMS, there was no other way to identify the specific cause of the delay. Also, it is not possible for someone to stand there while the process occurs and point out what is the delay is. We need a monitoring system. Through automatic monitoring we can look at a particular object, which might be a person or a tool for a particular process, and if a particular process is leading to a delay or not. In short, there was no old structure that could help in monitoring the activities and tell the delays. Delay can be very detrimental to any production facility as it slows down the production process, takes more time than needed, and so, the efficiency of the production goes down as a result. So, capturing the specific cause of delay might help in better work at each step of the process and would lead to better efficiency and save time.