Automation and orchestration: a perfect match?
Automation and orchestration are often put in a versus situation, but that is not the best approach. Let’s find out how automation and orchestration work together.
What is automation?
Automation is a loosely defined term, but most people think of robots or all-powerful algorithms. Following a broad definition, automation is a wide range of technologies that reduce human intervention in processes: from the automatic setting of temperature by your thermostat to regulating heat inside a nuclear reactor.
But when talking about automation in software development, we often find that this term refers to taking a simple task and taking it out of human hands. That might be a significant difference when comparing definitions: the most widely used assumes a whole process is automated; the one we use, only a task. The first will be reserved for orchestration.
To point out a common usage, I’ll bring a video you might be familiar with. Humans Need Not Apply is a video made by CGP Grey, and if you haven’t seen it yet, I vouch for it. Grey argues that we have gone from everyone needing to make food to almost no one—we automated most of the tasks to get food; hence, he says, we automated farming. That is the broader use of the definition. We could say that we automated farming tasks, but we actually orchestrate them as a whole.
Humans are smartly lazy: we will probably always figure out how to improve our wellbeing and reduce our work effort by bringing new tools to the table. You might think that this concerns only physical labor, but the services and software spheres do not get a free pass.
Automation seems to have taken a different path nowadays. That is the argument proposed by In a Nutshell on The Rise of the Machines. Automation no longer means clunky and stupid machines doing menial physical work. With an increased understanding of complex tasks, many engineers and developers find narrowly defined, repetitive tasks in many human activities, which you have probably guessed already; narrowly defined and repetitive tasks are what automation feeds on.
Does this mean that automation will replace humans in the software industry? We remain skeptical about it. It may be swimming against the current, but, as with many innovations, new problems arise from the creation of new technologies. We know that big tech companies are not employing as many people as their counterparts did a century ago. But we believe that with every new technology, new problems arise: you will always need people dealing with those.
What is orchestration?
You have probably heard the term orchestration, perhaps not associated with technology but mostly with music, which may not be the best metaphor. But if automation means having a robotic violinist, a mechanical cellist, and an artificial pianist, orchestration is putting them all together to play a well-adjusted tune.
Orchestration consists of managing a large-scale virtual environment or network. It works on a higher level as it encompasses many complex systems and services by managing many automated tasks. If you have paid attention to the definition of automation, you will notice that it refers to a single task, compared with the broad implementation of orchestration that arranges a group of tasks to optimize a workflow.
It is clear then that we are talking about two different levels of organization. While you are made up of cells that perform tasks; you are not just your cells, you are a complex system that cannot be reduced to its parts. Orchestration is putting all together to work in harmony. Does that sound right?
Automation vs. orchestration? What is the difference?
We have already discussed automation and orchestration, but how do they correlate? We stated that they are not the same so, is automation vs. orchestration a versus question?
As I said, automation means completing a task without human intervention. When well planned out, automation saves time, makes operations easy, and frees workers to perform other tasks. The scope of automation is not clear to this day, as the impact of new technologies is very ill predicted in the long run. We might be heading for a jobless world, but we are skeptical about it. Futurology is not our line of work.
On the other hand, orchestration lets you plan how any complex combination of automated tasks works together: it might be an assembly line for cars or developing software, but having these tasks alone is not enough.
The difference between orchestration and automation is suddenly evident. The first requires the second, and the second by itself has a very narrow application. Only by understanding the parts and the whole of a system can we work it out.
Open-source tools for automation and orchestration
We have already discussed in detail some open-source tools for automation. They have been chosen by our developers, who are constantly facing new challenges in the software development world. As a software factory, we prefer open-source programs as we believe in their effectiveness in the long run.
As we have seen in another post, DevOps culture provides a revolutionary framework to develop software. This approach, combined with open-source tools, can be a mix that powers up excellent software. Here are our top 10 open-source tools for DevOps automation.
GitLab is an open-source web-based DevOps lifecycle tool. It provides a Git-repository manager that, among other things, has features such as issue tracking, continuous integrations, a wiki, and a deployment pipeline. Not fully open source, GitLab uses an open-core model, where basic functions are open while extra functionality is proprietary.
“Where the world builds software.” If you are passionate about software, you are probably familiar with GitHub, and the giant does not need a long introduction. Thanks to it, millions of developers can build, manage, and ship their software. GitHub now reigns unchallenged as a development platform with its attractive features and easy-to-use UI. Among the services provided by GitHub, we find web hosting and version control in conjunction with Git.
Kubernetes, the one with the weird name, is an open-source container orchestration tool for automating deployment, scaling, and management. It was born out of Google and is now maintained by the Cloud Native Computing Foundation. Also referred to as K8s, Kubernetes is essential for apps packaged with various containers as it reduces isolation and packaging of clusters logically for easy deployment.
SonarQube is a platform for continuous inspection of code quality. This automation tool provides reviews with static analysis of code detecting bugs, security vulnerabilities, and code smells. SonarQube is designed to meet enterprise requirements and provide easy management of code quality and security at a business scale.
CircleCI is a Continuous Integration (CI) and Continuous Delivery (CD) platform that allows engineering teams to build and deploy software. The team validates code changes by connecting to a version control system like GitHub. With this, you get total control over your pipelines, plus you gain the ability to define de CI and CD processes. You can use workflows to set how and when jobs run to optimize them.
Prometheus is monitoring software that can also run on traditional servers. It has become the mainstream monitoring tool of choice in container and microservices infrastructure. As DevOps becomes increasingly more complex, Prometheus brings an automated monitoring tool to the table. By keeping this software running, you can constantly monitor all the services you are responsible for. It will alert you if any of them crashes, and you can set warnings to identify problems before they break out.
Ansible is a mass SSH deployment language that is very easy to approach. It is a leading open-source tool for software deployment, provision, and configuration management that allows you to build code infrastructure. Ansible has then become renowned as it simplifies software development automation across the complete lifecycle.
ELK stack is actually a combination of three open-source tools: Elasticsearch, Logstash, and Kibana. This mix provides the perfect tools for log management, allowing the Ops specialist to gather logs and search through them. While Logstash and Elasticsearch perform the work behind the scenes, Kibana coronates this combination by providing a data visualization tool that supplies easy-to-read reports and real-time summaries of streaming data.
Docker is a software development platform plus a sort of virtualization technology that lets you run apps no matter where they are. Docker provides a modular system for environment building, and each container is a neat package with everything you need to run it: system libraries, runtime environment, product code. Containers usually run one specific task and can be updated independently, giving you much-needed versatility.
Jenkins is an open-source and free server used for automating software development processes, including CI/CD, building, deploying, and testing. With an extensive plugin base, Jenkins can easily integrate with many CI/CD tools. Being a self-contained program in Java, Jenkins works in most operating systems, including Linux, macOS, and Windows. Its easy configuration and extension allow for error verification and to help you expand its functionality.
Automation, orchestration & DevOps
We’ve already talked about saving valuable time with DevOps, and you might remember that we mentioned automation and orchestration. That is because DevOps often relies on these practices to provide the best possible service and fast-track software development.
Many fear automation. With machine learning on the rise, no job seems secure from the grip of the machines, not even John Connor's. We should keep Amara’s law in mind: "We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.” While some people will see the potential loss of jobs, others will bask in the enhanced productivity. The answer might not please any of the involved.
It is true. DevOps automates many manual and repetitive tasks that people do. And yes, these people will not be doing those tasks anymore. But, will they be fired? Not necessarily. As machines replace humans in menial labor, those same workers have more free time to acquire further knowledge and get involved in new tasks. So people won't lose their jobs, which is a good thing, but there will be a shifting landscape in which employees and employers have to establish a contract with constant innovation and education as the cornerstones.
Incorporating human error: Serendipity
Where there were many complicated repetitive tasks, now there is a robot, an algorithm that never succumbs to human error. While this seems ideal business-wise, but, if you are in a management position, think twice before crossing out a human from your spreadsheet. Serendipity is a concept that has gained some relevance lately. In case you don’t know, serendipity is an unplanned fortunate discovery, a common occurrence throughout the history of product invention and scientific discovery. And serendipity needs human error to appear.
Serendipity shoul also be seen as a potential design principle. You probably know some weird story about how someone actually discovered the best family recipe when cooking something the wrong way—at least that is true for me. That is why, even with the perspective of introducing error, humans are necessary for the equation, be it DevOps, cooking, or any development culture.
At Awkbit, we understand the importance of articulating automation with orchestration; there is no wall without its bricks, and they are not useful by themselves, not for good purposes anyway. When working with automation, we'd better keep in mind that the humans behind development matter and the overall goal of technology. Is it here to replace us or to help us? This way, we can take advantage of uniquely human abilities while supporting ourselves in orchestrated automation. Are you into automation? Do you think anything can be automated? Should everything be?