If you’re not used to the term “DevOps”: Wikipedia defines DevOps as “a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality”. However, most of the people who are involved have a rough definition that can be best described as a bucket of associated words. So, whenever you hear DevOps, just think of Agile, Lean, Culture, Automation, Feedback and Business and you’ll be good.
Software is eating the World
When I went to the airport early in the morning on October 23rd, I was still a bit jet-lagged. However, after a week of cold, changeable weather in the US I was looking forward to a sunny contrast in Nice. I arrived without any delays, just in time for my “Certified Jenkins Engineer” exam. It was tough and in parts way more technical than I expected but I managed to earn my badge which allowed me to strut around like a peacock for the rest of the week!
The main event started on Wednesday. From the first keynote through most of the talks the leading premise was “Software is eating the World”. What may sound like a prophecy of doom when you first hear it mainly means that software is everywhere and software development is becoming a crucial success factor, even for companies that would not define themselves as a “software company”. Some go as far as to claim that “Every company will become a software company [or vanish]”. Discussing the latter usually ends up in a fight about the definition of specific terms (we techies we love to fight about details!) but the general message about the importance of software is undeniable.
So, if we agree on this premise, what does it mean for an automotive OEM or a supplier? It means that the quality of your software will be what sets your company apart from your competitors. It means that having a sophisticated process for maximum efficiency should no longer be considered a “nice to have” thing to make developers happy. It needs to become an integral part of your management strategy.
The A in DevOps stands for Automotive
With the “Software is eating the World” premise in mind everyone is looking for solutions. And the Jenkins Ecosystem is full of solutions. The DevOps community is full of demos, prototypes and success stories. But if you look closer, they usually end up deploying a web app to the cloud. Also builds and test run on fungible (=interchangeable) agents in the cloud, you can scale anything up to infinity. Got 100 Browser-Language-OS combinations? Just test them all in parallel. Did I mention that everything is just a microservice and runs in containers? This is the picture of the DevOps world that many of us know. Full of easy-to-demonstrate mini examples which use fancy technologies like Docker or Kubernetes that we’ve never even heard about.
In our daily work in automotive software development we deal with huge monsters of legacy software, target specific compilers, hardware resources, etc. Most things take longer compared to “normal” software development, especially regarding web applications. We apply model-based development to handle great complexity and the people involved in developing and testing need a combination of skill sets from software-, mechanical- and electrical engineering.
But fear not! This picture of DevOps approaches being limited to the web app paradise is merely a facade. Soon I saw some familiar faces and the schedule included four embedded talks (given that many talks were on a more abstract level anyway, I think four is a respectable amount). These talks, the discussions I had with my peers and the morning fun run quickly restored my faith into DevOps for our industry. DevOps can be leveraged in automotive, you just need to know which hints to follow and which to ignore.
Leveraging DevOps in Automotive Embedded Software Development
We have to consider the following challenges for our DevOps approaches in automotive:
1. Low Standardization for Development & Testing
- Challenges
- Tools and processes are highly customized
- Only in rare cases, two companies (or even departments) use the same set of tools and if they do, they have different customizations
- Model-based and hand-code projects use different processes, tools and require different skills
- Recommendations
- Group as many similar tasks together and define an abstract process for them
- Prefer tools that integrate well with your automation pipelines
- Try not to micro-manage: delegate responsibilities for details but also give the responsible (e.g. team leaders) the ability to satisfy their task-specific requirements (integrate specific tools) within the frame of this abstract process
- Communicate goals and best practices: you can’t control every detail but you can guide the people that take care of these details to consider the higher-level goals
2. Limited resources have to be handled efficiently
- Challenges
- Software licenses need to be managed for interactive users and automation pipelines
- Hardware resources (PIL evaluation boards, HIL systems, test benches) need a lot of advance planning
- Recommendations
- Especially in automated build pipelines licenses need to be managed as a lockable resource. Nothing is more annoying than coming to work and realizing that your pipelines have failed because someone forgot to close Tool XY on his PC and was blocking the license. License usage should usually be quite predictable.
- Consider bottlenecks coming from limited resources. If you identify a bottleneck (i.e. number of licenses or available hardware) react as soon as possible and consider this for the future. You can’t parallelize your HIL testing five times if you only have two slots available.
3. Feedback comes late or not at all
- Challenges
- During development and testing the long durations often lead prevent fast feedback
- While you can constantly monitor a webserver in production this is not common for the software in a car
- Some feedback will get back via the garages or car dealers it’s hard to track it to a specific piece of software
- Recommendations
- Feedback is crucial to continuously improve. It’s a core aspect of DevOps!
- Feedback during Development and Testing:
- For development and testing long durations are usually the result of too big functions. Split them into smaller chunks, which are easier to understand, faster to build and to test.
- Don’t test the complete functionality with complete code coverage on integration level. I often see tests for software components that are testing literally everything on this level. In addition to the long durations of builds, test executions and other analysis tasks the sheer complexity of a software component (functions with hundreds of inputs and outputs) is more than the human brain can handle. On an integration level the tests should focus on integration aspects, not the detailed functionality of all units.
- Delivering to / getting Feedback from Production:
- OEMs in particular have the possibility to move forward in this area. Deploying your software usually still means that it’s flashed to a piece of hardware which is then built into the car. With “over-the-air” solutions new possibilities are available.
- Consider these new possibilities to close the feedback loop for your software. Data from production is the most valuable feedback that you can get.
4. Changing culture and processes will take a lot of time
- Challenges
- During development and testing the long durations often prevent fast feedback
- While you can constantly monitor a webserver in production this is not common for the software in a car
- Some feedback will get back via the garages or car dealers it’s hard to track it to a specific piece of software
- Recommendations
- Don’t try to change everything at once
- Pick a reasonable pilot project, set your goals and define in advance how your success can be measured
- Allocate resources in advance. Managers need to be willing to put money on the table. This kind of task should not be done as a 3-hours-per-week side project.
These challenges need to be tackled so that your business can truly benefit from DevOps. Managers need to realize that the transformation of culture, workflows and decision processes is vital to stay relevant in today’s ever-changing world. My time in Nice was a pleasure and I learned a lot of cool new stuff which can easily overwhelm you. Luckily, I wasn’t the only piston head at Jenkins World which lead to great conversations that allowed us to connect the dots between the main principles, the benefits of the cool new stuff and the reality we experience every day. Speaking of benefits, if you’re ever in the area – Nice is definitely worth it! Here are some impressions:
The beautiful beach just south of the old town
The beautiful beach just south of the old town
Le Mirroir d’Eau at the Promenade du Paillon
Facing the Place Saint-François from the Promenade du Paillon close to the conference center
