Continuous delivery is one of DevOps governing principles. Professionals in that sector strive for automation, for seamless feedback loops, rapid testing and delivery between development and operations - but, it’s not always realistic and the human element, for better or for worse, always creeps back in.
But with COVID-19 forcing teams, that otherwise would be a tap on the shoulder away, to work away from each other having a robust DevOps system is more important now than ever.
We spoke to a handful of DevOps leaders, from Architects to Tech Leads, to see how they’re getting on with the ‘new normal’ and how it’s impacted their DevOps operations.
Engaging With DevOps Teams Remotely
COVID-19’s impact on the working ecosphere has been massive. Ripples from the impact have been felt in almost every aspect of our day-to-day with work being no exception. With teams forced to isolate away from each other, engagement with those teams has become priority number 1 for leaders who want to keep momentum.
Of course, the Tech workforce is no stranger to remote working. In fact, Buffer’s 2020 State of Remote Working revealed that Forty-one percent of those who took the survey work at organizations in the software space with IT and Services being the second highest (19.5 percent).
DevOps, however, resides in a limbo between those two sectors - borrowing working methodologies from both.
“We have sprint planning every week over video. We also have an informal hangout session twice a week, where we just chill and let off some steam. This replaces the very important coffee machine talk, that is lost during isolation. Before this started we were already proficient at using Slack.”
Johan Hernefeldt is a DevOps Architect at Telia. Johan and his Team are currently developing Telia’s Contact Center system with goals to expand from the Nordics into Europe.
“In my development team, I develop processes and automations. I have a team of architects, and together we improve the infrastructure so it’s better suited for DevOps. We are ISO27001 certified and follow GDPR very strictly. Our clients include public sector, insurance companies and banks, which makes security and customer integrity very important.”
In total, Johan and his team are now between 60-70 developers, testers and operations and the initiatives he works on, work towards the engagement of all of those individuals.
Prior to COVID-19, everyone in Johan’s team at Telia was able to work from home. The preferred option, however, was having everyone in the office where they could cooperate more easily. As a result of this, guidelines and documentation were created to help everyone do troubleshooting of build servers, test servers and different tools themselves. Getting access to everyone who needs it while working remotely posed a challenge Johan needed to address.
“We started to use Confluence for documentation between colleagues. We also have an automation script that pulls all the markdown from our git repositories and publishes it to Confluence. This way we get all the technical documentation and API specs for different branches, so developers, and testers can easily find the latest API spec on what they currently are working on.”
Having this standard of internal documentation, easily accessible, has positive repercussions on many aspects of the DevOps team. New hires, if there were to be any, would easily be able to get up to date information. More importantly, it also helps avoid communication overload for core members of the team.
“We also have ‘How-To Guides’ in Confluence for each system that is usually managed by one person. This helps to avoid situations where one person is getting calls from 8 different developers asking the same question.”
Having the tools to make communication seamless is a key enabler for teams regardless of discipline, but the communication still needs to be managed to make sure it stays conducive to a positive working environment.
Priy Werrij is the DevOps Team Lead at Crunchr, a cloud solution for people analytics and workforce planning. With clients across the globe, Priy and his Amsterdam based team are over 50% technical.
Priy’s role is to facilitate all teams in Crunchr. To help them thrive and feel enabled to do their work. It involves DevOps and tooling, but also involves the human aspect so often overlooked.
“I see myself as carrying a large can of fuel, anywhere I go I put some of that fuel in the engine to keep the (devops) machine running efficiently.”
Engagement with his teams was one of Priy’s top priorities as they transitioned away from the office and towards working from home. Similar to Johan, Priy operates daily stand-ups and meetings with his teams - but Priy has noticed the move to this format has improved communication in these meetings overall because, usually, it gives equal footing to each attendee.
“We all call in, only one person can talk at any one time so the team meetings we had planned so far are actually maybe better than before in terms of focus. It is important that everyone is live and alert on what’s happening. Especially because daily meetings and stand-ups are the only place now where you can see what everybody is working on and if somebody is struggling.”
As mentioned at the start of the article, those working in tech are no strangers to distributed work, that isn’t the problem. The loss of control and agency over the choice of where we can work, coupled with the immediate isolation some may be feeling is the problem. Addressing this head on is of vital importance for team leaders who may present a life-line for some individuals who have no means of interaction outside of work.
“It needs a lot of discipline.”
Ralf Lang is the Lead Developer at B1 Systems, a Linux/Open Source Consulting, Training, Managed Service & Support provider based out of Germany.
With customers who have onsite consultants who normally work at the customer's offices and therefore adhere to their policies on working remotely, B1 systems finds itself in a unique position when it comes to keeping their staff engaged.
“Normally when we have regular meetups or regular travel to one of the office locations you can go for lunch with the person and have a better talk on how things are going. This is completely missing and what I have begun doing is reserving a time slot in the week where I make phone calls to people on a one-to-one basis and discuss things generally focused on their feelings.
When people get in this it can feel like an interview. If you have a more informal type of conversation you find yourself having more open conversations.”
On a daily basis Ralf manages two teams. One is an operational team of 10 people and the other is a development team of 7 with the development team evolving into a DevOps team. The goal here is to evolve that team to the point that they shouldn’t need to rely on any other team for infrastructure, testing or production and will be able to build toolchains and to manage their own Kubernetes classes.
Evolving DevOps Processes For Working Remote
Priy and his team and Crunchr have worked hard on their processes but, like many companies, there will still be gaps that would normally be filled in with the human interaction we’d all be accustomed to in an office setting.
“If a developer has some code they want to push through in a specific way, they’d normally be able to come over to the operations team and explain. Now, that’s more difficult to do.”
In terms of integration, continuous testing and code review, Priy and Crunchr are doing exciting things with automation. But when it comes to deployments and releases, Crunchr prefers a more human approach. Naturally, this presents new challenges when a team is distributed.
“I think up until the code is merged with our master branch it is the same as before. What we do see, however, is that when this code is in the master branch and we need to make the release it is more of a manual process.”
For Crunchr, the gap in the process is something working in their favour. At face value, it means that developers can fully focus on the code quality because there are operational aspects to worry about. Holistically, Crunchr is operating with a product that is deployed strategically and not on a day-to-day.
“The features we’re building are broader and not something we change in a day. Our user base is also small. So, for now, the gap suits us.
On the other hand, in terms of development feedback, bridging that gap might be important. Sometimes, developers just want to build their nice feature and show it to their colleagues and they are happy. Some of them don’t care when or how it ends up in production, changing that mindset is necessary to see if it’s worthwhile to close that gap.”
Working with many vital services like Healthcare and Banking, there can be no margin for error on the process and productivity side. Johan and his team at Telia have certain internal processes that need to be followed, and now the focus is on drafting guidelines and embedding operational support in each SCRUM team.
“We just rolled out our first Kubernetes platform to the operations guys two weeks ago. We have worked on this platform for well over a year and worked hard on making sure it’s secure for our clients.”
The work Johan and his teams are doing has to follow strict regulations varying from country to country, therefore they’ve had to put limitations into their DevOps Workflow to keep the information secure. One way they’ve done this is by separating the Dev & Ops entirely, limiting the amount of users with access, whilst trying to keep as much automation as possible.
“Not everyone should have access. And for GDPR purposes there are only a select few who should have access.”
For Ralf and the consultative operation at B1 System, functions are partly reliant on the clients they work with. Still, an important part of evolving their processes during remote working has been on an increased emphasis on communication between everyone, developer included.
“We have some differences between the ways consultants work with clients. Some are quite manual processes, some are really embracing DevOps. The more developers communicate via the software they write. The more they can explain the terraform files and explain the setup process, please apply the state, the easier it will be for everyone.
What really helped us was Slack because one of the big differences is normally you can look over the desk and see the person speaking on the phone or having a conversation so you know not to distract them.”
Looking After the DevOps Teams
Everyone is experiencing the effects of COVID-19 differently and nowhere is this more the case than in our working situations. Some of us may have family around us, others may not. And despite these circumstances presenting different challenges, they’re challenges nonetheless and it’s up to business leaders to step up to provide support during this time.
And there is empirical evidence linking employees’ wellbeing to their individual performance.
But the answer, like Priy and the teams at Crunchr found out, doesn’t lie in a blanket, one-size-fits-all approach.
“We as a company organise virtual events, like drinks or lunches that are done together. The more technical people don’t always attend these centralised events. Instead, I’ve noticed that taking just a bit more time after a standup to ask some questions like what they have planned for the weekend can replace more centralised engagements.
But it’s a balancing act because there are those people who really do like to attend centralised activities.
We’re still looking at making it easier to organise interaction with colleagues besides working together on work-related matters. It’s a work in progress.”
Having a clear roadmap for success is something Ralf and B1 Systems hold as part of their culture and attribute to the long term well-being of those who work for them.
“We don’t have this culture of checking who is first in the office and who is last. We are very much concentrating on achieving milestones. Do we get the tickets done? Do we get the feature requests done? If not, then we of course question if there was any complications. This is something we didn’t change after the COVID-19 crisis.”