Andrew Harner & Tom Keschl

Transcript 

Speaker: Milan Bajic  00:39

Andrew, it’s been a while since you and I worked together and recently, we had a chance to catch up. And you were telling me kind of what you’re currently working on and the organization that you work for, and some of the challenges that you have with your team, as well as other teams that are working as part of the delivery pipeline. So maybe it will be good just to give the audience a quick overview of, type of work that you do, the company and maybe a little bit of background on the team that you joined. It’s been a couple of years now, right? But what was the situation when you joined the team? And maybe just give us a little bit of background on that.

Speaker: Andrew Harner  01:23

Yeah, absolutely. So quick introduction. I’m Speaker: Andrew Harner, self-proclaimed agility champion. I try to not pigeonhole myself or brand myself into any one framework or methodology, but just try to absorb as many tools as possible that I can use in organizations. I joined Wex back in 2018, December 2018. Wex is a financial services company that has some exposure in the fuel business and health care and business to business b2b, payment spaces. And, Tom…

Speaker: Milan Bajic  02:13

What do you guys do, what does Wex do? What type of products and services do you provide and who are your customers?

Speaker: Andrew Harner  02:21

We are payments company. We help facilitate payments in those areas so think, HSA payments, fuel cards, virtual credit cards, that type of stuff. And Tom and I currently, we work on an open loop issuing processor in the b2b portion of the company. I’ll pause there and let Tom just do a quick intro before we get into anything else.

Tom Keschl  02:51

I’m Tom Keschl, I’m a team lead on one of the development teams for that open loop processor, which is called tag or transact global.

Speaker: Milan Bajic  03:04

So what is opening processor? What does that really do?

Tom Keschl  03:08

So imagine the credit card that you have in your wallet. As long as it’s not tied to you, there’s two different kinds of things; there’s open loop and closed loop. Open loop is typically what the consumer is going to have in their wallet like a generic MasterCard, you can use it to pay at any vendor, any merchant location that accepts that card, as opposed to closed loop, which the business and the card, and the issuer all need to be in like a shared group together, and there’s benefits for that. So other parts of Wex, do closed loop processing, particularly around like fuel payments, fuel cards and stuff. And we do the open loop portion.

Speaker: Milan Bajic  03:54

Okay, awesome. Thank you for kind of giving, I knew a little bit about it. I think it’ll help the listeners also understand. So Andrew, you joined this company Wex in 2018 and then what happened?

Speaker: Andrew Harner  04:06

Yeah, so the intent for bringing me on was they were two Scrum teams at the time. And they were looking to scale up to a third team. And so they brought me on to be the scrum master for that third team. So joined in 2018, December before that 13 was really formed and like any good Scrum Master spent some time observing, trying to figure out what was going on with people and trying to build relationships. Through that observation period, I had discovered that, there were some blind spots that they had in their operating model, so to speak. The teams at the time were leveraging kind of relative story point estimation and velocity for their sizing and planning activities and that data was being maintained in a spreadsheet that was reviewed during and discussed during each retrospective. And one of the first things I noticed was kind of how this was implemented, and how these other teams were using these things. The work itself, the customer request was split in different ways. One of the ways was aligned to skill sets,  so we had a ticket that a developer would work on, and we had a ticket. Part of that ticket was a part of the requests, we had another ticket for a QA engineer to write the test cases for. And then we had another ticket that represented the deployment activities, the release activity. It wasn’t a holistic view of that work item, it was already kind of split. And then the work itself was only visualized up until a certain point in the value stream and so there was a series of release activities that included a regression cycle and deployment to our production environment. And those activities were not necessarily treated as team activities, they were treated as release activities and so those work centers weren’t visualized on the team sport. And so bring it all back, the estimating and planning that took place earlier in the sprint, didn’t have insights into those portions. And so we were estimating and planning to get work up through a certain portion of the valuation, but it wasn’t the complete picture.

Speaker: Milan Bajic  06:48

So in sense, your definition of done wasn’t really that strong. It was like a (inaudible 06:52)

Speaker: Andrew Harner  06:53

It was like Dev complete. We still didn’t have the final piece of feedback which was, does this cause a regression? Will this cause regression? And that activity was performed the last two days before the pressure started, so didn’t really leave as much time to get any fixes that would have been identified in before the scheduled prod release and so those are some other early observations. The next piece was, so I spent some time trying to bring the team together around that, I think it was the second or third month that I was there, I convinced enough people to come together for a Value Stream Mapping session. We sat down over a little one hour sessions over a couple of weeks, pulling all the people together that had a piece in that full process and we had mapped it out on a virtual whiteboard and it was really interesting conversations that took place. I remember Tom, I was asking our QA manager at the time, hey, how often are you finding bugs or finding defects when code makes it to this portion of the value stream? And she said, 99% of time we get bugs. And I’m like, so the development works better as a 1%, complete and accurate in there. And she’s like, well, yeah, and Tom’s face was like, oh, man, that is terrible. But the conversation that surrounded those were really powerful, it was the first time I think that the team had come together to see all the steps that it would take to get request out the door, wasn’t just I’m focused on my piece, they really painted that full picture. And so once I, as a newcomer had that full picture, I started to collect some data in a very manual way of work moving throughout the pipeline of the value stream and as I analyze that, and kind of historically went backwards and tried to click some stuff from previous sprints, previous iterations, I’d realized that the team really hadn’t made any significant improvements to their delivery capabilities in two years. 

Speaker: Milan Bajic  09:13

How do you feel to be probably the only person that understood that, because I’m assuming most of the organization was blind to that.

Speaker: Andrew Harner  09:22

Yeah. We’re months into this now and so I have been having conversations throughout these times, and I get this feeling like I’m thinking a little bit differently than other people are thinking in the organization and some people reacted really well to that and those were great and other others were a little bit curious and questioning, which is great but those conversations try to be a little delicate in their mind and sometimes that was successful or not in those times. But yeah, so that led to this realization and what do I do with this information now, and so, enter Tom stage left or right. And so, I’ve been trying to build relationships with Tom, I saw him as a great friend, personally, but also, he was really well respected professionally across the team. And so as I’m building this relationship, we’re playing super mario brothers at lunch and talking and getting to know each other and this led to like I don’t know how to hit it but it was really great debate and discussion that you followed us out of the building into the parking garage and it was like, I don’t know, six o’clock in January. And I was like, Tom, you guys haven’t made any improvements in two years? Prove me wrong. And so he spent that next weekend is trying to prove me wrong.

Tom Keschl  10:57

Yeah, so unlike Harner, I didn’t come from a background where I would ask a lot of those questions. I came up in that traditional agile scrum methodology, and I feel like I’d been on some good teams and some unsuccessful teams in the past and so I had a really good handle on it and I knew where I was going and so at this point in my life, I was King velocity. I was talking about story points estimates, I thought I had it all wrapped up and so we get into the parking garage and Harner challenges me with this.

Speaker: Milan Bajic  11:36

By the way, this is January in Maine and you guys are standing in a parking garage.

Tom Keschl  11:43

It was unheated, freezing cold. Like we’re using the concrete blocks of the wall to show how work items are moving. I mean, the conversation went everywhere. But when he said, when Andrew said, prove me wrong, show me what substantive changes you’ve made here, what concrete changes you’ve made to your delivery process, I listed off a handful right off the bat. And he’s like, well, how did that improve, how did that improve things, show me. So I went back home, and my wife was at work the next day, this is Saturday so I’ve got the kids to myself, working through all that. And I’m like thinking over in my head like, Man, I’m going to show this guy, what’s up. I pull out my laptop, I get all the kids huddled together around the counter. And we’re eating our food at the counter, which is a big deal for them, because they always eat at the dining hall table. But I’m sitting there with my laptop, trying to play with these numbers and stuff and fend off the kids and I’m pulling all the stuff out of our tickets system, I’m pulling all of the timing information I can, get it all into a spreadsheet, get some spreadsheet magic, learning spreadsheets at the same time, too and finally, I get something together that I share with Harner, around two o’clock in the afternoon and we probably spend the rest of the day going back and forth and we get a pretty primitive now, realization of lead times throughput, some work in progress statistics, all kind of carved out on this really, really choppy spreadsheet.

And it turns out that Harner was correct of our delivery capabilities. I spent 12 hours proving myself wrong. But yeah, our delivery capabilities over the past two and a half, I think it was at the time, years of working on the project had not meaningfully improved at all. In fact, the only time that we saw like a throughput increase, which is what we’re traditionally worried about, with the number of stories we got done, the number of story points we got done, the only time we even saw that was a month or two after we hired a new person. So that was like the only way in which we improve that delivery trajectory was hiring more people. Yeah, yeah, yeah. And so all of us know that’s not the way it’s supposed to be in the industry. The mythical man month is a pretty seminal text and stuff. So at that point, I was like, okay, there might be something to what Harner is talking about here.

Speaker: Milan Bajic  14:31

And then what happens, so you got at least one other person, Andrew to understand what you’re saying.

Speaker: Andrew Harner  14:38

Yeah. So, Tom and I kind of set out in this mission to help align the rest of the folks that we were working with, trying to help guide them towards this realization as well. So the first step was to take that work that we had done, that I did with the value stream mapping, and tried to reflect that and visualize that for the application and we’re lucky enough to work at a company that we have tooling that supports that. Some folks use manual things like stickies on a board, which is great but we have a technology that supports that so we could put that in there, and then have that help kind of facilitate some automation of collecting this data for us. And so that value stream mapping session that had led to us kind of rethinking how we’re visualizing our work in JIRA. So each of our columns now represented our work centers, and we had a pretty high level but solid understanding of all the tasks that are being performed in each of these work centers. And again, what is the exit criteria? What does it mean, when we move a ticket from this state to this state, and in that alignment, led us to having this data that Tom spent hours and hours and hours, and I had spent months trying to kind of curate manually, we had it available to us in real time, updated in real time for us. That was hooked in through an add on, called knave, so knave allows us to pull those flow value stream metrics out. But one of the biggest changes that the team made was extending the visualization from dev complete all the way to done right. So we brought those release capabilities, those release activities, onto the board, to have it visualized that we can make that work visible.

Tom Keschl  16:44

And the reason why we decided to do that, was because in the course of that 12 hour crunch time on Saturday, where we had that spreadsheet together, we realized that the two teams were defining those things slightly differently. The data integrity was only on a per team basis, and my team in particular, wasn’t capturing that for the rest of the things after it was dev complete. So we wanted to get to alignment for everybody that was participating in our value stream, all of the things that any given work item went through. So that was the first thing.

Speaker: Milan Bajic  17:23

Yeah, so what happened in the sense that you said it was done dev complete, were you deploying it, just not visualizing, what was going on, so why weren’t you?

Speaker: Andrew Harner  17:34

Interesting comment. What are the responsibilities of this our team. Some teams, they have handoffs, other teams responsible for that regression, or integration testing and system testing and release. But in tags context, the team owned all of that stuff, we own the complete ecosystem of our SDLC. So from the identification and prioritization of work all the way through that work hitting production is our activities that are performed within our teams. We just brought that in, visualize the team’s full process onto the board. Previously, because the release activities were treated not as team activities, even though it was performed by the team, they were left off the board, but we tried to get the team realize, hey, it’s still work that you’re doing, the work is not done yet. And so instead of having work pile up in a ready for regression testing, and that being kind of done, we wanted to have that buffer, that wait state visualized, as well as the time it was taken to get through regression, and then waiting for the production release and then finally moving to done. So the old board, Dev complete meant we’ve done our coding, we’ve done our functional acceptance testing, and now we’re just going to be waiting for the regression cycle to begin, which could be the following day. But if that was completed the first day of the sprint, it could be six days from now, seven days from now.

Speaker: Milan Bajic  19:19

So how do you get the team members involved? Like, in a sense, none of this can happen if Andrew and Tom are champions, I’m assuming and based on just small things that I picked up from earlier conversation as well as this, you had to get your team members involved. So how did that go? 

Tom Keschl  19:39

It was very iterative. Have you ever seen that video of I don’t remember what festival it was but there’s this guy that’s dancing by himself on a hill. We’re very, very fond of that video. We’re very fond of that because, like in that video, if you are the audience or whatever is familiar with it, there’s the second guy that comes in, we call him the green shirt guy because he’s wearing a green shirt and he comes in and he starts dancing with the dude and then not very long after that the entire hill is just a bunch of dancing people having fun.

Speaker: Milan Bajic  20:15

I thought you were going to say you gave some kind of (inaudible 20:19)

Tom Keschl  20:21

That may have been easier if we had a bunch of…

Speaker: Milan Bajic  20:22

You’re talking about the second follower…

Tom Keschl  20:26

We finally refer to that guy as, or me as the green shirt guy because once I come in, I’m like, I prove myself wrong so now I’m like, well, I want to do it the right way, I want to find some improvement here. So after we get everybody aligned, we brought in other Scrum Masters and like one or two members of the leadership team at first, once we said, hey, we want to align on measuring this stuff, we want to start capturing the full value stream in a uniform way. And everyone was like, well, okay, that’s not that big of a lift for us. Because of what Harner was just talking about, we already owned all those capabilities on the team. It’s just basically reorganizing what we decided to call things and making sure that’s standardized across everyone and so that was pretty easy left. So we got somebody say yes to that. Like, it was fine, everybody got on board. And then we let it sit for a while, honestly, we started just capturing the data, we cleaned up the data, and we started capturing it, we got the team’s familiar with like the new definitions, the simple definitions of things so that we could start getting some insight into the system. And then we waited about a month or two and we looked at our tool and to see what the data was telling us. And it was frankly, pretty astonishing at first. We started running an experiment based off that which I’ll let Harner talk about.

Speaker: Milan Bajic  21:56

Maybe just a pause here. I want to bring attention to something that I think there’s a misconception out there. So people joke around with how JIRA has become agile or is the most popular scaling. And I think it’s important to point out, just for you how the tool JIRA, and these are tools, were really to help you have quality data and to paint a picture. And right or wrong you used the tool in a way that actually helped you understand what was going on. Do you have any thoughts on that, because I think you both probably familiar with what I’m talking about, how people joke around the use of JIRA.

Speaker: Andrew Harner  22:51

That’s where I was going to call out next to. A lot of times the inverse of what we did is true, where, in the inverse being teams change their process to fit JIRA and they read that you should implement JIRA this way, and you kind of conform to it. And what we wanted to be very clear was that, I don’t want to change anything about how the team is working, There is a an Existing Value Stream. It is what it is and the teams are operating in that already. I just wanted to be able to have JIRA reflect that, reflect what reality was for them. And so Tom mentioned that kind of standardization or definitions, those things already existed, they just weren’t explicit. And so JIRA helped to surface those and make them visible to everybody.

And so work kind of naturally traverses this process already, and bring that forth, so really helped to enhance or amplify our abilities to measure our delivery capabilities, the things that already existed, right, we didn’t have to make any changes. We did change, but we changed the way we visualized it not the way we actually worked. And that was the key thing for us, is we getting everybody aligned, hey, we all agree, take JIRA out of it. We all agree this is how work flows through. And these are the types of work that we do. Now, if we are aligned on that, why don’t we have JIRA reflect those things so that we can use it to capture the data that we want? And so that was the easy sell that Tom said, it was fairly easy to get the teams to agree to that.

Speaker: Milan Bajic  24:42

So what happened when they started seeing that? I’m assuming it’s motivating to see this stuff and it’s also probably exciting for you. Now you get that whole hill dancing and it’s a party, right?

Speaker: Andrew Harner  24:59

Yeah. There wasn’t a whole lot of dancing at first. So Tom, he alluded we observed for a while. So we set this up, we put the hooks in, and then we just kind of backed off for a while I think it wasn’t a month, I think it was close to two, six to nine months that we kind of stepped back and just let the data start rolling in. And then finally, I think I’d almost forgotten about it for a bit. And then I pinged Thomas, Hey, we got like nine months of data. Now let’s go out there and check and see what’s going on. And so at that point, it reignited a flame that was like, wow, this is really powerful. We get we have all these different lenses that we can look through kind of our work. And it became very clear to us where our constraint was, it was like night and day. But there was no question that this is the area and this is why we hadn’t seen any improvements in two years. Because none of our maybe…

Speaker: Milan Bajic  25:57

Could you talk about some of those specific examples, what was like, Oh, holy shit, this is when everybody sees this, people will freak out, like… 

Tom Keschl  26:08

Well, the very first thing we noticed is that there was this weird pattern we’re seeing every couple of months, where work would just sort of all of a sudden, come in, in a huge batch. And we’d be working on a ton of stuff. And anecdotally at that time, the first time we started looking at the data, my team was also starting to complain and grumble about the number of things we had going on at once and we made like internal agreements about limiting that. At that point, nobody was really talking about the data, nobody was talking about the concept of, whip limits specifically, to solve that problem. It was all just sort of like, Hey, we’re really scattered, we need to focus up a little bit. And so we go, Harner and I go, look at the data, we see this giant spike in work in progress. And at that point, that’s when Harner reached out to his team to talk about the data and try to run an experiment.

Speaker: Andrew Harner  27:09

I brought this up, and I showed him some of the little charts and graphs and I said, every time we saw this spike in work in progress, we also saw corresponding spike in cycle time. So everything kind of we have these massive kind of tidal waves of sorts of work coming in and taking longer than we purge. And then work comes in, we take longer to purge. And I said, every time we have less work going on, those things are getting done in the time that they should take. They weren’t inflated. They weren’t bloated. And so I said, Hey, let’s run an experiment off this data. This was back in July, starting in August, let’s implement and enforce some strict whip limits. And the team agreed and then the conversation was, where do we start?

What is the appropriate whip limit? And there isn’t necessarily the right place to start, but it’s just let’s choose someplace and go from there. But we use the data to make that and so we saw that the team, it was seven, seven items was like kind of below where the team had normally been fluctuating. I think they were fluctuating anywhere from like 12 to eight at any given time. And so we said let’s exploit this, let’s run this experiment of limiting to seven. And so we did that. And then we sat back and watched for the next six weeks or something like that. And man, those tidal waves were gone. They were just ripples at that point. And so, that was the only change we made, we didn’t make any change to how we find our work, how we broke stories apart, we wouldn’t make any changes to how we developed or worked as pairs or it was nothing. That was the only change. And it was just a dramatic change.

Speaker: Milan Bajic  29:06

Did you guys notice anything because usually when you create constraints, like whip, usually it forces people like Hey, I’m now a developer, I need to test or did you guys experience any of that? What happened in that?

Speaker: Andrew Harner  29:19

Yeah, absolutely. So that definitely triggered some conversations about who’s responsible for certain things. And so we started to see more partnership take place between Dev and QA because if some test engineer was out on vacation, and now we went from two test engineers to one, one of the Devs would have to come up and help kind of take over some of those testing responsibilities. So we saw that kind of wall, which is an interesting thing, because we have developers, QA Engineers and operations engineers on a team but yet there’s still like these micro silos within the team of handoffs. And so we start to see those kind of artificial walls being pulled down, which is really, really cool. As a result of that…

Speaker: Milan Bajic  30:12

And that was all natural. Did you guys actually kind of nudge them or did that happen just by itself?

Speaker: Andrew Harner  30:19

I think it happened pretty naturally because now all the work was visualized. Everybody could see it every morning. We’ve got work piling up in front of QA and so is about to hit the limit here. So let’s lend on a helping hand. 

Speaker: Milan Bajic  30:38

It’s interesting. It’s almost like a lot of times what I see Scrum Masters and change agents kind of, I mean, not just a good word, it’s more like, tell people what to do or tell team members. And this is almost like you’re putting a mirror, using the data, putting a mirror to the team, and even that time, here’s what it is. And them self-organized. And I think that’s what I’ve seen naturally. And like what I’ve seen stick when it works, and when it doesn’t, is usually somebody enforcing these things or telling people. And it’s easier to do that, when you’ve seen it. You just want to jump and tell team what to do. But I also know apparently what transpired there with you teams is what is going to give you better results long term when people actually gone.

Tom Keschl  31:31

And we never really talked about that beforehand, I think Harner and I just sort of assumed that that’s the way we wanted to do this. We wanted to look at the data and we wanted to see what the data was telling us and any problems or challenges we came up with as a result of those conversations, we wouldn’t try to source the solution amongst ourselves, we would just raise the problem, we’d bring it to the team, we say, hey, we have this massive tidal wave of work in progress every two months, what do you guys want to do about it? Hey, we have this other problem, what do you guys want to do about it? And every time without fail, as we’ve started to take on these experiments, and refine them, and then reflect back on the results of those things with the team. We’ve seen the improvement and we’ve kind of gotten the team more and more involved in the process. And I think you’re right, I think it’s really, again, I don’t have much experience outside of this team with doing that but it’s a very powerful motivator, putting the data and the problem in front of the team and saying, how do we fix this? Here are some suggestions we have from our experience or whatever but how do you guys want to do this? And then, come back to it and see if we are successful or not. So after we did the whip limit, experiment, that was kind of the second time we got somebody to say yes, somebody to kind of get bought in a little bit more, get our hooks a little bit more into everybody. At this point too, Harner and I were meeting pretty much every day to learn about the data, to figure out what new we wanted to pull in to ask more questions because at that time, we were really only looking at lead times. And so we started every task…

Speaker: Andrew Harner  33:23

Every day. Yeah. Oh, what is this telling us?

Tom Keschl  33:27

Yeah, we didn’t even know how to read some of the charts. It was kind of crazy but and then we started thinking, well, what else can we get out of this? Is it just lead time? Are there other things? So we started looking at capturing a quality metric, we started trying to get all the DORA metrics and all that stuff. But the lead time picture, we hadn’t really talked about it with the teams, we did mention it, kind of almost coincidentally when we’re talking about whip, because people always want lower times, but Harner and I in the background, we were like, man, these blocks of time that we’re taking to do these things are crazy. And when we started visualizing how long work was waiting for the next phase, they got even more stark like the reality of the situation where we were living in, it turned out our regression times and waiting for regression because of the way that we structured the work. That is where all of the time was spent waiting, queued, like waiting for that activity to happen. And then spent where the work was done. There was like days, days of things just piling up and piling up and piling up there. So in our personal conversations, were looking at this and we’re like, Okay, well this is this is a little bit odd. And as we started looking at quality metrics, which we could also pull out of our ticketing system, it\ got even more obvious that, there were problems there. Our defect rate, our change failure rate as defined by DORA with hot fixes and stuff like that, those rates were actually higher than we felt were acceptable.

But what was interesting was that everyone on the team had been starting to get this feeling like our quality was starting to decrease, and you’d hear about it kind of like, not on the water coolers, because, you know, COVID, but in some of the one on ones that I was in with developers, you’d hear Oh, I’m starting to get worried about quality. And then, once we started looking at that particular data, here, we had a measure of how long this activity that was supposed to increase our quality was taking, and how not well we were doing increasing the quality of our deliverables. So that was the next thing and that was probably the one where Harner and I were like, we have to tell our boss. Well, actually, before I said that, I was like, we have to bury this, that was my immediate reaction. We cannot tell a single person about this, we just have to shutter the shop, pretend we didn’t look at it. But Harner only let me suffer that illusion for a couple of seconds, where it was like, well, let’s just bring our boss, wonderful man named Chris Browning. Let’s bring Browning into the conversation and show it to him. What was his reaction?

Speaker: Andrew Harner  36:34

It happened just like that. I was like, hey, Browning, you got a couple minutes to hop on this meet? And he was like, Yeah, sure. So we brought him in. And we walked him through kind of the logic of how we’re collecting this stuff and what it meant. And it was like, he got it. He’s like, Yeah, that makes a lot of sense. That’s probably where we should be focusing our efforts on our improvement. 

Speaker: Milan Bajic  37:01

Did he know, did you explain it to him in laymen terms, or was he familiar with the actual the Lean and Agile and these concepts, maybe even is you talk more, it’s like, you’re combining Scrum and Kanban. Or at least some of these lean, the specific lean practices and looking at it holistically, that was your manager. Because, essentially, what I’m getting at is a lot of times managers don’t fully understand this stuff. So as Scrum Masters, how do you actually get them on the board to understand this stuff? So it’s either through just using common sense, like this is where the things are or did you talk more in like, hey, look, what happened here when our, lead times increased or whatever.

Tom Keschl  37:52

I think it was a combination of that. I think Harner had been doing a great job coaching the people in leadership about kind of the DevOps and lean practices and what we could leverage here and there and to solve problems, but also the exercise we undertook, like months earlier, where we got very simple and concise definitions of, what is a defect? What is the task? What are all these different types of ticket types? Why do they matter? What’s our process? Why does that matter? I think that all made it common sense so that we could put the data in front of our boss, and because he knew what those definitions were, because he understood what the data was and because we had started to build confidence by running these very small experiments and showing the results of those things. It wasn’t like, well, I don’t trust the tool, I think this is wrong. It was like, Okay, now we don’t have to look for the problem anymore because we’ve just found it, this was our principal constraint for lead time, it was a principal constraint for the defect, the defect metric, the quality metric that we were looking at, too. So it was kind of, okay, whether you want to improve quality, or time the market or whatever this is the thing we have to focus on, because this is the activity that’s holding us up and that’s supposed to catch defects and give in to the acceptable rate that we would like.

Speaker: Andrew Harner  39:29

As we all know, agile development or iterative development, or whatever is about feedback loops, it’s about establishing that and condensing those feedback loops. So this regression activity, this regression suite that was taking 20 hours over two days to run, it took place the last two days of a sprint, was really the feedback loop for our developers to get that learning of, is my code going to cause a regression? And so we could make the argument that the principal constraint is the output of development. But we don’t really know if the output from the development work center was good or bad until it passed these suite of tests. And so creating the capability to condense that feedback cycle so that it was within hours of a code merger, a check in versus 10 days was even more validation for me that was where we should be focusing our efforts. But it’s been proven that to us, to Tom and I, it became very evident that is a very challenging and complicated and complex constraint to solve for. It has so many pain relievers, familiar with the value proposition canvas. If having this value proposition of optimizing our regression, and our testing capabilities, optimizing, this portion had so many pain relievers, we could get work through faster, we could learn quicker.

The way we are treating our tests, or thinking about our test strategy was test automation equals selenium equals cucumber. And so trying to break that mentality, and rethink how we can test this functionality, does it make sense to test all of this business logic in the UI layer? Can we strip it down and speed it up and test that logic at the API layer or via unit tests? It’s a very common, and then where do you start, you have a 20 hour regression suite? Like which of those tests do you start with and so that’s really where we’re at now. We’ve got our boots on, and we’re right in the thick of that right now, that conversation. And it’s phenomenal that the progress that we’ve made, and the feedback that we’re able to get from these changes with this data now. And so Tom and I have been monitoring each regression cycle since September, since we started refactoring and optimizing this regression suite. And at the time, it was about 20 hours. And up until last two sprints ago, we brought it down to about 16 hours, we shaved four hours off of it. Just that four hours was like a huge relief to our QA engineers. That’s a huge time savings for them. And then…

Tom Keschl  42:51

Sorry, the anecdotal evidence there at that time was, you would start hearing the QA engineers go, man,  this last regression cycle went pretty smooth, this one was pretty good, this one went pretty smooth. If they keep going smooth in the future, this is going to be all right.

Speaker: Andrew Harner  43:11

Yeah. And as we’re refactoring these tests, we’re bringing Dev and QA and product together and rethinking, the intended functionality of the application for each of these tests. So what is this cucumber test trying to do? And now that we all understand that we’ll rewrite it and so it’s not necessarily a one to one swap from a quality perspective, we’re actually seeing an increase in quality because we’re testing things in the right way, but also at the right time, quicker. And we’re confident that, it’s not the test that’s broken because we’re all aligned that it’s being executed the right way. So, we know that if the test fails, it’s a real failure, we should probably think about that. So anyways, we cut it down and this last one. So we were at 13-16 hours, we set a goal for q2, of 13 hours, we want to get under 13 hours. This last regression suite ran in six and a half hours. So you’re just shattered in that just you could feel when we started to talk about we don’t know if it’s an anomaly yet, because we don’t know if it’s going to continue. But the energy that came out of that conversation was infectious, everybody’s like, let’s try to get it down even more. Let’s see how far we can get this thing and so, we’re off chasing that right now to see what can we do to maintain that and drive it down even more?

Speaker: Milan Bajic  44:51

So maybe coming back to the manager again, your boss, can you talk about him a little bit? In what ways has he helped? What was his name? Sorry. Chris Browning…

Tom Keschl  45:07

In what ways has he helped?

Speaker: Milan Bajic  45:10

Well, in a sense, like, so a lot of times, here’s without putting Chris, as I said earlier, what I want to bring to attention is, a lot of times, people like Chris have no clue what’s going on. And when I work with these people, if you build trust, they’re like, Milan, what I’m trying to do is figure out, my role has changed. I’m trying to figure out how I can help the team without micromanaging. So Chris might be, exception, where he fully understand, but 98% of these managers, in my opinion, don’t have a holistic view, don’t know how to support scrum master. I just wanted to bring that to your attention, just maybe to like, what did Chris do that was helpful? What are things that he didn’t understand that you guys coached and helped them? Because I think, if somebody is listening to this, and they have a manager, or they’re the manager, they might find value in your example?

Tom Keschl  46:09

Yeah. So previous to any of this, I had just gotten my team lead position. I was a software developer. Before that, and I feel like a pretty productive one. And in those times, when Chris first came on board, one of the only ways, the only mechanisms we really had to figure out what technical projects we should be working on next, like what tech debt, what problems we had with the code we wanted to work on was to get everybody together in a room all the devs and be like, hey, what’s the worst problem in your opinion. And so we had a couple page document, where we would go through and then once we got everyone together, we kind of did this weird voting thing. And we prioritize that list. And we got some of that work done. And coincidentally, it was those improvements that we made, that prompted Harner in that January parking garage conversation, to ask, how have those fixes that you made those things that all the developers really believed were the worst problems that we solved in 2019? How did those meaningfully contribute or change our ability to deliver software? So back then that’s the only mechanism we had, once we started getting people to buy into the data, once we got good, reliable data, once we were made aware of it, the conversation completely shifted. We didn’t have to search for the problems anymore. When we focus on delivery capabilities, we could look at the data for our delivery capabilities. And we could see the problem.

Speaker: Milan Bajic  47:52

You could see where the bottlenecks are, and then point them rather than guess.

Tom Keschl  47:57

Exactly. Yeah. And it’s invaluable, like, everything became visible. Before, the only thing you had was a gut feeling. I feel like this, I feel like that, and the only time that was surface was in retro. QA would be like, oh, yeah, this regression really sucked or the developers would be like, yeah, these tickets over here, they really sucked, and we dig into why but we didn’t really, we would have that retro conversation and then like two weeks later, everybody forgot about the previous retro and maybe we’re complaining about some of the same things or maybe it was like four or five weeks later, where the problem was stuck up again. And so there wasn’t that consistent, holistic view with that historicity or that historical nature, where you could go back and say, well, here’s been our problem all along. It was very, like, set it, try to fix something, forget it and maybe if the problem comes back up, you didn’t even remember that you tried to solve that before. Whereas now it’s like, here’s the data, what is the data telling us? Let’s interrogate that.

Speaker: Andrew Harner  49:02

And specifically to Chris’s engagement, he always trusted us but we had helped to build his trust in the process in this information, and what it could do. And he was really the advocate for this kind of outward, he was kind of the conduit to our, up our leadership chain, and as well as the business partners. Our organizational structure is a little unique so our product and technology and QA kind of report up through a different hierarchy. And so, again, the conduit that kind of helped to bring the leadership channels together, and kind of be the voice of what we’re doing. So he’s helped to get these initiatives, these ideas kind of embedded into some of our OKRs and the kind of the discussions that happen at that level. So he’s really been that outward voice outside of kind of our organizational bubble, which has been really helpful. To help grow that, we’re starting to see some of the ripple effects and other teams showing interest in some of these things, which has been really, really helpful.

Speaker: Milan Bajic  50:28

So maybe that’s what we can discuss as a last topic. This is great across a couple of teams. You publicly traded company. How do you get others involved because that’s the tough part, you can’t just tell them do what we did, in a sense.

Speaker: Andrew Harner  50:54

But to a point, all we were doing is leveraging kind of industry proven patterns. Visualize your value stream, understand your types of work, be data driven by these really high level things. While each value stream is inherently kind of unique and special, right to the concept of visualizing your value stream end to end is something that should be applied, at least in my opinion, across any product or service. 

Speaker: Milan Bajic  51:31

It is, but you can say you could have another Andrew and Tom, that, for instance, one of the things that I noticed, for both of you, Tom has a technical background, I know, Andrew, you don’t, as far as I know, but you understand this stuff. So you can talk to developers, a lot of Scrum Masters don’t fully understand and that’s not their area that they’re comfortable with, so they avoid. So I think that’s one thing. You could say imagine like that you don’t have the technical back, you don’t have this and then you’re saying do value stream mapping, and you’re actually enforcing people to do it, you’re going to have a different result than what you guys did. So a lot of this comes to your leadership, your understanding, when to push when not to push and that’s the hard part to scale. The practices itself, like value stream mapping, like capturing the data, they can copy that, but it’s the soft skills, it’s all the things that you did, and that you keep navigating that what’s harder to scale. Because it’s the people side of them.

Speaker: Andrew Harner  52:38

I think it has to come from away. Your point in the words you use it force, if it’s a top down enforcement of you, you must do value stream mapping, and you must rethink everything. I think there’s going to be a natural resistance to that. I think throughout this process,  I was kind of poking and prodding and pushing Tom and others to help find what that trigger was, what the why was going to be. And each kind of chapter in this book, this story that we’re telling, we’ve identified another why, another reason why we should be continuing to drive forward. And I think that’s important to other areas. There’s other areas that we’ve partnered with and it’s predictability. That’s their why, like, they want to be more predictable. I want to understand why they’re missing deadlines. That’s a great, foot in the door of, hey, well, if we map out our value stream, we can kind of see where things are getting hung up, and we can see why we’re not predictable, we can see why we’re missing our deadlines. So understanding and trying to, I guess the right word is training empathy, establishing empathy with your customers, and also, as coaches or as, change leaders, we have to establish empathy with the internal business partners that we’re partnering with to see what’s going to get them to join the guy on the hill. So I’m out there dancing, I’m out there, I could be up there dancing for a while, but Tom felt bad for me dancing alone so that was his why. I don’t know.

Tom Keschl  54:29

Yeah. I think that you’re onto something there. We’re still very much in this. So it’s not like this is a solved problem for us. What we’ve tried to do is that every opportunity that we have something to share with somebody, we try to share it and be open and transparent. One of Harner’s favorite sayings, or at least the one that he tells me most often is that we need to be bold and we need to be vulnerable. Not everything in this journey has been rosy or sunny. But every time we’ve had a success, we want to talk about not just the success and how we got there, but also the pain points along the way. We’ve had audiences with, our boss’s boss and his peers, we’ve taken opportunities that we have with, newsletters to try to push various pieces of this thing, like when we first got our value stream mapped. And when we brought everybody in to agree on that, the next newsletter we tried to send out like, oh, yeah, we just completed this exercise. And one of the questions that came back is, what’s a Value Stream Map? Why is that important? Can you give us some background there. So we’ve tried to leverage every tool at our disposal talks, one on ones, conversations with our boss, trying to talk to the larger group of leadership on our project. Because everybody has connections everywhere. It’s a corporate company, but it’s a pretty small corporate company, and people know each other. So we’re trying to leverage all those relationships as well. And just kind of tell our story, tell the story over and over and over again, just like we got the teams to kind of buy in and be involved, just like we got our boss Browning to buy in and be involved. That’s what we’re trying to do now, and have seen some people get interested in that. And we’re working with now, various people throughout the organization, just as like a small community of practice. But trying to figure out where the problems are, what’s next, and how we can help the kind of other value streams at our company start to see some progress, like what we’ve seen, progress, it’s not even going to be the same journey in a lot of cases. 

Speaker: Milan Bajic  56:59

Is that not more of coaching, does it go beyond communities of practice? Do you guys actually coach, other Scrum Masters and other people outside of your value stream?

Speaker: Andrew Harner  57:11

Yeah, we got the network of change agents that we’re kind of growing. Our company uses G Suite and so we’ve got our avatars and slack. And so we’ve chosen I don’t know, we’re all Ninja Turtles, I’ve got different ninjas, we’ve got Master Splinter and so we can start to see kind of, who’s out there, who’s kind of working together and helping to grow, kind of just putting ideas out there getting people to think. But, yeah, I firmly believe it takes a need. One of the two most important works, literature works in our business I think is the goal and the Phoenix Project and the main characters in both of those, Alex and Bill, their company, or their plant, is on the brink of closing. And they have a need, right? 

Speaker: Milan Bajic  58:16

Was kind of urgency, right?

Speaker: Andrew Harner  58:19

Yeah, so to speak. And so how do we create that sense of urgency or that sense of need, so that it becomes a pull instead of a push in this information? Earlier, I got so frustrated, why don’t you understand this is so logical, why wouldn’t you want to do this? But you give them these books, but then the reality is, like those characters, they had a need, they were in a position of influence and of change, they could help really kind of turn things around and that coupled with, the sense of urgency that they felt from their plant closing or their team being shut down was really what drove them to help pull that in. It’s hard to establish that urgency or need when it seems like it’s just butterflies and rainbows. But you really never know when that next thunderclouds come rolling in, I mean, COVID hit. Nobody saw that coming and that probably established a sense of urgency for a lot of different companies. 

Tom Keschl  59:21

I don’t know, I think it’s been about the data to like when we were talking about things with velocity and story points, there’s that age old problem of how do you compare apples and oranges? How do you compare this team to that team or whatever, for whatever the reason is behind that desire it’s not really possible, but when we started talking about, we had a little internal technology talk about what we done and what we were measuring, and that five or 10 minutes part that Harner presents that information, it was very clear and logical to people. And there were a lot of bytes after that, a lot of people reaching out and getting really excited by what we were saying and wondering how they could leverage that and they’re part of the organization too. And really, it’s been the data that’s (inaudible 1:00:19), it may be an inappropriate term, but seductive. Those meetings that we had every day, when we were trying to just understand the data, it made me feel like I knew more about the project I was working on, than anyone else, it made me feel like I knew more about it than I ever had before, as we’ve gone from data unaware to data aware, and now I’m probably having a little bit of a data affair if that’s, probably silly but…

Speaker: Milan Bajic  1:00:50

No, I mean, it’s used both what you Andrew said, just here in the last couple of minutes, it reminded me of something that, as I’m listening to you guys, I can tell that naturally, you understand people, and if I had to guess at least I work with Andrew. So I know his people skills in his understanding, Tom, I never work with you so I don’t know that. But if I had to guess, you don’t fully understand psychology, but you understand people, right? And I think maybe just to bring it to full circle here because I think we’ve discussed a lot of different things here. And it’s like a lot of times Scrum Masters don’t fully understand the people, the culture and psychology of how do you create that sense of urgency, what sense of urgency for one person versus another. And if this is maybe just a message to aspiring coaches and Scrum Masters out there, I didn’t fully understand things till I started diving into psychology, understanding the people. And once I started reading, some of the things that you do naturally, that you just need to get feeling. Once you actually get good understanding people and how people think what motivates them, similar with the data, then you can start doing an influencing more, if you know what’s going on rather just and hold on to your gut feeling. So that’s what it is, as you guys were talking I think data is important understanding both Lean and Agile or DevOps. That’s the latest, but understanding that whole systemic view, using the quality data, having the right tools, but I would say a lot of people also forget about the people side of things. And that’s something at least that I’ve seen in an Agile community that, there’s more and more of need to understand that. 

Tom Keschl  1:02:50

This is for them. This is for all of us. This isn’t about well, it’s about bottom lines, and all that other stuff, too. But I think the literature and the studies have shown over and over and over again that if you treat your line workers well, if you make their lives better, your company will see success and that’s what it’s always been about for us. It’s been about making our lives better through making our coworkers lives better, through making our processes better, through delivering a better product and making our company better.