4.2 What's agile development? - Video Tutorials & Practice Problems

<v ->There is a particular process called Agile development</v> that especially if you're working with a data scientist could be a useful way of developing your AI models, but it's not the way that every company develops software. And so most companies develop software following a process like this. It's usually called the Waterfall Process because of this diagram where there's certain phases. And in fact, you might work at a company where they have this kind of stage gate process, they might call it, where they're releasing money for your project for you actually going through these various stages. And so this traditional Waterfall process has a few problems when it comes to doing AI projects. The first is that, it kind of assumes that you know what you're doing from the very beginning. It doesn't really lend itself to experimentation and it really puts different people in charge at different steps of the process. It's heavy on documentation and planning. It uses change requests and other type of change control procedures. So it's not really lending itself to a journey where you're not 100% sure where you're gonna end up and a process that you know you have to experiment to really get the answer. And AI really demands both of those things. So I don't recommend using the Waterfall process for AI projects because it just doesn't lend itself to that. What happens instead with Waterfall projects are the documents really become the handoffs from person to person. So you have each of these specialists that are working in kind of a silo, AI projects really don't benefit from that. Remember how I said how important it is for you, the domain expert, to be working with the data scientist? Well, there's other expertise that you need as well. Your experiments might have to do with changes to user interfaces, changes to processes, have everybody working together. Instead of trying to document everything and plan everything upfront, that actually works much better for AI. Because with Waterfall, the idea behind it is that if we really think real hard, we're going to know the answers to all the questions, we're going to deeply understand the requirements, write them all down. And you're also depending on every member of the project reading everything that was written down and understanding it. And we know that some people, that's not their strong suit, they're better at verbal communication. And even the hardest part and the part that really dooms you with AI is that you're hoping that nothing changes before you can finish the project. And with AI, it's almost guaranteed that things are gonna change if only because you're understanding the problem better. And so what's wrong with Waterfall is that it assumes that you understand everything upfront. It uses something they call BDUF Big Design Upfront. And it uses this step-by-step execution where the architecture is decoupled from the coding and the testing. Well in AI, that's not really how it works. There really isn't the same amount of coding, it's mostly experimenting with the feature analysis and the models. And it doesn't use different teams for different aspects of the project the way Waterfall does, Waterfall has different specialists that are in charge. And the other problem with Waterfall is it basically assumes kind of an all or nothing, that everything that we decided upfront we're gonna deliver or else we have to do change requests all the way along the way. And this process really isn't fun for anybody even when you're doing traditional software development. For AI, it's not just that it's not fun, it's not even really possible 'cause Waterfall demands perfection. And if you think about it and you say, could you really predict what you're gonna get done in the next six months on your job or in the next quarter or even in the next month. But if I said, could you predict then the next couple of weeks, maybe you'd get that right. Well, Waterfall really demands that you know months in advance how much you're going to get done. It demands that you really understand things really really well. So it would be hard to predict what your team's gonna achieve a long time from now, but even you personally it might be hard. And you know if you think you can do that then try writing them down and seeing if it actually happened. And ask yourself, what did you think you were gonna get done this time last year? And did that happen? There are all sorts of monkey wrenches that get thrown in because if it didn't happen, what happened? What was the thing that blocked you? Was it a reorg or a merger or a budget cut or a strategy change or personnel changes, or maybe the project turned out to be not even possible to do? And what did we do with those plans? What happened when those obstacles that we couldn't have foreseen jumped in the way? A lot of times we had to completely stop and replan everything. So Agile is actually the opposite of Waterfall. It's a different type of process and it started with something called the Agile Manifesto, where a group of people who were so sick to death of Waterfall got together and said, we need a different way of managing projects because the process has to change because the progress is not predictable. We don't know with perfect certitude what it is we're going to get done in three months or six months or nine months, maybe the best we can do is a week or two. And so instead of trying to focus on the highest priority being predictability of the plan, instead we said the highest priority should be to satisfy the customer and to early figure out what they need and continuously keep delivering and tweaking what you're delivering in order to make them happier. So changing requirements are normal. You don't need to change process, what you really need is to get feedback from the customer as to whether the problem is being solved. And so the process actually depends on getting feedback all along the way and changing your mind about what you're gonna do. So you deliver your working software frequently, might be every couple of weeks, maybe at the outside a couple of months and the shorter you can make that actually the better it works. And so Waterfall really focuses on process being the most important, Agile focuses on people being the most important. So it means that business people and developers have to work together every day. And so you make sure that you empower motivated individuals and you make sure that they are communicating with each other face to face if possible but definitely in conversation more than in writing. So you don't rely on these big long documents that people have to read, you rely on quick meetings where people are giving feedback and making decisions. And so there are short bursts of development. So you'll have what's called a product backlog. These are all of the potential features that you might wanna put into your product. These are all the functions that you might want in your product. And then you'll do something called a sprint. This is one form of Agile where you'll have perhaps a two week period in which you say, let's do these two things and you produce working software. So it's not a situation where people are working on software for months and months and months and then nothing operates. You always try to have working software at the end of the sprint. There's a lot of different types of Agile development. This one is called Scrum that we're showing on this particular slide, but you can also have Extreme programming, Lean development. There's lots of different religions of Agile. It doesn't really matter which one you pick as long as you pick something that allows you to continuously develop, continuously get feedback from the business team that the problem's really being solved better and better instead of having this long process where you hope something comes together at the end. Now I can prove to you that Agile is easier than Waterfall. And the way I'm going to prove it to you is that Waterfall is like baking and Agile is like making soup. So probably some of you know people who will bake but there are a lot of people who are happy cooks who say, ah, I can't bake. The reason is 'cause baking is actually harder. With baking you have to know exactly what you want upfront. You have to precisely measure the ingredients and prepare everything in an exact order and then you stick it in the oven and you wait and you wait and nothing is done until the end. It isn't even food in the middle. In fact, if you were to eat it, you could even get sick because it's got raw egg. And what happens if you stick it in the oven and you suddenly realize, hey I forgot the sugar. Oh, well, you start over again from scratch. You didn't make croissants, you made garbage. And so now you're going to scrap the whole thing, start over again. That's what Waterfall is like. Agile is more like making soup where you can experiment as you go. You can start putting a stock pot on the stove. You can start putting some vegetables in it, some water, start cutting things up. See what you've got, making vegetable soup. And halfway through, you can say, hey, you know what? We've got some cooked chicken in the freezer, if I throw that in the microwave and throw it in now, all of a sudden I have chicken vegetable soup. So you can even change things along the way and still not have to start over again. And what's really important is you can start tasting it. So you can taste your soup all along the way and you can taste and say, oh boy, that needs a lot more salt. Let me put the salt in there. And so it's much easier making soup this way than the way you bake. And the way I can prove this to you is that it is possible for you to make soup the same way you bake. You could decide what you want upfront. You can throw all the spices in, never taste it, put the cover on the stock pot, simmer it for three hours, ladle it into bowls at the table. You could do that but the problem is that it's a lot harder and the soup wouldn't taste as good. And the truth is that it's food all the time. If you decided that you were really hungry and there was nothing else in the house, you could eat the soup even after it was only simmering for an hour. The carrots might be a little crunchy but you could eat it. And that's what Agile was like. As soon as you have something better than whatever your current is, you can decide, hey this is the new current and you can never do that with Waterfall. And with AI, that's often a really important thing to be able to do. That you will be able to focus on making things better incrementally all the time. You'll be doing it when you first develop it and eventually get to the point that you say, hey, this is good enough for us to start using this in production but you're also gonna do it all along the way. You may come up with an improvement even a year after you're using it that makes it even better, and you'll follow the same process there as well. So how does Agile work? Well, you have very short projects. The system has to keep working all the time and the project ends on time. So at the end of that sprint, even if everything you did, that you decided you were gonna do is not complete and the product owner, and in many cases it might be you, the business person decides what we're gonna do, the development team decides how they're going to do it, and you work together to figure out when. And so there's a number of different players in an AI project, you have business people, you have subject matter experts. You might have a developer who's actually kind of trying to corral the data for you. A technical architect may be involved if it's a really complicated project. A data architect, they may know how the data works and where to get it from. You might have a user experience person if you've got a user interface in the project. Maybe a data operations person who's helping you pull the right data all the time. You probably are gonna have some type of data scientist or AI or machine learning expert. And you might have some type of interaction designer, that might be the person who is actually understanding what the process flows should be. And before even the user experience person gets involved, what does your user need to know? What are the things that they would understand in order to make this project really work? Now some of the people that you might work with in this project might not be so excited about Agile. They may feel like they already have their job wired with Waterfall. They know what they're supposed to do and they plan everything upfront and they do it, and now you wanna change the rules and they might be afraid they're not gonna succeed with the new rules. They might actually be responsible for making something work not just execute their specialty. And so this could be hard for them and so they might actually be afraid. And so a lot of people fear change, even if they don't admit it. And so one approach you can take is to actually scare them more about not changing. What happens if we don't change this? The project's gonna fail and they're gonna blame us all. Right? So that's one way to approach that. But some people struggle with change but it's really a certain type of change they struggle with because not all change is the same. There are actually two different types of changing; technical change and adaptive change. A technical change is just a change in behavior. So for example, if you just wanted people to start coming to the meeting 10 minutes early, that's really not that hard to do, but if you actually want them to prepare totally differently for the meeting, you want them to have actually read some pre-meeting document for an hour before they come, that's actually a change in the way they think about their job and what they have to do before they get to a meeting. That's kind of an adaptive change. So an example would be, how many times have you said to yourself, hey, I'm gonna do this new thing. I'm gonna write that book finally or I'm going to start exercising every day and I'm gonna get up an hour earlier every day. And which if you haven't thought it through very much, you might say, well, this is a pretty easy change to make, I just set the alarm for an hour earlier and I'll start getting up an hour earlier every day. The problem is that you're treating it as a technical change when it's actually an adaptive change. Now why is it an adaptive change? Well, because you have to change the way you think. You have to think about how you're going to get that sleep differently than you do now because you probably have to go to bed an hour earlier the night before in order for this to really work. Because what probably happened to you is you tried getting up an hour earlier every day, and then within a few weeks you said, oh boy, I really can't do it. My body clock just can't adjust to this. You told yourself a story of why you failed, but the real reason you failed is you treated it as a technical change when it was actually an adaptive change. And so what do you really have to do? Well, you have to decide what you're giving up from the night before. So maybe you were watching The Late Show the night before or maybe you were spending another hour with your significant other, and you're not gonna be able to do that anymore, you have to go to bed an hour earlier and give something up, giving something up is an adaptive change. That's the hard part about Agile is that you have to change the way you're thinking. And so if you have people who really like Waterfall and they're wired for it, you have to help them change their thinking so that they understand why Agile is really gonna be an easier way for us to work and a way that will actually help us to deliver these AI projects. Otherwise, you're gonna be doomed to failure because they are going to end up not being able to adapt and that would be the worst thing when you're trying to implement your Agile project.