puppets depeche mode

hola. me llamo doug. hablo un poquito espaã±ol. so that's about the extent you'll hear of it in this presentation.this is "i want it all and i want it now: configuration management and ci."if this isn't what you wanted, you're stuck here now because i will watch you and glareat you as you leave. no, i won't actually, so if you want to leave...hopefully you're in the right spot. i'll go ahead and start with a little bitabout myself. like i said, i'm doug dobrzynski.i am build manager and a drupal developer at promet source.we're based out of chicago, illinois in the united states.in the past, i actually got started with drupal

at the university i attended where i actuallybecame the student manager of the web help desk, taking care of hiring students, ticketing, managing all that allocation, the workflow. it started out with me begging for help becausei alone as a student could not be the web help desk, growing it to, at one point, havingeight other students working under me. i've actually made websites, though, sincefifth grade. although you might think i major in cs, computerscience, i actually was a history major and a russian, central, and easter european studiesconcentration, so it goes to show there are many paths to technology and drupal, and it'smore about passion and where you want to go. i wanted to get a little bit of an idea ofwho you are first as an audience.

how many of you are developers?so most of you. how many of you are sys admins?so there's some overlap of roles. how many are project managers?some more overlap, not uncommon. how many are upper management or are in moreleadership roles? several more as well, so a fairly good mixwith a lot of overlap, which will be very good, i think you'll find.so i wanted to start by saying, "why does this topic even matter?"why is configuration management and ci important? how many have played the "what's on staging?what's on production? who knows?" game? how many of you have been in that situation,trying to figure out what's on production

versus staging versus in development?so several. how many have had times where you've had toplay deployment roulette? oh, it worked this time in production. itworked this time. oh, this deployment blew up.any of you experiencing that? a couple more as well.how many have had or heard or said, "but it worked on my machine"?so that's a lot more common. and how many have played the "well, it wasfixed at one point" game? so several more now, i think.throughout those most, if not all, of your hands went up.this is why this matters.

this is why configuration and ci matters,to help deal and hopefully alleviate these problems or, at the very least, reduce them.i want to get started with a little bit more theory to think about what it even is, solet's start with configuration management. here's one definition: "configuration managementis a critical cornerstone of it automation, providing tools that allow you to centrallymanage the packages, configuration files, process state, firewall rules and other settingsthat equip servers to do their assigned jobs." "configuration management is the managementof configuration items, which are the things that you care about, the things that are essentialfor delivering an automated business service." "configuration management is unique identification,controlled storage, change control, and status

reporting of selected intermediate work products,product components, and products during the life of a system."the ieee institute of electrical and electronics engineers way back in 1987 in their guideto software configuration management actually included these four components: identification,control, status accounting, and audit review. so this isn't a new concept."software configuration management (scm, or just plain cm, configuration management) isany organizational framework--that is, a discipline--for managing the evolution of computer systemsthrough all stages of systems development." okay, who's confused?anyone in here confused, because i am and i'm supposed to be talking about this.as you can see, there is some debate and variation

over exactly what configuration management is. if you decide to take on configuration managementas a company, this is something that you will have to think about.what does configuration management mean to us?come to an agreed upon definition so that you're all talking about the same thing, andmaybe identify certain parts of it that are important, or more important, to your businessaspect. for our intents and purposes, i'm going todefine it as setup of the infrastructure used in order to get your software from the beginning,that is from programmers and developers, to the end, that is the customers.i think it's really important in there to

define it not as just the infrastructure setup,but why are you setting up that infrastructure? what's your ultimate goal so that you don'tforget where you're going and are blindly going down a path and then say, "oh, maybethis isn't exactly what we wanted"? it's always good to keep that end-point inmind, that you need to deliver a product, that developers may have pieces of the code,and somehow you need to get that together and out to the client and be able to havethat consistent. why should i even adopt cm?the beginning questions where everyone almost ended up raising their hands, thinking aboutthose scenarios, are part of the reason. puppet labs actually outlines three reasonsfor cm.

not automating configuration management causes pain. so if you look at this as setting up servers,if someone's going in and manually setting up your servers, all your vms, it's goingto take time. they'll probably forget something. if they'resupposed to set up five servers by tomorrow, chances are the developers or the client isgoing to go to use that server the next day and something will be missing.something will not be working. it might be, "oh, we forgot to set up dnson this. oh, the package installs actually failed," or "oh, we forgot a package."then you have to go back. if it's gone to your client, and you didn't even catch that,that's embarrassing. your client's not going

to be happy.it's frustrating for developers who say, "okay, here's all the work. let's push this out toa qa server," and now it doesn't work. they can't push it out.now qa's delayed. the project manager's probably going to startworrying and say, "oh, we have this much time to do qa, but we don't have somewhere to qain it. when is this going to happen now?" relatedly, everyone benefits from automatedconfiguration manager. the clients don't go to a server and finderrors. if you're setting up a server for the client'stechnical person to use, they don't go in then and find issues with it.your developers are able to push more consistently

without issues.qa can get it somewhere where they can actually qa.it makes people happier, and in that process, it saves time and gives you time to do otherthings that you want to do. now you can work on improvements and innovations.now what maybe took hours or maybe even days to set up x servers, you can automate, getdone in half a day. now you have time to say, "oh, what else canwe do? what can we do to improve our company? what can we do to be even better?"and you can deliver things faster to your client.what are some tools out there? i'm not going to go into a lot of depth inthis, but puppet, chef, ansible, and saltstack

are probably the four that you'll hear mostoften. i tried to find some metrics in relation inthe drupal community to how these were used. i didn't have much luck, so i searched anycommunity, just what are some metrics on the use of these?which are most popular? the bottom line is metrics are hard.i won't go into details on how it's measured, you know.do you measure downloads? do you measure installs?how do you measure that? do you trust people to report?how do you get enough people to report to get a good picture?there's an article that i'll link to, a blog

post on some of these challenges of gettingmetrics, which configuration management systems are out there.but some of the four ones that you'll want to look at: puppet, chef, ansible, saltstack.i can say from anecdotal experience, from what i've seen in talking to people withinthe drupal community, puppet and chef are probably the most common.so then it becomes a question of which do you use?these are questions that have been asked before, and there are some pretty good answers outthere, so i'm going to just share a couple of them.this comes from a blog post on just puppet versus chef.if you want to get into some really interesting

debate, just google "puppet versus chef,"and you can go down a mine hole of information and comments, but one of them ultimately concluded:"if you have time to try them both, then do. it's a bit of a case of horses for coursesand you may be surprised. doing one is light years better than doing neither, so if you'reagonizing over the decision, just stop, toss a coin, and get going today."bottom line: something is better than nothing. similarly, in a book "taste test" that comparesall four that i mentioned, matt jaynes says, "just using any of these tools is a huge win.having scripted, documented, replicatable servers can speed up your development andoperations dramatically." ideally you would want to pick probably acouple.

try them out in your situation.try them out on your projects. set a stake."we're going to implement this to this state on this project by this day using no morethan this hours" and see how far you can get with that.you may get there; you may not, but it'll give you a good idea.it may be, "this isn't the right tool for us."it might be, "maybe this is more than we can chew off in this time."it might be, "maybe we don't have the resources to do this."moving on then from ci, starting again back with the theory: what the heck is ci even?when talking about ci, of course you have

to look at martin fowler's blog post on ci.so martin fowler defines ci as "a software development practice, where members of a teamintegrate their work frequently. usually each person integrates at least daily, leadingto multiple integrations per day. each integration is verified by an automated build (includingtest) to detect integration errors as quickly as possible."let's go ahead and break that down a little bit more.there are only four aspects required for ci: version control, build script, some sort offeedback mechanism, and a process for integrating source code.obviously you're going to want your items, your configuration, your development intoa version control repository, a single version

control repository.i want to add, i was explaining this to someone. they're like, "yeah, we have that first stepmet." i'm like, "okay. point me to your versioncontrol repository." and their like, "well, it's on every developer'scomputer." having a single version control repositoryisn't the same as using a version control system individually. using it individually, every single developercould track changes they made, and they could say, "yeah, i made these changes. i made thesechanges. i made these changes," but there's no way to track what's being merged, and atthat point, the project managers, when they're trying to get a picture of what's done orwhat changed, or you're trying to debug what

might have happened, what are the changesin this time frame, it's going to take a lot of time to get your developers to report that.you might run into, then, "well, i'll leave out this one detail because it doesn't seemimportant and it's embarrassing," and it might turn out that's the one detail you needed.so you want to have a single version repository, the master where you know this is where everything'sgoing. this is the state that we can use.you're going to want to have a build script, some way to create your system.in this case, once you have a server set up, you're going to want to get drupal, get itinstalled, get the right modules enabled. if you're maybe doing updates and changes,you're going to want to get all those changes

applied, and you're going to want to scriptthat. you don't want to have to go in and say, "okay,i need this change, this change, this change. oh, this change requires i go into the uiand change all of these pieces." you're going to want to automate that so that,a., you don't miss pieces. it's more reliable that way, and, b., it saves time.you have that automation once. you run it. feedback mechanism.when you're doing tests, you want some way to know a test failed.when you integrate the code, say, whether it's on after github, or if you have an integrationserver, you want to know when those are failing. so someone's code that they're trying to mergein has issues with code that already exists,

you want to know when that's going to fail.ideally you want to send that out in some way, whether it be e-mail, a text message,a notification, in a chat, you'll want to think about what are the best mechanisms toget this out and who do i want to get this out to?one thing to be really careful in your feedback mechanisms is information overload.how many of you get github notifications for every single project or notifications forevery single ticket updated, even if you were the one who made that change?what happens when you get those e-mail notifications and you just see a ton of them?you don't see them. they disappear. so you want to find the right amount.it's probably not a good idea to send... if

tests are failing all the time for every individualdeveloper's prs, you might not want to send that out to all developers because they'regoing to start saying, "not me, not me, not me, not me. i don't need to pay attention.there's one there that i did see that was me. there's another one that was me, but noneof the first ones were me, so i'm not going to pay attention."you're defeating your purpose of getting feedback fast.you want that process for integrating your code, making sure there aren't conflictingchanges, making sure that when you put all of the different pieces together, they fitand don't break. martin fowler goes into a lot more depth intosome practices.

these entail the pieces that i talked about.you'll see a lot of the same items here. like i mentioned, you want that single sourcerepository. you want to automate the build, automate tests.you want frequent integration. your developers ideally should be making dailycommits and pushing these, and then all those commits should be merged together, say, onan integration machine. if you're looking at cases where a developeris not committing daily and pushing that, that could be one of two things, usually.there's probably an edge case that i'm not thinking of, but it's either, a., they'redoing a ton of work, and in three days they're going to push eight new features, or theirtasks are too big.

they're too big or too complex, andthere's some problem there. you might need to break it up into a smallerpiece. it's easier to complete and test a smallerpiece, and it might also be that what they're running into actually is something that'sreally complex. maybe it's a bug that could take way moretime, and it's raising a red flag, "hey, this probably is going to affect our timeline.we're supposed to have this one piece of functionality done, and it's not coming."so if you are expecting daily commits, frequent pushes, and all of a sudden those slow down,that's a good sign that something's not right and raises these red flags in the projectearlier rather than finding out later, "yeah,

i'm doing work. yeah, i'm doing work. so wait,what were you actually doing the past four weeks?" which can put project managers andtechnical leads in a really tough spot. another key item is if the build breaks, fixit immediately. if it's an integration where there are multiplepeople working on the code, it's probably good to stop work and get together and resolvethat. in fact, you actually should do that.you shouldn't continue work while the integration build is broken, because that means you'reworking against something that's broken, and when it gets fixed, you may have to go backand change what you implemented, whether it's a new feature or a bug fix, or you may breakthe build again and have to spend more time

fixing the build.if it's someone who... if you're running tests on every pr and it breaks thebuild, then that person should go back and look at their prc. "what did i do to breakthe build?" the other thing is you'll want to keep thebuild fast. projects get complex, so you may find thatyou have to break this up, look at different ways, whether it's throwing more resourcesat it or not running all the tests all the time.i was working on a project where we were running travisci with a bunch of tests, and it wasa huge project. we were building the site from scratch allthe time, so site install plus enable all

the modules being used, enable all of thefeature modules, do a bunch of configuration then, and run all the tests, and it got towhere it was taking 30 to 40 minutes. that's too long for a single person.it's slowing down your rapid feedback time, and then you may say, "okay, i completed thistask," and before you can even get it to development initially, where someonecan take a look at it, you're waiting maybe 40 minutes for a two-line fix, that maybeyou didn't have to run all these tests on just the pr.you'll want to work in a clone of production. this should be fairly standard practice now.you don't want to work in an environment that's not like production.this becomes especially important on staging

or pre-prod, wherever your clients are lookingat it and approving. you want that to match production.you don't want to push to production and then find out, "oh, maybe there was this one differencethat was important." like i've hit on before, you want reportsand transparencies, so ideally you would be able to say, "exactly these changes were pushedto the integration server at this point. these are the changes that are on staging rightnow. these are the differences from this time to this time. here are the tests that arefailing. oh, look. developer b's tests fail every single time. maybe we should look atthis," and it's going to improve your process. you can get feedback for helping your developers,improving that.

you can save time because your project managerdoesn't have to come every single day, maybe multiple times, and say, "what's on staging?what's on development? what's done? what's not done?"and then another key that you'll probably see over and over is automate your deployment.ideally you want to get where you can say maybe dev is built automatically when codeis merged into the main line. at this point it's cut and release, and we say, "deploystaging with this version." then you can say, "okay, deploy that versionthat's on staging to reduction," and the goal is to get to a one-step deployment with asmuch automation as you can. so hopefully you should have some answersright now to, "why should i even adopt ci?"

it reduces your risks.you catch errors sooner. there's less chance of these errors makingit to production and your client calling and saying, "why isn't this right?"it also reduces repetitive manual processes. this saves time.your developers can actually be doing what they're paid to do rather than running a bunchof commands or doing a bunch of pointing and clicking or setting up servers, whatever itmay be. they can work on other, more valuable tasks.you can generate deployable software at any time and any place, so you shouldn't haveto wait to get software out. you should be able to say, "okay, this isgood to go. we're going to deploy it," obviously

going through approval channels.you have a lot of qa done ahead of time. it enables better project visibility.you know when things are breaking. you know when you break things unintentionally.to have one example of project visibility, i actually was working on one page, or onespecific view, and that was used on a few sections of the site.i push it, and all of a sudden, our home page tests are breaking.every single landing page test is breaking. this affected one section.this other part shouldn't be breaking. if you had qa done, hopefully you would havehad a thorough enough manual qa process, if you're still doing that, to check other areasof the site, not just what work was being

done, but this was no time other than to writethese tests initially. i know everything's breaking.this saved us embarrassment. we didn't even get it to development or qa.i saw that i broke the build. i went back and looked and said, "oh, i onlyused one equal sign instead of two and set the view name on every single page."this is an example of something that could have been really embarrassing.if this were a hot fix, and we're pushing this one thing, you want this to productionrapidly, you go through qa,your qa person looks at the pages that's on, doesn't look at any other pages, you pushit for production,

your client's not going to be very happy ifhalf their site breaks, and that's feedback. i get the notification that i broke the build.i go and fix it. it's only me.i'm the only one looking at it. it saves this time from other people, andi know automatically without having to go in qa first, which is something i would sayprobably not a lot of developers are fans of.as that example shows, you can establish greater confidence in your software product from yourdevelopment team. when you have these tests in place, you can know, "this didn't break,all these other things that we're testing, and we didn't even have to look at this manually."you can say with a much greater chance, "my

changes aren't going to affect other placesor break unintended things." what are some tools out there?there are a ton of ci tools out there. you can get into continuous delivery and deploymenttools. these are probably some of the more well-knownones and more used tools. there's travisci, jenkins, bamboo, hudson,cruisecontrol. if you want to know even more, i stronglysuggest looking at the wikipedia page. they have a fairly good comparison, at leastas a starting point, of "these are all the tools out there," and it's huge.now let's take a step back a little. you may be asking, "why did we go throughall this theory? why did we got through all

this justification? isn't this supposed tobe you telling us how to do it?" yes and no.in a way, i am, because this is what you have to go through as a company when you want tostart building this culture of configuration management and ci.you have to define these aspects. look at all the tools out there.look at what do we want to accomplish? what's all the work it entails?and get people on the same page. you have to motivate people.there may be a project manager who's like, "look, yes, that's sounds all nice and fancyand cool and all that jazz, but i need my project. we don't have time to do this."and you can go through and give these

justifications, explain, "this is why it matters."yes, it may take more time up front. actually, it probably will take more time up front,especially if you haven't done this as a company on any projects, but you can look at "here'swhat it offers." i can offer you greater assurance that mycode, that my changes didn't break the front page, and i can save you time now that theqa wouldn't have to go in and just say, "there's no front page," which is a waste of the qatime. so you're saving resources on that project.if you haven't noticed, it's a lot more about just the tools that you use or maybe the processthat your developers implement. i've seen cases where they say, "hey, developera, this is admin a. work together. make ci

happen."no other company involvement. you can probably guess it doesn't always goso well. it's really something that you need to buildup as a culture and as a practice and get everyone on board to know why this is importantand to help hold each other accountable and help each other.if you have other people working with you as well that are supportive, it's a lot betterthan pounding away at it as a single person and saying, "i'm running into this problem.i'm running into this problem. i'm not making much progress" or, "i'm doing all this coolstuff and i'm helping, and nobody else cares." all of a sudden, that person starts gettingunhappy, they become frustrated, and they leave

like i just said, in probably the book on continuous integration called "continuousintegration: improving software quality and reducing risk," ci is not just a technicalimplementation. it is also an organizational and culturalimplementation. people often resist change, and the best approach for an organization may be to add these automated mechanisms to the process piece by piece.this leads into my advice for you, advice to think about as you're looking at usingconfiguration management and ci. i kept it actually really simple and havefound that you can get overwhelmed in a lot of it, which is kind of ironic and maybe hypocriticalthat i threw all this theory at you and probably

overwhelmed some of you, but the key is startsomewhere. look at one piece you can do and say, "oh,this is something we can do now." hey, we hired this new person who knows chef.let's have them use chef to configure all of our servers.talk about it. get other people involved.like i've hit on, it's not just about the tools that you're using.it's not just about the technical aspects. it's about culture and people communicatingand building that trust and getting it going as a company.it's not just a few people doing ci. make it a priority.this should probably be an obvious one, but

it's something that takes time and dedication.if you say, "okay, yeah, we'll use ci," it's probably not going to happen.i used "make" very importantly here. it's important not just to say, "yes, thisis a priority." there's work involved in making it a priority.maybe you have to take on less work to have some more time to implement this.maybe you bring on another resource. you have to take action as well to get thisstarted, get this implemented. making it a priority also means involvingother people and getting them the time. it's probably not going to go very well ifyou throw this on one person and say, "you have x hours to make this happen."relatedly, allow time.

not just time as in priority for people todo it, but time for this to come. don't expect, you know, that silver magicbullet. it's going to not happen at once.you may start making small, incremental improvements, so there's a lot to do.there's a lot of things you can go. you may look at, "this is where i want tobe down the line," and lose sight of what you're actually doing.you can say, "i want to be all the way in california when it's sunny."if you're aware of chicago weather, it's snowy, and i'm missing the fact that, "hey, the sidewalksare cleaned now." if you don't forget that, it takes time forthings to change.

one way you can get started that promet'sactually worked on is what we've called the drupal 7 framework.it's an easy way to start a drupal project. what we've dubbed the "promet way," it's byno means the only way. it's a way that we've started using.so working locally to get started, you'll have to have composer, but once you have composer,it is two steps, two commands to get a working installation of drupal.you use composer, create-project, promet/drupal7-framework pointing to the github repo, and then yourproject name. that will get all of drupal and several modulesthat are fairly common on every project, including views, items like that.then you run vagrant up, which, if some of

you are familiar, we use vagrant to createa vm instance locally to have the site. with those two commands, you can have drupallocally working, and you have several key modules you need.there's no need to go download drupal, get a virtual machine, do all of that.i'm going to dive in a bit deeper into several pieces that the drupal 7 framework pulls together.these are all open source tools that we have that either exist in the community or we havebuilt on or used. we use what we're calling drupal tangler.this actually uses composer to manage your dependencies and get the all files in thecorrect drupal structure. so getting your modules and sites/all/modules,getting your themes and sites/all/themes.

the nice thing with using composer...how many have actually used composer or are familiar? for those of you who haven't or have used composer and do code reviews, have you everhad that instance where you're reviewing someone's code, and they had to implement some functionality,and it's maybe 10, 50 lines of code buried in 5 modules files, and you're trying to reviewthat code. have any of you been in that situation?how fun is it? how easy is it to review their work?-[inaudible] -yeah, so there's a lot to grab, a lot ofline. it's difficult. it takes a long time. by using composer to manage your dependencies,you have one line that you have to change

or one line that you add to say, "i want toadd this module" in your composer.json, and then it will also modify the composer.lock,which locks the dependencies, the versions, the hashes used.the nice thing is github hides the lock file, so all you see is that one-line change andthen all of their change, so it makes it a lot easier for review.it also makes it a lot easier because now you don't have to go and download all thesemodules. it's also much better for version constraints.how many have taken on a support project, saw that all their modules were outdated,done updates, and then find out, "oh, they actually didn't want to update that one module"?this can help you on that because you can

say, "use this very specific version or onlyuse minor releases," and then you can look and say, "oh, this is pinned to version 7.x.1.2.there's a reason it's pinned to that." it saves you time there.it makes updating a lot easier. you can see what changed from version to versionin one file or looking at the composer.lock file.it uses drupal libraries installer plugin to get libraries that you may need and placethose into sites/all/libraries. we use drupal settings compile to load configurationfrom an arbitrary directory on a server instead of templating settings.php or versioning databaseconfiguration. we use drupal drop ship, which actually usesa fork of kraftwagen manifest to create a

reusable deployment composed of manifests.is anyone in here familiar with manifests? kraftwagen manifest?it's actually really hard to find a good definition of kraftwagen manifest.the goal of kraftwagen manifest is to provide an idempotent deployment, a deployment thathas the same results every time, but you basically define a manifest, name it, give it some information,and then you define what that manifest does. the way we use manifest, one that's a reallygood way is configuration that might be different on development servers or locally versus production.so one good example of this would be commerce settings.you want to make sure that you are always using the production, the live commerce clienton production.

you do not want to accidentally change thatback to dev and maybe cost them thousands of dollars, so this defines a kraftwagen manifestusing environment variables to make sure that those settings are always right on every deploymenton production. relatedly, make sure then that, once you definethem, that it sets up test settings on development staging, development integration server aswell as locally. my advice is if you start doing this, if youlook closer there's a default end file which you can copy and then override.set your defaults to non-production settings. that will save you a lot of time and is generallya safer thing to do. you can also use it to, say, configure, "howdo we handle e-mail on development settings?"

you don't want to actually be sending e-mailswhen you're testing. this is something that we've learned the hardway, but it's you'd think would be something every developer would set up.i set up development environment, this site sends e-mails.maybe we shouldn't send e-mails to everyone on the college campus with an e-mail addresswhen we're testing the e-mail send function. if you want to learn more about it, there'sa whole blog series that we have. we're actually right about at time.i want to make sure we have some time for questions.-have you noticed that there's been a lot of presentations and activity around ansiblein the drupal community more lately? there's

been a book written that includes drupal playbooksand stuff like that. i agree that if someone in the group knowschef or whatever, start where you can and that's it. it doesn't matter, really.but ansible seems to be a much easier approach. i don't know if you'll agree with that ornot. anyway, to discuss. -i have noticed that it is becoming more popular.i haven't worked with it. we've used chef, since that's what we havedone and what we've known, and sometimes your reason for using something may be. that'swhy. any other questions, thoughts?-when you want to talk your manager into getting into this and convincing them, it's not veryeasy to convince them with your words unless

you have enough power to do that.what i think may be very, very useful is to know worst cases or bad cases that have happenedto you. do you know of any that may be of particularinterest in the situation? in that case you can just prepare a good,very detailed argumentation of why you are proposing going with this methodology, whichis the right thing, of course. -that's a really big question.-like a serious situation or an accident, a serious accident [inaudible]-before it's gone manual? -[inaudible] there must have been cases where someone at some time [inaudible] -i can actually give one example, and theni'll have andy give another.

so in the very, very early stages with justusing build scripts, nothing nearly this complex, say everything needs to be in code, we usedbuild script to get the correct settings on dev versus staging versus production.someone decided that's too slow. we don't have time to make sure we defineall these things. the client comes and is like, "okay, we needto do updates." so we do updates, and their whole site breaksand, even worse, we were getting inconsistent results.sometimes it would take one action, and sometimes it would do something else, and we ended upfinding one thing that was breaking, which actually was the home page, was happeningonce every ten times.

this was because he decided we aren't goingto use a build script. we aren't going to make sure we're defining releases and havea proper channel for qa. it's client wants this on staging to reviewnow, so go in and cherry pick some stuff, and only that developer knows.i had to spend three months going through this project to fix it just so we could doupdates, just so we could run updates, and in the process i found out that there wasmissing work. if that's not an argument for process...one example. andy, do you have another?-so dougie and i work together. one argument that we had for it, we actuallyhad one of those projects that was pretty

risky.it had a very tight timeline. it had two teams, a very large client teamand a promet team working together, and it had a drop-dead deadline with a very complexe-commerce build. we decided to add the extra time to buildtest cases for every single feature, for every single item.yeah, in the beginning it was a little frustrating because everything took a little bit longer,but in the last month of that project when you had 20 developers working on this thingand they had to run the build themselves to make sure that things didn't break, that reallysaved our bacon. there's no way we would have been able todeliver if every one of the developers would

have to build this thing from scratch, makesure it runs on their machine before they deployed it, and you had an automated testrunning all the test cases. yes, it did take longer, but when you're workingon larger teams, it does save your bacon. so there's one case for it.any other questions, thoughts? -demo.-demo? you guys want to see a demo here?we'll see how this goes, since this is completely for andy's humor. like i said, this is all open source. i'm going to go ahead and find that commandjust to save my typing time.

we'll see how this goes.just doing composer, create-project. that's the path to the repository on github,and i'm going to call it drupalcon, and we'll see if it'll actually go. and it looks... my internet. it looks like, because i don't have internet,it's not going to go. i won't troubleshoot that in front of you,but i actually did do that in preparation, debating doing the live demo, and it doeswork. it creates that directory in the folder thati was in with, in this case, drupalcon as a name, has allthe projects, run vagrant up, and there you go.

it always fails in live demo, especially when you don't prep it.any other questions, thoughts? -my wifi is not working either. i think it's...-okay. -yeah, it's not working.-yeah, it's the wifi, but unfortunately i can't show you the magic. if there aren't any last questions, i'll go ahead and put up my contact.i'm dobrzyns on drupal. my twitter handle is kind of long. i shouldfigure out something better, but if you want you can find these slides and all the linksand resources in there.

i'll be around at this time as well.i have some business cards if you want to come up, chat about this, have more questions.feel free to try this out. check it out.check out drupal 7 framework. see if it's something that you like.provide some feedback for us. maybe there are some ways that we can improveit. we're happy to entertain those and work withthe community as well. thank you.