There are various roles internal audit can play in major system implementations and one of the most common is post-implementation reviews. Some things can’t be changed, or are very expensive to fix, after a go-live. Instead of coming in and bayoneting the wounded at the end of the project, there are several other approaches we can take that actually provide much more value to the organization.
Tom Harris, the Vice-President of Internal Audit for National CineMedia joins me to provide a case study on how he took an approach of embedding members of his internal audit team into the project team, and even helped drive the user-setup portion of the project. We explore the risks and benefits of taking different approaches. While embedding internal audit into the project team may limit objectivity and assurance projects for a time, it just may be one of the best way to provide long-term value to the organization and deepen relationships with management.
Transcript
1
00:00:01.709 –> 00:00:09.330
Welcome everybody to another episode of jamming with Jason. Hey. Today I am excited because I have Tom Harris with me.
2
00:00:09.840 –> 00:00:16.770
And just a little background on, Tom. He is the Vice President of internal audit at National Center media.
3
00:00:17.400 –> 00:00:27.720
So he’s a chief audit executive there and we we’ve known each other for a few months now started talking about actually system implementations so wanted to have Tom on
4
00:00:28.170 –> 00:00:36.390
So that we can kind of talk about how internal audit can kind of interact in a major system implementation.
5
00:00:36.930 –> 00:00:44.460
Because there’s a couple of different options that people have and so will kind of share what worked really well for Tom just recently.
6
00:00:44.880 –> 00:01:01.500
Kind of the other approach as well and just kind of go through, talk, talk about share some war stories and some other things like that so that those of you that are going through this can kind of see what some of the different options are that are out there. So, Tom, welcome aboard.
7
00:01:02.940 –> 00:01:05.760
Well, good morning. How are you, Jason. I’m doing well.
8
00:01:06.210 –> 00:01:17.130
Now just, just a little bit of trivia. And just to kind of put put some spotlight on people, you know, a lot of people maybe haven’t heard of national center media.
9
00:01:18.150 –> 00:01:32.520
You know, I didn’t. I hadn’t actually heard of it before I before I met you, but it’s like it’s kind of like companies that I used to work for where everybody uses or sees the product, but they just don’t realize the company. So do you want to just kind of tell everybody just
10
00:01:32.520 –> 00:01:33.570
Quickly kind of what
11
00:01:33.600 –> 00:01:38.580
What your company does, because they’re all going to go, oh, yeah, okay. I know what you guys do.
12
00:01:39.570 –> 00:01:49.410
All right. Well, certainly. And, and I wasn’t aware of national center media until I started looking into the organization. And it’s a great company. It really is.
13
00:01:49.920 –> 00:02:04.860
Our motto is, we connect brands to movie audiences. We have two primary markets. One is the cinema market and and when you go to the movies and you show up early. There’s a pre show the Nuvi pre show is NCS
14
00:02:05.700 –> 00:02:16.350
Product and we reach 750 million moviegoers a year. We’re in 21,000 screens. The top 160 DMA doesn’t mean a market areas.
15
00:02:16.740 –> 00:02:23.370
So we’ve got a lot of different advertising opportunities within the different theaters throughout the US.
16
00:02:23.940 –> 00:02:31.380
The other market is the digital market and and the tagline, there is going beyond the big screen and we we
17
00:02:31.980 –> 00:02:44.340
Are in the digital space we’re kind of young we’re small we’re growing. There’s a lot of competitors out there right now the big names are Google and even Amazon is is an advertising as well as Facebook driving
18
00:02:44.700 –> 00:02:54.930
Digital ads to individual unit users. We’ve got for owned and operated digital properties newly calm, it’s, it’s kind of like Fandango
19
00:02:55.320 –> 00:03:03.510
It’s your go to digital destination for trailers and showtimes and we also have shuffle. It’s a mobile movie at trivia game.
20
00:03:04.320 –> 00:03:17.790
Movie arcade, which is a really cool augmented reality app, and then fantasy movie league as popular as the other fantasy leagues are we figured we should have one movie. That’s pretty cool. Yeah.
21
00:03:19.260 –> 00:03:28.560
And our digital market is a collector and user of consumer data. And I think we are all aware that that regulatory landscape is changing quite a bit.
22
00:03:30.270 –> 00:03:31.410
Well, in the whole enter the
23
00:03:31.500 –> 00:03:32.700
Whole entertainment space.
24
00:03:32.730 –> 00:03:34.530
Is changing significantly, too.
25
00:03:34.530 –> 00:03:47.310
So I mean it’s I think it’s, it’s always interesting to kind of hear what what companies are doing. And then you know how some of this stuff kind of kind of fits in here. I know, I know, you know, mainly our topic today is kind of talking about system implementations, but
26
00:03:47.760 –> 00:03:50.010
I think it’s important for people to realize to you.
27
00:03:50.010 –> 00:03:52.980
Know it’s like everything is changing out there, right.
28
00:03:53.010 –> 00:03:56.310
I mean system implementations are one thing that changes.
29
00:03:57.090 –> 00:04:04.740
But there’s lots of stuff you know some of those names that you listed up there, you know, Amazon, Facebook, Google,
30
00:04:05.940 –> 00:04:11.700
You know, they’re starting to actually develop entertainment content to I mean they’re not just doing ads.
31
00:04:11.700 –> 00:04:13.470
But they’re like moving heavy into the
32
00:04:13.470 –> 00:04:19.890
Content space. So going to be a real shake up entertainment, but I know that that’s not what we’re here to talk about today, but I kind of
33
00:04:23.610 –> 00:04:25.440
So thanks for indulging me on that.
34
00:04:26.910 –> 00:04:27.360
Now,
35
00:04:29.070 –> 00:04:37.050
You know, maybe to kind of kind of start with, you know, just kind of give a we can give a little, a little background, maybe on
36
00:04:37.470 –> 00:04:47.400
It because to me it seems like. And I don’t know if this is the way you see it but but internal audit usually kind of takes one of two approaches when it comes to system implementations
37
00:04:48.480 –> 00:04:55.710
And so here we’re usually talking about like a big system implementation enterprise resource planning the RP kind of tool so you
38
00:04:55.710 –> 00:04:57.030
Know, something that goes
39
00:04:57.090 –> 00:04:59.370
You know, way out into the organization.
40
00:04:59.370 –> 00:05:10.020
Effects lots of users and I’ve usually seen either kind of this standoffish let’s watch what happens. And then go in and audited afterwards and tell them what they did wrong.
41
00:05:10.590 –> 00:05:22.440
Or there’s the let’s jump in and kind of embed ourselves in with the team and help work through the process as well. Those are kind of the two that I’ve seen. Have you seen any other ones. Besides that, or
42
00:05:23.580 –> 00:05:40.410
Yeah, actually I throughout my career, I think I’ve participated in in about a half a dozen different types and some of them are limited scope. Some of them are full scope. I think the two that you mentioned would be the two full scope the post implementation review and that’s that’s
43
00:05:41.760 –> 00:05:46.680
That’s essentially when auditors go in after the battlefield and speed wounded.
44
00:05:49.710 –> 00:05:58.920
We’re here to help. And now we’re back. Yeah, that’s right. The two biggest lies in business, right now we’re from internal audit. We’re here to help. And we’re so glad you’re here. Right.
45
00:05:59.850 –> 00:06:09.900
But, but there’s some other ones to consider as well. And these are more limited one would be a methodology assessment and that’s where you kind of focus on the process, not a particular project. You look at
46
00:06:10.290 –> 00:06:17.070
Compliance the methodology effectiveness of the methodology and you can do this pre during or post implementation.
47
00:06:18.120 –> 00:06:30.840
Another would be a Project Risk Assessment where you you break that particular project down into the types of risks, it’s it’s facing how the business is addressing and mitigating those risks.
48
00:06:31.650 –> 00:06:38.280
And the trouble without one, however, is audit typically isn’t involved and follow through with the risk mediation.
49
00:06:38.940 –> 00:06:49.980
Risk mediation until after post go live. So you have some issues where you’ve identified risks and then you come back later to see how they were mitigated. But even then it’s too late.
50
00:06:50.340 –> 00:07:00.510
Yeah, another another limited scope is kind of a pre launch readiness assessment. So the developments done you at is done and then you sit down and say, okay, do we check all the boxes.
51
00:07:01.440 –> 00:07:18.900
And that is less effective because oftentimes if there needs additional development work or more focus on user access or security you kind of have to halt the project and then go back and redo some things you really don’t get an opportunity to fix them ahead of time.
52
00:07:19.980 –> 00:07:35.580
And then another one that I think you kind of touched on a little bit would be the key phases review and at different checkpoints or milestones, you take a look at where things are at. It’s a little more involved. But again, it is kind of standoffish. If you get to a
53
00:07:36.870 –> 00:07:46.380
checkpoint and you realize something isn’t done right, it’s like, Okay, stop. Let’s go back and redo it. So, those, those are I think all very useful.
54
00:07:46.920 –> 00:08:05.160
The post implementation review, I think, is probably the most, the one that I’ve seen most of my career and its biggest drawback is you don’t you don’t apply those lessons learned. Until next time, which with an AARP could be a 10 years later. Well, no, and 10s of millions of dollars to
55
00:08:05.760 –> 00:08:06.300
You know, yeah.
56
00:08:06.330 –> 00:08:07.380
Absolutely. I think that’s
57
00:08:07.380 –> 00:08:07.950
That’s where
58
00:08:08.040 –> 00:08:17.910
It kind of, you know, becomes interesting that way. So I thought what I do for some of the US or listeners that aren’t really familiar with the system development lifecycle.
59
00:08:18.270 –> 00:08:29.250
You know that’s that’s kind of a process that IT departments kind of go through, right. So if you took the CPA exam, you know, because I’ve taught. Lots of people, you know, to help them pass the CPA exam.
60
00:08:29.790 –> 00:08:35.670
This is one of the big concepts. It’s in there. So some of you may be familiar with this already, but if not, I’ll just kind of run
61
00:08:36.060 –> 00:08:40.590
Run through kind of the steps really quick because I think this goes back to
62
00:08:41.100 –> 00:08:50.610
You know what you were talking about Tom on you know what kind of a scope, are we going to do is it’s going to be a full scope is it going to be partial. Is it going to be, you know, whatever. So
63
00:08:51.390 –> 00:08:56.400
Kind of just start with there’s like this system planning right where people realize hey you know what
64
00:08:57.390 –> 00:09:04.890
We need a different tool, but we’ve got is not working for us. And so we have to start planning and thinking about what it is that we need
65
00:09:05.310 –> 00:09:12.540
Then you start kind of analyzing or assessing what systems are out there. And then you can decide, am I going to buy something.
66
00:09:12.870 –> 00:09:25.470
You know, which would be like a system selection or am I actually going to develop something myself so I’m going to build something myself. And so if you build it, then there’s this whole programming concept that comes into it.
67
00:09:26.010 –> 00:09:38.280
If you buy it, then you’ve got to configure it. Right. And so again, in any of those steps so far. You can see where audit could be involved or not involved, right, you get to the end of it you do the testing.
68
00:09:38.580 –> 00:09:48.930
You know your test to make sure it’s doing what it’s supposed to do. And then there’s this whole conversion and implementation phase that you go through where it gets rolled out into the company.
69
00:09:49.410 –> 00:10:01.680
And then, you know, as you go through, you’re going to be, you know, looking at refinement and how the operations are working and other things like that until you need to go back to the system planning phase again. Right.
70
00:10:02.460 –> 00:10:06.510
And so I think you know like you talked about some of these different ways that we can
71
00:10:06.510 –> 00:10:15.630
Do it. We kind of plug ourselves in two different phases of this s DLC process.
72
00:10:16.110 –> 00:10:31.890
And so maybe let’s let’s talk first about this post implementation review because I think, like you said, this is the one that I’ve seen auditors do the most. And it seems kind of like traditional audit work. We go in afterwards. And we are back and letting people sometimes
73
00:10:32.250 –> 00:10:34.830
Because they didn’t get everything right and then we’re critical of
74
00:10:34.830 –> 00:10:41.910
Them at the end. And I don’t think that’s the most effective way of doing it right.
75
00:10:43.230 –> 00:10:47.520
And so again, I mean, I don’t know what your experience has been on that. If you want to maybe share as
76
00:10:47.550 –> 00:10:50.220
A story or if you have a story about that. I know I do.
77
00:10:51.420 –> 00:10:54.810
On on what can go wrong when we just kind of jump.
78
00:10:54.810 –> 00:10:55.470
In at the end.
79
00:10:56.730 –> 00:11:01.890
Yeah, absolutely. And, and I’ve seen system implementation reviews, where
80
00:11:03.300 –> 00:11:10.740
Audit has come in to look at how user access was set up how user security was set up, which is a critical piece of earpiece.
81
00:11:12.090 –> 00:11:23.100
And oftentimes, not frequently, but oftentimes that go live will be the user access is set up so that everyone has administrative
82
00:11:23.520 –> 00:11:30.960
Privileges and the thinking is that they don’t want security to get in the way of the functionality until they fully vetted
83
00:11:31.380 –> 00:11:40.710
The functionality of course in a you at environment, you can do that, but in a live environment where you’ve got real data that poses a huge risk to an organization.
84
00:11:41.250 –> 00:11:51.900
And you don’t know about that. You’re not involved about that it with that as an audit until you’ve already gone live and you recognize that for a six week period.
85
00:11:52.350 –> 00:11:57.990
Everybody had full rights to do whatever they want, you know, all of the high and critical
86
00:11:58.710 –> 00:12:11.910
Segregation of duty issues are unmitigated all of those risks are just fully exposed to me. I’ve seen this a couple of times in my career and it’s always around when audit plans, a post implementation review.
87
00:12:12.480 –> 00:12:22.290
If you’re involved during the entire project, you can put the brakes on and say, no, we’re not going to do that. We just can’t do that and then work with the business to set those security roles up
88
00:12:22.740 –> 00:12:39.030
So that in you at not a problem in go live when when you’ve got related. You’ve got to have big problem they so I think that’s probably the most significant risk that I’ve seen with Post Implementation reviews is in not being involved up to go live. Yeah.
89
00:12:39.480 –> 00:12:41.010
Well, and you see it. You see this all the
90
00:12:41.010 –> 00:12:47.310
Time and because I actually what you just described, kind of goes into my story. So I think it
91
00:12:48.450 –> 00:12:55.020
Maybe one point to kind of make to because as you were talking right you said, especially like in UAE people think, hey,
92
00:12:55.410 –> 00:13:06.480
We don’t want security to slow us down. So let’s just, you know, leave it open. Let’s go through, let’s do what we need to do. And we’ll worry about security later and and there
93
00:13:06.510 –> 00:13:09.300
Because there’s always like this tension between
94
00:13:09.330 –> 00:13:21.360
The IT group who is trying to satisfy the customer, they want they want the system to be able to, you know, not slow down the user. And then there’s this other group, usually called information security.
95
00:13:21.870 –> 00:13:28.050
Right. That does want to or needs to, you know, deal with the user security issues and other stuff like that.
96
00:13:28.650 –> 00:13:43.080
And when those two are separated. You know, usually it becomes kind of a check and balance if information security is just kind of a part of it, then usually it’s left until the last minute. And that was my story. Okay.
97
00:13:43.440 –> 00:13:47.880
I came in. I came into an organization to set up an internal audit department.
98
00:13:48.810 –> 00:14:00.090
There had been what they call them internal auditor there before me. But, you know, it really wasn’t internal audit. I honestly couldn’t figure out what she did most of the time.
99
00:14:01.410 –> 00:14:10.830
Afterwards, and when she left. I came in and was starting a department and exactly what you described of security and, you know, kind of
100
00:14:11.250 –> 00:14:23.190
Getting left until the end is exactly what happened. So I came in right after the system implementation and honestly it. So, this the it was scheduled to go live on a Monday.
101
00:14:24.210 –> 00:14:34.560
And on Friday, the story that I heard because I wasn’t there yet, but the the team had gotten together. And on Friday BEFORE THE MONDAY. GO LIVE said, oh, there’s this whole security.
102
00:14:35.910 –> 00:14:41.310
component that we didn’t really address before, what should we do about that.
103
00:14:42.060 –> 00:14:42.870
And so
104
00:14:43.530 –> 00:14:44.520
You know, it’s like
105
00:14:44.610 –> 00:14:47.490
You shouldn’t get that far, and not realize right
106
00:14:47.490 –> 00:14:49.920
So, so one person like
107
00:14:49.950 –> 00:14:57.900
took home the book, I guess, right, and read over the weekend as well. I think we need to flip the switch to this and it was, it was kind of that all open
108
00:14:58.350 –> 00:14:59.100
Sort of thing.
109
00:14:59.850 –> 00:15:01.950
And so you fast forward, you know, six months or a
110
00:15:01.950 –> 00:15:10.110
Year. When I come in. Well, I also get information security is one of my responsibilities very quickly after this little cluster that happened.
111
00:15:10.800 –> 00:15:24.990
And found out that that decision that they made made my life in our group tremendously difficult because now we had to like completely customize every user role.
112
00:15:25.410 –> 00:15:32.700
And shut everything down and make sure that we’ve shot all of the right things down instead of like a shot and then open it.
113
00:15:33.120 –> 00:15:39.930
Kind of a thing. So I think that point that you bring up, you know, if we just wait until the post implementation review.
114
00:15:40.410 –> 00:15:49.950
Sometimes things have happened that we really can’t fix. At that point, you know, like you said, it’s like, well, shoot, I guess we got to do it next time. But that might be five years from now.
115
00:15:51.960 –> 00:16:00.630
So yep and and the things that can’t be fixed are really difficult to get around for instance data conversion accuracy.
116
00:16:01.260 –> 00:16:08.820
The completeness and accuracy controls over ensuring that the data in the old system is converted into the new system.
117
00:16:09.810 –> 00:16:18.930
It once you do that if you don’t have those controls set up kind of got to go through a UA T before you go through a go live to validate the process of the conversion
118
00:16:19.260 –> 00:16:34.890
If that isn’t done up front and and if the business isn’t ready and aware that they need to essentially evidence that the business verified completeness and accuracy over that data, you can undo it once once once you’ve converted. It’s done.
119
00:16:36.060 –> 00:16:40.110
And that was another thing that the business was very receptive with, you know,
120
00:16:40.650 –> 00:16:49.080
And I was very fortunate here at NCR because the development team was really focused on doing things right and the business was really focused on doing things right.
121
00:16:49.920 –> 00:17:03.420
And, you know, the point is that once you get past go live. There’s stuff. You just can’t go back and fix and the stuff that you can is very expensive and takes a lot more time.
122
00:17:04.800 –> 00:17:19.800
Yeah, so you know and like I said, I think that, historically, I think most auditors are doing this post implementation review but but you did, you know, I think the word you use before it was kind of embedded you embedded yourself into
123
00:17:20.310 –> 00:17:21.690
The team itself right so
124
00:17:21.690 –> 00:17:22.380
You were kind of
125
00:17:23.460 –> 00:17:25.680
Involved all along the way, which
126
00:17:25.890 –> 00:17:37.650
Which I think is a good thing. Now, a lot of audit purists would say, well, now when you do that, right. You’re effectively helping to build the system so you lose your objectivity and now you can audit it
127
00:17:38.370 –> 00:17:45.870
Okay, so I’m sure you know those thoughts must have been going through your head too. Right. I mean, how would you respond to somebody that says that
128
00:17:47.070 –> 00:17:59.790
Well, it’s certainly a risk of this approach, and it’s something that that you need to be candid about with your audit committee and with your CFO and with the project sponsors and
129
00:18:00.180 –> 00:18:06.900
In this particular instance, I absolutely did lose my objectivity over user access and setting up security.
130
00:18:07.380 –> 00:18:18.150
That the business. The project had gotten to a point where the business needed some to make some decisions on what the risk ratings were for the sod conflicts and how to mitigate those
131
00:18:18.600 –> 00:18:26.760
What kind of user roles could be set up how things could be configured. They needed somebody to come in and just really sit with them and work through it.
132
00:18:27.270 –> 00:18:35.550
So I did that, recognizing that you know at this point, I can’t perform an independent audit over user access for a year, year and a half.
133
00:18:36.120 –> 00:18:48.660
That the approach then became finding a partner, a third party partner who could independently come in and conduct an audit and there’s lots of there’s there’s lots of firms out there that are capable of doing this.
134
00:18:49.890 –> 00:18:55.860
But you know he required the support from the CFO, because it was budget that we hadn’t planned for required
135
00:18:56.310 –> 00:19:04.440
Knowledge by the audit committee to say okay this is what happened. We, we knew this was a risk and this is how we’re going to mitigate it and
136
00:19:05.160 –> 00:19:13.920
And I think having all of those pieces connected was critical to being able to step in, you know, it was an anticipation of what happens if
137
00:19:14.460 –> 00:19:22.200
To be able to step in to keep the project going to set the user access up correctly for other parts of the project.
138
00:19:22.800 –> 00:19:30.630
I was able to maintain my independence more of a consultative basis. Well, you know, when you set up workflows. This is what it should look like.
139
00:19:31.080 –> 00:19:43.290
And that, but not actually do the work, but help and assist what those workflows should look like. And that’s really, you know, looking at controls and assessing the controls that they’re putting into place before they put them into place.
140
00:19:43.950 –> 00:19:54.720
In, in that respect, I was able to maintain independence and we’re very confident going forward for the next year so that we can conduct these audits and do them in a way that maintains our objectivity.
141
00:19:55.170 –> 00:20:06.420
So absolutely. It’s, it’s probably one of the biggest risks. Another huge risk taking this approach is it takes a little big time commitment. There’s a lot of meetings that you have to attend.
142
00:20:07.200 –> 00:20:19.500
And of course the corollary is if you’re spending your time on this, what other things aren’t being done that might be emerging risks that audit still has to respond to. Yeah, well,
143
00:20:19.530 –> 00:20:21.180
So those are two things that you do have to
144
00:20:21.180 –> 00:20:28.110
Think about as you’re going into it. Right. I mean, if you’re the chief audit executive and and you’re looking at the amount of work that you can do.
145
00:20:28.590 –> 00:20:38.670
This next year, you have to make some of those decisions right is is this important enough and it’s a strategic initiative for the organization, we’re going to be spending 10s of millions of dollars on this.
146
00:20:39.210 –> 00:20:49.020
And like I said, if we don’t get it right, there’s some things that can’t be fixed, you know, should we be spending our time they’re realizing again if we get
147
00:20:49.770 –> 00:20:55.830
Deep into it. We might have an objectivity or independence issue and may not be able to audit that
148
00:20:56.730 –> 00:21:01.590
But, you know, in my opinion, I think it’s better for us to participate.
149
00:21:02.190 –> 00:21:11.130
On the team and provide value that way instead of the value of the audit afterwards because like you said, you can always hire somebody to come in.
150
00:21:11.730 –> 00:21:24.930
And do that afterwards. I think it, it just provides more value. And one thing that I hope everybody listening gets is the audit report is not the only value.
151
00:21:24.930 –> 00:21:27.390
That we provide to our organizations, right.
152
00:21:27.930 –> 00:21:30.060
So, okay.
153
00:21:30.240 –> 00:21:44.220
To choose to be on the team and actually help throughout. In fact I would argue in something like this with the system implementation, you add more value by being a part of the team then coming in afterwards.
154
00:21:45.060 –> 00:21:45.330
So,
155
00:21:46.470 –> 00:21:47.640
So, so maybe let’s
156
00:21:47.760 –> 00:21:51.840
Because I know this actually really went successful for you. And so I wanted to kind of
157
00:21:53.010 –> 00:21:58.950
break it apart a little bit for others that are kind of, you know, maybe about to embark on
158
00:21:58.950 –> 00:22:00.540
This kind of a journey and have to
159
00:22:00.540 –> 00:22:03.390
decide, you know what they’re what they’re going to do.
160
00:22:04.680 –> 00:22:16.740
Because again, I’m guessing. This was probably kind of a new approach in your organization where you were more on the team. So maybe what were some of the things that you needed to do.
161
00:22:17.430 –> 00:22:18.480
To be able to get the
162
00:22:18.480 –> 00:22:28.260
Support and and to, you know, really, kind of, you know, persuade or influence people that this really was the best use of your time.
163
00:22:29.640 –> 00:22:38.130
Yeah, absolutely. And it is a different approach when and I’ve been with SEM for about a year now and when I interviewed for the role.
164
00:22:38.580 –> 00:22:50.640
My audit committee chair in the interview process mentioned that the prior GL implementation 810 years ago didn’t go very well and it couldn’t happen again.
165
00:22:51.990 –> 00:22:52.200
DING
166
00:22:53.400 –> 00:22:55.080
Ding, ding, ding, a big red flags. Right.
167
00:22:55.230 –> 00:23:05.610
Yeah and you know to me that told me a couple things. One is I would find it highly unusual that the audit committee chair would would still have that in front of his mind after so many years.
168
00:23:05.940 –> 00:23:17.490
You know they’re they’re at a governance level where yeah we knew we had a new GL system and we knew across this much, but that’s about it. Right. He was able to articulate specific issues with the implementation.
169
00:23:17.850 –> 00:23:28.230
That he wanted. He wanted it done differently this time. And when I met with the CFO, as well as, you know, she mentioned that the same thing. She wasn’t here with the company at the time.
170
00:23:28.620 –> 00:23:44.310
But she did note that the prior GL implementation had quite a few issues with it and it took some time for the organization to recover from that. And this, you know, this time and needed to be different. It’s like, okay, it’ll be different. Yeah.
171
00:23:45.480 –> 00:23:53.010
And I’m a big believer that internal audit should only go where they’re invited and I know that is a bit controversial.
172
00:23:53.730 –> 00:24:03.420
Because if if somebody is doing nefarious things, then naturally you wouldn’t invite audit into take a look at it. But on the other side of the coin.
173
00:24:04.020 –> 00:24:17.490
I think the CFO and the audit committee chair will invite you wherever you need to go right. So I think by invitation only for this type of a project is is critical. And it’s not just well
174
00:24:18.720 –> 00:24:25.290
Audit Committee said, I’m going to go there. Somebody’s gonna do it. It’s also talking with the project sponsors. How would you like me to approach this.
175
00:24:25.980 –> 00:24:33.570
What kinds of concerns do you have, what kinds of things can audit do to address those concerns.
176
00:24:34.080 –> 00:24:44.580
And taking that very service approach that that mentality that we are here to provide a service. We’re here to help the business to make it better, that’s ultimately what audits goal is
177
00:24:45.180 –> 00:24:56.310
And just that attitude in talking with the project sponsors initially and finding out what they’re worried about much like we do the risk assessments every year and then figuring out
178
00:24:56.820 –> 00:25:09.180
creative and collaborative ways that we can address those things and and by having those conversations pretty early on, it became obvious that the the team, the development team, the stakeholders.
179
00:25:10.620 –> 00:25:21.330
They were, they were very much wanting audit to get involved and even to the point where it’s it was, you know, we were looking at a vendor right now.
180
00:25:21.870 –> 00:25:27.300
A couple of vendors, we’ve gone through this RFP process. This is where we’re at. We’re not convinced we have the right
181
00:25:27.720 –> 00:25:38.910
The right resources to help with this development effort. What do you think we should do those kinds of conversations early on in the project and then being able to deliver some some answers to some difficult questions.
182
00:25:39.870 –> 00:25:52.590
Since audit has this independent set of eyes and we’ve got completely different business experiences throughout our career oftentimes we think an obvious answer is right there that for some reason.
183
00:25:53.520 –> 00:26:03.810
The business and the people on the team. They just don’t see it. So being able to articulate those things and talk through those things in a very consultative advisory capacity.
184
00:26:04.320 –> 00:26:14.430
really sets the tone for how you’re going to operate with the organization as a whole. And I think those are really key in getting the approval, if you will, the invitation.
185
00:26:15.660 –> 00:26:23.790
To be invited in. And that’s something that that speaks as much about the organization and and the people that you’re working with.
186
00:26:24.120 –> 00:26:36.300
As it does about governance in the audit department itself. So, those, those things really setting that tone, letting people know that you actually are there to help you not just going to stand back and criticize it afterwards.
187
00:26:37.230 –> 00:26:38.130
Well, I think, I think.
188
00:26:38.160 –> 00:26:38.760
You know what
189
00:26:38.790 –> 00:26:48.300
What you brought up there kind of refers back to I know a lot of times we use the word insight that we provide insight. Right. But that’s, that’s exactly what you’re describing, there is
190
00:26:49.110 –> 00:26:56.520
You know when when you’re in the day to day activities, you know, you’re the you’re the the project sponsor or somebody else on the team.
191
00:26:57.450 –> 00:27:05.430
It’s, it’s often easy to miss something that’s right in front of your face because we’re so worried about whatever else we’re doing
192
00:27:05.940 –> 00:27:17.100
And so that outside perspective from audit really can provide that inside when you just bring up some of those things. Right. And that’s, that’s, I think, where we where we can end up adding a lot of value.
193
00:27:18.360 –> 00:27:33.180
In our organizations. Now, I thought it was interesting because I was going to ask you a follow up question. You know, it’s like, obviously, you had support from the audit committee and the CFO. But how did you get the support from the project sponsors and the team.
194
00:27:33.690 –> 00:27:35.010
But it, but it sounds like when
195
00:27:35.010 –> 00:27:36.450
You when you went to talk to
196
00:27:36.450 –> 00:27:49.020
Them, you know, and kind of say, Hey, you know, we know this thing is coming up. What are you worried about instead of going to them and saying, Hey, the audit committee wants me to audit you or check out and make sure everything is right.
197
00:27:49.530 –> 00:27:50.880
You kind of went in with the
198
00:27:51.030 –> 00:27:54.030
Hey, what are you worried about. And as a natural part
199
00:27:54.030 –> 00:27:58.170
Of that conversation. They’re like, dude, we totally want you to come help us right
200
00:27:58.500 –> 00:28:00.600
Is that kind of the sense that came out of this right
201
00:28:01.890 –> 00:28:14.220
Yeah. Absolutely. And I think that that talking through what they wanted from the project and what what’s not working for them now and what is working for them now and just getting an understanding of that.
202
00:28:14.700 –> 00:28:20.070
Really the natural conclusion was can you know, can you come in and help us with these things.
203
00:28:21.780 –> 00:28:30.960
And it was vendor selection hadn’t been made yet, but the project design the product requirements had been identified and there were
204
00:28:31.530 –> 00:28:43.560
A list of seven things seven enhancements that are controller wanted and they were. I mean their basic control stuff that we, the organization should have had, you know, years ago workflow has been around for what a couple of decades and
205
00:28:44.220 –> 00:28:56.850
We’re just now getting to implement it, so talking through those things and our controller and get to the point where she was willing to give up some of those benefits. Some of those improvements and enhancements.
206
00:28:57.270 –> 00:29:11.610
Just to get the project done because of costs and budget and it’s like, well hey before you go that way. Let’s see what we can do about finding the right vendor about driving a better budget that our CFO approved.
207
00:29:12.870 –> 00:29:19.650
And working through that process with all of the different stakeholders, I think, was really beneficial. I would call that one of the very early wins.
208
00:29:20.070 –> 00:29:26.460
And it demonstrated that that yeah audit can bring things to the table that they that they hadn’t thought possible yet.
209
00:29:26.760 –> 00:29:37.890
Not because we’re so great. But just because we have a different perspective. And we have different experiences throughout our career that we, that we can say, hey, wait. I’ve seen this before. Let’s try it this way.
210
00:29:38.790 –> 00:29:47.130
So I think the early wins and the kinds of conversations that we had was just really critical to forming a project team and forming
211
00:29:47.580 –> 00:30:00.150
Trust with with the group so that we could we could you know two months later when we’re duking out what a workflow should look like. We still have those common elements that we built trust on
212
00:30:01.440 –> 00:30:02.250
Well, and I think
213
00:30:02.760 –> 00:30:14.730
I think that approach that you took that this is a big lesson learning for people, you know, is, is there’s two, there’s two ways you could have gone in and talk to the project sponsors right and i and i think
214
00:30:14.820 –> 00:30:16.230
A lot of times people have the
215
00:30:16.230 –> 00:30:29.190
Tendency to do this. This first approach, which would be, Hey, you know you’re on our audit plan. This next year because the audit committee and the CFO are concerned that we can’t have a mistake like happened last time, blah, blah, blah, blah, blah. Right. So you’re going to be audited.
216
00:30:30.060 –> 00:30:32.910
And I think that’s kind of the knee jerk reaction of a lot of
217
00:30:32.910 –> 00:30:39.930
People is, you know, oh well you know they want me to do this. So I’m going to, we’re going to come on it right and you come in with kind of that.
218
00:30:40.590 –> 00:30:50.880
That attitude which is very off putting to other people right versus what you did, which is coming in and saying hey you know you, we got this new project coming up here, what
219
00:30:51.330 –> 00:31:02.910
What are you, what are you worried about, you know, and just start to have that conversation because as people start bringing that stuff up right some some magic words that people can use is
220
00:31:04.110 –> 00:31:11.130
Once somebody kind of expresses what they’re worried about or what they need, then you could turn around and say, Well, would you like some help with that.
221
00:31:12.210 –> 00:31:12.990
And if they say
222
00:31:13.020 –> 00:31:14.760
Yes, it’s like
223
00:31:15.510 –> 00:31:18.090
Now you’re being invited right
224
00:31:18.120 –> 00:31:19.590
And so you’re not
225
00:31:19.620 –> 00:31:26.940
There like forcing them to do it and and especially like that first attitude you know of coming in. We’re going to audit.
226
00:31:27.600 –> 00:31:40.530
A lot of the, you know, some of those people on the team might have been there for that previous GL implementation. So if you go in kind of bad mouthing the old thing or the stuff that’s not working. Currently, they’re probably going to take offense at it anyway.
227
00:31:40.830 –> 00:31:44.250
Yep begin with and that’s going to damage your relationships.
228
00:31:44.250 –> 00:31:49.470
So great job on that. See, that’s why I wanted to talk to you today. You did all these things.
229
00:31:49.500 –> 00:31:51.990
Are great and I want everybody else to hear this too.
230
00:31:52.740 –> 00:31:53.370
So,
231
00:31:54.480 –> 00:31:57.540
So, so you get in your you’re on the team. So maybe kind of
232
00:31:57.780 –> 00:32:06.750
Talk about, you know, because like you said this is a it is a big time commitment you know for you and for certain members of your team.
233
00:32:07.980 –> 00:32:23.400
Because in doing this, you’re obviously not doing some other things but but kind of maybe explain some of the ways that you were involved in the team and kind of through the process. You know, because I’m sure there were other people on your staff probably that maybe we’re
234
00:32:23.460 –> 00:32:25.170
We’re helping out with this too, and
235
00:32:25.170 –> 00:32:25.590
So,
236
00:32:25.890 –> 00:32:32.640
Some of the different roles that you played in that and how how that kind of unfolded as as you went through.
237
00:32:33.930 –> 00:32:50.400
Sure, absolutely. And it I think attending the the weekly stand ups, we use an agile methodology we do use s DLC that that kind of waterfall approach where it’s linear and sequential so you got to finish one phase before you move on to the next.
238
00:32:51.450 –> 00:33:02.100
And in the development phase here, we, we, we, the agile methodology would have weekly stand up meetings where we come in. Talk about blockers work through issues.
239
00:33:02.460 –> 00:33:13.140
And the idea is to continue to work on things right to keep it on pace and we would have sprints off to the side, which is another agile methodology to get certain things done.
240
00:33:14.190 –> 00:33:28.170
And the, the stand up meetings, I think we’re probably the most important. A lot of the breakout sessions where we would sit down with the dictator stakeholders and developers and and work through individual functionality.
241
00:33:28.620 –> 00:33:40.710
And again, independence is key attending the meetings, providing that that consulting, they had asked questions about how is this structured and I could ask questions back. Well, if you do it this way.
242
00:33:41.310 –> 00:33:54.570
What about that and and if you take this approach, what’s the impact over here. And a lot of those questions lead to dialogues that that were really productive in terms of how the product was going to be configured.
243
00:33:55.350 –> 00:34:06.420
Particularly when it came to the custom code within revenue and deferred revenue our model has some very peculiar things when it comes to revenue recognition, the timing of when we deliver
244
00:34:07.170 –> 00:34:15.720
The, the expectation within the market about the flexibility of delivery time different costs CPM costs.
245
00:34:16.410 –> 00:34:22.740
Different market areas get changed and all of that can have an effect on revenue recognition. So our
246
00:34:23.310 –> 00:34:36.720
Revenue module required quite a bit of customization and working through that, not wanting to replicate what was done before, wanting to make it better. I think was was the focus of what those breakout sessions were and
247
00:34:37.800 –> 00:34:46.470
A couple of key points I talked a little bit about where I rolled up my sleeves when it came to user access and and the security roles.
248
00:34:47.280 –> 00:34:56.280
There were some other points with data conversion that the testing of the completeness and accuracy of the data coming from the old system to the new system.
249
00:34:56.640 –> 00:35:11.880
Providing a three lines of defense so that when Deloitte our external auditor came in, we were able to deliver a package that says here’s how we addressed the data conversion and you know they spent a few hours looking at it and said this is fantastic.
250
00:35:13.350 –> 00:35:26.130
Which which I think was reflects on the team because they were willing to go through and put in the due diligence to make sure that that the testing and the validation of that conversion
251
00:35:27.210 –> 00:35:41.400
Met expectations, not just complete and accurate, but also being able to evidence it in a way that our external auditor felt really good about having accurate data. So I did. I did have a senior it auditor.
252
00:35:42.480 –> 00:35:53.400
very experienced gentlemen who who’s very strong all the way around. He did the we call it a validation of the assessment of the conversion, which
253
00:35:53.850 –> 00:36:03.510
Kind of gives you an idea of what that picture looks like he did it at you at which was a full validation for all of the data conversion and then we did it again at go live.
254
00:36:04.200 –> 00:36:13.290
In both of those instances we developed a methodology for presenting what was done. And then we also issued a memo.
255
00:36:14.190 –> 00:36:21.690
Copying our external as well as the audit committee and the CFO, saying, you know, here’s, here’s what they here’s the business did. Here’s how we validated it
256
00:36:22.410 –> 00:36:29.610
We connected with our external pretty early on to make sure our approach was good. And so, those, those are two memos, we issued
257
00:36:30.360 –> 00:36:41.010
There weren’t a lot of formal communications, you know, typically you’ll have a status meeting or an update memo or no report at the end and a planning memo at the beginning.
258
00:36:41.700 –> 00:36:51.510
Because of the way this project evolved. There was very little bit of that form communication. There was a lot of informal day to day discussions with
259
00:36:52.080 –> 00:36:58.350
With the stakeholders, with the team meetings with the audit committee on a quarterly basis.
260
00:36:59.070 –> 00:37:10.200
Occasionally a separate call to the audit committee chair just to, you know, let them know what we’re doing. And this is one of the things that I’m I don’t know how well I did this part.
261
00:37:10.740 –> 00:37:16.170
Because I would have liked to have seen more frequent formal communication command of it.
262
00:37:16.620 –> 00:37:24.390
And it just didn’t develop it was there was a lot of moving targets and in my mind, I’m thinking how could I have done that better. I’m just not sure yet.
263
00:37:25.170 –> 00:37:29.880
Well, we’ve got this independent audit of user access and security coming up.
264
00:37:30.570 –> 00:37:40.110
We’ll have a formal report out of that. But as I look back over the last nine months and I look at the amount of time that I put into it.
265
00:37:40.830 –> 00:37:55.020
That that my my senior internal auditor put into it and the end just the lack of formal communication. I’m kind of bothered by that nobody else seems to me, but I’m looking at it going, How could. How could I have done that better. And I’m and I’m not sure.
266
00:37:55.710 –> 00:38:00.030
Well, and actually, because it’s funny, as you were saying that because I was going to ask you, kind of a
267
00:38:00.030 –> 00:38:02.550
Follow up coaching question, if you will, from that but
268
00:38:03.060 –> 00:38:04.710
You almost kind of answered your own
269
00:38:04.710 –> 00:38:05.700
Question. Okay.
270
00:38:06.120 –> 00:38:12.180
But, but I think it’s, it’s an important point to bring up and I’m glad that you did because I’m sure that a lot of people listening.
271
00:38:12.750 –> 00:38:18.360
Would be or could be feeling the same way. Right. It’s like, I mean, in order, we get used to, you know, we go into
272
00:38:18.630 –> 00:38:27.930
We go and do a project and there’s, you know, pre communications and during communications and afterwards we issue this final report, we’ve got this PowerPoint deck and we’ve got blah, blah, blah, blah, blah. Right.
273
00:38:28.350 –> 00:38:30.210
And we get used to the more formal
274
00:38:30.210 –> 00:38:31.170
Communication.
275
00:38:32.340 –> 00:38:34.950
And sometimes we spend way too much time on those two
276
00:38:34.950 –> 00:38:35.970
By the way, but that’s
277
00:38:36.150 –> 00:38:37.500
That’s another point for another day.
278
00:38:39.240 –> 00:38:49.470
But, you know, especially because as you were talking to me. You guys are taking an agile approach to the project and formal communications under an agile approach.
279
00:38:49.830 –> 00:38:55.110
Don’t really exist. That’s what your stand up meetings are that’s some of your, you know, other stuff that’s kind of built into it.
280
00:38:55.860 –> 00:39:07.410
But throughout the process. You obviously were communicating what you were doing because like you said you were a little bit bothered by it, but nobody else seemed to be bothered by it. So to me, that means
281
00:39:07.830 –> 00:39:13.800
Well, if nobody else in your organization was bothered by it, including the audit committee your executive
282
00:39:14.070 –> 00:39:15.150
You belong. The team.
283
00:39:15.450 –> 00:39:19.020
If they didn’t feel the need for you to put out a
284
00:39:19.020 –> 00:39:22.440
Formal report, I think, good on you, mate. It’s okay.
285
00:39:22.860 –> 00:39:25.440
We’re still communicating right so
286
00:39:25.470 –> 00:39:26.310
So it’s okay.
287
00:39:26.370 –> 00:39:27.330
So comfortable with that.
288
00:39:28.830 –> 00:39:31.560
Okay, I’ll think about, think about
289
00:39:33.690 –> 00:39:34.770
I think that’s
290
00:39:35.820 –> 00:39:47.490
Like you said, I probably should talk more about that on some other stuff, but it’s it’s it has, as we move from kind of, you know, traditional auditing kind of what we’ve done to more
291
00:39:48.120 –> 00:40:00.000
You know quicker speed. I mean, some people using the term agile auditing and trying to incorporate those processes into it. I think it’s, it is going to make some of us feel a little uncomfortable like
292
00:40:01.050 –> 00:40:03.480
I did all this work, but what do I have to show for it.
293
00:40:03.990 –> 00:40:04.980
Right. It’s like, yeah.
294
00:40:05.250 –> 00:40:08.070
I mean, I grew up working with my hands. My dad was a
295
00:40:08.070 –> 00:40:14.370
Contractor I built furniture and did all kinds of stuff, you know, at the end of the day when you’re using your hands to make something
296
00:40:15.210 –> 00:40:25.860
You can see the progress at the end of the day, and you can feel good about it. You know, or it’s like my dad would drive me around. He’s like, yep, I built that house over there in 1974 right or whatever kind of thing.
297
00:40:26.910 –> 00:40:28.410
In our job.
298
00:40:28.650 –> 00:40:35.190
We don’t get some of that same satisfaction because there’s no, like, tangible end product.
299
00:40:36.030 –> 00:40:42.780
And so sometimes it can feel a little awkward like that, but like, I’m telling you, we probably need to get, get comfortable with it because
300
00:40:43.170 –> 00:40:53.700
If your audit committee didn’t have a problem with it if your executive didn’t have a problem with it. And if your, you know, project team didn’t have a problem with it. I think you guys still did a pretty good job of communicating
301
00:40:55.110 –> 00:40:55.800
That fair enough.
302
00:40:57.180 –> 00:40:59.700
And I know you said you got to noodle on it a little bit, but
303
00:40:59.730 –> 00:41:16.200
I think that’s probably a reality that we’re moving into as a profession. We’re gonna have we’re gonna have to learn how to get comfortable and find other ways maybe to check in to make sure that we’re doing what we’re supposed to be doing. But if everybody’s happy we probably are so
304
00:41:16.590 –> 00:41:18.480
Okay, anyway. Sorry.
305
00:41:18.960 –> 00:41:20.010
I’ll get off my soapbox now.
306
00:41:22.620 –> 00:41:23.430
But, uh,
307
00:41:23.970 –> 00:41:38.970
Yeah so. So that was actually kind of like a lesson learned, I guess, too, because I wanted to kind of wrap up our talk you know today. I mean, obviously you guys you did some great things, people were happy. It sounds like the project was a success, you know,
308
00:41:40.170 –> 00:41:47.580
Besides this one that we just talked about, you know, around formal communication and should you have done something better around formal communication.
309
00:41:48.090 –> 00:41:55.830
I guess, or there’s some other things that you learned, kind of going through this or maybe things that were aha moments for you as you as you were going through
310
00:41:57.600 –> 00:42:10.590
Yeah, there’s, there’s a handful of things we we got a little creative in you it and and let me, let me just say that I do think that it was a success. I, you know, a B plus A minus, which for
311
00:42:11.610 –> 00:42:21.690
project of this size is pretty good. We did have one three week slip in schedule and three weeks on a nine month project is is nothing good.
312
00:42:22.560 –> 00:42:32.100
That’s nothing. Yes, we came in at budget little under I think just a hair under overall budget, which is pretty phenomenal.
313
00:42:32.820 –> 00:42:41.580
And we’re getting the functionality that we wanted. So I think, I think overall, it was, it was a success, we had, we did have to get creative to meet that
314
00:42:42.240 –> 00:42:50.760
To meet some of those objectives within you at we actually set up a separate two different utilities, which
315
00:42:51.600 –> 00:43:02.250
ended up creating some issues for us after go live that we didn’t see and we did this because we hadn’t gotten a clean data conversion yet. So we wanted to be able to
316
00:43:02.580 –> 00:43:16.800
Have a UT you at environment where we could do just data validation and of course if you have a UA T environment for functionality, changing the data. So, so we had two different UHT set up at one point in time going concurrently.
317
00:43:17.970 –> 00:43:27.330
In the custom code for our revenue and deferred revenue was being tested. And we’re flushing out quite a few functional issues in that one environment.
318
00:43:27.990 –> 00:43:35.880
And then we did our validation data validation. We have a couple of different times where we had to re initiate the data conversion
319
00:43:36.330 –> 00:43:42.600
And make changes and how we configure it, and some of the sequencing and we finally got to the point where we got a clean conversion
320
00:43:43.230 –> 00:43:55.710
And then we pull those to you at environments together, but we didn’t really test some of the functionality in that combined environment.
321
00:43:56.250 –> 00:44:08.370
We thought it was good. Right. Okay. Yeah. After, after go live. We started seeing some snuggly ISSUES. AND THEN THE SNUGGLY issues were with invoicing and it was all about.
322
00:44:09.060 –> 00:44:18.360
Printing right it’s like the you could go in and check the invoice data and it was clean. But when it came out as an invoice that would go to our customer. It wasn’t
323
00:44:20.610 –> 00:44:25.440
Or they would just disappear. Right, we just didn’t have any invoices, it’s, you know, the data is there.
324
00:44:25.710 –> 00:44:32.550
So that. So the core logic is there, but what we do with the data to get the invoice out something was going wrong, something was up. We couldn’t figure it out.
325
00:44:33.330 –> 00:44:43.500
And the, the, the sneakily issues started to raise and they’re things that would would slow us down in our clothes process.
326
00:44:44.100 –> 00:44:57.330
Nothing major or monumental but but they just started to, you know, essentially, we had to throw some more resources at it to start cleaning them up and getting on top of it because we weren’t closing them out as quickly as we were accumulating them.
327
00:44:58.380 –> 00:45:14.850
So, so, so I think the duel you at may have contributed to those things. I think our testing scripts within you. It could have been better but they weren’t bad and and I think those are some tweaks, we, we would do next time. Right.
328
00:45:16.140 –> 00:45:32.550
And we also had to make some some decisions about functionality that we were going to push post go live full AP workflow is something that we had to push post go live because Microsoft had an issue that wouldn’t be fixed until the later revision.
329
00:45:33.810 –> 00:45:37.830
We also pushed moderate risk side conflicts to after go live.
330
00:45:39.630 –> 00:45:53.130
Addressing all high and and and knowing what the moderate ones were we just weren’t going to fix them until later, you know, we took a look at him and said, Yeah, we can live with this for a week or two. And then we also had
331
00:45:54.750 –> 00:46:01.830
Some, some and I’m trying to think. There’s another functionality that we had to push doesn’t come to mind right now.
332
00:46:02.460 –> 00:46:11.790
And another thing that we had to prepare for was a mandatory update from Microsoft. Two months after go live to go from 8.4 to 10 point four.
333
00:46:12.300 –> 00:46:18.660
So those were things that were we had to prepare for if we’d have pushed the go live.
334
00:46:19.530 –> 00:46:36.690
Point for later. You know, another period or two later, we would have to start things over because only the new version would have been available for us to go live so that that I think really drove us to look at you at a little bit differently with the two different environments.
335
00:46:37.380 –> 00:46:46.530
So I think those are those are all lessons learned, maybe one or two of those are pitfalls. But ultimately, these were decisions that were that were made.
336
00:46:47.160 –> 00:46:56.520
And I think the decision making process was very effective because we talked about risks we talked about how to mitigate it. We talked about, you know, do we really want to do this. Do we want to take these other approaches.
337
00:46:57.330 –> 00:47:12.180
We didn’t know everything at the time, but the discussions and the conversations around those decisions were were really centered on what’s the risk. What’s our mitigation and and why do we really want to do it. I think there were very effective conversations
338
00:47:12.960 –> 00:47:16.410
Well, and that’s and that’s the important thing for people to remember. Anytime we’re
339
00:47:16.470 –> 00:47:23.370
We’re, we’re dealing with risks we don’t have all the information when you have to make a decision. So
340
00:47:24.030 –> 00:47:25.650
You have the good conversations
341
00:47:25.710 –> 00:47:29.820
Right, and then you make the best decision. You can, in hindsight, you know,
342
00:47:29.970 –> 00:47:33.210
Okay, next time you might do something different. But, I mean, it sounds like again.
343
00:47:33.240 –> 00:47:38.010
A minus or b plus effort that’s that’s still pretty good. That’s still a little
344
00:47:38.040 –> 00:47:38.670
Better. Yeah.
345
00:47:39.720 –> 00:47:43.050
We’re pretty pleased. Yeah, yeah. Now one of the things, actually, that I that
346
00:47:43.140 –> 00:47:48.870
Because when we were talking beforehand to probably to bring up to, because this kind of gets back to
347
00:47:50.580 –> 00:48:00.420
Cloud computing and the fact that, you know, a lot of people are moving into this space and in one of the things around software is updates and patches and things like that. Right.
348
00:48:01.200 –> 00:48:01.860
And school
349
00:48:02.220 –> 00:48:06.480
Historically, you know, each organization has kind of decided, well, do
350
00:48:06.480 –> 00:48:15.960
I want to do the patch or the update. Do I want to wait, you know, three months. Do I want to wait until the next version, you know that comes out.
351
00:48:16.530 –> 00:48:23.880
And and when you were in a you know client server environment, you could make those decisions because everything was running off of your own servers.
352
00:48:24.660 –> 00:48:26.760
In fact that was sometimes why you made those
353
00:48:26.760 –> 00:48:29.040
Decisions. Right. It’s like our servers can’t
354
00:48:29.070 –> 00:48:29.940
handle that.
355
00:48:29.970 –> 00:48:31.290
Update or whatever we got
356
00:48:31.290 –> 00:48:36.870
To fix some hardware things first. Okay. In the cloud environment, you don’t have that choice to you.
357
00:48:39.270 –> 00:48:50.250
Know, not really. And we were Microsoft does updates on a monthly basis. And initially, we thought that we were required to take every single update
358
00:48:50.670 –> 00:49:07.350
And they’ve kind of soften that a little bit so that there are certain updates we can skip and certain that are mandatory they do recommend that you don’t skip two in a row, because the changes will be significant or I should say more significant than if if you did everyone
359
00:49:08.940 –> 00:49:18.510
It does require a bigger focus on the updates. How are you going to test it, particularly when it comes to the custom code.
360
00:49:18.990 –> 00:49:28.440
We haven’t gone through one yet. We’re, we’re right now preparing to go through one and hopefully it’ll address the AP workflow stuff. It’ll, it’ll
361
00:49:29.040 –> 00:49:35.610
address some of the other functional things that we’ve seen that that we kind of want to get cleaned up, nothing major.
362
00:49:36.030 –> 00:49:44.430
But the turnaround time on those updates are really quick, you typically have a week from the time you get the code until it’s going to be pushed and
363
00:49:45.120 –> 00:49:59.190
We’re kind of trying to keep an open mind about what that looks like. And we’ve done some things internally where we’ve moved projects out to give our developers or we call them the finance application team.
364
00:50:00.300 –> 00:50:05.970
To give them more time in their calendar. For instance, the independent art of user access
365
00:50:06.690 –> 00:50:17.460
We push that out to September to give our developers, an opportunity to look at the code and to kind of free up their, their schedule and again it’s collaborative right say hey we got to do this.
366
00:50:18.000 –> 00:50:25.020
What’s your calendar look like, you know, the response was, Well, we can fit it in if we have to, like, Well, you know, what does that mean
367
00:50:25.830 –> 00:50:32.970
Well here’s all this. Here’s all the stuff we’re working on that week. Oh, okay. How about if we move it out a month. Oh, that’d be great.
368
00:50:33.420 –> 00:50:43.920
So it’s, it’s part of the of that collaborative approach. So I think that’s really key with all of these changes and particularly with the cloud, just being aware
369
00:50:44.370 –> 00:50:53.580
Of what’s being pushed and the very tight turnaround cycle that the team has to do the testing to check for functionality and then prepare for that update
370
00:50:54.150 –> 00:50:58.380
Yeah, well, because it does obviously change some of the processes that you have to do both.
371
00:50:58.410 –> 00:51:05.700
Both your team as well as it, you know, to be able to prepare for these if they’re coming out, monthly, you know, where
372
00:51:06.030 –> 00:51:22.170
You know, again, it used to be. Maybe quarterly or six months, you know, is when when the updates used to come out and now, like you said, that they’re coming up much more frequently. So you just have to have to have a better process to be able to help you get through those as well. So
373
00:51:22.320 –> 00:51:35.130
Yeah. Yeah, absolutely. And right now, our department is looking at a consulting project to implement DevOps principles within our different product development life cycles.
374
00:51:36.030 –> 00:51:47.790
We have core product yeah and and it’s really pretty interesting. There’s s s DLC is very good at being pragmatic and controlled predictable.
375
00:51:48.690 –> 00:52:00.960
But it’s not very good when you need speed to market and product evolution is critical. And of course, with our digital products, you know, we need to get the market quick we
376
00:52:01.260 –> 00:52:01.950
Adapt
377
00:52:03.000 –> 00:52:12.240
You know, these are all web based and mobile app based things that we’ve got to quickly turn around. So our digital products group has
378
00:52:12.630 –> 00:52:28.920
Implemented some of these DevOps principles pretty still low to medium maturity. For most of them. But if we can take these ideas and use them within our core business where we have our IT operations and we have all of our center media products.
379
00:52:29.190 –> 00:52:31.500
Then I think will be more nimble.
380
00:52:32.760 –> 00:52:35.580
Company and our products will come out quicker.
381
00:52:36.120 –> 00:52:48.000
The coral. The corollary to that is using the same kinds of DevOps principles for these cloud pushed upgrades, particularly with with finance and operations module.
382
00:52:49.050 –> 00:52:50.100
Automated testing.
383
00:52:51.450 –> 00:53:03.060
Understanding what dependencies are within our custom code. I think those principles are going to be really key to quickly adopting whatever changes get pushed to us.
384
00:53:04.710 –> 00:53:10.590
Well, I think that’s that is going to be the future things are things are going to continue to happen at a quicker pace.
385
00:53:11.730 –> 00:53:24.480
And we have to figure out how to be able to do it at a quicker pace, you know, that is one of the challenges I think for our profession in general as business speeds up, we have to figure out ways of speeding up as well. So
386
00:53:26.610 –> 00:53:26.700
I
387
00:53:27.060 –> 00:53:27.750
Completely agree.
388
00:53:28.350 –> 00:53:33.180
Good discussion Tom and like normally at night, we get to talk in and holy smokes, the time goes by.
389
00:53:35.220 –> 00:53:35.670
I know
390
00:53:36.060 –> 00:53:37.890
You probably need to get to another meeting.
391
00:53:38.550 –> 00:53:47.910
But hey, I really, I really appreciate you coming on and talking about this. I think it’s some really practical stuff that a lot of people are dealing with and
392
00:53:48.330 –> 00:53:57.300
You know, hopefully, again, as they listened through here that they get some ideas for you know how they can improve what they’re doing. Maybe take this kind of an approach going forward.
393
00:53:57.840 –> 00:54:13.110
Because, you know, just like you were talking about s DLC is. It’s good. It’s a good process in general, but it’s very rigid, I think, you know, a lot of our traditional audit methodologies like you know going in and doing a post implementation review.
394
00:54:14.400 –> 00:54:20.250
Fine, but it probably doesn’t match as much the business environment that we’re in right now.
395
00:54:20.730 –> 00:54:26.670
And so we’re going to have to start doing things a little bit differently. And like I said, I really appreciate you coming on and sharing
396
00:54:27.120 –> 00:54:40.770
What you’ve done that is a little bit different and kind of the good things that happened. Yeah, there were some learnings. But, you know, overall, like you said this was an A minus p plus kind of effort. So, gotta say, Good on you, mate. You guys did did good on that one.
397
00:54:41.790 –> 00:54:45.570
I will, thank you. And it’s been a pleasure to talk about it and I really enjoyed the conversation.
398
00:54:45.960 –> 00:54:46.920
Alright, well thanks
399
00:54:47.070 –> 00:54:47.640
Thanks again.
400
00:54:48.630 –> 00:54:50.160
You bet. Cheers. Cheers.
