When I read this article recently, I was instantly reminded of the problems my customers face when SAP makes yet another unannounced change. The facts in the article are:
- SAP made a change to the customer's cloud system that the customer did not authorize.
- SAP claims a human error was the reason for this error. (At this point, I was wondering whether the human error happened during the update or whether SAP did not test enough before the update to detect it.)
The article is about cloud platform, but my C4C customers face similar issues with SAP. The difference is that SAP usually does not apologize but instead often claims (especially if custom developments are involved) that there was no change and that the system has always behaved that way. In the beginning, I often questioned the sanity of myself and that of my customers, as I always do and document a test before handing a change over to the customer, the customers always test as well, and the end users use the functionality, often for months, without issue until something does not work anymore (usually after an upgrade). So I've taken to assume that SAP do not keep track of their changes themselves and just claim that nothing changed - this helps me keep confident and sane.
Anyway, regular updates are part of being in the cloud - in that respect we all knew what we signed up for. After all, regular updates are a good thing with regards to security and they may be a good thing with regards to new features. But the downside is that you have to test with every upgrade. With SAP C4C, the interval is every three months, your test systems are upgraded two weeks before the productive systems, which gives you 2 weeks to test and is quite ambitious. So they better do a good job of helping you along.
Let's take a look at the options your vendor has here:
- being transparent so you know what to test
- testing thoroughly so all of your own tests are successful
- resolve any errors that you find before the release
Being transparent
If your vendor is very transparent, they will facilitate testing by listing all changes they made since the last upgrade so that you know exactly what to focus on in your testing.
Unfortunately, SAP does not do that with C4C. Sure enough, there are the Release Briefings and the "What's new" section in the documentation. But this is very high level and is more marketing than a true changelog.
Compare this with the on-premise support packages where SAP provides a (very long) list of every note that is part of the package and you know what I mean.
Adding to this, the between-release upgrades (a.k.a. Hotfixes or HFC) are not documented at all. So whenever you check the About window of C4C and notice that the version number has yet again been incremented, you find yourself wondering just what has changed.
While this is mostly a nuisance when the SAP Standard is concerned, it becomes a real problem with custom developments. I've had more than one instance where SAP changed the backend logic (a favorite being when they changed the finalize ordering, i.e. the sequence in which BO events are called), didn't document or communicate it and all of a sudden our custom developments started failing.
Testing thoroughly
Okay, so if thorough documentation or changelogs are not an option on SAP's part, then maybe they're at least testing everything thoroughly before going live.
As you may have guessed, they are not. I believe they are testing alright, but not thoroughly enough.
I would even go as far as saying that it is impossible for SAP to test a system that allows custom developments thoroughly enough, which makes good documentation and changelogs even more important.
But often, even the basics are not tested well enough. A favorite example being that the new GetFromDB() method (which allows you to compare the buffer of an object with the database) they released with 1902(?) dumped when you called it on an object that was just being created (i.e. did not exist on DB). In the dump log you could see that the retrieval function from DB was not the problem. The retrieved object was not bound. However, the subsequent code did not check this but just accessed the object - dump.
This tells you two things:
- ABAP Beginner mistakes are still happening in SAP Standard (they can happen to anybody working under time pressure, but you'd expect that SAP have a review system in place, especially for important/often used functionality like this)
- The scenario "executing GetFromDB() when creating a new object" was not tested. This is not excusable, in my view.
Granted, SAP is not alone with severe bugs like those. In times of increased competition companies can save a lot by cutting down on their test departments and relying on customers to do the testing for them. This is usually framed in a way like "early access to new features", but all you're really doing is doing the testing for your vendor for free. SAP is no exception here. You can request early access to new features via incident if you like.
Resolving errors
In the "olden days" of on-premise, if you observed issues during testing, you ultimately had the option to postpone your upgrade until the errors were resolved.
In times of cloud-software you don't have that option, but you can expect your vendor to postpone an upgrade if a new version is flawed and cannot be fixed before production upgrade.
How does that work with C4C? SAP claims that you have until Thursday before the upgrade (night from Saturday to Sunday for most customers) to report release-critical incidents.
Let's cut it short: Even if you report a simple issue with the new release on the first day after your test systems are upgraded, the best answer we have yet been able to get from SAP was that they would fix it with the first Hotfix (HFC), about 2 weeks after the upgrade.
If you have a trickier issue that involves your custom developments, you are lucky if SAP even acknowledge that there is an issue before the release. Don't expect a resolution to your issue before upgrade, though.
Why the talk about "you can report until the Thursday before go-live"? No idea. Appears to be marketing only.
And just so you know, postponing the upgrade by SAP in case of errors has also never happened in my time.
So in short: If errors occur during the test period, you're mostly out of luck. I don't say that SAP is unable to fix the errors before go-live - I just haven't seen them do it yet.
FYI: When I reported the error with GetFromDB() 3 days after test upgrade, I included the one-line ABAP Fix (check object is bound) in the incident. SAP still needed a few weeks(!) to fix the issue, while we had to apply a production hotfix directly after upgrade. Imagine me telling my customers I need a few weeks to make that hotfix and you will likely imagine me looking for a new job next.
What you can do
Based on my C4C upgrade experience of the past 4 years, I usually recommend the following to my customers:
- Do not mix a production upgrade with a deployment of your custom developments. If you deploy at the same time as upgrading, SAP is sure (in my experience) to claim that your deployment is the source of the error and you can add an extra week (if you're lucky) to the resolution time to prove otherwise. Deploy your custom developments to production well before your test system is upgraded.
- Keep your developments clean. Make sure you have an experienced developer at hand (Hello! 😉), who knows SAP Best Practices (largely not published or scattered across SCN). That way SAP will be less likely to complain that your developments are the fault.
- Make sure you have a clear test plan and test everything, including areas where SAP did not announce any change and especially the areas where you've noticed errors before (though, granted, there is in my experience not "the one" area that is particularly prone to upgrade errors).
- Make sure you document your tests with screenshots so you can prove to SAP that it worked before (in most cases, SAP will still claim that nothing changed but it might boost their efforts in trying to help you)
- If you're still in the 2 week test period, try (if you can) to show SAP that the issue only occurs in your upgraded test tenant but not in your (not yet upgraded) productive tenant.
- Know your support people at SAP. I have list of helpful SAP support consultants and every time I raise a critical incident I ask it to be assigned to the person that I know can help me best. SAP usually follows along with that request.
- If your incident is not being processed adequately, request a screen-sharing session and if that doesn't help, request a critical incident manager. This is more for ease of mind, really, but can speed up processing a bit. You'll have somebody who will call you 3 times a day and update you on the status of your incident.
- If you're a large customer: Use your C4C issues as leverage in negotiations with SAP. (One of my customers had us record all the time we spent dealing with SAP Incidents separately, so they could make a list for SAP.)
Outlook
From my many dealings with SAP regarding C4C over the last years, I conclude that C4C was not designed to be a heavily customized system, nor one where you have complicated developments. It appears to have been designed for users that want to implement a CRM in the iconic 4 weeks. Setup, add a few custom fields. Done.
SAP Core Developers have hinted in phone calls at their own frustration with the way the backend currently behaves and that it needed to change.
It really pains me to see that SAP are so often standing in their own way of delivering what is basically a really great product. C4C has great extensibility and easy integration with other SAP software. That's a core strength no other vendor delivers.
Fortunately, my impression is that SAP are internally recognizing that they have issues (though they'd probably never admit it officially) and that they are working hard to improve. (From end of 2018 to mid 2019 we did not see any major new features for developers, so I take it the focus was on stability in that time.)
As long as C4C is not the system it can be, though, I call for better mitigation of the issues at hand. This consists, in my opinion, of:
- having a test period that is long enough to allow SAP to fix any errors reported before the production upgrade (it currently seems that customers should be given 4 weeks to test and then SAP should give themselves 4 weeks to resolve any issues reported)
- providing detailed changelogs that include backend changes so customers know what to test in particular
- SAP improving their product testing so that "beginner mistakes" as the ones described above do not make it into production. If you'd take as much time testing your software as you take resolving errors (see GetFromDB() example above), that would be a start 😉.
C4C is a great CRM solution and outperforms its competitors easily. And as customers care about a stable product at least as much as about new features, upgrade management needs to improve to keep them happy. Implementing the points just mentioned does not cost the world but would be a big step forward.