Oracle Software and Solutions

We use Oracle Solaris to develop and support our VDCF (Virtual Datacenter Cloud Framework) management and monitoring software. Several virtual machines (LDoms and Zones) are used on SPARC and x86 Servers.

We are able to deploy new environments very quickly and securely. Using the virtualization features, we can migrate the environments very flexibly between our servers.

Solaris includes two virtualization solutions: LDoms for SPARC and Solaris Zones. Both solutions can be combined to create private clouds. Solaris Zones is ideal to separate applications and to migrate from older to current hardware. LDoms is very efficient because it uses the hardware hypervisor of the SPARC servers.
Both technologies increase Security, because they separate the applications from each other. Using the Security Compliance Framework we are sure the systems are setup properly

Patching without downtime would be nice.

Update 08/2021: Live Paching of Kernel is now available. We applied IDRs successfully on several servers.

I have been using the solution for more than 14 years now.

We had no issues with the stability.

Solaris is very robust and scalable. No issues so far scaling it.

Oracle offers a good online support portal called "My Oracle Support", which includes a big knowledge base. Because Oracle is a very large organization, it sometimes takes a bit too much time for support requests to reach the right support engineer.

Neutral

Used Solaris 10 before, but Solaris 11 is much easier and faster with patching.
Based on BootEnvironments and ZFS Solaris 11 always offers a failback.

The initial setup is straightforward. Oracle has in-depth admin manuals. To speed up deployments, we implemented our own deployment and management framework VDCF.

We implemented it with our in-house team based on Oracle's best practices documents. With virtualization, we recommend to first define a standard on how to deploy and then to deploy using that standard, avoiding any variation. There are so many options, but our favorite is the fully-virtualized LDom with applications installed into Solaris Zones.

When buying a server from Oracle, all the software is included -- OS, virtualization and patches. There are no hidden costs. We like the long life cycle of Solaris and the SPARC servers. There's no need to replace the hardware every two to three years, and we have a life cycle of five years and more.

Use deployment tools for automation and avoid doing everything manually. Deployment tools help to avoid errors and create a standardized environment.

Provide 24x7 monitoring of application system availability for both on-premise, and on-cloud application systems, and forecasting data relating to overall system utilization (or under-utilization.)

With over 2,700 different target components deployed and in-use, it would be impossible for 4 DBA's to otherwise monitor and assess the condition of this number of separate application and database systems by individually logging into separate administration consoles for each product. The automation elements reduce the training time required to get our junior operators up to speed doing routine patching or cloning, and allow the senior staff to have more resource time available for strategic planning and ensuring overall availability of the critical systems without sacrificing proper monitoring and security awareness checking.

Recently, including new off-premises (cloud- based) application re-hosting, presents the challenge of whether to monitor cloud from on-premise, or vice-versa. We are currently experimenting with a cloud-watches-cloud plus on-premise-watches-on-premise approach.

The 13cR5-PG (Patch Group) 6 updates to the OEM family, strongly integrate Cloud (off-site, hybrid and on-premise) services providing a seamless way to see all of your resources regardless of where they are deployed.

PG6 adds enhancements for EMCLI (Command Line Interface) extensions to allow additional automation and scripting of common OEM tasks (Blackout begin/end, Patch availability monitoring, cloning enhancements.)

The 12c series of Oracle Enterprise Manager products, version 12.1.5.0.x introduced Cloud (both public and private) support for both monitoring, and lifecycle management. This allows individual components, or entire systems to be resident either in-house on conventional hardware/VM's, or ported into virtualized datacenter environments, off-site, whether for fault tolerance and disaster recovery, or simply to reduce the cost of non-Production systems by using a pay-to-play methodology (reducing investment in specific hardware just to support disposable and transitional software systems development?) Consolidated and uniform monitoring of all Oracle-related technologies. Ability to enable centralized control and administration of Oracle products from a single console. Ability to see at a glance, all conditions of all products throughout the enterprise. Version 13c includes hybrid cloud support making transparent the transfer of provisioning systems between Oracle's Cloud and on-premise equipment. All of the cloud systems and in-house systems appear in logical groups according to their common function (Applications, Middleware, Databases, etc.) The configurations are also snapshot-friendly and made comparable to quickly isolate differences between similar systems.

In the Oracle Cloud Infrastructure (OCI) based Oracle Observability and Management OAM (formerly Oracle Management Cloud (OMC)) 1.53.0 you have many subscription-based options, like Log Analytics and Application Performance Management that you can activate quickly, try them on for size, and keep them or decomission them for relatively less cost and time than doing so, on-premises. These features also avail you access to technologies requiring higher horsepower (CPU/RAM/etc.) found in the OCI Exadata-based Virtual Machines (VMs) that may be beyond your current investment level. Newly added features, include Java deployment central administration, logging analytics, 21c and 23c compatibility, and  enhancements for the new cloud service extensions.

Continued development of the Cloud Management suite has integrated further dynamic container and vCPU management options to the cost management side of the Integrated Lights-Out features of the cloud service providing better management of your capacity planning, and thus cost mitigation.

Reporting and statistical charting is largely still left up the end-user to develop custom solutions. The newest releases support reporting through the Cloud Analytics, SaaS service. Reliance upon BI Publisher will eventually be depreciated. The actual product inventory discovery and configuration process has improved, but is still fairly convoluted and requires multiple pre-requisite setup steps to be completed, requiring numerous Cancel, Go back and set something else up, then Return to the process you were performing types of process flows. It works really well if all the technology stack layers are current releases, but the more heterogeneous the architecture is, the more you will spend more time configuring outlying systems, or systems that aren't quite up-to-date.

The OAM array of offerings is quite numerous, so when you first try to navigate the menu offerings, you'll experience 7 to 10 layers of click-throughs to find what you may be looking for in the mass number of options. There is currently a hybrid UI between the Cloud Classic menus and the revised OCI v2.0 menus that has many duplicates, but are found using different navigation paths, which can be confusing. While the main dashboards are fairly clear, there are often click-throughs that lead you down not so clear breadcrumb trails during your navigation. One thing that is particularly annoying is when a Cloud Management component itself fails (such as a Cloud Agent, or a discovery job) you will still need a 2nd monitoring system, such as OEM to watch for the failed Cloud service target. Sometimes simple operations like checking for a failed backup set in OAM are not at all straight-forward and need to be programmed individually by the customer, rather than being available as a simple template to be activated, as expected.

More than 15years.

During upgrades, plugins take on 2 different architectures - some are built as standard WebLogic composites, allowing hot upgrades and installation (meaning you don't have to have an outage for the OEM Management System (OMS) in order to complete the installation. Others are more complex requiring specific outage windows during which time the OMS will not be able to monitor targets (often the OEM Agents are still operating, and collecting information, but you need to develop standalone OS-based alerting on the individual boxes to leverage an individual Agent's ability to provide outage notifications when the OMS is down.) These OMS-downtime upgrades and installations can also be the most prone to potential failure during installation requiring complete system rollback to a backup point.

OAM is updated frequently and only incurs momentary downtime when sortware homes are being switched between pre- and post-patch versions. Agent patching is similar to on-premise, requiring no downtime.

Both on-premise and OMC are scalable vertically and horizontally; you would just include the licensing costs of high-availability for both the middle-tier and database layers (or number oF VMs for clustering) and the added storage or incremental costs to include additional targets or resiliency.

Customer Service:

7 of 10 for customer service (often various contacts are disconnected from each other communication-wise regarding the profile and installation data of customers).

Technical Support:

8 of 10 for technical support (My Oracle Support is well-integrated into the OEM platform and guides the user quickly to relevant solutions.)

Neutral

Previously used prior release of OEM (version 9.2.0.6) - these older versions are no longer supported by Oracle, and are prone to many security, stability and incompatibility issues with most newer product releases (for example, 10g (10.2.0.x) is the only client still capable of backwards-compatible connections to 9i and 8i databases, and also forwardly compatible to 11g and 12c databases - but it is not supported anymore, patch or bugfix-wise.)

Straight-forward - the basic deployment simply requires an adequately sized and certified hardware platform with sufficient storage to hold the software, and database repository, and sufficient CPU to run a web-based application based upon the number of potential monitored targets and users.

OAM is a little different in that there are a number of security-related prerequisites to enable the OAM side to communicate securely with the OCI VMs being monitored.  All of the process, though lengthy, is reasonably documented, and typical of Oracle documentation, a little fragmented and jumps around a bit.

In-house implemented. 3 days from initial download to production operation.

The typical acquisition and year-to-year support maintenance costs equate roughly to a person-year of a typical DBA. You can expect that given the ability of the existing staff to either become more precise and strategic at preventing issues from arising without adding additional staff, makes this an easy pitch. e.g. You could either add 3 additional dedicated DBA's whose job is to login and monitor every application in a typical 4-5 major application environment (EBS, HR, BI, Webapps, Mobile, SOA, etc.) 24x7, or you can install OEM and configure it to provide the requisite alerts and projection reports needed to ensure the same SLA requirements, without adding staff.

Based upon 3 days of implementation by a single person, plus licensing costs would be approximately $60,000, including the virtualized hosts. Day-to-day is very roughly $100 for routine patching and maintenance. High-availability significantly may double or even triple these expenditures, but for some environments that's inevitable. If one system needs that level of oversight, then all of the infrastructure will be managed to the same level of oversight by OEM. There is also an augmented set of separate, but useful features being added under the Oracle Management Cloud suite of products designed to focus more on analytical problem identification (such as when one issue triggers a seemingly unrelated set of other symptoms) and leverages the power of mass target data agglomeration at the expense of having costs driven by the volume of data being recorded. But this is typical of most cloud-based solutions, where bandwidth utilization is a cost driver.

Oracle Management Cloud is basically a VM environment spun up in the region containing your systems to be monitored and managed. Oracle do not support cross-region management using OMC (however, you can do that with additional plugins to OEM on-premise) so if you run VMs actively in more than one region, you'll end up with multiple OMC systems, and the associated costs.  Be aware, you are charged in many different levels, but mostly to license the product itself, the cost and storage to run the VM, and then storage to injest and store however much logging and data you want to retain for the puposes of monitoring, troubleshooting, auditing, and forecasting growth. This is quite different from on-premise monitoring costs which usually are limted to a base license, plus support subscriptions, and then incremental spends for increasing base storage for the hosts. It's best to proceed slowly and incrementally when adding systems to OAM to monitor, so you can baseline and measure how much each system costs in-total to monitor, and determine what level you want to monitor each system (Enterprise and Standard monitoring templates are available and can be dynamically switched for each system, or the whole OAM target inventory.)

ELK, HP OpenView, BMC Patrol

Analyze your daily monitoring and service intervention needs, based upon the available (or target) headcount, otherwise you may find yourself inundated with more alerts and notifications than can actually be handled by your staff. Don't just turn on all the alerting and notifications out of the box. Start with what you know you want and are capable of responding to, and closing effectively. Enable those. Then work outwards towards the nice-to-know, or discovery alerts.

Currently, depending on your implementation and use timeframe, you may also want to evaluate the Oracle Observability and Management Cloud solutions, which are similar, but different in that they employ a pay-as-you-go approach to the monitoring and management cycle. The On-Premise OEM solution works best for dynamically growing environments (that tend to get bigger and bigger over time) or for stable complex environments that will be in-place for the next 10 or more years.

Primarily OLTP but report is done against a combination of Materialized Views and transactional tables.

We have a number of statistics collected before cutover on our legacy environment compared to Exadata. Without doing anything other than copying the data across, we saw significant performance gains for most key processes. We receive feedback from users stating how fast the performance is compared to other systems. Performance issues are few and far between. Our database environment is extremely stable compared to the legacy DB configuration. We upgraded from a X2-2 quarter rack to a X5-2 eighth rack and experienced significant performance gains. We recently performed another technology refresh to a X7-2 so obviously, we've been very pleased with the initial investment. For this deployment, we decided to virtualize the Exadata configuration, providing some additional flexibility to our operational environment.

We primarily run OLTP with some reporting. With that being said, the feature that provides us the most performance gains is the Smart Flash Cache for the OLTP databases. The "offloading" capabilities provide the biggest performance gains for Reporting such as smart scans and storage indexes. There is a new security feature which allows disabling ssh to the storage servers which will make my security folks very happy. Also, there is a STIG script for hardening storage servers and Database Nodes which can be implemented as a report only or actually implement security settings. Would advise running report first to assess the results and then manually modify, as needed.

My biggest gripe has been patches which has dramatically improved since our initial Exadata was delivered (January 2011). The only issues we periodically experience are with non-default RPMs on the database nodes. These may fail during the pre-req check which means opening a SR with support. This has become the exception, not the norm so overall not much to complain about. The X2-2 used to experience frequent disk failures but now, that is a thing of the past. 

eleven years

There is redundancy built throughout the Exadata so even when we've experienced a disk failure, it's a very low stress situation. Early on we had some performance issues with DBFS and a node eviction problem. DBFS was resolved through a combination of settings changes and a quarterly patch. The node eviction was resolved through a one-off patch that eventually got rolled into a quarterly patch. I would chalk up these issues to being early adopters. We do have an occasional bug but I can't think of any that would be unique to Exadata with the database software. At least this provides some degree of comfort that Exadata is not the source of the issue.

The key for scalability is selecting the appropriate disk configuration and the proper size rack configuration. The two options are High Capacity and High Performance. If ever in doubt, always go with High Capacity. The performance difference is negligible at best, however having the extra space allows for more consolidation. That's the entire point of Exadata, to consolidate databases. We've added a few databases to the Exadata since we originally started to use the environment and there has been no performance impact. In our case, a Quarter rack was appropriate but for larger environments, this may not be enough.

Customer Service:

In terms of overall Oracle customer service, we've had good experiences on this front. Oracle has provided us access to their experts and continually check to see how things are going. Whenever an issue comes up, they treat the problem seriously. Since we support a government customer, Oracle is extra motivated to ensuring we have a successful experience. Since 2011, there have been significant improvements with support. Occasionally we do hit issues which it seemingly takes support a longer period of time to provide a patch or workaround but these namely involve additional features, not core technology so it's a matter of exhibiting patience.

Technical Support:

On the hardware side, customer service is quite good. Any disk failures get replaced in a day and with triple redundancy for disk, it's not been a concern. Software customer service has improved over the years. Early on was a little rough as I will say the software wasn't fully mature. As the product has matured, so has the software support's capability to resolve issues more quickly. We can't take advantage of ASR, however this seems like a major improvement for customer service in terms of responsiveness.

We didn't switch, we were doing a technology refresh and went with Exadata instead of building out our own Oracle RAC configuration. We previously had a combination of Dell servers, Red Had Linux OS, Oracle Cluster File System on EMC Storage with Juniper switches. This configuration had lots of performance issues, node evictions, and constant headaches. Since moving to Exadata, all those pain points went away.

There is a definite learning curve initially. We had to learn about migration options, shared mount point options, how to integrate with Cloud Control, patching, health check, how to optimize, and how to harden the Exadata environment. Since we went live, many more folks use Exadata so there's more how to's and best practice documents available so the learning curve isn't nearly as steep. We learned a lot in the process and now have a tremendous amount of expertise in setting up, configuring, optimizing and maintaining the Exadata.

We implement Exadata in-house and have gone through several migration methodologies.

We had ACS perform the initial Startup Pack, however there are companies that can do this much cheaper to lower the original setup cost, such as ours. Day-to-day cost is greatly reduced compared to our legacy environment as we no longer have to serve as "fire fighters." In terms of pricing, Exadata is probably not going to be the lowest cost option. There is a price to pay for performance and stability. With that being said, I have not heard of any customers who have regretted the purchase and/or looking to get off the technology. On the contrary, I can't imagine going to another solution at this point and trying to justify this with the user community in terms of why the system performance degraded. Can't imagine that would go over too well.

We had a custom solution and evaluated Exadata versus the custom solution. Exadata was actually a cheaper solution due to the number of cores. Oracle software licenses are based on processor so if comparing a Quarter Rack versus a 4+ four node custom solution, Exadata may win out from this perspective. We were looking at a 5 node RAC which would have doubled the cost of our software licenses when compared to the equivalent with a Quarter rack of Exadata. Besides, the performance metrics indicated Exadata would easily outperform the custom solution which made our decision a no brainer.

Exadata is a powerful solution. As I mentioned there is a learning curve. Working with a company that has experience with Exadata can help avoid potential pain points and maximize the ROI.