Tidal by Redwood Reviews

In our organization, we are running scheduled jobs in batches. There are three different uses for Tidal, which are primarily development, testing, and production. It's like the environment that allows you to run batch jobs in those three environments.

I do production, like systems administration, and the development of the batch environment. So, I install Tidal, maintain it, upgrade it, apply hotfixes, or do any type of system admin function. Also, I set up batch environments for development, QA, and production which run in Tidal. I also train people on how to use it.

In terms of deployment, we're on-prem. We've adapters and virtual machines, but everything is on-prem.

It streamlined the batch environments. It got everybody on the same page, and people haven't had to monitor their own jobs. We've been able to centralize the running of batch environments. We've been able to migrate various schedulers into one central location. It has helped with reducing risk by being aware of what's running and making sure that what needed to run did run versus something breaking somewhere and nobody knowing about it. Here, if something breaks, everybody would know about it, and then action would be taken. It has reduced risk and streamlined and centralized operations.

It's very versatile. We generally have SQL jobs. We've SQL, SSIS, PowerShell, C#.NET, etc. So, whatever a program is written in, and whether it runs on Windows, UNIX, Linux, or mainframe, there are just a wide variety of jobs it runs in different programming languages.

I've been an administrator for other schedulers over many years, which includes CA, AutoSys, Control-M, etc. Of all of them, Tidal is the easiest one to use. It's very GUI oriented, whereas a lot of the other schedulers have been command-line-based. It's by far the easiest scheduler to use for creating objects, which would be jobs, events, and things like that. It's all graphical. It's Windows-based. When building a job, most features are predefined. You just have to select where the executable is to run from and the time or event-based settings. You don't have to use the command line. Everything is kind of predefined for you. So, you just go through Windows and fill out different features in different jobs. It's simple. I've trained a lot of people on how to use it, and people who use Tidal for the first time are able to make jobs, whereas, with other schedulers, there are other things that they need to know, such as operating system-level commands. With Tidal, almost everything is already defined. You just tell it where to run the job from, which folder it's in, what's the name of the executable, and then the time when you want it to run. It can be learned within an hour. It's pretty cool.

I've used Tidal Automation to connect and integrate with a lot of different software and systems. I've used PeopleSoft, Informatica, BI, SQL Databases, VMware, Azure, and others. I haven't had any issues. So far, everything has been great. They have a compatibility matrix that ensures that different versions of Tidal will work with different software. They're good about keeping that up to date. Occasionally, you have to apply hotfixes to different software levels, but I've had good luck with the interfaces.

It is easy to integrate other technologies and processes via the REST API. All you need is access and a log-in ability, and it integrates well. You don't really need to do a lot of configuration.

It has increased capacity by manyfold in terms of the number of jobs and/or integrations. I've gone from 500 jobs to 1,500 in just six months. That's really in the first year of setting up Tidal. It worked that way at other companies too, with the same kind of jump.

Our use of Tidal is mostly file-event driven. We use it to manage our ingestion, processing, and loading of data. Tidal has a hook and it runs ETL for us. It runs jobs and SQL and some of our database appliances like IIAS, the new version of Netezza Teradata.

We have a file gateway that receives a file and drops it in a location. That file event picks it up and drops it over to the ETL tool. The ETL tool will run and aggregate a number of source files and turn it into a properly formatted input file. That file then goes through data hygiene and data analysis. Then it goes through a matching process. It is then put back out and runs an ETL process to stick it into a SQL database. And then there are a number of jobs that are run in the SQL database to manipulate that file.

We don't have a lot of calendared events or scheduled windows.

We have a central location for Tidal in our data center, and then we have client-hosted solutions where we run smaller instances of Tidal, and those are in the cloud. We use AWS, Azure, and GCP.

It reduces our administrative costs. As much as people are in a DevOps model, we can create dashboards for tier-one people to monitor multiple jobs and then alert or call when things fail. It helps us with reliability and managing SLAs.

It has also helped to reduce weekend and overtime hours due to the fact that you can have a single person manage multiple jobs. If we didn't have the single pane of glass and that visibility, people would have to manually look at logs to determine the progress of a job. So it reduces headcount. But when you run 24 by seven and 365 you still have people working weekends.

We run 70,000 Tidal jobs a day. it would take a mountain of people months to run that many jobs manually.

We're running jobs on a global scale. Being a global company, we're running scheduled jobs and ad hoc jobs across different regions. Jobs cover backend processing, financials, and the like. We're running on an SAP ERP system and we're also running Informatica for data warehouse. We're running BusinessObjects web reports as well as a lot of straight Windows and Unix command-line things. We run FTP processing, PGP encryption processing, and data services jobs. We're running about seven or eight of the different adapter types that Tidal has available.

We have it on-prem. Both our test and production environments are on fault-tolerant setups.

When I started here, they had already been on Tidal for about five years. So I'm not really sure where they were before Tidal. They did a lot of mainframe things in the past. From what I've heard from people here from the "old school," once they globalized and got everything into Tidal, the ability to see everything across the scheduling universe was a huge improvement. They didn't have to give different people different access to different systems and check four or five things, just to make sure something was running correctly.

The solution helped to reduce weekend and overtime hours. We're a 24 by 7 support model. Regarding the Tidal application, the one thing that we try to explain to anybody, from a support or monitoring standpoint, is that jobs trigger through Tidal, but not physically in Tidal. So if we have, hypothetically, an SAP job failure, it's not a Tidal failure, it's an SAP failure. So it goes right to SAP support, which saves time. In the environment I came from, they didn't have that mentality. So if, hypothetically, an ERP job failed, they'd call the Tidal person first instead of the ERP support. That type of understanding, as a whole, really helps from a support standpoint. The admins don't get a lot of calls unless there's an actual issue with the Tidal application itself.

In the time I've been here, we've definitely increased staff availability. From a business standpoint, we've started utilizing file monitors more, for what they call "file events" within the application. In the past, when an end-user would drop a file in SAP, for example, they'd contact our operations team, or send an email saying, "Run in this job." There isn't a real need for that in many cases. We've implemented a lot of file events that will actually only run jobs if they need to, if a file's available. Along the same lines, we had processes that would run a process in SAP, and even though it didn't create a file, there were other jobs downstream that would be hanging out and waiting for a file that never showed up. So not from just a staff availability point of view, but in terms of resource availability, it has definitely improved things a lot. From an operator standpoint, I would estimate Tidal is saving us 15 to 20 hours per week, just in manual interaction with inserting jobs on a request, since a lot of that stuff was implemented at our end.

Regarding job counts, we're pushing over seven million a year. That varies, obviously, depending on request jobs and other things. There are some processes that we shut down for year-end processing, so they stop running for a week or two. But from an expansion standpoint, we are constantly looking to see where else we can use Tidal, for new applications that are coming online or things that people are running on their own where they haven't even thought about Tidal's scheduling. In 2019, we did 7.7 million jobs. In 2018, we were at 7.1 million. In 2017, we were at 6.1 million. So with Tidal we're adding on the order of half-a-million jobs per year.

We have multiple ERP systems, and we use it to schedule all of our jobs in those systems. Our biggest use case for Tidal is to automate jobs that we submit through our JD Edwards ERP. 

We also have a lot of integrations using FTP file events to move files around, and we are also using the software to automate all of our manual stopping and starting of services and patching of systems.

Our MRP job stream is very complex, and it has a mission-critical report that needs to run on a daily basis. One of the first things that Tidal gave us was visibility into that report so that we would know how long it runs, and we would be able to monitor it. If there were any issues, we could resolve the issues and not affect the next day's manufacturing because we took care of the issue. 

Awas that the team had a requirement that they needed to submit multiple jobs after a single job ran. They would put them all into one big single-threaded queue, and all those jobs that were dependent on that first job would get queued up and they would run single-threaded. That's because, with jobs, you can't go from single-threaded to multi-threaded and back. With Tidal, we are able to have one job run. As soon as that job is done, we can submit about a hundred jobs. Those hundred jobs spawn and do hundreds of other jobs. After all those jobs are done, we have another single-threaded single job. We would've never been able to handle such complexity with any other scheduler that I know of. So, we are able to solve the business' needs and improve the performance of MRP just by going to Tidal.

Another example is that a lot of times, the jobs were submitted every five minutes. If you asked the reason for that, the answer was that a file might show up, and as soon as that file shows up, I want to kick off the process, and process that file. We changed it from every five-minute job. It now only submits when that file shows up. Using file events removed the confusion of when that job is going to run where somebody would go out there and submit it, and if it was every five minutes, they'd have to wait for five minutes. Now, they submit it, or they put the file out there, and within a minute, they get a response back that says, "We're done with the job, and it is processed." They were surprised at how that was possible. It is just a file event, and it only runs when it is needed. It is not run every five minutes. So, we don't have all this extra processing time or extra jobs out there doing absolutely nothing because there is nothing to do. So, that's another way it has changed the business.

It has made things more efficient. We went from one set of jobs running every five minutes to running just ten times a day, which was the max I had seen that job run. So, we've seen the need go down because we're able to be more efficient with the jobs that need to be processed. An increase in the number of jobs would only be because we've been able to take more jobs that someone was manually submitting. We showed them how to set up the process in Tidal, and all of a sudden, with Tidal, it is just easier to automate it.

We use the Dashboards feature. We have opened up Tidal. Previously, a lot of our other schedulers were very much IT-only. We're the ones who got in there and created weekly jobs, whereas now, we've pushed some of that functionality back to the business. They go out there and submit a job. We taught them how to schedule to submit that job, and then, they maintain that job. If there is an issue, we monitor it for them and help them to resolve it, which makes it much more of a team effort between the business and IT, rather than just IT supporting these jobs.

It is very easy to use the Dashboards feature. Basically, they log in and see their dashboards, and/or they can see all the jobs that are running. One of the key things that we need to do is secure it. I can't have somebody log in and see another system's jobs for two main reasons. The first one is that it would be confusing, and the second one is that I can't have that. I need to have it secured. So, on top of it being easy, it is also very secure where when they log in, they only see their set of jobs and their dashboards.

The Dashboards feature gives information about a job’s status. A lot of times, when there is an issue with a job, this is the starting point to figure out what the issue was. You can then go and see job logs. A lot of times people call and say, "I have a job, and when is it going to be done?" With Tidal, it is very easy to go look at Tidal and say, "This is when it started, or this is the expected time it is supposed to start, and here is the average that it has done for all the other jobs that it has run." So, there is a lot of information that people can get at this all-in-one spot. If they had to manually go look at it, they would've to go to multiple different spots to get all the information. Even then, how you read the data isn't exactly consistent? For example, I submit a job, but it goes into waiting status. According to JD Edwards, it is running, and then, at some point, it'll go into processing status. That's the actual time that it takes to run. If you look at the start time and the stop time, that'll be two hours. If you look at the amount of time it really took to process, it would be only about 10 minutes. It could have spent the rest of the time waiting. That's where Tidal gives you the ability to see the actual processing time of how long it is going to take to run.

The Dashboards feature offers self-service for business users with customized content. As it has become available, it has been used much more. They never had it before. So, they didn't know what they didn't have. Now that we're able to give it to them, they're liking what they see. Being able to customize it gives them more flexibility in what they want to see.

Tidal has a lot of adapters. We use the SQL adapter. We use it to connect to an iSeries. We use it to connect to FTP. We use it to connect to all of our Windows Servers. There is also an agent for ServiceNow. We don't use that yet, but one of our future plans is to use it. We have JD Edwards ReportsNow that, instead of using ReportsNow scheduler, is using Tidal for scheduling jobs and reporting. The power of Tidal is the fact that I can have one scheduler to schedule everything. I don't have to have a scheduler on each one of these systems. I can, but then I have to maintain it on all of those systems. Having it on one, and being able to control everything, makes integrations a lot easier because I can stop processes, do patching, and bring processes back up. If it is all in one system, it is nicer and easier to do.

It is very easy to integrate it with other technologies and processes. A lot of times, there are a couple of different ways to do the same thing, such as with JD Edwards Orchestrator. We found three different ways to integrate into orchestrations. One was by running a command line that makes use of the SoapUI. They also have a web service that's more of a generic web service, and then, they came out and wrote a specific interface into orchestration so that we could schedule orchestrations through Tidal. Because there are so many different ways to do it, that means it is pretty flexible. It was fairly easy to figure out a way to do it and test it. When they came out with the actual adapter for JD Edwards Orchestrator, that made it super easy.

Another big advantage is the fact that when we schedule a job we know that it will submit, and if there are any errors we will be notified and able to resolve them. That way we're being proactive instead of reactive.

If there's an error with a job, there is an alert with a PDF. In that alert, we can customize messaging to assist people in resolving it. If there is a specific file location that they need to look at, we can put a link to that file location. If it's something with logs, we can attach the logs to the email so they have one place to start looking. We can even attach work instructions to that email notification: "Hey, if this job errors out and you received this email, here are the steps to resolve." People don't have to go looking for that information. They can just start resolving it right away.

Tidal has also definitely helped to reduce at-night hours. It's able to monitor itself. If a job fails, we're able to resolve the job and let the customer know, instead of the customer calling us and saying, "Some job failed. Go fix it," and having to research it. It could save my team about an hour's worth of work in each of those situations.

Overall, it saves us about 20 hours of work each week, hours where we would have been stuck trying to determine what the issue was instead of having an alert that tells us exactly what the issue is.

It's a company-wide batch scheduler.

It runs tons for us. It runs Windows, Unix/Linux. We connect with a lot of databases: Oracle, SQL, Sybase. We have BusinessObjects BI adapters, we scan emails, and we incorporate it with TriZetto Facets healthcare solutions. There's so much. It's our core enterprise scheduler.

It helps because we have brought in a lot of other applications and systems where we're able to use an enterprise-level scheduler that is consistently monitored and backed up and has a ton of redundancy so that we don't have any downtime. We're pretty close to 99 percent uptime on our scheduler.

It has reduced some of our weekend and overtime hours. For us, it's all based on the programming around the scheduler. For some teams, it has greatly reduced weekend and night hours, but for some people it hasn't because they babysit the process.

Tidal has also helped us increase capacity in terms of the number of jobs. Over the last three years we've added between 10,000 and 15,000 jobs.

Our largest use cases are for the execution of SAP and JD Edwards jobs. Then, there are a lot of other technologies, however, in terms of the Pareto principle, really that's the bulk of our processing. SAP is what we use for our manufacturing and operational type stuff with the actual products. JD Edwards is a lot more of financial reporting and projections and things like that.

We use the solution to run SAP and JD Edwards. Windows and Unix hosts are probably the second most common use case, as well as web services. Any REST API would apply - and we use a lot of REST API technology. Protocol, really.

Primarily the biggest thing is giving visibility to what, in a lot of places I've worked in, are transparent, like invisible processes. You have this massive batch and, unless you have someone watching that and you have a place to have a single point of truth and say this is successful or not, it makes it very hard to trace things. With Tidal, you can say, "Hey, this is the objective notifications," and all the ones you explicitly want are checkboxes. It's easy to add those things in.

It's mostly used for batch processing across our entire environment. We have approximately 16,000 servers and about 36,000 jobs defined and we run about 10 million jobs a month.

It's mostly VMs that we're using it on. Our physical footprint is pretty small.

I don't think we could work without Tidal or a similar solution. We do tens of millions of jobs a month, and for somebody to try to manually launch them and watch them would be physically impossible. Given that it was 20 years ago that we implemented this, we haven't lived that life in a long time, and we've grown significantly since then.

We use Tidal to run jobs across multiple application platforms, such as SAP, ECC, PDN, and Informatica, as well as jobs that run in Azure cloud. We also use it for several warehouse management jobs with OS/400 and AS/400 connectors. We have a lot of different types of connectors, then we are bringing all these jobs into Tidal so we can set up dependencies between jobs that run, e.g., an SAP job and a OS400 job may be dependent on each other in some way, allowing a cross-platform job flow.

We are currently on the most recent version.

We are using it for cross-platform workloads. That is probably the biggest reason that we are using it. The solution is generally good. Over the years, we have needed to do our own learning about how to manage it in terms of understanding dependencies and successors, then setting up times and so forth. However, this is the type of stuff you would have to learn with any scheduling app. We find it to be really useful. I'm hoping with the Explorer tool that they'll have better reporting so we can do some full cross-platform job stream reporting that they haven't really done much in the past. Therefore, we should be able to see some of that. In terms of managing it, I find it very useful other than the learning curve.

We use cross-platform management for so many things. We use it a lot for our warehouse management replenishment type things: to and from SAP. Once we implemented our job stream flow, things gets sorted out of house for delivery and can be update in SAP (and vice versa). Having the job stream has been helpful. Also, having it all automated makes a difference to replenishment. 

We use the ability to enable admins and users to see the information relevant to them specifically in our production environment. We can, but don't always, limit someone to only seeing data that they need to see. Then, they are not overwhelmed by other data. We do allow most of our users to see all the other data just for information and to understand the environment. However, you can begin to narrow in on what you need, if you're using policies and work groups correctly. Depending on how we use it, especially in production, it lets users only be able to do what they should be doing in production. They should only be managing their jobs, possibly see other jobs, and understand if there is a delay upstream which could be impacting them. They won't be able to manage those jobs. They need to contact the right people who understand those jobs to manage them. The solution lets them work within their lanes and do the work correctly without having a negative impact upstream, and hopefully, not downstream. 

There is an awareness that we are scheduling across the multiple applications and understanding that all applications don't live in their own silos. There is an impact across the organization. It gives us that holistic awareness, in general.

In the past couple of years, I have done education and we have leveraged creating alerts that go to the right people. It has allowed us to do that. Therefore, I don't get alerts for something that I shouldn't be dealing with. Now, people who own the jobs get the alerts and they can figure out if there is a problem with the application that they need to work with or if it is something with Tidal. Then, if necessary, they can elevate it up to me. Fortunately, that doesn't happen as much anymore, which makes me very happy. It gives us the alerts in time so we can handle things ideally before they become critical, and hopefully, we're doing our jobs so the right people are contacted.