As a Splunk engineer, I collect data from various sources, including Universal Forwarder, Heavy Forwarder, DB Connect, and Syslog to monitor application logs. This data is used to create dashboards that visualize application health and identify potential security incidents. Additionally, I configure alerts to notify teams via Slack or email when CPU or memory usage reaches critical thresholds, allowing for prompt resolution. Furthermore, I use Splunk to create KPIs and NDDs for various aspects of the organization, including a custom ITSI service for Microsoft 365. This service monitors child entities like Teams, Outlook, and Edge within a parent application, tracking metrics like team member logins and meetings, CPU usage, and memory usage. All this information is consolidated into a three-page ITSI report.
Splunk Enterprise Security helps us detect malicious activity, such as failed login attempts by unauthorized users. These attempts, whether brute-force attacks or phishing attempts, trigger alerts with detailed information about the incident mapped to the MITRE ATT&CK framework. This allows our security team to investigate and take appropriate action quickly.
We managed Splunk's large clustered environments, I oversaw data collection from roughly 750 applications via universal deployment clients. This experience, coupled with my nearly six years of Splunk expertise, made monitoring application logs and creating Splunk knowledge bases straightforward tasks. While processing task cut-off tickets from the application team could be time-consuming, the actual monitoring itself was easy to manage.
The end-to-end visibility provided by Splunk is important because our company uses applications like K-Connect and Splunk to monitor user activity across different sectors. Having previously worked in both healthcare and finance, I'm familiar with how this process works. We access user information including personal data to track their activity from start to finish within our systems. Splunk allows us to mark specific user data points for further analysis, ensuring we have a full view of user or patient activity within each organization we serve.
Splunk helps me find security events across multi-cloud and on-prem platforms. I would identify missing data by checking the last hour's timeframe (span=1h). If on-prem or cloud data was missing, I'd investigate which logs weren't being ingested, whether an indexer was down, or if a forwarder wasn't sending data. Additionally, I'd check if the application or event log volume was overwhelming the universal forwarder, requiring a queue to process the data effectively.
Splunk improves our ability to handle data from applications. This data is often unstructured or unavailable in a usable format. To make it usable, we used to normalize the logs manually through back-end commands and edit various Splunk consoles and platforms. This process transformed the data into a structured, human-readable event format, allowing us to extract the information we needed.
We can identify potential malicious activity through Splunk by analyzing database logs with SQL queries. For instance, a high number of failed login attempts within a short timeframe could indicate unauthorized access attempts. Additionally, with multi-factor authentication systems like Duo, a user logging in from two geographically distant countries within a short period might be suspicious. To address this, I've developed SQL queries that check for logins within a one-hour timeframe across different countries. These queries trigger alerts on a dashboard, allowing IT to investigate the user's IP address and determine if the login is legitimate.
Splunk has significantly improved our business resilience by providing a single pane of view for all our data. This visualization allows us to monitor for anomalies, including unusual application activity, unauthorized executables, and suspicious shell scripts running on both Linux and Windows servers. By triggering alerts for these events, Splunk empowers our organization to proactively identify and address potential threats, ultimately improving overall stability.
Splunk allows us to easily check the data for malicious activity. It also helps reduce the alert volume by allowing us to set thresholds for alerts. For example, we only receive an alert when the CPU usage exceeds 90 percent or the number of failed logs is more than 15.
Splunk helps us investigate by providing relevant context from system logs. We can search the Splunk logs for specific applications and timeframes, and then examine all the data fields for suspicious activity, failed login attempts, or any other anomalies.
It helps security teams investigate threats faster by providing a central platform to collect and analyze data from various security applications. This focus on enterprise security allows teams to identify and respond to threats across the organization, leveraging frameworks like MITRE ATT&CK to match attacker techniques and tactics.
Splunk's strength lies in its single-page view. This interface allows us to explore all our data, build dashboards with alerts, and visualize real-time information through various charts like column, bar, and pie formats, providing a full user experience.
Due to its high licensing cost, Splunk is out of reach for many organizations. Making their licensing more affordable would open up Splunk's solution to a wider range of users.
I have been using Splunk Enterprise Security for six years.
Splunk Enterprise Security is a stable solution.
Splunk Enterprise Security has excellent scalability.
The technical support is good.
The deployment is complicated because our organization works with on-prem servers. All the data needs to be duplicated and all the searches and indexes need to happen properly.
The Splunk licensing is high.
While more affordable, alternative SIEM solutions lack the flexibility and in-depth visualization capabilities offered by Splunk.
I would rate Splunk Enterprise Security nine out of ten.
While Splunk Enterprise Security offers a user-friendly interface, its true power lies in its ability to create highly customized dashboards that streamline investigations and reporting.