Here’s the short version: security is a big data problem because enterprises (and plenty of smaller firms) ingest enormous, messy, fast-moving streams of logs and alerts, and the hard work is separating a real incident from background noise before it is too late. When IT people say that, they are not being dramatic. They are describing a queue that never empties.
Every device, app, and user interaction adds another stream. SIEM logs, support tickets, endpoint alerts, cloud audit events: the river only gets wider as you add SaaS and remote work. Teams who live in Splunk, Microsoft Sentinel, or CrowdStrike know the feeling. The product names change. The pile does not.
This is the operations cousin of our unified cybersecurity platform post. That page is the hub. This page is why the hub has to swallow an ocean. Secure Techies is an MSP, not a data-platform vendor. We will tell you which streams to keep. We will not invent a “millions of alerts a day” number for a shop we have not measured.
Why security is a big data problem: the four V’s
Classic big-data talk uses volume, variety, and velocity. Security adds a fourth that analysts feel in their bones: veracity. Not every signal is true, complete, or worth a page.
Volume. Laptops, servers, firewalls, identity providers, email, SaaS admin centers, and backup jobs all emit events. NIST’s Guide to Computer Security Log Management (SP 800-92) is old and still right: you need an infrastructure and a process, not a folder of syslog. If you keep everything “just in case,” you pay to store noise and you search slower when it matters.
A 40-person professional office in the Valley still sees enough Defender and sign-in noise to miss a real forwarding rule if nobody owns the queue. Large enterprises do see millions of events. That scale is real for them. It is not a number we will paste onto every client.
Variety. Data arrives structured (JSON sign-in logs), semi-structured (syslog), unstructured (ticket comments, email bodies), and in vendor-specific schemas that refuse to match. Endpoint events do not look like Entra logs. A camera NVR does not look like either. Correlation is a translation job.
Velocity. New logs arrive every second. Cloud and remote work sped that up: a sign-in from a phone on LTE is an event whether the office is open or not. Detection that waits until Monday is not detection. 24/7 coverage is how velocity becomes a staffing question.
Veracity. Not all signals are clear or trustworthy. False positives, incomplete telemetry, clocks that drift, agents that went quiet, and “informational” alerts that are actually the start of ransomware. Separating real threats from noise is the work. Alert fatigue is what happens when that work is understaffed.
Retention is not the same as ingest
Keeping a log for 180 days is not the same as alerting on it. Microsoft Purview’s audit solutions make that split obvious: Audit (Standard) retains searchable Microsoft 365 activity for 180 days so you can investigate; that is not a SIEM, and it is not a pager. Mix the two jobs and you either drown in pages or you have a nice archive nobody searches until the insurer asks.
NIST’s Cybersecurity Framework 2.0 puts Detect next to Govern, Identify, Protect, Respond, and Recover. Collection without a detect use-case is storage. Collection without a named responder is a report.
Why the pile keeps growing
You added EDR. Good. You added cloud. Good. You added a second SaaS CRM. Each one is a new schema and a new default of “alert on everything.” Verizon’s DBIR keeps showing attackers using both people (phishing, stolen credentials) and systems (vulnerabilities, ransomware). Defenders respond by turning on more sensors. Sensors without a diet plan become the second incident.
Identity is a high-value stream: who signed in, from where, with MFA or not. Email is a high-value stream: payload, headers, impersonation. Endpoint is a high-value stream: process trees, ransomware behavior. Firewall denies are usually a low-value stream at small scale. If you ingest denies first because they are easy, you taught the SIEM to cry wolf.
IBM’s Cost of a Data Breach work keeps tying shorter detection and containment to lower average cost. That is an industry research number, not a quote for your office. It is still why drowning in alerts is not a vanity metric. Missed time is the expensive part.
Remote work added another firehose. Home routers, personal phones in Outlook, and after-hours sign-ins from travel all look slightly wrong until you have a baseline. Without that baseline, every new country is an incident and every real travel day is noise.
What this looks like in tools people actually run
SIEM (Splunk, Sentinel, and peers) exists because logs had nowhere to live together. The core job is still the same: aggregate, correlate, alert, report. Correlation is the only reason to pay the ingest bill. If you turned the SIEM on and never wrote a correlation rule that matches an incident you have actually had, you bought a search engine.
EDR / XDR (CrowdStrike and others, or Microsoft Defender) is dense, high-value endpoint data. CrowdStrike’s own EDR explainer is blunt: the product records endpoint behavior so you can see incidents that prevention missed, then contain the host. Our CrowdStrike outage write-up is a reminder that the agent is also an operations dependency. Big data includes the day the sensor itself fails.
Tickets are unstructured and often the only record that a human looked. If the SIEM and the help desk never meet, you have two histories of the same Tuesday. Business email compromise cases almost always have a ticket trail if anyone opened one. If they did not, the SIEM is guessing.
Cloud logs (Azure Activity, AWS CloudTrail, Google Cloud Audit) are easy to forget until someone asks who made a bucket public. They are high volume. Filter them or they will own the budget.
CISA’s Logging Made Easy is a no-cost log-management option aimed at smaller shops that otherwise have little detection. It is not a replacement for a named owner. It is proof that “we cannot afford logs” is often a process problem, not only a license problem.
High-value streams: identity, email, endpoint
If you can only afford three sources this quarter, pick the three that show up in real incidents we still see in Southern California offices.
- Identity. Entra sign-ins, MFA results, Conditional Access failures, new inbox rules created via Graph, consent grants. This is how ransomware crews and BEC crews both start: a valid login.
- Email. Payload, impersonation, forwarding, and the moment a user reported (or did not report) the message. Mail is still the front door.
- Endpoint. Process trees, ransomware behavior, USB, and “the agent went quiet.” EDR is the DVR on the laptop.
Everything else (printer syslog, verbose firewall denies, info-level SaaS) waits until those three land in one searchable place and someone tunes them weekly. That is the diet. It is also how you avoid paying ingest prices for noise you will never query.
CISA’s Known Exploited Vulnerabilities catalog is a small, high-signal list. Use it to prioritize patches. That is big-data hygiene: a short list that beats a long one.
Alert fatigue is the human failure mode
Security analysts find themselves awash in alerts, struggling to sift actionable threats from routine noise. That is the client’s sentence and it is accurate. Fatigue is not laziness. It is a rational response to a queue that never pays off. Ask anyone who has waded through a Sentinel, Splunk, or CrowdStrike queue that was never tuned.
What actually reduces it:
- Drop sources that never produce a real case. Be able to name the last time a firewall-deny alert became an incident.
- Correlate before paging. Sign-in plus inbox rule plus EDR is one case, not three pages. That is the same job our SIEM detection and response post describes from the product side.
- Tune weekly. A rule that fired 400 times and never mattered is a bug.
- Name an owner after hours. A beautiful lake of data with nobody swimming at 2 a.m. is a report, not a SOC.
Incident response is the playbook. Endpoint detection is one high-value stream. Unified platforms are how streams meet. None of them replace a person who can say “this one is real.”
Tickets are data too
A closed ticket with “user clicked, we reset MFA” is often more valuable than 10,000 firewall lines. If that note lives only in someone’s head, the next analyst starts from zero. Land security cases in the same help-desk system you already use, with the host, the user, and the last three related events already on the ticket. That is unstructured data with a job: the next human can act.
How a 30-person firm should think about “enterprise” scale
Do not copy the enterprise ingest bill
You do not need a data lake. You need:
- Identity, email, and endpoint logs landing somewhere searchable
- Retention that matches investigations and any legal hold (180 days of Microsoft 365 audit is a start, not a theology)
- A weekly 30-minute tune of whatever still pages
- MDR or managed cybersecurity if you cannot staff nights
Do not copy an enterprise SIEM architecture because a vendor demo used one. Ingest cost will punish you. Start with the three streams above. Add cloud-admin logs when you have a person who will query them. Add firewalls last, and only the accepts and VPN events you can name a use for.
A 30-person firm still has a big-data problem in miniature: too many formats, too fast, not enough truth in each line. The answer is the same as the enterprise answer, scaled down. Diet. Owner. Correlation. Not another dashboard.
A simple pipeline that respects the four V’s
| Stage | Question | Failure mode |
|---|---|---|
| Collect | Do we need this source? | Ingesting because we can |
| Normalize | Can we join this to a user and a host? | Vendor JSON nobody maps |
| Detect | What would make us act at 11 p.m.? | Rules that never fire, or always fire |
| Respond | Who is on the hook? | Shared inbox |
| Review | What did we close as junk last month? | Never tuning |
Microsoft Sentinel’s own docs describe collecting across users, devices, apps, and infrastructure, then using analytics to group alerts into incidents. That grouping is the point. A wall of raw events is not insight.
What “solved” looks like (honestly)
You will not “solve” security big data. You will put a diet and an owner on it.
Solved enough:
- You can answer “show me this user’s last 24 hours” without opening four portals
- High-severity pages a human who is awake
- You can name three detections that paid off last quarter
- You are not paying to store firewall denies you never query
- Tickets and SIEM tell the same story for the same Tuesday
Not solved:
- “We turned on the SIEM” with no tune
- A promise that AI will read every log so you do not have to
- Keeping seven years of everything “for forensics” with no legal driver and no index that works
- Counting “millions of events” as a security program
Enterprises must wrangle not just massive data volume, but also the complexity and speed that come with it. Solving security today means building strategies, automation, and tools that can keep up with big data, so you can catch real risks before they get lost in the flood. That is the client’s closer. It holds if you start with identity, email, and endpoints, and you name who is on the hook at 2 a.m.
If your queue is already a joke and you want a ranked list of what to keep, contact Secure Techies. We will start with identity, email, and endpoints, not with a lake.
Frequently Asked Questions
