UNIT-V: Log Analysis using ArcSight

ArcSight Enterprise Security Manager

• What is ArcSight?
• ArcSight is an ESM platform which stands for Enterprise Security
Manager.
• Tool that is designed and implemented for
• managing the security policies within an organization.
• It is used in
• detecting, analysing, and resolving cyber security related threats within a
short duration of time.
• The ESM platform has the products for
• collecting the events,
• real time event management,
• log management,
• automatic response, and
• compliance management. 2
ArcSight Components
• ArcSight describes the components of the security model consisting of security
monitoring features and functionalities.
• ArcSight resolves the problems of several requirements by
• collecting and storing the data for long term use cases.
➢The features include
➢ storage,
➢ reporting,
➢ searching,
➢ monitoring the use cases and
➢ finding the correlation among the products.
➢It collects the information of log events from
➢ applications, clouds, network environments, endpoints, systems, security products.
➢ArcSight correlates the data and stores the information for compliance use
cases.
➢Generating the automated alerts for the investigation work that is carried out
on the data.
3
ArcSight Components Classification
1. Arcsight SIEM Platform
• The Arcsight SIEM Platform environment includes
• the security and visibility operations which leverage the monitoring platform
infrastructure.
• The platform captures, normalizes and categorizes all the events and logs
from network and security devices.
2. ArcSight ESM
• The ArcSight ESM has the capability of collecting the broad log information
combined with the powerful correlation engine which can
• detect the threats from multiple products and
• alerts the customers to take action on the vulnerabilities.

4
Contd..
3. ArcSight Logger
• The ArcSight Logger provides
• the log management and storage capabilities with automated compliance reporting.
• It can store upto 42TB of log data that
• can search for multiple events per second over structured and unstructured data.
• It supports automated reporting for SOX, PCI DSS, and other regulations.
4. ArcSight Express
• The ArcSight Express includes the technologies of
• real-time correlation and log management from ESM and logger.
• The Express is referred to as
• “security expert in a box” which has
• several built-in correlation rules, dashboards and reports.
• It provides the deployment and low-cost monitoring solutions for the
infrastructure.
5
Contd..
5. ArcSight SmartConnectors
• The ArcSight SmartConnectors collect
• the event data from network devices and
• normalizes the data structure into schema.
• The connectors can
• filter the data,
• save the network bandwidth and
• storage space.
• The SmartConnectors improves the efficiency by
• aggregating the events to reduce the quantity of the same type.
• The events can be categorized into readable format which makes
• it easier for using the events to build the filters, rules and reports.
6
ArcSight Latest Version
• ArcSight ESM version 7.0,
• ArcSight Express version 5.0,
• ArcSight Investigate version 2.20, and
• ArcSight Data Platform version 2.31 (containing ArcSight's Logger,
ArcMC, and Event Broker technology) were all launched in January
2019.
ArcSight ESM Network model
• The ArcSight ESM Network model is the combination of network and
assert models together builds the correlation criteria.
• The network model represents the nodes and characteristics of the network.
• The assert model represents the attributes of asserts.

8
Contd..
• The elements of network model consists the following resources.
• Asserts - It represents the nodes on the network consisting the servers, routers
and devices.
• Assert Ranges - It represents the set of network nodes with block of IP address.
• Zones - It represents the portion of network categorized by block of addresses.
• Networks - It differentiates the two private address spaces.
• Customers - These are the business units associated with the networks.

9
Asserts
• The Asserts resources identifies any network endpoint within an
• IP address, MAC address, host name, external ID.
• An assert resource is the specification of network identity which
includes
• Assert name.
• Network IP address.
• MAC address.
• Host name.
• External ID.

10
Assert Ranges
• An Assert Ranges is the group of asserts attached to a network that
• uses a block of IP address.
• When an event is processed by the SmartConnector, its endpoints are
identified as
• a single asset or
• an asset that belongs to a particular assert range.
• A reference to an assert or assert range identifier is populated in the
event schema.

11
Zones
• Zones usually represent a functional group within a network or a subnet
such as
• LAN, VPN or DMZ identified with the block of IP address.
• Every assert or address range is associated with a zone.
• ESM is already configured with the global IP address which helps in
• resolving without setting up any additional zones.
• The address range in the zones in the same network cannot overlap.
• When SmartConnector processes an event, it evaluates
• each IP address in an event and
• tries to locate the zone associated with the IP address among the ordered list of
networks.
• If a matching zone is found then the search is over, if not it moves with the
order specified in the next network during SmartConnector configuration.
12
Networks
• Networks are ArcSight resources that are used
• to differentiate between the zones when IP ranges overlap.
• Local and Global are two standard networks configured for ESM.
• Network designations will enable the SmartConnector
• to tag the events with the correct zone such that
• the manager can find the correct model for assert events.

13
Customers
• The customers tagging is a feature that
• is developed to support Managed Security Services Provider (MSSP)
environments.
• A customer will be considered as
• an “owner” of an event instead of considering it as a source or
• target of an event.
• The customer variables can be either
• a fixed string or
• a velocity template variable.

14
Event life Cycle in ArcSight
Data
collection
and event
processing
Network
Event model lookup
archival and priority
evaluation

Event
life
Incident
analysis and
Cycle Correlation
evaluation
reporting

Monitoring
Workflow and
investigation

15
Event life Cycle in ArcSight
• There are seven event life cycle in ArcSight ESM
1. Data collection and event processing:
The data is gathered from various sources and then it is
processed.
2. Network model lookup and priority evaluation:
Apply the logical setup of a network with the naming
and structures so as to understand the environment,
location, and then is set for priority evaluation.
3. Correlation evaluation
In this phase, the correlations will be evaluated and
then will move to monitor and investigate.
16
Contd..
4. Monitoring and investigation:
The scenarios have to be properly understood to know
• what it is in order to monitor and
• allowed for investigation from an analyst so as to move to the workflow.
5. Workflow:
In this phase, the workflow process model is implemented.
6. Incident analysis and reporting:
To report the data and provide the analysis for what is obtained
or received.
7. Event archival:
Finally, the events will be archived into an external storage
environment. The data can be stored for an extended period of
time. An event is passed from all these seven stages.
17
What is Correlation and Aggregation in
ArcSight
• Aggregation
• At the SmartConnector level, aggregation limits the number of events consumed by
the destination device (ESM / Logger).
• Suppose a SmartConnector is receiving events from a firewall device, for example.
• In that case, it will aggregate (i.e., summarize) similar circumstances over a defined
period and deliver a single event to the destination.
• This can save you a lot of money in terms of bandwidth, storage, and processing.
• Correlation
• Correlation is a technique for determining the correlations between events.
• ESM's correlation engine, for example, employs the rules you create (or those
provided by ESM) to correlate base and aggregated events coming in from
SmartConnectors to identify if something of interest has occurred.
• For example, a failed login event on an endpoint may not be of interest in and of
itself, but if the same failed login event occurs several times in a short period, it could
indicate a brute force login attempt. This type of action can be monitored by a rule,
which will generate a correlation event that can act.
Event Schema and Lifecycle
• Event Schema
• Event Lifecycle
Event Schema Overview

• -17 Schema Groups • File

• Root • Old File
• Category • Request
• Threat • Original Agent
• Device • Final Device
• Attacker • Event Annotation
• Target • Device Custom
• Agent • Flex
• Source
• Destination
Derived Fields
• A derived field is not set at the connector, but derived based on
another field value.
• Attacker fields are typically derived from the source fields.
• Target fields are typically derived from the Destination field
• Fields request Protocol, request URL Authority, Request URLHost,
request URLPort, request URL FileName and requestURL Query can
be derived from the rquestURL field.
• requestURL format:
<protocol>//<authority>@<host><port>/<filename>?<query>
• Request URL or the derived fields can be set at the connector, but not
both
Attacker or Source / Destination or Target
• Attacker and Target represent the threat direction (Logical)
• Source and destination represent the network traffic flow (Physical)
• There are some occasions where the typical perspective may differ the
physical perspective, requiring ArcSight to adjust the values
• Ex: A user at the HostA clicked on a malicious weblink,. As the result the HostA has
initiated connection with the destination HostB and is communicating with the
destination. So from the network physical perspective the HostA is the source and the
HostB is the destination. However form the security (logical) perspective it is reverse:
HostA is actually a target (victim) and the HostB is the attacker.
• Logger does not include fields for attacker and Target! You will have to use
source and destination for your searches.
• In most cases, attacker=source; target=destination
Fields Processed by the Framework
• i.e. Fields not handled by the parser
• The following field categories are populated by the Connector
Framework
• Agent
• Original Agent
• Category fields are handled by the categorization file built along with
the log parser.
• Category fields are set based on the parsed values of the log message.
Fields Processed by the Manager
• Threat- Field are populated based on calculations made during threat
level formula calculation
• Event Annotation-Fields that can be set by the user or the system and
persisted with event
What Time is It?
• Device Receipt time
• Start Time
• End Time
• Agent Receipt Time
• Manager Receipt Time
Search Basics
• Unstructured Data
• The more exact you can be, the accurate the results will be
• If you enter only an IP address, logger must search every field foe every event
to try and match that address, this is both time and resources consuming.
• Example:
• 10.10.10.10
• Windows
• Unix
Keywords Search (Full-Text Search)
Building a Search
• Structured Data
• Syntax; fieldname operator “property”
• Examples:
• Destination Address=“10.10.10.10”
• Destination Port =“80”
• Name CONTAINS “buffer Overflow”
• When you start typing field name, a dropdown will be open with a choice
of fields
• Depending on the field you are searching, there are several operators
available
• -,=, !=, IS NULL, CONTAINS, STARTSWITH, ENDSWITH, IS NOT NULL, IN, BETWEEN, >,<
• After choosing the field name, press the space bar and a list of operators
for that field will be displayed in drop down.
Field-Based Search
Key Elements of a Search
• Time
• Static v Dynamic Time
• Static = Specific Time Ranges
• Dynamic leverages variables
• Syntax: <current_period> [+/- <units>]
• The <current_period> always starts with a ‘$’ and consists of a word, case sensitive, with
no spaces
• Example
• Last two hours of activity: start:$Now – 2h End:$Now
• Last day of activity: Start: $Now-1d End:$Now
• Last 10 minutes of activity: Start: $Now- 10m End:$Now
Key Elements of a Search
Building a Search
Search Case and Syntax
Search tips and tricks
Operators-Chart
• Displays search results in a chart based on specified fields
• Can Span Time: bucket size for grouping events,. Use d for day, h for
hour, m for minute, s for second
• Can Show Mathematical Function: count, sum, avg, min, max, stdev
• Title, Chart Type and Display Limit can be modified
• Example:
• Logger| chart count by deviceEventCategory name
• Destination Address is not null | chart count by destination Address
Chart Operator
Operator-Sorting
Adding Sort to Our Chart
Operators-DEDUP
Before DEDUP
After DEDUP
Operator Head
Head Example
Operators-Tail
Tail Example
• Tail Example
Operators- Top
Top Example
Operators-Rare
Rare Example
Arc Sight Logger
• ArcSight Logger is a log management solution that
• optimized for extremely high event throughput,
• efficient long-term storage, and rapid data analysis.
• An event is a time-stamped log entry, such as
• a syslog message sent by a host, or
• a line appended to a log file.
• Logger
• receives and stores events;
• supports
• search,
• retrieval, and reporting; and
• forward selected events for correlation and
• analysis to destinations such as
• a syslog server
65
Arc Sight Logger-Contd..
• HP ArcSight Logger delivers
• industry-leading, cost-effective log management
solution that
• unifies searching,
• reporting,
• alerting, and
• analysis across any type of enterprise machine data for
• IT GRC, IT Operations, SIEM, and log analytics.

66
How Logger Works?
• Logger stores time-stamped log entries and
• called events at high sustained-input rates.
• Logger compresses raw data, but can always
• retrieve unmodified data on demand, for forensics quality litigation data.
• Logger can receive data in the form of normalized CEF events from
• ArcSight SmartConnectors,
• Syslog messages, and
• Log files directly from a device.
• Logger can then forward received events to a syslog server or ArcSight
ESM.

67
 Contd..
• SmartConnectors are
• the interface between
Logger and devices on your
network that
• Generate events you want to
store on Logger.
• SmartConnectors
• collect event data and
• normalize it into a Common
Event Format (CEF).
68
Security Event Processing inside the Connector
Processing
Normalization
Sanity Verification
Map File Processing
Categorization
Categorization (Contd..)
User Name Splitting
Name Resolution
Name Resolution
Customer and Zone Population-Customer
Customer and Zone Population-Zone
Scanner Event Processing
Connector Filtering
Time Correction
Field Based Aggregation
Other Important Components
Protection against Denial of Service
91
Contd..
• Once events stored on a Logger, do the following:
• Search for events that match a specific query.
• Generate reports of events of interest.
• Generate alerts when a specified number of matches occur within
a given time threshold.
• Alerts can notify you by e-mail, an SNMP trap, or a Syslog message.
• Establish dashboards that display events that match a specific
query.
• Forward selected events to ArcSight ESM for correlation and
analysis.
• Forward events to TH or other tools.
92
The need for a universal log management
solution
• Logs provide an audit trail that can be analyzed to detect
and conduct
• detailed forensic analyses of cyber attacks,
• streamline regulatory audits,
• assist in application development, and improve IT service levels.

93
HP ArcSight Logger features for downloadable
and enterprise versions

94
ESM- Started
• ESM is a comprehensive software solution that combines traditional
security event monitoring with
• network intelligence,
• context correlation,
• anomaly detection,
• historical analysis tools, and
• Automated remediation.

95
Starting the ArcSight Console
• Start the Console:
• Depending on the chosen shortcuts during installation, start the
Console using any of these methods:
• Using the Console desktop icon
• Selecting from the system tray
• Selecting from the Start menu Alternatively, open a command window in the
Console’s bin directory and type arcsight console

96
Log in
• The login mechanism varies according to the type of authentication
you have set up during Console installation.
• If you are using SSL authentication, set it up and import the
certificate .
• After the certificate is imported, you can start the Console
without entering a user ID or password.
• If you are using password authentication, log in with your user ID
and password.
• Certificates are imported automatically.
• If you have selected “Password or SSL Authentication,” you
choose which way to log in, each time.

97
Working in the Console
• Navigating
• Use the Navigator panel on the Console
• to locate and manage security resources, and
• the Viewer and Inspect/Edit panels to analyze resource data and
• view or adjust the attributes of the resources producing the data.

98
 The Navigator panel showing the Dashboards
resource tree
The resources available in the Navigator panel
can be affected by permissions set for your
user type. On the Navigator panel, you can:
• Choose a group or a specific resource from
the resource tree.
• Expand (+) and collapse (-) resource groups
to locate particular subgroups or individual
resources. You can also use the keyboard
right arrow key to expand and left arrow key
to collapse the Navigator resource trees.
• Right-click groups or individual resources to
choose from their context menus.
99
Navigator
Panel
Resource
Tree
Batch Editing
• Batch-Editing Cases or Connectors
• Where:
• Navigator > Resources > Connectors, or
• Navigator > Resources > Cases
• To batch-edit cases or connectors:
• 1. Ctrl+click or Shift+click to select a set of individual cases or
SmartConnectors in their respective resource trees.
• 2. Right-click the selected items and choose Edit.
• 3. Make changes to the appropriate common fields, such as Description or
Owner.
• 4. Click Apply to record your changes and leave the editor open, or click OK to
save and close.
• Saving affects only the fields you have changed, in each of the selected resources.
101
Contd..
• Locking Case Groups
• Use the Lock Case check box to lock and unlock cases in batches. See "Viewing
Group Cases in a Grid View" on page 620.
• Note: If a rule action is configured to update a case, and the case is locked at the
time the rule
• triggers, then the case will not be updated. See "Applying Rule Actions on Cases"
on page 517. SmartConnector Reminders
• Batch changes affect only default configurations, not alternates. However, you
can add new alternate configurations by batch editing.
• Note that if you make changes under the Filters tab, the entire tab's contents are
saved to the selected SmartConnectors.
• You can batch-edit connectors only of the same version.
Reconnecting to the Manager
• If your Console loses its connection to the Manager, a dialog popup
enables you to
• Retry the connection,
• Relogin, or to Cancel the connection.
• Try these options in this order.
• A connection to the Manager cannot be re-established
• if the Manager is restarted or
• if a network problem prevents communication with the same Manager.
• In such cases,
• click Cancel and start the Console again,
• using an appropriate Manager host name.
103
Changing the Console Display

104
Contd..

105
Changing User Preferences
• Preferences dialog box through the Edit>Preferences menu command.
• Changing Your Password
• Administrators create users and assign passwords.
• After logging in with your administrator-created password,
• you must change it for security reasons.
• Where: Edit > Preferences > Password
• 1. Enter your old password, new password, and confirm the new password.
• 2. Click OK.
• By default, passwords require
• a minimum of 6 characters, can contain a maximum of 20 characters, and
• contain numbers and/or letters.
• Ask your system administrator about any special requirements for your
site.
106
Viewing
• The Viewer Panel: You see the results of security-event analyses in the Viewer
panel, which can display several different types of views.
• Although there are some views that display information about resources,
most views are active channels, which are continuously evaluated collections
of security-event data
• Tip: Here are some Viewer Panel features you can use.
• To show a resource (like a particular dashboard or active channel) in the viewer, right-click it in the
Navigator tree and select Show <resource>.
• To close individual views quickly, Shift+click their name tabs. (You can also right-click a view’s name
tab and select Close from the popup menu.)
• To float the Viewer panel, click the Float icon at the top left of the Viewer.
• The Viewer tabs in the Viewer panel have a live link at the top. You can click
these links to open the contents in an external, fully functional browser
window.
 Inspect/Edit Features and Utilities
In the Inspect/Edit panel, you can:
l Choose Window > Inspect/Edit
Panel to open or restore the panel, if
it already has inspectors or editors in
it. If no inspectors or editors are open,
the panel does not display anything.
• l If no editors or inspectors are
open, or to work with different
ones, double-click an event in a
grid
• view or right-click an item in a
Navigator panel resource tree and
choose Show <resource>.
• l To clear an editor from the
Inspect/Edit panel, right-click its
tab and choose Close.
Click the Field Set Selector dropdown menu (defaults to Event
Inspector at Console startup) to use your field set of interest.

Note: If you have not exited the Console for a day or more,
you may notice that the Field Set
Selector no longer displays a list of available field sets. If this
happens, right-click on any field
under the Name column and choose Select a Field Set. The
dropdown works with newly-started
Consoles. As a good practice, exit the Console if you are done
using it.
l Click the Hide Empty Rows button ( ) to see only populated

fields.
l Click the New Field Set button ( ) to create a new field set.

l Click the icon toggle button ( ) to show/hide icons next to

each field entry.