Python27

README.md

@@ -1,43 +1,20 @@
-[![Gitter chat](https://img.shields.io/badge/gitter-join%20chat-brightgreen.svg)](https://gitter.im/CiscoSecurity/Lobby "Gitter chat")
+https://developer.cisco.com/site/license/cisco-sample-code-license/
 
-# License
+Cisco Sample Code License, Version 1.1
 
-Copyright (c) 2017 by Cisco Systems, Inc.
+These terms govern this Cisco Systems, Inc. (“Cisco”), example or demo source code and its associated documentation (together, the “Sample Code”). By downloading, copying, modifying, compiling, or redistributing the Sample Code, you accept and agree to be bound by the following terms and conditions (the “License”). If you are accepting the License on behalf of an entity, you represent that you have the authority to do so (either you or the entity, “you”). Sample Code is not supported by Cisco TAC and is not tested for quality or performance. This is your only license to the Sample Code and all rights not expressly granted are reserved.
 
-[Cisco EULA](http://www.cisco.com/c/en/us/about/legal/cloud-and-software/software-terms.html)
+1. LICENSE GRANT: Subject to the terms and conditions of this License, Cisco hereby grants to you a perpetual, worldwide, non-exclusive, non-transferable, non-sublicensable, royalty-free license to copy and modify the Sample Code in source code form, and compile and redistribute the Sample Code in binary/object code or other executable forms, in whole or in part, solely for use with Cisco products and services. For interpreted languages like Java and Python, the executable form of the software may include source code and compilation is not required.
 
-    ALL RIGHTS RESERVED. THESE SOURCE FILES ARE THE SOLE PROPERTY
-    OF CISCO SYSTEMS, Inc. AND CONTAIN CONFIDENTIAL  AND PROPRIETARY
-    INFORMATION.  REPRODUCTION OR DUPLICATION BY ANY MEANS OF ANY
-    PORTION OF THIS SOFTWARE WITHOUT PRIOR WRITTEN CONSENT OF
-    CISCO SYSTEMS, Inc. IS STRICTLY PROHIBITED.
+2. CONDITIONS: You shall not use the Sample Code independent of, or to replicate or compete with, a Cisco product or service. Cisco products and services are licensed under their own separate terms and you shall not use the Sample Code in any way that violates or is inconsistent with those terms (for more information, please visit: www.cisco.com/go/terms ).
 
-# eStreamer eNcore
-The Cisco eStreamer client. 
+3. OWNERSHIP: Cisco retains sole and exclusive ownership of the Sample Code, including all intellectual property rights therein, except with respect to any third-party material that may be used in or by the Sample Code. Any such third-party material is licensed under its own separate terms (such as an open source license) and all use must be in full accordance with the applicable license. This License does not grant you permission to use any trade names, trademarks, service marks, or product names of Cisco. If you provide any feedback to Cisco regarding the Sample Code, you agree that Cisco, its partners, and its customers shall be free to use and incorporate such feedback into the Sample Code, and Cisco products and services, for any purpose, and without restriction, payment, or additional consideration of any kind. If you initiate or participate in any litigation against Cisco, its partners, or its customers (including cross-claims and counter-claims) alleging that the Sample Code and/or its use infringe any patent, copyright, or other intellectual property right, then all rights granted to you under this License shall terminate immediately without notice.
 
-The Cisco Event Streamer (also known as eStreamer) allows you to stream System intrusion,
-discovery, and connection data from Firepower Management Center or managed device (also
-referred to as the eStreamer server) to external client applications.
+4. LIMITATION OF LIABILITY: CISCO SHALL HAVE NO LIABILITY IN CONNECTION WITH OR RELATING TO THIS LICENSE OR USE OF THE SAMPLE CODE, FOR DAMAGES OF ANY KIND, INCLUDING BUT NOT LIMITED TO DIRECT, INCIDENTAL, AND CONSEQUENTIAL DAMAGES, OR FOR ANY LOSS OF USE, DATA, INFORMATION, PROFITS, BUSINESS, OR GOODWILL, HOWEVER CAUSED, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
 
-eStreamer responds to client requests with terse, compact, binary encoded messages – this
-keeps it fast.
+5. DISCLAIMER OF WARRANTY: SAMPLE CODE IS INTENDED FOR EXAMPLE PURPOSES ONLY AND IS PROVIDED BY CISCO “AS IS” WITH ALL FAULTS AND WITHOUT WARRANTY OR SUPPORT OF ANY KIND. TO THE MAXIMUM EXTENT PERMITTED BY LAW, ALL EXPRESS AND IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTY OR CONDITION OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, SATISFACTORY QUALITY, NON-INTERFERENCE, AND ACCURACY, ARE HEREBY EXCLUDED AND EXPRESSLY DISCLAIMED BY CISCO. CISCO DOES NOT WARRANT THAT THE SAMPLE CODE IS SUITABLE FOR PRODUCTION OR COMMERCIAL USE, WILL OPERATE PROPERLY, IS ACCURATE OR COMPLETE, OR IS WITHOUT ERROR OR DEFECT.
 
-eNcore is a new all-purpose client which requests all possible events from eStreamer, parses
-the binary content and outputs events in various formats to support other SIEMs.
+6. GENERAL: This License shall be governed by and interpreted in accordance with the laws of the State of California, excluding its conflict of laws provisions. You agree to comply with all applicable United States export laws, rules, and regulations. If any provision of this License is judged illegal, invalid, or otherwise unenforceable, that provision shall be severed and the rest of the License shall remain in full force and effect. No failure by Cisco to enforce any of its rights related to the Sample Code or to a breach of this License in a particular situation will act as a waiver of such rights. In the event of any inconsistencies with any other terms, this License shall take precedence.
 
-# Support
-This is a beta version of eNcore. Before the General Availability release this will be
-updated with details of paying for and receiving support.
+Last updated January 26, 2018
 
-# Quick install
-* Download the release: eStreamer-eNcore-X.YY.tar.gz
-* Navigate to the directory you want to contain eStreamer eNcore
-* `tar -xf eStreamer-eNcore-X.YY.tar.gz`
-* `cd eStreamer-eNcore`
-* Run eNcore: `./encore.sh`
-* Run a connectivity test: `./encore.sh test` (and enter the pkcs12 password)
-* View the log output `tail -f estreamer.log`
-* `./encore.sh foreground` - run in the foreground
-* `./encore.sh start` - starts a background task
-* `./encore.sh stop` - this will stop the background task
-* `./encore.sh restart` - this will restart the background task

encore.sh

@@ -194,6 +194,11 @@ stop() {
     fi
 }
 
+clean() {
+    # Delete data older than 12 hours -> 720mins
+    find ../../data -type f -mmin +720 -delete
+}
+
 restart() {
     stop
     start
@@ -217,6 +222,10 @@ main() {
             diagnostics
             ;;
 
+        clean)
+            clean
+            ;;
+
         foreground)
             foreground
             ;;

estreamer/__init__.py

@@ -34,4 +34,4 @@
 from estreamer.configure import Configure
 from estreamer.pidfile import PidFile
 from estreamer.controller import Controller
-__version__ = '3.5.3'
+__version__ = '3.8.4'

estreamer/adapters/binary.py

@@ -31,6 +31,7 @@
 from estreamer import ParsingException
 
 from estreamer.definitions import TYPE_BYTE
+from estreamer.definitions import TYPE_UINT8
 from estreamer.definitions import TYPE_UINT16
 from estreamer.definitions import TYPE_UINT32
 from estreamer.definitions import TYPE_UINT64
@@ -141,15 +142,17 @@ def _parseDiscoveryHeader( self, data, offset, record ):
          eventSubtype, ignore2, ignore3 ) = Binary.unpackDiscovery(
              data[ offset : offset + ( 4 * 10 ) ] )
 
+        recordLength = len( data )
+        recordType =  record[ 'recordType' ]
         record[ 'deviceId' ] = deviceId
-        record[ 'legacyIpAddress' ] = self._ip2str(
-            socket.AF_INET,
-            data[ offset + 4 : offset + 8])
+
+        headerLength = int( record[ 'recordLength' ] )
+
+        record[ 'hostIpAddr'] = self._ip2str( socket.AF_INET6, data[56:72] )
 
         record[ 'macAddress' ] = Binary._formatMacAddress(
             mac1, mac2, mac3, mac4, mac5, mac6 )
 
-        record[ 'hasIpv6' ] = hasIpv6
         record[ 'eventSecond' ] = eventSecond
         record[ 'eventMicrosecond' ] = eventMicrosecond
         record[ 'eventType' ] = eventType
@@ -158,8 +161,6 @@ def _parseDiscoveryHeader( self, data, offset, record ):
         offset += 40
 
         if hasIpv6 == 1:
-            ipv6 = self._ip2str( socket.AF_INET6, data[40:56] )
-            record[ 'ipv6Address' ] = ipv6
             offset += 16
 
         return offset
@@ -198,6 +199,9 @@ def _parseBlock( self, data, offset, attribute, context ):
             elif isinstance( attribute['block'], int ):
                 blockKey = attribute['block']
 
+            if self.logger.isEnabledFor( logging.TRACE ):
+                self.logger.log( logging.TRACE, "_parseBlock offset={0}:attribute={1}:blockKey={2}".format(offset,attribute, blockKey) )
+
         elif 'list' in attribute:
             if attribute['list'] == BLOCK_AUTO:
                 ( blockKey, ) = Binary.unpackUint32( data[ offset : ( offset + 4 ) ] )
@@ -217,6 +221,9 @@ def _parseVariable( self, data, offset, attribute, context ):
         blockLength = context[ lengthSource ]
         attributeName = attribute[ 'name' ]
 
+        if self.logger.isEnabledFor( logging.TRACE ):
+            self.logger.log (logging.TRACE, '_parseVariable - offset={0}:attributre={1}:context={2}:data={3}'.format(offset,attribute,context,data) )
+
         if 'adjustment' in attribute:
             lengthAdjustment = attribute[ 'adjustment' ]
 
@@ -260,6 +267,10 @@ def _parseAttributes( self, data, offset, attributes, context ):
         recordLength = len( data )
 
         for attribute in attributes:
+
+            if self.logger.isEnabledFor( logging.TRACE ):
+                self.logger.log( logging.TRACE, "_parseAttributes RecordType={0}:offset={0}:attribute={1}".format(recordType, offset, attribute) )
+
             attributeName = attribute[ 'name' ] if 'name' in attribute else None
 
             if self.logger.isEnabledFor( logging.TRACE ):
@@ -344,7 +355,8 @@ def _parseAttributes( self, data, offset, attributes, context ):
                 else:
                     if attributeType == TYPE_BYTE:
                         byteLength = 1
-
+                    elif attributeType == TYPE_UINT8:
+                        byteLength = 1
                     elif attributeType == TYPE_UINT16:
                         byteLength = 2
 
@@ -356,12 +368,40 @@ def _parseAttributes( self, data, offset, attributes, context ):
 
                     else:
                         raise ParsingException( 'Unknown type: {0}'.format( attributeType ) )
+
+                    if recordType == 98 :
+                        maxLen = len(data)
 
-                    context[ attributeName ] = struct.unpack(
-                        '>' + attributeType,
-                        data[ offset : offset + byteLength ] )[ 0 ]
+                        context['id'] = struct.unpack('>'+TYPE_UINT32, data[16:20])[0]
+                        context['protocol'] = struct.unpack('>'+TYPE_UINT32, data[20:24])[0]
 
-                    offset += byteLength
+                        #24-28  Type always 0
+                        recLenBytes = struct.unpack('>'+TYPE_UINT32, data[28:32])[0]
+                        nameLength = int( recLenBytes - 8 )  # 24 - 8
+                        maxLength = int(nameLength + 32)
+                        name = struct.unpack('>'+str(nameLength)+'s',data[32: maxLength])[0]
+                        self.logger.log ( logging.TRACE, 'username(len): {0}|size: {1}|data: {1}'.format (recLenBytes, nameLength, data[32: maxLength])  )
+
+                        context['username'] = name.decode('utf-8')
+                        if (len(name) > 0 ) :
+                            username = str(context['username'])
+                            context['username'] = username.rstrip(username[-1])
+
+                        self.logger.log( logging.TRACE, 'username : {0}'.format(name) )
+                        offset = maxLength
+
+                    else:     
+                        try:
+                            self.logger.log( logging.TRACE, 'unpacking binary data {0}'.format(attributeName) )
+                            context[ attributeName ] = struct.unpack(
+                                 '>' + attributeType, data[ offset : offset + byteLength ] )[ 0 ]
+                            offset += byteLength
+
+                        except struct.error:
+                            hData = binascii.hexlify( data[ offset: offset + byteLength ] )
+                            hexData = binascii.hexlify( data )
+
+                            raise ParsingException('Error Decoding binary for rec_type={0} attr={1} type={2} data={3} data_full={4}'.format( recordType, attributeName, attributeType, hData, hexData ) )
 
             elif 'list' in attribute:
                 oldOffset = offset
@@ -392,6 +432,8 @@ def _parseAttributes( self, data, offset, attributes, context ):
                     block = context[ attributeName ]
 
                 offset = self._parseBlock( data, offset, attribute, block )
+                if self.logger.isEnabledFor( logging.TRACE ):
+                    self.logger.log( logging.TRACE, 'Parsing Attribute (Block Type) :-: attr name={0}:attr={1}:offset={2}:block={3}:value={5}:offset={6}'.format(attributeName, attribute, offset, block, block, offset) )
 
         return offset
 
@@ -400,17 +442,18 @@ def _parseAttributes( self, data, offset, attributes, context ):
     def _parse( self, data, offset, record ):
         recordType = record[ 'recordType' ]
         recordLength = len( data )
-        try:
-            #if recordType == 95:
+
+        if recordType == 62 :
+            record['length'] = struct.unpack('>L', data[ 4 : 8 ] )[0]
+            record['id'] = struct.unpack('>L', data[ 8 : 12 ] )[0]
+            record['name'] = data[12:recordLength].decode('utf-8')
+
+            offset = recordLength
+
+        else :
             attributes = RECORDS[ recordType ][ 'attributes' ]
             offset = self._parseAttributes( data, offset, attributes, record )
 
-        except (AttributeError, ValueError) as ex:
-            hexRecord = binascii.hexlify( data )
-            self.logger.error( 'AttributeError: Record={0}'.format( hexRecord ) )
-            self.logger.exception(ex)
-            return
-
         if offset != recordLength:
             msg = '__parse(): Offset ({0}) != recordLength ({1}) for recordType {2}'.format(
                 offset,
@@ -500,6 +543,7 @@ def parse( self ):
         eStreamer and loads it into a common dict format which is used
         everywhere else
         """
+
         if not self.isParsed:
             if self.recordType not in RECORDS:
                 self.logger.warning( '__decode(): Unknown record type {0}.'.format(

estreamer/adapters/cef.py

@@ -20,6 +20,8 @@
 import copy
 import time
 import socket
+import estreamer.adapters.pretty
+import estreamer.crossprocesslogging as logging
 import estreamer
 import estreamer.adapters.kvpair
 import estreamer.definitions as definitions
@@ -303,6 +305,47 @@ def __packetData( data ):
         },
     },
 
+    definitions.RECORD_INTRUSION_EXTRA_DATA: {
+        'sig_id': lambda rec: 'INTRUSION_EXTRA:110:1',
+
+        'name': lambda rec: 'INTRUSION EXTRA DATA',
+
+        'severity': lambda rec: 7,
+
+        'constants': {
+            'cs1Label': 'eventId',
+        },
+
+        'lambdas': {
+            'rt': lambda rec: rec['eventSecond'] * 1000,
+            #'start': lambda rec: rec['packetSecond'] * 1000,
+            'deviceExternalId': lambda rec: rec['deviceId']
+#            'cs1': lambda rec: __packetData( rec['eventId'] )
+        },
+
+       'fields': {
+            'deviceId': 'dvchost',
+            'eventSecond': 'eventSecond',
+            'eventId': 'eventId',
+            'externalId': 'externalId',
+            'blob.blockLength': '',
+            'blob.blockType': '',
+            'checksum': '',
+            'recordLength': '',
+            'archiveTimestamp':'',
+            'blob.data': 'data',
+            'blockLength': '',
+            'blockType': '',
+            'type': 'type',
+            'blob': 'blob'
+        },
+
+        'viewdata': {
+            View.SENSOR: 'dvchost'
+        },
+
+    },
+
     # 112
     definitions.RECORD_CORRELATION_EVENT: {
         'sig_id': lambda rec: 'PV:112:{0}:{1}'.format(
@@ -498,7 +541,7 @@ def __packetData( data ):
 
     # 400
     definitions.RECORD_INTRUSION_EVENT: {
-        'sig_id': lambda rec: '[{0}:{1}]'.format(
+        'sig_id': lambda rec: 'INTRUSION:400:{0}:{1}'.format(
             rec['generatorId'],
             rec['@computed.renderedId']
         ),
@@ -692,14 +735,13 @@ def __init__( self, source ):
         self.record = estreamer.common.Flatdict( source, True )
         self.output = None
         self.mapping = None
+        self.logger = logging.getLogger( self.__class__.__name__ )
 
         if 'recordType' in self.record:
             if self.record['recordType'] in MAPPING:
-                self.mapping = MAPPING[ self.record['recordType'] ]
+                self.mapping = MAPPING[ self.record['recordType'] ] 
                 self.output = {}
-
-
-
+
     @staticmethod
     def __sanitize( value ):
         """Escapes invalid characters"""
@@ -710,11 +752,10 @@ def __sanitize( value ):
         value = value.replace('\\', '\\\\')
         value = value.replace('"', '\\"')
         value = value.replace(']', '\\]')
+        value = value.replace('|', '\|')
 
         return value
 
-
-
     def __convert( self ):
         """Writes the self.output dictionary"""
 
@@ -762,6 +803,10 @@ def __cefMessage( self ):
         # $hostname =~ s/\.+$//;
         hostname = socket.gethostname()
 
+        #logger
+        self.logger = logging.getLogger( self.__class__.__name__ )
+
+
         # http://search.cpan.org/~dexter/POSIX-strftime-GNU-0.02/lib/POSIX/strftime/GNU.pm
         # # Get syslog-style timestamp: MAR  1 16:23:11
         # my $datetime = strftime('%b %e %T', localtime(time()));
@@ -789,7 +834,7 @@ def __cefMessage( self ):
             CEF_DEV_PRODUCT,
             CEF_DEV_VERSION,
             sigId,
-            name.replace('|','\|'),
+            name,
             severity,
             data,
             SYSLOG_NUMERIC,