diff --git a/src/pages/en/actionable_detection.md b/src/pages/en/actionable_detection.md deleted file mode 100644 index 762434c..0000000 --- a/src/pages/en/actionable_detection.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -title: Detection Engineering -description: Detecting anomalies and load or suspicious activity inside the Kubernetes network. -layout: ../../layouts/MainLayout.astro ---- - -## Detection - -[Kubeshark](https://kubeshark.com) scripting in conjunction with hooks provides mean to programmatically detect suspicious network behaviors. Using helpers can trigger actions as a result and by that reduce the incident response time. - -For example: -```js -function onItemCaptured(data) { - if (data.response.status === 500) { - // Your code goes here - } else if (kfl.match(data, 'request.headers["Authorization"] == r"Token.*" and src.ip != "192.168.49.2"')) { - // Your code goes here - } -} -``` -A few more detection examples: -- Abnormal API throughput -- Suspicious payload matching a regex -- Incoming communication from bad IPs - -## Actions - -Actions are divided to three segments: -- Alerts -- Forensics -- Telemetry - -### Alerts - -You can send a message to Slack, to a console log, to the WebUI or use a webhook to send anything anywhere. - -Alerts can be used to notify that a certain action was completed (e.g. PCAP was generated and upload) or to provide a real-time notification of a programmatically identified network behavior. - -### Forensics - -Forensics generation can be triggered programmatically using hooks and/or Jobs. - -The following forensic types are available: -- Network snapshots in the form of PCAP files -- Name resolution history -- User-generated files - -Forensics can be uploaded to an immutable datastore like AWS S3 or Google Cloud STorage with existing helpers or by using a Webhook. - -### Telemetry - -[Kubeshark](https://kubeshark.com) enables you to send metrics and logs to your favorite telemetry or logs provider and enjoy dashboards and alerts. - -> Read more in the [InfluxDB & Grafana](/en/integrations_influxdb) and [Elasticsearch](/en/integrations_elastic) sections. diff --git a/src/pages/en/agent_create.md b/src/pages/en/agent_create.md deleted file mode 100644 index 91a11e6..0000000 --- a/src/pages/en/agent_create.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -title: Creating Network Agents -description: A guide for creating, editing, and managing custom network agents in Kubeshark, using templates, local scripts, and the GenAI-powered assistant. -layout: ../../layouts/MainLayout.astro ---- - -You can use an existing agent template, extend a template, or build a new one using our Custom GPT. - -Each **Network Agent** has its own logic, defined by a [JavaScript script](/en/automation_scripting), which you can inspect, approve, or customize. - -**Network Agents** observe traffic using [hooks](/en/automation_hooks) and can trigger actions via [helpers](/en/automation_helpers). - -## New Agent Creation - -### Using an Agent Template - -[Kubeshark](https://kubeshark.com) provides fully functional Agent templates. These can be used as-is, customized to suit your needs, or used as inspiration for creating new agents. - -![Automatic activation of the Redact Sensitive Data Agent](/redact_agent.png) - -### Using Custom GPT - -While there are multiple ways to create new Agents, the easiest method involves using our trained [OpenAI-based custom GPT](https://chatgpt.com/g/g-6815c948b00c81918f1157b5a3cc87b2-kubeshark-network-agent). - -> This GPT is not connected to your running instance of [Kubeshark](https://kubeshark.com) and has no awareness of your identity or usage history. It is a pure code-generation assistant actively trained by the [Kubeshark](https://kubeshark.com) team. - -### Persistency - -Any Agent created in the dashboard will persist for the lifetime of the [Kubeshark](https://kubeshark.com) deployment. When [Kubeshark](https://kubeshark.com) is uninstalled, any Agent logic will be erased from memory. - -To ensure Agent logic persistency, save locally in JS files and use the [`kubeshark scripts` command](/en/automation_scripts_cmd) to synchronize those with any running deployment of [Kubeshark](https://kubeshark.com). - -## Post-Creation Agent Logic Modification - -Agent logic scripts can be modified after creation using different methods based on your use case. - -### Via the Dashboard - -#### Manual Editing - -The fastest route to feedback is to change the script directly in the dashboard. The dashboard includes a full-fledged VS Code–like code editor. - -#### Via GenAI Assistant - -You can use our GenAI agent to change the script based on a prompt. The benefit here is that a single prompt can handle complex code changes that are returned ready to execute. - -You can always use the same functionality to fix any code changes you've introduced (just use the prompt: `fix`). - -Simply highlight the code you want to have inspected or changed, and use our inline code assistant. - -![In line code assistant](/code_asst.png) - -### Local File Editing - -If you keep your files locally and use the [`kubeshark scripts` command](/en/automation_scripts_cmd) to synchronize them with your running deployment of [Kubeshark](https://kubeshark.com), any changes to the local file will be reflected in [Kubeshark](https://kubeshark.com). - -### Autonomous Agent Logic Modification - -The Agent can also modify its own logic, assuming you've explicitly enabled it to do so. \ No newline at end of file diff --git a/src/pages/en/agents_run.md b/src/pages/en/agents_run.md deleted file mode 100644 index 88363eb..0000000 --- a/src/pages/en/agents_run.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Activating Network Agents -description: This guide explains how to activate Kubeshark Network Agents automatically or on demand, using templates, scripts, or the Dashboard. -layout: ../../layouts/MainLayout.astro ---- - -**Network Agents** can be activated either on demand or automatically in several ways: - -1. Using Agent templates -2. Using the [`kubeshark scripts` command](/en/automation_scripts_cmd) -3. Via the [Kubeshark](https://kubeshark.com) Dashboard - -## Activating Agents Using Agent Templates - -[Kubeshark](https://kubeshark.com) provides fully functional Agent templates. These can be used as-is to fulfill automation tasks, customized, or used as inspiration to create new agents. - -![Automatic activation of the Redact Sensitive Data Agent](/redact_agent.png) - -### Activating an Agent Template Automatically - -To automatically run one or more Agent templates, add the agent name(s) to the `scripting.active` Helm value. - -For example, to activate the `Redact Sensitive Data` agent each time [Kubeshark](https://kubeshark.com) starts: - -```yaml -scripting: - active: - - Redact Sensitive Data -``` - -### Activating Agents Automatically - -To run agents by default, add their names to the `scripting.active` value. -For example, if the *agent's script* begins with: - -```javascript -// script 1 -``` - -Then add the following to your configuration: - -```yaml -scripting: - active: - - script 1 -``` - -## Running from the UI - -You can also run agents directly from the [Kubeshark](https://kubeshark.com) UI. -**Note:** Scripts created in the UI are not persistent and will be lost when [Kubeshark](https://kubeshark.com) restarts. - -To run a script: - -1. Start a new script in the UI. -2. Paste the *agent's script* code. -3. Activate the agent. \ No newline at end of file diff --git a/src/pages/en/api_endpoints.md b/src/pages/en/api_endpoints.md deleted file mode 100644 index a016bbf..0000000 --- a/src/pages/en/api_endpoints.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Websocket API Endpoints -description: Use a Websocket API endpoint to extract the traffic you need. -layout: ../../layouts/MainLayout.astro -mascot: Hello ---- - -[Kubeshark](https://kubeshark.com) offers a Websocket API endpoint, enabling external applications to access captured data. These endpoints facilitate real-time data retrieval through Websocket connections. - -Two options are available: -1. A single central endpoint via the Hub with access to all nodes -2. Direct node access via the Worker - -## Hub Websocket API Endpoint - -To demonstrate, this example uses [wscat](https://github.com/websockets/wscat), a command-line tool, to establish a WebSocket connection to the Hub. The tool simplifies interactions with WebSocket servers, making it ideal for testing and development purposes. - -```shell -wscat --connect ws://127.0.0.1:8899/api/wsFull -``` -Once open, the connection awaits a KFL (Key Filtering Logic) to start sending traffic. The KFL instructs the Hub on which data to filter and send back. - -Data will continue streaming until the connection is closed. - -Here's an example where we send the KFL: `response.status > 300` once the connection is open. The returning stream will include traffic with response status codes that are greater than 300. - -```yaml -➜ ~ wscat --connect ws://127.0.0.1:8899/api/wsFull -Connected (press CTRL+C to quit) -> response.status > 300 -< {"dst":{"endpointSlice":null,"ip":"169.254.169.254","name":"","namespace":"","pod":null,"port":"80","service":null},"elapsedTime":0,"entryFile":"000000019561_pcap-0_entry.json","error":null,"failed":false,"id":"10.0.41.65:30001/000000019561.pcap-0","index":0,"node":{"ip":"10.0.41.65","name":"ip-10-0-41-65.ec2.internal"},"outgoing":false,"passed":false,"protocol":{"abbr":"HTTP","backgroundColor":"#416CDE","fontSize":12,"foregroundColor":"#ffffff","layer3":"ip","layer4":"tcp","longName":"Hypertext Transfer Protocol -- HTTP/1.1","macro":"http","name":"http","ports":["80","443","8080"],"priority":0,"referen -``` - -Another example for retrieving the content of a certain node: -```yaml -➜ ~ wscat --connect ws://127.0.0.1:8899/api/wsFull -Connected (press CTRL+C to quit) -> response.status > 300 and node.name == "ip-10-0-41-65.ec2.internal" -< {"dst":{"endpointSlice":null,"ip":"169.254.169.254","name":"","namespace":"","pod":null,"port":"80","service":null},"elapsedTime":0,"entryFile":"000000019561_pcap-0_entry.json","error":null,"failed":false,"id":"10.0.41.65:30001/000000019561.pcap-0","index":0,"node":{"ip":"10.0.41.65","name":"ip-10-0-41-65.ec2.internal"},"outgoing":false,"passed":false,"protocol":{"abbr":"HTTP","backgroundColor":"#416CDE","fontSize":12,"foregroundColor":"#ffffff","layer3":"ip","layer4":"tcp","longName":"Hypertext Transfer Protocol -- HTTP/1.1","macro":"http","name":"http","ports":["80","443","8080"],"priority":0,"referen -``` - -Another example for running in a non interactive mode: - -```yaml -➜ ~ wscat --connect ws://127.0.0.1:8899/api/wsFull -x "response.status > 300 and node.name == \"ip-10-0-41-65.ec2.internal\"" -w 1000 -Connected (press CTRL+C to quit) -< {"dst":{"endpointSlice":null,"ip":"169.254.169.254","name":"","namespace":"","pod":null,"port":"80","service":null},"elapsedTime":0,"entryFile":"000000019561_pcap-0_entry.json","error":null,"failed":false,"id":"10.0.41.65:30001/000000019561.pcap-0","index":0,"node":{"ip":"10.0.41.65","name":"ip-10-0-41-65.ec2.internal"},"outgoing":false,"passed":false,"protocol":{"abbr":"HTTP","backgroundColor":"#416CDE","fontSize":12,"foregroundColor":"#ffffff","layer3":"ip","layer4":"tcp","longName":"Hypertext Transfer Protocol -- HTTP/1.1","macro":"http","name":"http","ports":["80","443","8080"],"priority":0,"referen -``` -## Controlling the Returned JSON Structure -TBD \ No newline at end of file diff --git a/src/pages/en/authentication.md b/src/pages/en/authentication.md deleted file mode 100644 index e597bad..0000000 --- a/src/pages/en/authentication.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: Authentication -description: Learn how to enable authentication for Kubeshark. -layout: ../../layouts/MainLayout.astro ---- - -**Kubeshark** provides email and social authentication right out of the box. Authentication is a standalone feature that can be toggled on or off. -![Social and IDP Authentication](/authentication.png) - -## SAML - -To integrate your organization's SAML provider, begin by signing up on the [License Portal](https://console.kubeshark.com/). Afterward, reach out to us on [Slack](https://join.slack.com/t/kubeshark/shared_invite/zt-3jdcdgxdv-1qNkhBh9c6CFoE7bSPkpBQ), use our [contact-us](https://kubeshark.com/contact-us) form or send an email to info@kubeshark.com with the following details: - -1. Company name (tenant name) -2. Approved corporate domains list -3. The email of the tenant admin who registered on the [License Portal](https://console.kubeshark.com/) - -After receiving your details, we'll set you up as a tenant admin and provide you with a link to configure your SAML settings. - -## Configuration - -For those not using SAML, we offer a two-step process: -1. Authentication - which can be performed via email or through social authentication providers (like Google, Microsoft). -2. Authorization - this involves setting configurations in the **Kubeshark** config file. - - -```shell -tap: - auth: - enabled: true - approveddomains: - - kubeshark.com - - some-other-domain.com - approvedemails: - - me@gmail.com - - they@yahoo.com -``` -To enable authentication, set the `tap.auth.enabled` field to `true`. To disable it, set it to `false`. - -In the `approveddomains`, list the domains that should be authorized to view the dashboard. - -In the `approvedemails`, list the emails that should be authorized to view the dashboard if they are outside the `approveddomains` list. diff --git a/src/pages/en/automation_helpers.md b/src/pages/en/automation_helpers.md deleted file mode 100644 index 8de205a..0000000 --- a/src/pages/en/automation_helpers.md +++ /dev/null @@ -1,730 +0,0 @@ ---- -title: Helpers -description: The reference page for the helpers provided in scripting and jobs features of Kubeshark -layout: ../../layouts/MainLayout.astro ---- - -Helpers represents actions supported by the various available integrations. They enable writing custom-logic scripts in conjunctions with hooks that can trigger actions based on a programmatic decision. - -The following script example shows the use of the Slack helper to trigger a Slack alert every time the response status equals 500. - -```js -function onItemCaptured(data) { - if (data.response.status === 500) - vendor.slack( - env.SLACK_WEBHOOK, - "Server-side Error", - JSON.stringify(data), - "#ff0000" - ); -} -``` - -This page contains information about the helpers, objects and overall behavior of the scripting API. - -The same API is available inside the JavaScript functions defined as jobs. - -All of the arguments in function calls are automatically converted from dynamically-typed [JavaScript ES5](https://262.ecma-international.org/5.1/#sec-8) -to statically-typed [native Go types](https://go.dev/ref/spec#Types). - -> The JavaScript function signatures are hinted using TypeScript-style hints -> for easier explanations. Although the runtime is **not** TypeScript. - -## Console - -The `console.*` helpers provide a way to print messages or debug variables in the console. -You can access this console through the [Kubeshark](https://kubeshark.com) dashboard or the `kubeshark console` command. - -### `console.log(...args: string[])` - -Takes N number of `string` typed arguments, concatenates them and prints them to the console with " "(whitespace) as separator between -the arguments. - -##### Example: - -```js -console.log("The variable x is:", x) -``` - -### `console.error(...args: string[])` - -Takes N number of `string` typed arguments, concatenates them and prints them to the console **as error messages** with " "(whitespace) as separator between -the arguments. - -##### Example: - -```js -console.error("Something is not right and the value is:", value) -``` - -## Test - -The `test.*` helpers are useful for implementing test rules and manipulating the dashboard. - -### `test.pass(data: object): object` - -Takes a single argument which is a JavaScript object, sets its `passed` field to `true` like; `data.passed = true` and returns that object. If you use it inside the [`onItemQueried`](/en/automation_hooks#onitemquerieddata-object) hook and return the modified `data`, it will order the dashboard to mark that item as **green** on the left-pane. - -##### Example: - -```js -function onItemQueried(data) { - if (data.response.status == 200) { - return test.pass(data) - } -} -``` - -### `test.fail(data: object): object` - -Takes a single argument which is a JavaScript object, sets its `failed` field to `true` like; `data.failed = true` and -returns that object. If you use it inside the [`onItemQueried`](/en/automation_hooks#onitemquerieddata-object) hook and return the modified `data`, it will order the dashboard to mark that item as **red** on the left-pane. - -##### Example: - -```js -function onItemQueried(data) { - if (data.response.status == 500) { - return test.fail(data) - } -} -``` - -## Vendor - -The `vendor.*` helpers provide integrations to other software and services. They are useful for alerting other systems -and pushing data to them when a certain event occur or periodically through jobs. - -### `vendor.webhook(method: string, url: string, body: string, object: headers)` - -This helper executes an HTTP request to a webhook (the HTTP endpoint) that's defined by HTTP `method` and URL in the `url` -argument with the HTTP body as the string in the `body` argument. YOu can add headers to the request by providing a key/value pair set as the last argument. - -The webhook returns the response as value. - -##### Example: - -```js -response = vendor.webhook( - "POST", - "https://webhook.site/a42ca96d-4984-45dc-8f72-a601448399dc", - JSON.stringify(data), - { - "content-type": "application/json" - } -); -``` - -### `vendor.slack(webhookUrl: string, pretext: string, text: string, color: string)` - -Sends a Slack message to the Slack webhook in `webhookUrl` argument. -It's especially useful for **alerting** a group of developers about **an issue detected through the network traffic**, such as -*HTTP 500 response status code:* - -##### Example: - -```js -function onItemCaptured(data) { - if (data.response.status === 500) - vendor.slack( - env.SLACK_WEBHOOK, - "Server-side Error", - JSON.stringify(data), - "#ff0000" - ); -} -``` - -### `vendor.slackBot(token: string, channelID: string, pretext: string, text: string, color: string, fields?: object, files?: object)` - -Sends a Slack message to the Slack channel in `channelID` argument using the provided access token in `token` argument. - -`fields` optional argument is a list key-value pairs which describes the fields that are going to be added to the Slack message. -The keys are field names. The values should be string. There is always a *Timestamp* field added to the list of fields automatically. - -`files` optional argument is a list of key-value pairs which describes the list of files that are going to be attached into the Slack message. -The keys are file names. The values are file paths. - -It's especially useful for **alerting** a group of developers about **an issue detected through the network traffic**, such as -*"HTTP 500 response status code:"* - -##### Example: - -```js -function onItemCaptured(data) { - // Check if it's an HTTP request and the response status is 500 - if (data.protocol.name === "http" && data.response.status === 500) { - var files = {}; - - // Get the path of the PCAP file that this stream belongs to - var pcapPath = pcap.path(data.stream); - files[data.stream + ".pcap"] = pcapPath; - - // Dump the `data` argument into a temporary JSON file - var dataPath = file.temp("data", "", "json"); - file.write(dataPath, JSON.stringify(data, null, 2)); - files["data.json"] = dataPath; - - // Send a detailed Slack message with 2 attached files - vendor.slackBot( - SLACK_AUTH_TOKEN, - SLACK_CHANNEL_ID, - "Server-side Error in Kubernetes Cluster", // Pretext - "An HTTP request resulted with " + data.response.status + " status code:", // Text - "#ff0000", // Color - { - "Service": data.dst.name, - "Namespace": data.namespace, - "Node": data.node.name, - "HTTP method": data.request.method, - "HTTP path": data.request.path - }, - files - ); - - // Delete the temporary file - file.delete(dataPath); - } -} -``` - -### `vendor.influxdb(url: string, token: string, organization: string, bucket: string, measurement: string, data: object, tags?: object)` - -Pushes the data into a `bucket` the InfluxDB instance at URL `url` argument using the provided access token in `token` argument. -InfluxDB, as **a timeseries database**, is useful for collecting data or measurements about the Kubernetes network. -InfluxDB can also be added as [**a data source to Grafana**](https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-influxdb/). -It's logical to **aggregate data into a global variables** from hooks and **push it to InfluxDB through a job**: - -##### Example: - -```js -function pushDataToInfluxDB() { - // Print the data - console.log("Data:", JSON.stringify(data)) - - // Push the data - vendor.influxdb( - env.INFLUXDB_URL, - env.INFLUXDB_TOKEN, - env.INFLUXDB_ORGANIZATION, - env.INFLUXDB_BUCKET, - "Example Measurement", // Measurement - data, // Payload - {"example":"tag"} - ); - - // Reset the data - data = {}; -} - -// Call the JavaScript function `pushDataToInfluxDB` every minute -jobs.schedule("push-data-to-influxdb", "0 */1 * * * *", pushDataToInfluxDB); -``` - -### `vendor.elastic(url: string, index: string, data: object, username?: string, password?: string, cloudID?: string, apiKey?: string, serviceToken?: string, certificateFingerprint?: string)` - -Pushes the data into an Elasticsearch `index` inside an instance at URL `url` argument using various different authentication strategies -provided by Elasticsearch: - -- Set `username` and `password` for [Basic Authentication](https://www.elastic.co/guide/en/elasticsearch/client/go-api/master/connecting.html#auth-basic). -- Set `serviceToken` for [HTTP Bearer authentication](https://www.elastic.co/guide/en/elasticsearch/client/go-api/master/connecting.html#auth-token). -- Set `url` to empty string, `cloudID` to [Cloud ID of your Elastic Cloud deployment](https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html) and `apiKey` to the [API key that you have generated in the Elastic Cloud](https://www.elastic.co/guide/en/cloud/current/ec-api-authentication.html). - -##### Example: - -```js -function pushDataToElasticsearch() { - // Print the data - console.log("Data:", JSON.stringify(data)) - - // Push the data - vendor.elastic( - "", // URL is ignored for Elastic Cloud - env.ELASTIC_INDEX, - data, // Payload - "", // Username is ignored for Elastic Cloud - "", // Password is ignored for Elastic Cloud - env.ELASTIC_CLOUD_ID, - env.ELASTIC_API_KEY - ); - - // Reset the data - data = {}; -} - -// Call the JavaScript function `pushDataToElasticsearch` every minute -jobs.schedule("push-data-to-elastic", "0 */1 * * * *", pushDataToElasticsearch); -``` - -### `vendor.s3.put(bucket: string, path: string, region: string, keyID: string, accessKey: string): string` - -Uploads a file to an AWS S3 `bucket` on AWS `region` using the either AWS authentication. AWS authentication can be achieved using: -- Specific credentials: `keyID` and `accessKey` arguments -- Shared configuration (e.g. IRSA, kube2aim) - -The S3 path of the file is set based on this pattern: `__/`. -Returns the URL of the S3 location once the file is successfully uploaded. - -#### Example: - -```js -location = vendor.s3.put( - env.S3_BUCKET, - filePath - env.AWS_REGION, - env.AWS_ACCESS_KEY_ID, // optional. will default to shared configuration - env.AWS_SECRET_ACCESS_KEY // optional. will default to shared configuration -); -``` - -#### Using IRSA or kube2iam - -`vendor.s3.put(bucket: string, path: string, region: string): string` - -[IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) is a method for not using specific credentials but rather use a role associated with a service account. -To use IRSA, you'd need to: -1. Provide annotation of the IAM role -2. Use shared configuration - -For example, when using help (or CLI), add the following property: - -```shell ---set-json 'tap.annotations={"eks.amazonaws.com/role-arn":"arn:aws:iam::7456....3350:role/s3-role"}' -``` - -And as the helper use: - -```js -location = vendor.s3.put( - env.S3_BUCKET, - filePath - env.AWS_REGION -); -``` -### `vendor.s3.clear(region: string, keyID: string, accessKey: string, bucket: string)` - -Clears the content of the folder `_/` in the AWS S3 `bucket`. -The folder is simply owned by the [Kubeshark](https://kubeshark.com) worker/node. -It can be called through a job to do a periodic clean up. - -##### Example: - -```js -vendor.s3.clear( - env.AWS_REGION, - env.AWS_ACCESS_KEY_ID, - env.AWS_SECRET_ACCESS_KEY, - env.S3_BUCKET -); -``` -### `vendor.gcs.put(bucket: string, path: string, saKeyObj: json): string` - -Uploads a file to a GCS `bucket` on GCP using a service account key. - -The GCS path of the file is set based on this pattern: `__/`. -Returns the URL of the S3 location once the file is successfully uploaded. - -#### Example: - -```js -location = vendor.s3.put( - env.GCS_BUCKET, - filePath - JSON.parse(env.GCS_SA_KEY_JSON) -); -``` -### `vendor.gcs.clear(bucket: string, , saKeyObj: json)` - -Clears the content of the folder `_/` in the GCS `bucket`. -The folder is simply owned by the [Kubeshark](https://kubeshark.com) worker/node. -It can be called through a job to do a periodic clean up. - -##### Example: - -```js -vendor.s3.clear( - env.AWS_REGION, - env.AWS_ACCESS_KEY_ID, - env.AWS_SECRET_ACCESS_KEY, - env.S3_BUCKET -); - -## PCAP - -The `pcap.*` helpers provide certain functionality revolving around **the PCAP-based -network traffic storage mechanism of Kubeshark**. - -### `pcap.nameResolutionHistory(): object` - -The helper immediately returns a map of **the name resolution history of the given Kubernetes node**. Which -its **keys are UNIX timestamps** and values are yet another map with keys are IPs and values are names and Kubernets namespaces. -The UNIX timestamps mark a change in the name resolution throughout the history. - -##### Example: - -```js -var nameResolutionHistory = pcap.nameResolutionHistory(); -``` - -### `pcap.snapshot(selectedPcaps?: string[], pcapsDir?: string): string` - -It **merges all the PCAP files (TCP/UDP streams) into a single PCAP file** and saves it under the root folder. -The returned `path` has always this pattern: `.pcap`. Then you can supply this file path -to other helpers that accept a file path as argument such as; [`vendor.s3.put`](#vendors3putregion-string-keyid-string-accesskey-string-bucket-string-path-string-string), [`file.move`](#filemoveoldpath-string-newpath-string) or [`file.delete`](#filedeletepath-string). - -You can supply a list of PCAP filenames (the base names of TCP/UDP streams) in the optional `selectedPcaps` argument to specify -the list of PCAP files to merge. For example; `data.stream` in [`onItemCaptured(data)`](/en/automation_hooks#onitemcaptureddata-object) hook is a PCAP filename. - -You can specify a custom directory using the optional `pcapsDir` argument which contains PCAP files. -By default the merge happens in the internally managed folder of [Kubeshark](https://kubeshark.com) that contains all the PCAP files. -This argument should be used in conjunction with [`file.mkdirTemp`](#filemkdirtempname-string-dir-string-string) and [`file.move`](#filemoveoldpath-string-newpath-string) to collect PCAP files -into a directory. - -##### Example: - -```js -var dir = file.mkdirTemp("snapshots"); -var snapshot = pcap.snapshot(dir); -``` - -### `pcap.path(tpcOrUdpStream: string): string` - -Returns the full path of a given TCP/UDP stream then you can supply this file path -to other helpers that accept a file path as argument such as; [`vendor.s3.put`](#vendors3putregion-string-keyid-string-accesskey-string-bucket-string-path-string-string). - -> It's advised against modifying these files using helpers like [`file.write`](#filewritepath-string-content-string), [`file.append`](#fileappendpath-string-content-string), [`file.move`](#filemoveoldpath-string-newpath-string) or [`file.delete`](#filedeletepath-string) -> because the TCP/UDP streams are internally tracked, written and updated files -> that emerge from the Kubernetes network traffic capture. -> Modifying these files can break the core functionality of [Kubeshark](https://kubeshark.com). - -##### Example: - -```js -function onItemCaptured(data) { - // Get the PCAP file path of the TCP/UDP stream - var pcapPath = pcap.path(data.stream); -} -``` - -## File - -The `file.*` helpers provide file system operations on the worker-level (per Kubernetes node) to manipulate files, -aggregate data and temporarily or permanently store data in the disk. - -### `file.write(path: string, content: string)` - -Writes the string `content` into file at `path`. Throws an error if the file does not exist on given `path`. - -##### Example: - -```js -file.write("example.txt", "hello"); -``` - -### `file.append(path: string, content: string)` - -Appends the string `content` into file at `path`. Throws an error if the file does not exist on given `path`. - -##### Example: - -```js -file.append("example.txt", " world"); -``` - -### `file.move(oldPath: string, newPath: string)` - -Moves the file at `oldPath` to `newPath`. Throws an error if the operation fails. -If `newPath` is a directory then the file is copied to that directory by preserving its -base name. - -##### Example: - -```js -file.move("example.txt", "hello.txt"); -file.move("example.txt", "directory/"); -``` - -### `file.copy(srcPath: string, dstPath: string)` - -Copies the file at `srcPath` to `dstPath`. Throws an error if the operation fails. - -##### Example: - -```js -file.copy("example.txt", "example2.txt"); -``` - -### `file.delete(path: string)` - -Deletes the file at `path`. Throws an error if the operation fails. - -##### Example: - -```js -file.delete("hello.txt") -``` - -### `file.mkdir(path: string)` - -Creates a new directory satisfies `path`, along with any necessary parent directories. Throws an error if the operation fails. - -##### Example: - -```js -file.mkdir("example/folder/here") -``` - -### `file.mkdirTemp(name?: string, dir?: string): string` - -Creates a new temporary directory in the directory `dir` and returns the path of the new directory. The new directory's name is generated by adding a random string to the end of `name`. Throws an error if the operation fails. - -##### Example: - -```js -var tempDir = file.mkdirTemp("foo", "example/folder/here") -``` - -### `file.temp(name: string, dir: string, extension: string): string` - -Creates a new temporary file in the directory `dir` and returns the path of the new file. The new directory's name is generated by adding a random string to the end of `name`. The file extension is set to `extension`, default is `.txt`. Throws an error if the operation fails. - -##### Example: - -```js -var tempFile = file.temp("bar", "example/folder/here", "json") -``` - -### `file.tar(path: dir): string` - -Generates a `.tar.gz` with its name `kubeshark_.tar.gz` based on the files under the directory `dir`. Throws an error if the operation fails. - -##### Example: - -```js -var tarFile = file.tar("example/folder/here") -``` - -## Jobs - -The `jobs.*` helpers provide certain functionality revolving around **the jobs system of Kubeshark** which augments the scripting system by -scheduling or triggering jobs that you write as JavaScript functions. - -### `jobs.schedule(tag: string, cron: string, task: function, limit: number, ...argumentList: any[])` - -Schedules a job using the `cron` statement and function given by `task`. The job is uniquely identified by its `tag` and scheduled on all workers (Kubernetes nodes) simultaneously. - -`limit` sets the limit for how many times it has to run. (per node) - -`argumentList` is passed to the function given by `task`. - -##### Example: - -```js -function exampleJob() { - // Your code goes here -} - -// Schedule a job that calls `exampleJob` function every 5 seconds -jobs.schedule("example-job", "*/5 * * * * *", exampleJob) -``` - -### `jobs.remove(tag: string)` - -Removes the job scheduled by `tag` from all nodes. - -##### Example: - -```js -jobs.remove("example-job") -``` - -### `jobs.removeAll()` - -Removes all jobs from all nodes unconditionally. - -##### Example: - -```js -jobs.removeAll() -``` - -### `jobs.list(): string[]` - -Returns the list of all scheduled jobs (tags). - -##### Example: - -```js -var tags = jobs.list() -``` - -### `jobs.run(tag: string)` - -Runs the job scheduled by `tag` from all nodes. - -##### Example: - -```js -jobs.run("example-job") -``` - -### `jobs.runAll()` - -Run all jobs from all nodes unconditionally. - -##### Example: - -```js -jobs.runAll() -``` - -### `jobs.scheduler.isRunning(): boolean` - -Returns `true` if the job scheduler is running. Otherwise returns `false`. - -##### Example: - -```js -var status = jobs.scheduler.isRunning() -``` - -### `jobs.scheduler.start()` - -Starts the job scheduler. - -##### Example: - -```js -var status = jobs.scheduler.start() -``` - -### `jobs.scheduler.stop()` - -Stops the job scheduler. - -##### Example: - -```js -var status = jobs.scheduler.stop() -``` - -## KFL - -The `kfl.*` helpers provide functionality around the [**Kubeshark Filter Language (KFL)**](/en/filtering#kfl-syntax-reference). - -### `kfl.match(query: string, data: object): boolean` - -Checks whether the KFL `query` matches to given `data` or not. Returns `true` or `false` accordingly. - -##### Example: - -```js -function onItemCaptured(data) { - if (kfl.match("http and response.status == 500", data)) { - console.log("HTTP 500!") - } else { - console.log("Seems OK.") - } -} -``` - -### `kfl.validate(query: string): boolean` - -Verifies whether a given KFL `query` is a syntactically valid KFL statement. Returns `true` or `false` accordingly. - -##### Example: - -```js -if (kfl.validate("http and response.status == 500")) { - console.log("Valid KFL.") -} else { - console.log("Gibberish!") -} -``` - -## ChatGPT - -The `chatgpt.*` helpers lets you use [OpenAI's ChatGPT](https://openai.com/blog/chatgpt) in your scripts. - -### `chatgpt.prompt(apiKey: string, prompt: string, maxTokens?: number): string` - -Prompts ChatGPT using [OpenAI's API](https://platform.openai.com/docs/api-reference/introduction) with given `apiKey`. -You build the `prompt` string argument by starting with a question and include some network data to get a response from ChatGPT. -`maxTokens` optional argument lets you set the "the maximum number of tokens to generate in the completion", the default value is `1024`. - -##### Example: - -```js -function onItemCaptured(data) { - if (data.protocol.name == "http") { - // Delete internally used fields to not confuse ChatGPT - delete data.passed - delete data.failed - - var payload = JSON.stringify(data); - - var response = chatgpt.prompt( - env.OPENAI_API_KEY, - "Did the HTTP request failed in this HTTP request-response pair? " + payload - ); - console.log("ChatGPT:", response); - - var score = chatgpt.sentiment(response); - if (score.pos > 0.4) { - console.log("ALERT! ChatGPT is detected a failed HTTP request:", response, "Payload:", payload); - } - } -} -``` - -### `chatgpt.sentiment(text: string): object` - -Does [sentiment analysis](https://en.wikipedia.org/wiki/Sentiment_analysis) on a given `text` input and returns the -score object below: - -```go -type Score struct { - Negative float64 `json:"neg"` - Neutral float64 `json:"neu"` - Positive float64 `json:"pos"` - Compound float64 `json:"compound"` -} -``` - -> This helper is supposed to be used in conjunction with [chatgpt.prompt](#chatgptpromptapikey-string-prompt-string-maxtokens-number-string) helper -> in such a way that you pass the **response of ChatGPT** to this helper to get a **sentiment analysis score**. -> Using this score, you can detect whether the ChatGPT's judgement is **positive or negative**. -> Alternatively, you can tell ChatGPT to respond only using **"Yes" or "No"**. In that case, **you don't require** `chatgpt.sentiment` helper -> but **you lose the context** in the ChatGPT's response. - - -## Environment Variables - -The global object `env` holds all of the constants defined in the `scripting.env` field of `$HOME/.kubeshark/config.yaml` -or `kubeshark.yaml` in your current working directory. -Printing this variable allows you to debug the availability of those constants in the runtime. - -##### Example: - -Suppose you have; - -```yaml -scripting: - env: - SLACK_AUTH_TOKEN: "foo" - SLACK_CHANNEL_ID: "bar" -``` - -in your `kubeshark.yaml`. Then the JavaScript code below; - -```js -console.log(JSON.stringify(env)); -``` - -would print; - -``` -{"SLACK_AUTH_TOKEN":"foo","SLACK_CHANNEL_ID":"bar"} -``` - -into the console. diff --git a/src/pages/en/automation_hooks.md b/src/pages/en/automation_hooks.md deleted file mode 100644 index 155210f..0000000 --- a/src/pages/en/automation_hooks.md +++ /dev/null @@ -1,335 +0,0 @@ ---- -title: Hooks -description: Provided hooks including OSI L4 and L7 hooks that enable running functions whenever a packet is captured or a new application-layer message is dissected. -layout: ../../layouts/MainLayout.astro ---- - -Hooks are pre-defined JavaScript functions in the scripting system that are being attached to a certain point in the network packet **capture - dissect - query** pipeline of [Kubeshark](https://kubeshark.com). - -[Kubeshark](https://kubeshark.com) provides [OSI](https://en.wikipedia.org/wiki/OSI_model) L4 and L7 hooks that enable running functions whenever a packet is captured or a new application-layer message is dissected. - -Hooks function arguments with type **object** can be printed to console with the purpose of seeing the data structure of that argument, like; - -```js -console.log(JSON.stringify(data)) -``` - -A couple of examples can be found at the [end of this section](/en/automation_hooks#http-data-record). - -Hook description includes an **Arguments** section that explains the hook's arguments' Go data structure. These **Go data structures provide hints about the typings** that emerge in the JavaScript runtime. Please refer to the [JSON and Go](https://go.dev/blog/json) page to understand how JSON encoding works in Go language if you're not familiar with the Go language. - -## `onPacketCaptured(info: object)` - -```js -function onPacketCaptured(info) { - // Your code goes here -} -``` -The `onPacketCaptured` is an OSI L4 network hook. It is called whenever a new network packet is captured by [Kubeshark](https://kubeshark.com). - -On a **busy cluster**, the call frequency of this hook can go more than a **10000 times per second**. Because of that; a poorly optimized `onPacketCaptured` implementation can have a **performance impact** on the network traffic capture speed of [Kubeshark](https://kubeshark.com). Therefore it's logical to not call certain helpers (e.g. [`console.log`](/en/automation_helpers#consolelogargs-string) or [`vendor.slack`](/en/automation_helpers#vendorslackwebhookurl-string-pretext-string-text-string-color-string)) and instead use this hook to aggregate data into a global variable and then handle the aggregated data in another hook or a job: - -```js -var packetCount = 0; -var totalKB = 0; - -function onPacketCaptured(info) { - // Example of data aggregation, packet and KB counter - packetCount++; - totalKB += info.length / 1000; -} - -function yourJob() { - // Handle `packetCount` and `totalKB` here -} - -// Schedule a call to function `yourJob` every minute -jobs.schedule("example-job", "0 */1 * * * *", yourJob); -``` - -> The call to hook `onPacketCaptured` does not depend on user's action, it's sourced from network traffic capture. - -### Arguments - -Arguments of `onPacketCaptured` hook: - -#### `info` - -```go -type Info struct { - Timestamp time.Time `json:"timestamp"` - CaptureLength int `json:"captureLength"` - Length int `json:"length"` - Truncated bool `json:"truncated"` - Fragmented bool `json:"fragmented"` -} -``` - -`timestamp` is the UTC timestamp of the moment that packet is captured. - -`captureLength` is the length of the capture. - -`length` is the actual length of packet. - -`truncated` whether the packet is truncated or not. See [packet truncation](https://www.cisco.com/c/en/us/td/docs/net_mgmt/xnc/nexus_data_broker/use-case/39/packet-truncation.html) for more info. - -## `onItemCaptured(data: object)` - -```js -function onItemCaptured(data) { - // Your code goes here -} -``` - -The hook `onItemCaptured` is an OSI L7 network hook that is called whenever a TCP/UDP stream is captured, reassembled and successfully dissected by one of the protocol parsers of [Kubeshark](https://kubeshark.com). - -This hook is triggered less compared to the `onPacketCaptured` hook, since multiple packets translate into a protocol-level message (e.g. an HTTP request-response pair, a Kafka publish, a Kafka consume or an AMQP exchange declare). - -The item's scope is determined by the corresponding application layer protocol as one can imagine. For example; an HTTP request-response pair contains a lot parameters while an AMQP exchange declare is quite a simple message. - -> The call to hook `onItemCaptured` does not depend on user's action, it's sourced from network traffic capture. - -### Arguments - -Arguments of `onItemCaptured` hook: - -#### `data` - -> See the updated data structure: https://github.com/kubeshark/api/blob/master/api.go#L273 - -## `onItemQueried(data: object)` - -The hook `onItemQueried` is called whenever an already captured and stored TCP/UDP stream is queried or fetched through the dashboard. -All of its other aspects are same with the [`onItemCaptured`](#onitemcaptureddata-object) hook. - -> The call to hook `onItemQueried` depends on user's action, it's **not** sourced from network traffic capture. - -### Arguments - -Same as [`onItemCaptured`](#onitemcaptureddata-object) hook. - -## `onJobPassed(tag: string, cron: string, limit: number)` - -```js -function onJobPassed(tag, cron, limit) { - // Your code goes here -} -``` - -The hook `onJobPassed` is called whenever a job passes. - -### Arguments - -`tag` is job tag. - -`cron` is the cron statement. - -`limit` is the limit of job runs. - -## `onJobFailed(tag: string, cron: string, limit: number)` - -```js -function onJobFailed(tag, cron, limit, err) { - // Your code goes here -} -``` - -The hook `onJobFailed` is called whenever a job fails. - -### Arguments - -`tag` is job tag. - -`cron` is the cron statement. - -`limit` is the limit of job runs. - -`err` is the error message. - -## Example values of the `data` argument - -Here are some example values of the `data` argument that being used in -[`onItemCaptured`](#onitemcaptureddata-object) and [`onItemQueried`](#onitemquerieddata-object) hooks: - -### HTTP - -```js -{ - "dst": { - "ip": "10.0.0.99", - "name": "", - "port": "8086" - }, - "elapsedTime": 23, - "failed": false, - "id": "192.168.49.2:8897/000000025258.pcap-0", - "index": 0, - "namespace": "default", - "node": { - "ip": "192.168.49.2", - "name": "my-cluster" - }, - "outgoing": false, - "passed": false, - "protocol": { - "abbr": "HTTP", - "backgroundColor": "#326de6", - "fontSize": 12, - "foregroundColor": "#ffffff", - "layer4": "tcp", - "longName": "Hypertext Transfer Protocol -- HTTP/1.1", - "macro": "http", - "name": "http", - "ports": [ - "80", - "443", - "8080" - ], - "priority": 0, - "referenceLink": "https://datatracker.ietf.org/doc/html/rfc2616", - "version": "1.1" - }, - "request": { - "bodySize": 124, - "cookies": {}, - "headers": { - "Accept-Encoding": "gzip", - "Authorization": "Token edO4nlXbD5cfx8nw9_94LMO4tvGJ_xeMQMiFc6J_DNYFAGRe0YIgG8gz98UwDKOa6otCzml3SNw_c5TiDuB4eA==", - "Content-Length": "124", - "Host": "10.0.0.99:8086", - "User-Agent": "influxdb-client-go/2.12.2 (linux; amd64)" - }, - "headersSize": -1, - "httpVersion": "HTTP/1.1", - "method": "POST", - "path": "/api/v2/write", - "pathSegments": [ - "api", - "v2", - "write" - ], - "postData": { - "mimeType": "", - "params": [], - "text": "PerformanceKPIs,namespace=sock-shop,path=/basket.html,service=front-end.sock-shop latency=51,status=200 1678675066165748213\n" - }, - "queryString": { - "bucket": "Metrics", - "org": "[Kubeshark](https://kubeshark.com)", - "precision": "ns" - }, - "targetUri": "/api/v2/write?bucket=Metrics&org=[Kubeshark](https://kubeshark.com)&precision=ns", - "url": "/api/v2/write?bucket=Metrics&org=[Kubeshark](https://kubeshark.com)&precision=ns" - }, - "requestSize": 428, - "response": { - "bodySize": 0, - "content": { - "encoding": "base64", - "mimeType": "", - "size": 0 - }, - "cookies": {}, - "headers": { - "Date": "Mon, 13 Mar 2023 02:37:46 GMT", - "X-Influxdb-Build": "OSS", - "X-Influxdb-Version": "2.6.1" - }, - "headersSize": -1, - "httpVersion": "HTTP/1.1", - "redirectURL": "", - "status": 204, - "statusText": "No Content" - }, - "responseSize": 114, - "src": { - "ip": "192.168.49.2", - "name": "kubernetes.default", - "port": "36848" - }, - "startTime": "2023-03-13T02:37:46.172360504Z", - "stream": "000000025258.pcap", - "timestamp": 1678675066172, - "tls": false, - "worker": "192.168.49.2:30001" -} -``` -### DNS - -```js -{ - "dst": { - "ip": "172.17.0.3", - "name": "kube-dns.kube-system", - "port": "53" - }, - "elapsedTime": 0, - "failed": false, - "id": "192.168.49.2:8897/000000025277_udp.pcap-0", - "index": 0, - "namespace": "kube-system", - "node": { - "ip": "192.168.49.2", - "name": "my-cluster" - }, - "outgoing": false, - "passed": false, - "protocol": { - "abbr": "DNS", - "backgroundColor": "#606060", - "fontSize": 12, - "foregroundColor": "#ffffff", - "layer4": "udp", - "longName": "Domain Name System", - "macro": "dns", - "name": "dns", - "ports": [], - "priority": 4, - "referenceLink": "https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml", - "version": "0" - }, - "request": { - "opCode": "Query", - "questions": [ - { - "class": "IN", - "name": "session-db.sock-shop.svc.cluster.local", - "type": "A" - } - ] - }, - "requestSize": 154, - "response": { - "answers": [ - { - "class": "IN", - "cname": "", - "ip": "10.101.46.75", - "mx": "", - "name": "session-db.sock-shop.svc.cluster.local", - "ns": "", - "opt": "", - "ptr": "", - "soa": "", - "srv": "", - "ttl": 24, - "txts": "", - "type": "A", - "uri": "" - } - ], - "code": "No Error" - }, - "responseSize": 154, - "src": { - "ip": "172.17.0.1", - "name": "", - "port": "33305" - }, - "startTime": "2023-03-13T02:37:49.958762876Z", - "stream": "000000025277_udp.pcap", - "timestamp": 1678675069958, - "tls": false, - "worker": "192.168.49.2:30001" -} -``` diff --git a/src/pages/en/automation_introduction.md b/src/pages/en/automation_introduction.md deleted file mode 100644 index e27f67e..0000000 --- a/src/pages/en/automation_introduction.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: Actionable Network Insights -description: Users can automate network tasks with unlimited agents that detect threats, export data, and optionally connect to an LLM for continuous insight and logic refinement. -layout: ../../layouts/MainLayout.astro ---- - -## With Network Agents - -Users can create and use an unlimited number of **Network Agents**, each designed to perform a specific network-related automation task. - -Agents can detect anomalies and threats, generate reports, export metrics, traces, and logs, enforce network policies, record traffic, and more. - -### Optional Connection to a Custom LLM - -Connecting to an external LLM is **optional** and can be fully disabled to support air-gapped environments. - -Agent will run and you can still get help creating new Agents without any connection between [Kubeshark](https://kubeshark.com) and an external LLM. - -If enabled, each **Network Agent** will maintain a persistent connection with a custom LLM. This connection will allow: - -1. Continuous, bidirectional communication—typically sending data objects (JSON maps) over time and receiving actionable insights. - **Example use case:** detect anomalies across a time range. -2. Ongoing improvement of the agent’s logic based on LLM feedback. - -## How to Disable Access to External LLM - -If you prefer [Kubeshark](https://kubeshark.com) not to access an external LLM, you can configure either or both of the following settings: - -```yaml -aiAssistantEnabled: false # Disables the connection to the external LLM. -internetConnectivity: false # Blocks any cluster-external connectivity (for air-gapped environments). -``` \ No newline at end of file diff --git a/src/pages/en/automation_jobs.md b/src/pages/en/automation_jobs.md deleted file mode 100644 index d7f319f..0000000 --- a/src/pages/en/automation_jobs.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Jobs -description: Jobs are functions that are automated to run on a schedule. When functions don't depend on certain network behaviors, they can be automated to run on a schedule. -layout: ../../layouts/MainLayout.astro ---- - -Jobs are functions that are automated to run on a schedule. When functions don't depend on certain network behaviors, they can be automated to run on a schedule. -Jobs are added by adding one line of code at the script scope. - -```js -jobs.schedule("push-status-codes-to-influxdb", "0 */1 * * * *", pushStatusCodesToInfluxDB); -``` - -[Kubeshark](https://kubeshark.com) uses a [crontab](https://crontab.guru/) mechanism to schedule Jobs at the node level. It furthers provides complete control over the scheduled Jobs via the dashboard. - - -## The Jobs Dashboard -The jobs dashboard enables complete control over schedule jobs enabling the following operations: -- Stop a specific Jobs or all Jobs -- Start / Re-start a specific Job or all Jobs -- Delete a specific Job or all Jobs - -![Jobs Dashboard](/jobs-dashboard.png) - -## Running Jobs Programmatically - -While you can manually control the Jobs from the jobs dashboard, you can also control them programmatically from a script using the [`jobs.run`](/en/automation_helpers#jobsruntag-string) helper. - -> Visit the [helpers](/en/automation_helpers#jobs) page to read the complete list of helpers related to Jobs. diff --git a/src/pages/en/automation_logs_alerts.md b/src/pages/en/automation_logs_alerts.md deleted file mode 100644 index e4c3ec8..0000000 --- a/src/pages/en/automation_logs_alerts.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -title: Logs & Alerts -description: Kubeshark provides ample of ways to send log messages and alerts. -layout: ../../layouts/MainLayout.astro ---- - -[Kubeshark](https://kubeshark.com) provides ample ways to send log messages and alerts, some are inherent and some require integrations: - -- Console log and error messages -- Dashboard alerts -- Slack alerts -- Send log messages to Elasticsearch -- Use a webhook to send anything anywhere - -## Console Log & Error Messages - -The [`console.log`](/en/automation_helpers#consolelogargs-string) helper enables writing log messages that can be read using the `kubeshark console` CLI command. -The [`console.error`](/en/automation_helpers#consoleerrorargs-string) sends a message to [`stderr`](https://linux.die.net/man/3/stderr). - -This script example calculates and sends telemetry information once per minute. - -```js -var packetCount = 0; -var totalKB = 0; - -function onPacketCaptured(info) { - packetCount++; - totalKB += info.length / 1000; -} - -function logPacketCountTotalBytes() { - if (packetCount === 0) { - console.error("Received no packets."); - } - - console.log("Captured packet count per minute:", packetCount); - packetCount = 0; - console.log("Total KB captured per minute:", totalKB); - totalKB = 0; -} - -jobs.schedule("log-packet-count-total-bytes", "0 */1 * * * *", logPacketCountTotalBytes); -``` - -When used in conjunctions with `kubeshark console` you can expect the following console log output: - -![Console Log](/console-log-1.png) - -Redirecting the command's output to STDOUT will redirect only the results of [`console.log`](/en/automation_helpers#consolelogargs-string) and omit error messages that were sent to [`stderr`](https://linux.die.net/man/3/stderr). - -The following CLI command redirects the console log output to a file. - -```shell -kubeshark console > /tmp/log.txt -``` - -## Dashboard Alerts - -The [Kubeshark](https://kubeshark.com) dashboard can show alerts using the [`test.pass`](/en/automation_helpers#testpassdata-object-object) and [`test.fail`](/en/automation_helpers#testfaildata-object-object) helpers. The [`test.pass`](/en/automation_helpers#testpassdata-object-object) will color a traffic entry **green**, where the [`test.fail`](/en/automation_helpers#testfaildata-object-object) helper will color the traffic entry **red**. You can for example; call these helpers through a JavaScript conditional statements that acts as the test criteria: - - -As an example, use the L7 hook `onItemQueried` in conjunction with the `test.*` helpers to detect response code `500` and show alerts in the dashboard: -```js -function onItemQueried(data) { - if (data.response.status === 500) - return test.fail(data); - else - return test.pass(data); -} -``` -![Dashboard Alerts](/dashboard_alerts.png) - -> Read more about the `test.*` helpers in the [helpers](/en/automation_helpers) section. - -## Slack Alerts - -Use the Slack helper to send Slack alerts. - -> Read more in the [Slack integration](/en/integrations_slack) section. - -## Send Logs to Elasticsearch - -Use the Elasticsearch helper to send schema-free JSON documents to Elasticsearch. - -> Read more in the [Elastic integration](/en/integrations_elastic) section. - -## Webhooks - -The Webhook helper enables you to send any payload anywhere that supports a webhooks. - -> Read more in the [Webhook integration](/en/integrations_webhook) section. diff --git a/src/pages/en/automation_scripting.md b/src/pages/en/automation_scripting.md deleted file mode 100644 index 324149c..0000000 --- a/src/pages/en/automation_scripting.md +++ /dev/null @@ -1,166 +0,0 @@ ---- -title: Agent Logic -description: Custom-logic scripts use hooks and helpers to trigger actions, supported by the available integrations, and based on programmatic decisions and/or on a schedule. -layout: ../../layouts/MainLayout.astro ---- -## With Network Agents - -Users can create and use an unlimited number of **Network Agents**, each designed to perform a specific network-related automation task. - -Agents can detect anomalies and threats, generate reports, export metrics, traces, and logs, enforce network policies, record traffic, and more. - ---- - -## Agent Logic Structure - -Each agent's business logic is written in a [JavaScript ES5](https://262.ecma-international.org/5.1/) script and consists of the following components: - -1. **Hooks** – Triggered by specific events in the traffic flow. -2. **Helpers** – Used to invoke integrations and specific [Kubeshark](https://kubeshark.com) functions. -3. **General Code** – Performs calculations and stores information in memory. - -Agents process network traffic, perform custom logic, and trigger actions such as generating metrics, exporting logs, producing reports, or automating workflows. - ---- - -## Hooks - -Hooks are functions that react to specific network events. They allow the execution of user-defined JavaScript code within [Kubeshark](https://kubeshark.com)’s Golang backend. - -For example, the `onItemCaptured` hook is triggered every time an API call is reassembled. The following example prints the metadata of each captured API call: - -```js -function onItemCaptured(data) { - // Print API call metadata - console.log("Msg:", data); -} -``` - -Hooks run continuously in the background, regardless of whether the dashboard is open or closed. - -### Common Hooks - -| Hook | Triggering Event | Executed On | Description | -|----------------|--------------------------------------------|------------------|--------------------------------------------------------------------------------------------------| -| `onItemCaptured` | When an API call is reassembled | Workers | Triggered for each reassembled message. Useful for quick analysis, calculations, or tagging. | -| `onHubAction` | When `hub.action(action, object)` is called | Hub | Used for transferring or processing objects in the Hub. | -| `onPodEvent` | On pod events (e.g., restart, crash) | Hub and Workers | Enables response actions like traffic capture before a crash. | - ---- - -## Helpers - -Helpers trigger actions and integrations, such as sending messages to Slack or exporting data to AWS. They provide access to [Kubeshark](https://kubeshark.com)’s backend features. - -Example: sending data to a webhook: - -```js -vendor.webhook( - "POST", - "https://webhook.site/a42ca96d-4984-45dc-8f72-a601448399dc", - JSON.stringify(data) -); -``` - -### Common Helpers - -| Helper | Purpose | Executed On | Description | -|---------------------|------------------------------------|------------------|--------------------------------------------------------------------------------------------------| -| `console.log` | Logging messages | Hub and Workers | Used for debugging or generating custom logs and reports. | -| `jobs.schedule` | Scheduling recurring tasks | Hub and Workers | Runs a function based on a cron expression. Suitable for periodic processing. | -| `vendor.kinesis.put` | Exporting data to AWS Kinesis | Hub and Workers | Sends data to Kinesis for external processing or storage. | - ---- - -## Complete Script Example - -This script calculates and logs the number of packets and total traffic (in KB) every minute using the `onPacketCaptured` hook and a scheduled job: - -```js -var packetCount = 0; -var totalKB = 0; - -function onPacketCaptured(info) { - packetCount++; - totalKB += info.length / 1000; -} - -function logPacketCountTotalBytes() { - console.log("Captured packet count per minute:", packetCount); - console.log("Total KB captured per minute:", totalKB); - packetCount = 0; - totalKB = 0; -} - -jobs.schedule("log-packet-count-total-bytes", "0 */1 * * * *", logPacketCountTotalBytes); -``` - ---- - -## Script Storage - -Scripts are stored in the `kubeshark-config-map`. They can be developed locally and synced using the [`kubeshark scripts` command](/en/automation_scripts_cmd), which automatically detects changes. - -> Scripts must be compliant with [JavaScript ES5](https://262.ecma-international.org/5.1/) and use the `.js` file extension. - -**Example:** -```bash -kubeshark scripts --set scripting.source=/path/to/your/local/folder -``` - ---- - -## Viewing and Editing Scripts in the Dashboard - -All scripts stored in the config map are accessible via the **Scripting** section in the [Kubeshark](https://kubeshark.com) dashboard: - -![Accessing the Scripting Dashboard](/scripting_menu.png) - -### Script Examples - -Example scripts are available in the dashboard and can be loaded via the "Examples" dropdown: - -![Script Examples](/script-examples.png) - ---- - -## Environment Variables - -Define script-specific environment variables using the `env` section in your configuration: - -```yaml -scripting: - env: - SLACK_AUTH_TOKEN: "xo.." - SLACK_CHANNEL_ID: "C0.." - WEBHOOK_URL: "https://webh.." - INFLUXDB_URL: "https://us-e.." - INFLUXDB_TOKEN: "_9r.." - INFLUXDB_MEASUREMENT: "st.." - INFLUXDB_ORGANIZATION: "a.." - INFLUXDB_BUCKET: "al.." -``` - -Reference these variables in your scripts using the `env.` prefix: - -```js -vendor.influxdb( - env.INFLUXDB_URL, - env.INFLUXDB_TOKEN, - env.INFLUXDB_ORGANIZATION, - env.INFLUXDB_BUCKET, - "Example Measurement", // Measurement name - data, // Payload - {"example": "tag"} // Tags -); -``` - ---- - -## Variable Scopes - -[Kubeshark](https://kubeshark.com) scripts support the following scopes: - -- **Function Scope** – Defined within a function, accessible only there. -- **Script Scope** – Defined outside functions, accessible throughout the script. -- **Global Scope** – Defined using `this`, shared across all scripts. \ No newline at end of file diff --git a/src/pages/en/automation_scripts_cmd.md b/src/pages/en/automation_scripts_cmd.md deleted file mode 100644 index a0fe472..0000000 --- a/src/pages/en/automation_scripts_cmd.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: Synchronizing Local Files -description: Guide for syncing local ES5 JavaScript agent scripts with a running Kubeshark instance using the CLI. -layout: ../../layouts/MainLayout.astro ---- - -Users can create and manage unlimited agents using local ES5 JavaScript files (the *Agent's script*). These can be tracked and synced with a running [Kubeshark](https://kubeshark.com) instance using the CLI. - -### Prerequisites - -Install the `kubeshark` CLI using a method from the [installation guide](/en/install). - -Assume a file named `anomaly-agent.js` is located at: - -``` -/Users/me/agents/anomaly-agent.js -``` - -### Track All Files in a Folder - -```shell -kubeshark scripts --set scripting.source=/Users/me/agents/ -``` - -### Track Individual Files - -```shell -kubeshark scripts --set scripting.sources[0]=/Users/me/agents/anomaly-agent.js -``` - -Any change to a tracked file will be synchronized with the running [Kubeshark](https://kubeshark.com) instance. -Adding a new file or deleting an existing one is also treated as a change and will be synced with the deployed [Kubeshark](https://kubeshark.com) instance. - -### Determining the Agent Name - -The CLI extracts the agent name from the first line of the *Agent's script* (as a comment). -Example name: `API Call Anomaly Detection Script` - -![Script Name](/script_name.png) - -## Environment Variables - -*Agent scripts* can access environment variables set in the configuration. - -Example: - -```yaml -scripting: - env: - ZAP_SERVER_URL: https://a6a4...-free.app - ZAP_APIKEY: shshh - AWS_REGION: us-east-1 - S3_BUCKET: vol-ks...d-demo - # AWS_ACCESS_KEY_ID: "AK...N" - # AWS_SECRET_ACCESS_KEY: "xZ...ie" - KINESIS_STREAM_NAME: "stream1" - KINESIS_MIN_BATCH_SIZE: "10" -``` \ No newline at end of file diff --git a/src/pages/en/automation_wrappers.md b/src/pages/en/automation_wrappers.md deleted file mode 100644 index c6f204f..0000000 --- a/src/pages/en/automation_wrappers.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -title: Wrappers -description: Wrappers are a kind of helpers who wrap a certain piece of JavaScript code for ease of use. -layout: ../../layouts/MainLayout.astro ---- - -Wrappers are a kind of helpers who wrap a certain piece of JavaScript code for ease of use. They are like bigger functions that perform multiple tasks and hide the complexity of the code behind them. Wrappers start with the `wrapper.*` prefix. - -## `wrapper.kflPcapS3` - -This wrapper receives a list of KFL queries as input, monitors traffic and generates PCAP repositories that match any of the KFL entries. The compressed PCAP repositories are uploaded to AWS S3 and optionally sends a Slack notification. - -Here's an example of how to use the helper: - -```js -var KFL_PCAP_S3_KFL_ARR = [ - "http and response.status == 500", - "dns", -]; - -function onItemCaptured(data) { - wrapper.kflPcapS3(data, { - kflArr: KFL_PCAP_S3_KFL_ARR, - }); -} -``` - -- `http and response.status == 500` - HTTP traffic only where response status is 500 -- `dns` - all DNS traffic - -### The Input Object - -The `wrapper.kflPcapS3` expects the following input object: - -```go -{ - kflArr: string[], // the only mandatory field - awsRegion: string, // default: env.AWS_REGION - awsAccessKeyId: string, // default: env.AWS_ACCESS_KEY_ID - awsSecretAccessKey: string, // default: env.AWS_SECRET_ACCESS_KEY - s3Bucket: string, // default: env.S3_BUCKET - slackWebhook: string, // if doesn't exist, no slack message will be sent - slackAuthToken: string, // if doesn't exist, no slack message will be sent - slackChannelId: string, // if doesn't exist, no slack message will be sent - active: bool, // default: true - verbose: bool, // default: false - maxMinutes: int, // default: 60 - maxL4Streams: int // default: 100000 -} -``` - -Here's an example of how to use the wrapper with a complete input object, overriding all defaults: - -```js -wrapper.kflPcapS3(data, { - kflArr: KFL_PCAP_S3_KFL_ARR, // Mandatory - /* AWS S3 credential must be present, either here or in the config file as env variables */ - awsRegion: env.AWS_REGION, - awsAccessKeyId: env.AWS_ACCESS_KEY_ID, - awsSecretAccessKey: env.AWS_SECRET_ACCESS_KEY, - s3Bucket: env.S3_BUCKET, - /* Optional: A slack message is fired only if these properties are provided. There's no default value */ - slackWebhook: env.SLACK_WEBHOOK, - slackAuthToken: env.SLACK_AUTH_TOKEN, - slackChannelId: env.SLACK_CHANNEL_ID, - /* The rest of the properties are optional */ - active: true, // set to false to deactivate this helper - verbose: false, // set to true to see verbose log - maxMinutes: 60, // maximum time for a single PCAP file - maxL4Streams: 10000, // maximum L4 streams for a single PCAP file -}); -``` - -[Kubeshark](https://kubeshark.com) uploads the compressed PCAP repositories to AWS S3, making them available -in your AWS S3 console: - -![S3 FIles](/kfl-pcap-s3.png) - -### Progress Log File - -`wrapper.kflPcapS3` maintains a progress log file in the AWS S3 bucket. Here's the progress log file matching the above example: - -```json -[ - { - "file": "kfl_0_kubeshark_1678808288.tar.gz", - "kfl_index": 0, - "kfl_query": "http and response.status == 500", - "s3_url": "https://kubeshark-helper-test.s3.us-east-2.amazonaws.com/my-cluster_192.168.49.2/kfl_0_kubeshark_1678808288.tar.gz", - "time": "Tue, 14 Mar 2023 15:38:26 GMT" - }, - { - "file": "kfl_1_kubeshark_1678808543.tar.gz", - "kfl_index": 1, - "kfl_query": "dns", - "s3_url": "https://kubeshark-helper-test.s3.us-east-2.amazonaws.com/my-cluster_192.168.49.2/kfl_1_kubeshark_1678808543.tar.gz", - "time": "Tue, 14 Mar 2023 15:42:53 GMT" - }, - { - "file": "kfl_0_kubeshark_1678808758.tar.gz", - "kfl_index": 0, - "kfl_query": "http and response.status == 500", - "s3_url": "https://kubeshark-helper-test.s3.us-east-2.amazonaws.com/my-cluster_192.168.49.2/kfl_0_kubeshark_1678808758.tar.gz", - "time": "Tue, 14 Mar 2023 15:46:18 GMT" - }, - { - "file": "kfl_1_kubeshark_1678808954.tar.gz", - "kfl_index": 1, - "kfl_query": "dns", - "s3_url": "https://kubeshark-helper-test.s3.us-east-2.amazonaws.com/my-cluster_192.168.49.2/kfl_1_kubeshark_1678808954.tar.gz", - "time": "Tue, 14 Mar 2023 15:49:32 GMT" - } -] -``` - -### Optional Slack Alerts - -To get Slack alerts upon new PCAP repository available in S3, include Slack credentials as part of the input object. If Slack credentials are not provided, no Slack message will be sent. - -### PCAP Repository Content - -The PCAP repositories include additional meta-data information that accompanies the PCAP files. - -![The content of each repository](/pcap-s3.png) -The repository file name includes the KFL query index and the UNIX timestamp it was generated. - -The repository includes a `content.json` file that includes some meta-data about the PCAP file. The meta-data includes: - -- KFL query index -- KFL query -- The L4 streams -- The PCAP file name -- The time of creation - -It looks something like this: - -```json -{ - "kfl_index": 1, - "kfl_query": "dns", - "l4_streams": [ - "000000007221_udp.pcap", - "000000007222_udp.pcap", - "000000007223_udp.pcap" - ], - "pcap_file_name": "1678819560.pcap", - "time": "Tue, 14 Mar 2023 18:46:13 GMT" -} -``` - -Next in the repository is the `name_resolution_history.json`. As L4 streams contain only IPs and no identities, this file includes a history log and timing of all the name to IP resolutions made at the node where the PCAP was generated. - -The file looks something like this: - -```json -{ - "1678818739": { - "10.103.193.44": { - "FullAddress": "kubeshark-front.kubeshark", - "Namespace": "kubeshark" - }, - "10.103.193.44:80": { - "FullAddress": "kubeshark-front.kubeshark", - "Namespace": "kubeshark" - }, - "10.106.97.98": { - "FullAddress": "kubeshark-hub.kubeshark", - "Namespace": "kubeshark" - } - } -} -``` -You can view the content of the repository using [Kubeshark](https://kubeshark.com) with the `kubeshark tap --pcap ` - -> Read more in the [PCAP](/en/pcap#view-the-pcap-snapshot) section diff --git a/src/pages/en/aws_ingress_auth copy.md b/src/pages/en/aws_ingress_auth copy.md deleted file mode 100644 index 318e708..0000000 --- a/src/pages/en/aws_ingress_auth copy.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -title: AWS EKS with TLS Termination -description: This article describes how to self host Kubeshark using Ingress and an IDP. -layout: ../../layouts/MainLayout.astro ---- - -We recommend using AWS [NLB](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) on EKS for best results. Classic and Application Load-balancers ([CLB](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/introduction.html) and [ALB](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html)) aren't likely to work. Follow these steps to self host [Kubeshark](https://kubeshark.com) on EKS, using and AWS Load Balancer, Ingress Controller with TLS termination. - -1. Install the AWS LoadBalancer Controller Add-on -2. Install the Nginx Ingress resource and controller of type NLB - -#### Installing the AWS LoadBalancer Controller Add-on - -1. Create an IAM policy -2. Create an IAM role. -3. Install the AWS Load Balancer Controller. -4. Check everything was installed correctly. - -Follow the steps in this [article](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) to install the AWS LB Controller add-on. - -Download an IAM policy for the AWS Load Balancer Controller that allows it to make calls to AWS APIs on your behalf. Once downloaded, use the AWS CLI to create an IAM policy: - -```shell -curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.7/docs/install/iam_policy.json - -aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam_policy.json -``` - -Now use `eksctl` to create an IAM role: - -```shell -eksctl create iamserviceaccount \ - --cluster= \ - --namespace= \ - --name=aws-load-balancer-controller \ - --role-name AmazonEKSLoadBalancerControllerRole \ - --attach-policy-arn=arn:aws:iam::78......10:policy/AWSLoadBalancerControllerIAMPolicy \ - --approve \ - --region us-east-2-this-is-an-example -``` - -With the IAM policy and role, you can use Helm to install the AWS Load Balancer Controller: - -```shell -helm repo add eks https://aws.github.io/eks-charts -helm repo update eks - -helm install kubeshark-ingress eks/aws-load-balancer-controller -n \ ---set clusterName= \ ---set serviceAccount.create=false \ ---set serviceAccount.name=aws-load-balancer-controller \ ---set region=us-east-2-this-is-an-example \ ---set vpcId= -``` -When you're done, verify all was installed correctly: - -```shell -kubectl get deployment -n aws-load-balancer-controller -``` - -Results should resemble this output: - -```shell -NAME READY UP-TO-DATE AVAILABLE AGE --aws-load-balancer-controller 1/1 1 1 40m -``` -#### Installing the Nginx Ingress Resource and Controller of Type NLB - -Firstly, add the Ingress resource: - -```shell -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml -``` - -Now download the file: - -```shell -curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml -``` - -Once downloaded, open in an editor and replace the following annotations: - -```shell -service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true" -service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-2:71.......90:certificate/73......7 -service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" -service.beta.kubernetes.io/aws-load-balancer-type: "external" -service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance" -service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" -``` - -Make sure you change the AWS Certificate Manager (ACM) ID as well: - -```shell -service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-2:71.......90:certificate/73......7 -``` - -Find this line and replace with the EKS cluster's VPC CIDR: - -```shell -proxy-real-ip-cidr: XXX.XXX.XXX/XX -``` - -Deploy the edited manifest: - -```shell -kubectl apply -f deploy.yaml -``` - -#### Troubleshooting - -Use [this document](https://repost.aws/knowledge-center/load-balancer-troubleshoot-creating) to troubleshoot. diff --git a/src/pages/en/aws_ingress_auth.md b/src/pages/en/aws_ingress_auth.md deleted file mode 100644 index 2748b2c..0000000 --- a/src/pages/en/aws_ingress_auth.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: AWS EKS, ALB with TLS Termination -description: This article describes how to self host Kubeshark using Ingress and an IDP. -layout: ../../layouts/MainLayout.astro ---- - -If you're using an AWS Application Load Balancer (ALB) as the Ingress Controller, there are specific considerations and adjustments you need to make in your environment and configuration. Below, I'll outline the key steps: - -## Install and Configure AWS ALB Ingress Controller - -1. Ensure you have a Kubernetes cluster running on AWS. - -2. Install the AWS ALB Ingress Controller by following the instructions provided in the [official documentation](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html). - -**2.1.** Another useful documentation can be found in [ArtifactHub](https://artifacthub.io/packages/helm/aws/aws-load-balancer-controller). - -3. Verify that the ALB controller is up and running and connected to your cluster. - -## Adjust Ingress Configuration - -1. Set the ingress class for the AWS ALB. In your configuration file, change the ingress class configuration to point to the ALB controller: - -```yaml -tap: - ingress: - annotations: {} - classname: "alb" -``` - -Some annotations should be required to properly make the ingress available. Should be something like this: - -```yaml -tap: - ingress: - annotations: - alb.ingress.kubernetes.io/target-type: ip - alb.ingress.kubernetes.io/scheme: internet-facing -``` - -## TLS Termination on AWS ALB - -1. To enable TLS using an AWS-managed certificate, you can configure the TLS section in your configuration file. Ensure that the domain configured in the `hosts` matches the domain associated with your ALB in AWS: - -```yaml -tap: - ingress: - tls: - - secretName: kubeshark-tls-secret - hosts: - - kubeshark.local -``` - -Another way to do so is to include the certificate's ARN in the ingress' annotation, so the certificate will be injected by the load balancer automatically: - -```yaml -tap: - ingress: - annotations: - alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:::certificate/XXXXXXXXXXXXXXX - classname: "alb" -``` - -## Install Kubeshark with Adjusted Configuration - -1. Install [Kubeshark](https://kubeshark.com) using the adjusted Helm Chart for the AWS ALB. Use the following command: - -```bash -helm install kubeshark kubeshark/kubeshark -f -``` - -Be sure to adjust `` to the actual path of your custom configuration file. - -## DNS and Security Group Considerations - -1. Ensure the domain you're using in the ingress (e.g., `kubeshark.local`) is pointing to the ALB's DNS in AWS. - -2. Review the ALB's Security Group settings to ensure the required ports are open for the appropriate traffic. - -Remember to refer to the official AWS ALB Ingress Controller documentation and AWS documentation for more detailed and up-to-date guidance, as details may change based on current versions and features. - -## Further notes - -When using the AWS ALB Ingress Controller, you typically don't need to add annotations directly to your Ingress resources for basic functionality. However, there are scenarios where you might want to use annotations to configure specific behaviors or features related to the ALB. Here are some common scenarios where annotations might be useful: - -1. **SSL Policy**: You can use the `alb.ingress.kubernetes.io/ssl-policy` annotation to specify the SSL policy for your ALB listener. This can control the supported SSL/TLS protocols and ciphers. - -2. **Target Group Attributes**: Annotations like `alb.ingress.kubernetes.io/target-group-attributes` can be used to define specific target group attributes, like stickiness or slow start settings. - -3. **Health Checks**: Annotations like `alb.ingress.kubernetes.io/healthcheck-*` can be used to configure health checks for your target groups. - -4. **Authentication**: For scenarios requiring AWS Cognito or other authentication mechanisms, you might need to use annotations to set up authentication. - -5. **Rules Priority**: If you have multiple Ingress resources and you want to control the priority of the rules, you can use `alb.ingress.kubernetes.io/conditions` annotation. - -Remember that annotations are controller-specific, and their usage might change with different versions of the ALB Ingress Controller. Always consult the official documentation for the most up-to-date information regarding annotations and their usage with the AWS ALB Ingress Controller. \ No newline at end of file diff --git a/src/pages/en/bsl_1_1.md b/src/pages/en/bsl_1_1.md deleted file mode 100644 index 6890cd7..0000000 --- a/src/pages/en/bsl_1_1.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Kubeshark Business Source License (BSL) 1.1 -description: -layout: ../../layouts/MainLayout.astro ---- - -**License**: BSL 1.1 - -**Licensor**: Kubeshark, Inc. - -**Licensed Work**: Kubeshark's Hub, Worker or Front, including the software components, or any portion of them, and any modification. - -**License**: License to use the Licensed Work. License can be obtained either by browsing to https://kubeshark.com/pricing, the [License Portal](https://console.kubeshark.com/) or by running `kubeshark pro` from the terminal. - -**Change Date**: December 31st, 2028 - -**Change License**: Apache License, Version 2.0 (https://www.apache.org/licenses/LICENSE-2.0), as published by the Apache Foundation. - -**Additional Use Grant (Grant):** - -This Grant outlines the terms and conditions for limited production use of the Licensed Work. By exercising the rights granted under this Grant, you ("Licensee") agree to be bound by its terms and conditions. - -**1. Limited Production Use Without Obtaining a License:** - -1.1 Licensee is hereby granted permission to use the Licensed Work, without obtaining a License, in a limited production environment, subject to the terms and conditions of the BSL 1.1 License and this Grant. - -1.2 The limited production use includes testing, evaluation, and deployment of the Licensed Work in a live environment. - -1.3 Licensee may use the Licensed Work for internal purposes and may provide access to the Licensed Work to its employees, contractors, or agents involved in the limited production use, subject to their compliance with the terms and conditions of the BSL 1.1 License and this Grant. - -1.4 Licensee can not use, provide, distribute, or make available the Licensed Work as a commercial offering, product, hosted, or managed service, that allows third parties (other than your own employees and contractors acting on your behalf) to access and/or use the Licensed Work or a substantial set of the features or functionality of the Licensed Work. - -1.5 Any additional use of the Licensed Work in a production environment requires Obtaining a License from the Licensor. - -**2. No Warranty:** - -2.1 Licensee acknowledges and agrees that the Licensed Work is provided on an "as-is" basis, without any warranties or guarantees of any kind, whether expressed or implied. - -2.2 Licensor disclaims all warranties, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non-infringement. - -2.3 Licensee understands that the Licensed Work is undergoing active development, and therefore may contain bugs, errors, or other issues that could result in system failures, data loss, or other adverse effects. - -**3. Use in Production:** - -3.1 Any use of the Licensed Work requires obtaining an Enterprise License. - -3.2 An Enterprise POC license, a trial or a Community license do not qualify as an Enterprise license and do not grant permission to use the Licensed Work in prodction. - -3.3 Licensee may choose to use the Licensed Work in a production environment at their own discretion, despite the active development status in condition, once Licensee has obtained an appropriate license from the Licensor. - -3.4 If Licensee decides to use the Licensed Work in a production environment, Licensee assumes all risks and responsibilities associated with such use. - -3.5 Licensee agrees that Licensor, shall not be held liable for any damages, losses, or liabilities arising from the use of the Product in a production environment, including but not limited to any interruptions, failures, or data inaccuracies. - -**4. Governing Law and Jurisdiction:** - -4.1 This Grant shall be governed by and construed in accordance with the laws of California. - -4.2 Any disputes arising out of or in connection with this Grant shall be subject to the exclusive jurisdiction of the courts of California. - -By using the Licensed Work in accordance with the terms and conditions of the BSL 1.1 License and this Additional Use Grant, Licensee acknowledges its acceptance of the limitations and disclaimers set forth herein. - -Please note that this Additional Use Grant is an extension to the BSL 1.1 License and should be read in conjunction with the original license. In the event of any conflict or inconsistency between the BSL 1.1 License and this Grant, this Grant shall prevail. - -## Text of BSL 1.1 - -The Licensor hereby grants you the right to copy, modify, create -derivative works, redistribute, and make non-production use of the -Licensed Work. The Licensor may make an Additional Use Grant, above, -permitting limited production use. - -Effective on the Change Date, or the forth anniversary of the first -publicly available distribution of a specific version of the Licensed -Work under this License, whichever comes first, the Licensor hereby -grants you rights under the terms of the Change License, and the rights -granted in the paragraph above terminate. - -If your use of the Licensed Work does not comply with the requirements -currently in effect as described in this License, you must purchase a -commercial license from the Licensor, its affiliated entities, or -authorized resellers, or you must refrain from using the Licensed Work. - -All copies of the original and modified Licensed Work, and derivative -works of the Licensed Work, are subject to this License. This License -applies separately for each version of the Licensed Work and the Change -Date may vary for each version of the Licensed Work released by -Licensor. - -You must conspicuously display this License on each original or modified -copy of the Licensed Work. If you receive the Licensed Work in original -or modified form from a third party, the terms and conditions set forth -in this License apply to your use of that work. - -Any use of the Licensed Work in violation of this License will -automatically terminate your rights under this License for the current -and all other versions of the Licensed Work. - -This License does not grant you any right in any trademark or logo of -Licensor or its affiliates (provided that you may use a trademark or -logo of Licensor as expressly required by this License). - -TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON AN “AS IS” -BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS, EXPRESS -OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND -TITLE. diff --git a/src/pages/en/cloud_forensics.md b/src/pages/en/cloud_forensics.md deleted file mode 100644 index 84967c6..0000000 --- a/src/pages/en/cloud_forensics.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Traffic Recording & Offline Analysis -description: Record Kubernetes traffic and perform offline investigations to hunt down performance and security issues with ease -layout: ../../layouts/MainLayout.astro ---- - -When the culprit doesn't readily reveal itself during observation, you can choose to record traffic based on a schedule or the occurrence of specific events or behaviors. -Traffic is recorded in PCAP format and uploaded to immutable file storage. PCAP files can be retained for extended periods and analyzed offline at the viewer's discretion. - -You can record traffic based on specific patterns and make this recorded traffic available for offline analysis. - -#### Historic Traffic Snapshot Analysis - -[Kubeshark](https://kubeshark.com) can retain the captured traffic over a long period of time, enabling [Kubeshark](https://kubeshark.com) to present a historic traffic snapshot. - -The example below presents traffic captured between two timestamps: - -![Historical Traffic](/history1.png) -> Read more about it in the [Traffic Recorder](/en/traffic_recorder) section. \ No newline at end of file diff --git a/src/pages/en/collaborative_incident_diagnosis.md b/src/pages/en/collaborative_incident_diagnosis.md deleted file mode 100644 index 1df7ffe..0000000 --- a/src/pages/en/collaborative_incident_diagnosis.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: Secure Collaborative Incident Diagnosis -description: Empower developers to participate in the production incident diagnosis process by granting them a secure, front-row view of live API traffic. -layout: ../../layouts/MainLayout.astro -mascot: Hello ---- - -Provide developers with secure and direct access to live API traffic, enabling them to engage in real-time diagnosis of production incidents. This access facilitates the replication of production issues in development and testing environments, thus streamlining the process. - -This approach reduces the burden on DevOps teams to replicate evidence of bugs and instances of suboptimal performance. - -## Secure - -Provide a secure gateway for users to access [Kubeshark](https://kubeshark.com) via their web browsers. This is authenticated through their corporate IDP, without requiring kubectl access. Authorization is managed to allow visibility of only those namespaces they are permitted to see, with sensitive data appropriately redacted. - -## Traffic Investigation Developer Portal - -Allow developers to troubleshoot issues in KUbernetes, via the web and without having to have `kubectl` access. - -## SSO or Social Authentication - -Authenticate users through with your corporate Identity Provider (IDP) using Security Assertion Markup Language (SAML). - -## Namespace-based Authorization - -Implement namespace-based authorization rules and sensitive data redaction to ensure users only access permissible content. - diff --git a/src/pages/en/config.md b/src/pages/en/config.md deleted file mode 100644 index 8844db5..0000000 --- a/src/pages/en/config.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: Configuration -description: Kubeshark configuration explained in detail. -layout: ../../layouts/MainLayout.astro ---- - -[Kubeshark](https://kubeshark.com) reads configuration elements both from the **CLI** and from a configuration file that can be located either in `$HOME/.kubeshark/config.yaml`. - -## The Config Command - -`kubeshark config` - The `config` command generates a ​**​[Kubeshark](https://kubeshark.com)​**​ config file with default values. -`kubeshark config -r` will read the existing config file, if one exists, and will merge its values into a new config file and save to the config file location (`~/.kubeshark/config.yaml`). - -## Popular Configuration Elements - -### Dashboard IP and Accessibility - -For security reasons, the default address of the proxy host for the [Kubeshark](https://kubeshark.com) dashboard is set to `127.0.0.1` that allows opening only local connections. - -```shell -tap: - proxy: - host: 127.0.0.1 -``` - -Alternatively, use the shell config option: -```shell -kubeshark tap --proxy-host 0.0.0.0 -``` - -Consider changing this address to `0.0.0.0` or any other publicly accessible IP, to allow public address. Keep in mind that access to the dashboard isn't encrypted or authenticated. - -When you run [Kubeshark](https://kubeshark.com) on a remote server, make sure ports `8898`-`8899` are open for external connections. - -### Kubeshark Resource Assignment Limits - -Use to change the amount of resources assigned to [Kubeshark](https://kubeshark.com). - -```shell -tap: - resources: - worker: - cpu-limit: 750m - memory-limit: 1Gi - cpu-requests: 50m - memory-requests: 50Mi - hub: - cpu-limit: 750m - memory-limit: 1Gi - cpu-requests: 50m - memory-requests: 50Mi -``` - -### Run Kubeshark Headless - -By default, the [Kubeshark](https://kubeshark.com) dashboard opens automatically. If you are running [Kubeshark](https://kubeshark.com) on a headless machine and you'd like NOT to open the dashboard, set `headless` to true. - -```shell -headless: true -``` - -Alternatively, use the shell config option: -```shell -kubeshark tap --set headless=true -``` - -### Public IP and Headless - -This is a very popular option that enables running [Kubeshark](https://kubeshark.com) on a public IP and without opening the its dashboard. - -```shell -tap: - proxy: - host: 0.0.0.0 -headless: true -``` -Alternatively, use the shell config option: -```shell -kubeshark tap --proxy-host 0.0.0.0 --set headless=true -``` -You can now access the [Kubeshark](https://kubeshark.com) dashboard from a remote server. - -As stated above, when you run [Kubeshark](https://kubeshark.com) on a remote server, make sure ports `8898`-`8899` are open for external connections. - -### Scripts - -Information related to [Kubeshark](https://kubeshark.com)'s scripting engine. For example: - -```shell -scripting: - env: - VAR-1: "VALUE" - VAR-2: 77 - source: "/path/to/script/folder" -``` - -#### Source Folders - -```shell -scripting: - source: "/path/to/script/folder" -``` -A path for the scripts folder. If not empty and pre-populated with scripts, all script will run when [Kubeshark](https://kubeshark.com) starts. - -#### Environment Variables - -This section includes variable that can be used inside of scripts. - -```shell -scripting: - env: - VAR-1: "VALUE" - VAR-2: 77 diff --git a/src/pages/en/data_persistency.md b/src/pages/en/data_persistency.md deleted file mode 100644 index befb0e5..0000000 --- a/src/pages/en/data_persistency.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: Data Persistency -description: -layout: ../../layouts/MainLayout.astro ---- - -### Storage Persistency -By default, [Kubeshark](https://kubeshark.com) employs ephemeral storage, as opposed to persistent storage. Consequently, all content will be lost in scenarios such as pod eviction, OOMkilled events, or any other circumstances that lead to pod restart. - -While completely optional, to mitigate the risk of data loss, it is advisable to utilize persistent volume claims, thereby ensuring the use of persistent storage: -```yaml -tap: - persistentStorage: true - storageClass: -``` -Having said the above, [Kubeshark](https://kubeshark.com) will still record traffic the same way without persistent storage. \ No newline at end of file diff --git a/src/pages/en/dns.md b/src/pages/en/dns.md deleted file mode 100644 index 85958d5..0000000 --- a/src/pages/en/dns.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: DNS Support -description: Kubeshark provides UDP layer visibility into Kubernetes’ DNS traffic by capturing and dissecting all UDP streams that include DNS traffic -layout: ../../layouts/MainLayout.astro ---- - -**Kubeshark** provides UDP layer visibility into [Kubernetes’ DNS traffic](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) by capturing all UDP streams that include DNS traffic. Once captured, DNS traffic is dissected and become available as any other protocol supported by **Kubeshark**. - -### DNS Log - -Use **Kubeshark** to view a DNS log and export into a PCAP file. To view only DNS entries, use: `dns` in the [filter](/en/v2/kfl2) input. Use the [export to PCAP button](/en/pcap#manual-pcap-export) to export the DNS traffic to a PCAP file. - -![DNS Log](/dns-log.png) - -### DNS Investigation - -As with any other protocol supported by **Kubeshark**, you can use a [KFL2](/en/v2/kfl2) query in conjunction with any property in the DNS payload to trace down the root of any incident. - -![DNS Investigation](/dns-investigation.png) - -### Service-to-DNS Connectivity Map - -Use the **Service Map** in conjunction with a DNS [KFL2](/en/v2/kfl2) query `dns` to see all the services that are connected to all the DNS providers in the cluster (e.g. kube-dns). - -![DNS Service Map](/dns-map.png) - -### DNS Stress Test - -**Kubeshark** stores the UDP stream that includes the DNS traffic. The stream can be used to replay the traffic aimed at the server destination. - -When used with the **load testing** option, it can be used to stress test your DNS provider (e.g. kube-dns) and see where it scales or congests. - -![DNS Replay](/dns-replay.png) - -To activate the stress test, put any number in the **replay count** and select the **Replay the UDP streams concurrently. (load testing)** option. diff --git a/src/pages/en/envoy.md b/src/pages/en/envoy.md deleted file mode 100644 index 187f2c1..0000000 --- a/src/pages/en/envoy.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: Istio & Envoy Support -description: Explore the various ways to receive support. -layout: ../../layouts/MainLayout.astro ---- -Tracing traffic and connections in an Istio-enabled cluster can be complex due to mTLS. [Kubeshark](https://kubeshark.com) provides full support for Istio/Envoy with one small caveat. - -![Istio enabled cluster](/envoy_description.png) - -In clusters where Istio is installed, sidecar containers with the Envoy proxy are injected into the targeted pods. These proxies act as network gateways. When mTLS is in STRICT mode, the proxies encrypt egress traffic and decrypt ingress traffic. - -For each pod, we have two types of communication: -- The **`service`** <==> `sidecar`: This includes the original service ingress and egress traffic. -- The `sidecar` <==> `sidecar`: This is a duplicate of the original **`service`** <==> `sidecar` communication. This traffic is encrypted when mTLS is in STRICT mode. - -## Layers of Traffic Visible in Kubeshark - -#### L4 Layer - -[Kubeshark](https://kubeshark.com) will show all L4 traffic related to Istio, including encrypted and mTLS traffic. - -#### L7 Layer - -#### **`container`** <==> `sidecar`: Protocol Support - -Protocol messages will be visible, and [Kubeshark](https://kubeshark.com) will show all reassembled API calls based on its protocol support. - -#### **`container`** <==> `sidecar`: TLS / HTTPS - -[Kubeshark](https://kubeshark.com) will display the **`container`** <==> `sidecar` traffic in clear text (decrypted), which includes the original service ingress and egress traffic, based on [Kubeshark](https://kubeshark.com)'s TLS termination library support. - -> Read more about TLS termination library support in the [TLS/HTTPS section](/en/encrypted_traffic). - -#### `sidecar` <==> `sidecar`: Unencrypted - -When mTLS is not in STRICT mode, this traffic will be visible. - -#### `sidecar` <==> `sidecar`: mTLS (Encrypted) - -When mTLS is in STRICT mode, the mTLS traffic will show as encrypted, and there will be no attempt to decrypt it, as it is assumed to be a duplicate of the original **`container`** <==> `sidecar` traffic. diff --git a/src/pages/en/filtering.md b/src/pages/en/filtering.md deleted file mode 100644 index 5bac863..0000000 --- a/src/pages/en/filtering.md +++ /dev/null @@ -1,206 +0,0 @@ ---- -title: Display Filters (KFL - Kubeshark Filtering Language) -description: The filter input enables filtering results using Kubeshark Filter Language (KFL). -layout: ../../layouts/MainLayout.astro ---- - -> KFL is a rich filtering language (not a query language) - -Inspired by Wireshark's display filters. Its primary purpose is to filter network traffic according to specific rules. Instead of searching for the needle in the haystack, it filters out the haystack to reveal the needle. - -**Kubeshark Filter Language (KFL)** is a language that enables you filter traffic that matches a boolean expression. For example; to only see the items with HTTP response status codes (`400` – `499`), enter: - -```python -http and response.status == r"4.*" -``` - -and click the **Apply** button. Your traffic stream will look like this: - -![Filter example](/filter-applied.png) - -## KFL vs. Pod Targeting (Display vs. Capture Filters) - -KFL should not be confused with [Pod Targeting](/en/pod_targeting) as they serve different purposes. KFL statements only affect the data presented in the Dashboard, hence they are referred as Display FIlters. Pod Targeting rules constitute CApture Filters. These rules determine which pods are targeted and, consequently, which traffic is tapped. - -For those familiar with Wireshark, KFL can be likened to Wireshark's Display Filters, and Pod Targeting to Wireshark's BPF (Berkeley Packet Filter) filters. - -## Queryable UI Elements - -When you hover over UI elements and they display a green plus sign, it means this element can be added to your query. Selecting an element with a green plus sign will add this element to the query. For example, selecting this queryable element: - -![Queryable UI Elements Example](/filter-ui-example.png) - -adds `response.status == 201` to your query and only displays `HTTP 201` responses in the live traffic streaming. - -## KFL Syntax Reference - -**Kubeshark Filter Language (KFL)** is the language implemented inside [`kubeshark/worker`](https://github.com/kubeshark/worker) that enables the user to filter the traffic efficiently and precisely. - -```python -http and request.method == "GET" and request.path != "/example" and (request.query.a > 42 or request.headers["x"] == "y") -``` - -The language as a whole evaluates into a boolean outcome, always. Such that the record which makes the boolean `true` is a record that matches the filter. - -There are certain helper methods that can do more than reduce into a boolean value. - -### Literals - -The language supports the following literals: - -- Nil `nil` -- Boolean `true` or `false` -- Number `42`, `3.14` etc. -- String `"hello world"` -- Regex `r"prefix.*"` - -### Operators - -Operations can be grouped (precedence) using parentheses `(...)` - -> *Note: Operators are evaluated from left to right, excluding parentheses `(...)`* - -The language supports the following operators: - -#### Logical - -`and`, `or` - -> *Note: `false and`, `true or` are evaluated fast* - -#### Equality - -`==`, `!=` - -#### Comparison - -`>=`, `>`, `<=`, `<` - -#### Unary - -`!`, `-` - -### Helpers - -Helpers in KFL are method invocations that enable filtering capability which cannot be provided through the syntax. These are the available helpers in KFL: - -#### `startsWith(string)` - -Returns `true` if the given selector's value starts with the `string`. e.g. `brand.name.startsWith("Chev")` - -#### `endsWith(string)` - -Returns `true` if the given selector's value ends with the `string`. e.g. `brand.name.endsWith("let")` - -#### `contains(string)` - -Returns `true` if the given selector's value contains the `string`. e.g. `brand.name.contains("ro")` - -#### `datetime(string) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time that's provided by the `string`. If the given date-time string is not in a recognized format then it evaluates to `false`. The format must be same as `"1/2/2006, 3:04:05 PM"` (`en-US`) e.g. `timestamp > datetime("10/19/2021, 6:29:02 PM")` - -It's equal to the `time.Now().UnixNano() / int64(time.Millisecond)` in Go. e.g. `1635190131000` - -#### `limit(integer)` - -Limits the number of records that are streamed back as a result of a query. Always evaluates to `true`. - -#### `json()` - -A decoding helper that decodes the given JSON field if it's possible. It's used through chain calls and the JSONPath that's wanted to be accessed can be chained to the end like; `response.body.json().brand.name == "Chevrolet"` - -#### `xml()` - -A decoding helper that decodes the given XML field if it's possible. It's used through chain calls and the dot-notation path that's wanted to be accessed can be chained to the end like; `response.body.xml().brand.name == "Chevrolet"` - -#### `redact(string...)` - -A record altering helper that takes N number of `string` typed arguments. The arguments are dot-notation paths and it replaces the value on each matching path with `[REDACTED]` string. - -The `json()` and `xml()` helpers are supported inside the arguments of the `redact` helper. - -#### `now() integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time now. - -#### `seconds(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` seconds before or after from now according to the sign of the argument. - -#### `minutes(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` minutes before or after from now according to the sign of the argument. - -#### `hours(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` hours before or after from now according to the sign of the argument. - -#### `days(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` days before or after from now according to the sign of the argument. - -#### `weeks(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` weeks before or after from now according to the sign of the argument. - -#### `months(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` months before or after from now according to the sign of the argument. - -#### `years(integer) integer` - -Returns the UNIX timestamp `integer` which is the equivalent of the time `integer` years before or after from now according to the sign of the argument. - -> *Note: Calling an undefined helper makes the whole expression collapse and evaluate to `false`.* - -### Selectors - -Selectors in KFL are JSONPath(s) that refer to the path in a JSON document. In KFL, every record is a JSON document. Any selector that does not match a path is evaluated to `false`. - -Following are some selector examples: - -- `brand.name` basic selector -- `request.path[1]` index selector -- `request.headers["a"] == "b"` key selector - -Selectors can be combined with a subset of helpers such that they evaluate into a boolean without any operator. e.g. `brand.name.startsWith("Chev")` These are the list of helpers that can be used with selectors: - -- `startsWith(string)` -- `endsWith(string)` -- `contains(string)` - -Such that instead of writing a boolean expression like `brand.name == "Chevrolet"`, one of the following can be written: - -```python -brand.name.startsWith("Chev") -brand.name.endsWith("let") -brand.name.contains("ro") -``` - -A selector can be compared to another selector as well. Such that for these given JSON document, filters like those can be written: - -JSON: `{"model":"Camaro","brand":{"name":"Chevrolet"},"year":2021,"salesYear":2021}` Filter: `year == salesYear` - -JSON: `{"model":"Camaro","brand":{"name":"Chevrolet"},"year":2021,"salesYear":2020}` Filter: `year != salesYear` - -> *Note: A selector (JSONPath) that couldn't be found in the JSON document makes the whole expression collapse and evaluate to `false`.* - -#### Wildcard Selector - -Wildcard(`*`) selectors are evaluated into arrays and the elements in the array are checked using the respective operator against whether any or all elements are matching the given criteria or not. So, for example; - -- for a JSON like `{"request":{"path":["api","v1","example"]}}` the query `request.path.* == "v1"` returns `true` -- for a JSON like `{"request":{"path":["api","v1","example"]}}` the query `request.path.* == "v2"` returns `false` -- for a JSON like `{"request":{"path":["api","v1","example"]}}` the query `request.path.* != "v2"` returns `true` -- for a JSON like `{"request":{"path":[1, 2, 3]}}` the query `request.path.* > 2` returns `true` -- for a JSON like `{"request":{"path":[1, 2, 3]}}` the query `request.path.* > 4` returns `false` - -The wildcard selector can be used to compare two JSON arrays. So, for a JSON like `{"request":{"path":[1, 2, 3]},"response":{"header":[1, 2, 3]}}` the query `request.path.* == response.header.*` checks for full value equality and returns `true`. - -Empty arrays are evaluated to `false`, the rest is `true`: - -- for a JSON like `{"request":{"path":[{"x":1}, {"x":2}, {"x":3}]}}` the query `request.path.*.x and true` returns `true`. The part of the query `request.path.*.x` evaluates to `[1, 2, 3]`. -- for a JSON like `{"request":{"path":[]}}` the query `request.path.* and true` returns `true` - -Please check out [the unit tests](https://github.com/up9inc/basenine/blob/main/server/lib/eval_test.go) of the language for more examples. diff --git a/src/pages/en/genai.md b/src/pages/en/genai.md deleted file mode 100644 index 23c2993..0000000 --- a/src/pages/en/genai.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: Generative-AI Assisted Network Insights -description: -layout: ../../layouts/MainLayout.astro -mascot: Hello ---- - -[Kubeshark](https://kubeshark.com) monitors all traffic at all time. -It triggers custom code that can process each API caall and can result in triggering an action. -For example, the custom code can search for a JAON field in an API call response, advance a count and perioidically send the count as a custom metric to grafana. - -Network processor \ No newline at end of file diff --git a/src/pages/en/half_connections.md b/src/pages/en/half_connections.md deleted file mode 100644 index 9857602..0000000 --- a/src/pages/en/half_connections.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: Network Error Detection & Analysis -description: Kubeshark is useful for detecting various network-related errors -layout: ../../layouts/MainLayout.astro -mascot: Hello ---- - -[Kubeshark](https://kubeshark.com) is useful for detecting various network-related errors, such as: - -- [TCP (Connection) Errors](#tcp-connection-errors) -- [Timeout Errors](#timeout-errors) -- [Packet Loss](#packet-loss) -- [ICMP](#icmp) -- [Dissection Errors](#dissection-errors) -- [L4 Stream Capture Timeout](#l4-stream-capture-timeout) -- [Half Connections](#half-connections) -- [Error Filter](#error-filter) -- [Tip: Error Dashboard](#tip-error-dashboard) - -In addition to offering details about detected errors, [Kubeshark](https://kubeshark.com) allows you to record raw traffic for in-depth analysis using tools such as Wireshark - -> Learn more in the [Traffic Recorder](/en/traffic_recorder) section. - -## TCP (Connection) Errors - -Connection errors, identified as part of the TCP protocol, include: - -| Error | Description | -| --- | --- | -| SynSent | The client attempted to establish a connection, but the connection was refused. SYN sent, but ACK not received. | -| CloseWait | One side closed the connection with FIN, but the final ACK confirmation is pending. | -| LastAck | One side received a FIN from the peer, sent an ACK, sent a FIN, and is waiting for the final ACK from the peer. | -| Reset | A RST packet is seen, indicating that one side will neither accept nor send more data. | - -![TCP Errors](/tcp_error.png) - -## Timeout Errors - -[Kubeshark](https://kubeshark.com)'s L4 stream capture times out after 10 seconds. If an L4 stream capture doesn't complete within this timeframe, the stream is marked stale and dropped. THe timeout time is configurable and can be set using this configuration value: - -```yaml -tap: - misc: - tcpStreamChannelTimeoutMs: 10000 # in milliseconds -``` -or use: `--set tap.misc.tcpStreamChannelTimeoutMs=10000` - -## Packet Loss - -Packet loss can cause [Kubeshark](https://kubeshark.com) L4 streams to timeout and parser errors. Additionally, lost packets might result in no symptoms and no signals. - -Potential causes of packet loss: - -| Cause | Description | -|---|---| -| Insufficient CPU resources for [Kubeshark](https://kubeshark.com) | Packet loss can occur when [Kubeshark](https://kubeshark.com) is unable to read packets quickly enough, often due to a lack of CPU resources. | -| AF-PACKET vs PF-RING | AF-PACKET is prone to packet loss, whereas [PF-RING](/en/performance#af-packet-and-pf-ring) is less likely to cause it.| - -## ICMP - -[Kubeshark](https://kubeshark.com) intercepts, dissects, and presents [ICMP](https://datatracker.ietf.org/doc/html/rfc792) messages. ICMP is an L4 protocol used for error reporting and network diagnostics. For instance, if a router cannot forward a packet because the destination is unreachable, an ICMP message is sent back to the sender indicating the problem. - -Partial list of errors reported by ICMP messages: - -**Destination Unreachable**: Sent when a packet cannot be delivered to its destination for various reasons, with subcodes including: - -- Network Unreachable -- Host Unreachable -- Protocol Unreachable -- Port Unreachable -- Fragmentation Needed and DF (Don't Fragment) set -- Source Route Failed -- Destination Network Unknown -- Destination Host Unknown -- Source Host Isolated -- Network Administratively Prohibited -- Host Administratively Prohibited -- Network Unreachable for Type of Service -- Host Unreachable for Type of Service - -**Time Exceeded**: Indicates that the Time to Live (TTL) field of the packet has reached zero, necessitating discarding the packet. This is commonly used in network route tracing. - -**Source Quench**: A deprecated message requesting the sender to decrease the message sending rate due to router or host congestion. - -**Redirect Message**: Sent by routers to indicate a more efficient packet routing path. - -**Parameter Problem**: Indicates an error in the IP packet header fields, preventing processing. - -Router Advertisement and Router Solicitation: Used for routing information discovery and announcement. - -![ICMP reported errors](/icmp_error.png) - -## Dissection Errors - -Dissection errors are reported by [Kubeshark](https://kubeshark.com) protocol parsers: - -| Error | Description | -|---|---| -| unexpected EOF | TCP connection closed unexpectedly. | -| Parser error | [Kubeshark](https://kubeshark.com)'s application layer protocol parser reports an invalid payload according to the protocol definition. | - -## L4 Stream Capture Timeout -[Kubeshark](https://kubeshark.com) will drop entire Layer 4 (L4) streams if not all packets are captured before the process times out. - -The default timeout is set to 10 seconds. This can be modified by setting the `tap.misc.tcpStreamChannelTimeoutMs` parameter. - -## Half Connections - -Half connections represent incomplete transactions where either a request or a response is missing. - -![Half Connection Example](/half_connections.png) - -[Kubeshark](https://kubeshark.com) aims to identify the reasons for these incomplete transactions. - -## Error Filter - -Use [Kubeshark](https://kubeshark.com) Filter Language (KFL) to selectively display or filter out various errors. For example: - -```yaml -!error # Filters out any half-connections -!icmp # Filters out any ICMP entries -!tcp # Filters out any TCP level errors -``` - -## Tip: Error Dashboard -Enter the following KFL in the KFL box: -`response.status > 400 or error or icmp or tcp`, apply, copy and enter to favorite. -The link should look like this: -```yaml -http://:/?q=response.status%20%3E%20400%20or%20error%20or%20icmp%20or%20tcp -``` -It will look even better on a 60" screen. \ No newline at end of file diff --git a/src/pages/en/headless.md b/src/pages/en/headless.md deleted file mode 100644 index 9a49c9c..0000000 --- a/src/pages/en/headless.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Running Kubeshark Headless -description: -layout: ../../layouts/MainLayout.astro ---- - - -[Kubeshark](https://kubeshark.com) can run headless as well as serve a rich GUI dashboard. When running headless (no dashboard), the following functionalities can still remain active: -1. Monitoring traffic using scripting. -2. Continuous PCAP recording. - -## Monitoring Traffic Using Scripting - -Using [scripting](/en/automation_scripting), [Kubeshark](https://kubeshark.com) can be employed for monitoring purposes and triggering actions when certain network behaviors are detected. - -> Read more in the [scripting section](/en/automation_scripting). - -## Continuous PCAP Recording - -Additionally, running [Kubeshark](https://kubeshark.com) headless allows all captured traffic to be recorded in PCAP files, which can be exported on demand. Recorded traffic will include everything captured by [Kubeshark](https://kubeshark.com), considering the [capture filters](/en/pod_targeting), and will also include [TLS traffic](/en/encrypted_traffic), as well as traffic from Envoy or Istio. - -> Read more in the [PCAP Dumper section](/en/pcapdump). - -## Low Resource Consumption - -When running headless, [Kubeshark](https://kubeshark.com) consumes significantly fewer resources than when there is an active dashboard connection. - -## Opening Dashboards On-Demand - -You can still open any number of dashboards, even when [Kubeshark](https://kubeshark.com) is running headless. The dashboard operation is independent of the services provided by the [Kubeshark](https://kubeshark.com) backend (e.g., scripts and recording). - -## Important Configuration Values - -It's important to consider the following configuration values when planning to run headless: - -```yaml -tap: - capture: - dissection: - enabled: true # Ensure dissection is active | default is `true` - # Capture filters instruct [Kubeshark](https://kubeshark.com) on what traffic to capture - regex: catal.* # Only traffic from pods matching the regex will be captured | default is `.*` - namespaces: # Capture from these namespaces | default is ALL - - ns1 - - ns2 - excludedNamespaces: # Exclude these namespaces | default is none - - ns3 - bpfOverride: net 10.10.0.0/16 # Or simply use an override BPF filter | default is none -pcapdump: # Ensure traffic is recorded at all times (optional, as this is the default) - enabled: true # Store captured traffic in PCAP files (optional, as this is the default) - maxTime: 1h # Discard files older than 1 hour (optional, as this is the default) - maxSize: 500MB # Discard old files if storage exceeds 500MB (optional, as this is the default) - -scripting: # If you're using scripts - env: - VAR1: -``` \ No newline at end of file diff --git a/src/pages/en/install_helm.md b/src/pages/en/install_helm.md deleted file mode 100644 index c02c156..0000000 --- a/src/pages/en/install_helm.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Install with Helm -description: Install and run Kubeshark inside your Kubernetes cluster in seconds. -layout: ../../layouts/MainLayout.astro -mascot: Cute ---- - -> Read the [Helm section](https://github.com/kubeshark/kubeshark/blob/master/helm-chart/README.md) for most up-to-date instructions - -While the [CLI](/en/install) is a great option for on-demand usage and running on dev & test clusters, for a more permanent deployment, you can use [Helm](https://helm.sh/) and add [Kubeshark](https://kubeshark.com)'s Helm repository: - -## Official - -Add the Helm repo for [Kubeshark](https://kubeshark.com): - -```shell -helm repo add kubeshark https://helm.kubeshark.com -``` - -then install [Kubeshark](https://kubeshark.com): - -```shell -helm install kubeshark kubeshark/kubeshark -``` - -## Local - -Clone the repo: - -```shell -git clone git@github.com:kubeshark/kubeshark.git --depth 1 -cd kubeshark/helm-chart -``` - -Render the templates - -```shell -helm template . -``` - -Install [Kubeshark](https://kubeshark.com): - -```shell -helm install kubeshark . -``` - -Uninstall [Kubeshark](https://kubeshark.com): - -```shell -helm uninstall kubeshark -``` -## Configuration - -For complete and up-to-date configuration options, view the repo: https://github.com/kubeshark/kubeshark/tree/master/helm-chart - -## Port-forward - -Do the port forwarding: - -```shell -kubectl port-forward service/kubeshark-front 8899:80 -``` - -Visit [localhost:8899](http://localhost:8899) diff --git a/src/pages/en/install_manifest.md b/src/pages/en/install_manifest.md deleted file mode 100644 index c02c156..0000000 --- a/src/pages/en/install_manifest.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Install with Helm -description: Install and run Kubeshark inside your Kubernetes cluster in seconds. -layout: ../../layouts/MainLayout.astro -mascot: Cute ---- - -> Read the [Helm section](https://github.com/kubeshark/kubeshark/blob/master/helm-chart/README.md) for most up-to-date instructions - -While the [CLI](/en/install) is a great option for on-demand usage and running on dev & test clusters, for a more permanent deployment, you can use [Helm](https://helm.sh/) and add [Kubeshark](https://kubeshark.com)'s Helm repository: - -## Official - -Add the Helm repo for [Kubeshark](https://kubeshark.com): - -```shell -helm repo add kubeshark https://helm.kubeshark.com -``` - -then install [Kubeshark](https://kubeshark.com): - -```shell -helm install kubeshark kubeshark/kubeshark -``` - -## Local - -Clone the repo: - -```shell -git clone git@github.com:kubeshark/kubeshark.git --depth 1 -cd kubeshark/helm-chart -``` - -Render the templates - -```shell -helm template . -``` - -Install [Kubeshark](https://kubeshark.com): - -```shell -helm install kubeshark . -``` - -Uninstall [Kubeshark](https://kubeshark.com): - -```shell -helm uninstall kubeshark -``` -## Configuration - -For complete and up-to-date configuration options, view the repo: https://github.com/kubeshark/kubeshark/tree/master/helm-chart - -## Port-forward - -Do the port forwarding: - -```shell -kubectl port-forward service/kubeshark-front 8899:80 -``` - -Visit [localhost:8899](http://localhost:8899) diff --git a/src/pages/en/integration_kubeshark_logs.md b/src/pages/en/integration_kubeshark_logs.md deleted file mode 100644 index e37ff0f..0000000 --- a/src/pages/en/integration_kubeshark_logs.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: Aggregate Custom Logs -description: The easiest way to aggregate logs is by using Kubeshark logs. -layout: ../../layouts/MainLayout.astro ---- - -## Aggregate Custom Logs - -The easiest way to aggregate logs is by using [Kubeshark](https://kubeshark.com) logs. You can aggregate custom logs to either the Hub's log or the Workers' logs. - -Using a helper, you can print any string, JSON object, or a combination of both to [Kubeshark](https://kubeshark.com) logs with a custom log level. - -### Example - -Here’s a quick example of how to use the various methods: - -```javascript -if (utils.nodeName() == "hub") { // Send to the Hub's logs - aggregation.msg("AP0", "Hello world " + i + " " + d); - aggregation.json("AP1", "Interface", { - "key": i, - "data": d - }); -} else { // Send to all Workers' logs - aggregation.jsonWithMsg("AP2", "Interface", { - "key": i, - "data": d - }, "Hello world " + i + " " + d); -} -``` - -### Log Output Example - -Here’s an example of how such logs might appear in [Kubeshark](https://kubeshark.com) logs: - -```shell -2024-12-14T19:02:45-08:00 APP Interface={"data":6,"key":12} -2024-12-14T19:02:45-08:00 APP Interface={"data":11,"key":12} -2024-12-14T19:02:45-08:00 APP Hello world 12 6 Interface={"data":6,"key":12} -2024-12-14T19:02:45-08:00 AP0 Hello world 12 2 -2024-12-14T19:02:45-08:00 APP Hello world 12 11 Interface={"data":11,"key":12} -2024-12-14T19:02:45-08:00 AP1 Interface={"data":0,"key":12} -2024-12-14T19:02:45-08:00 APP Hello world 12 13 Interface={"data":13,"key":12} -2024-12-14T19:02:45-08:00 APP Interface={"data":17,"key":12} -2024-12-14T19:02:45-08:00 APP Interface={"data":16,"key":12} -2024-12-14T19:02:45-08:00 APP Hello world 12 17 Interface={"data":17,"key":12} -2024-12-14T19:02:45-08:00 AP2 Hello world 12 3 Interface={"data":3,"key":12} -2024-12-14T19:02:45-08:00 AP0 Hello world 12 14 -2024-12-14T19:02:45-08:00 APP Hello world 12 16 Interface={"data":16,"key":12} -2024-12-14T19:02:45-08:00 AP2 Hello world 12 7 Interface={"data":7,"key":12} -2024-12-14T19:02:45-08:00 AP1 Interface={"data":2,"key":12} -``` diff --git a/src/pages/en/integrations_aws_s3.md b/src/pages/en/integrations_aws_s3.md deleted file mode 100644 index 0703f49..0000000 --- a/src/pages/en/integrations_aws_s3.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Upload Files to AWS S3 -description: Kubeshark enables you to generate files (e.g. PCAPs) and upload them to an immutable datastore (e.g. AWS S3). -layout: ../../layouts/MainLayout.astro ---- - -The AWS S3 integration provides an immutable datastore option in case you want to export files outside of the Kubernetes cluster. - -You can read the [helper](/en/automation_helpers) section to learn more about the available AWS S3 helpers. - -The most common helper would be the [`vendor.s3.put`](/en/automation_helpers#vendors3putregion-string-keyid-string-accesskey-string-bucket-string-path-string-string) helper that uploads a file using the provided credentials. - -## Use Specific Auth Credentials -```js -vendor.s3.put( - env.S3_BUCKET, - tarFile, - env.AWS_REGION, - env.AWS_ACCESS_KEY_ID, - env.AWS_SECRET_ACCESS_KEY -); -``` - -> Read more about the required AWS credentials [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). - -## Shared Auth - -In cases, when shared authentication is used (e.g. IRSA, kube2iam), S3 helper will use the default authentication and not require specific AWS Auth credentials. - -```js -vendor.s3.put( - env.S3_BUCKET, - tarFile, - env.AWS_REGION -); -``` - -> Read more about [IRSA](/en/irsa) in this article. diff --git a/src/pages/en/integrations_azure.md b/src/pages/en/integrations_azure.md deleted file mode 100644 index b8a7738..0000000 --- a/src/pages/en/integrations_azure.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: Upload Files to Azure Blob Storage -description: Kubeshark enables you to generate files (e.g. PCAPs) and upload them to an immutable datastore (e.g. Azure Blob Storage). -layout: ../../layouts/MainLayout.astro ---- - -The Azure Blob Storage integration provides an immutable datastore option in case you want to export files outside of the Kubernetes cluster. - -You can read the [helper](/en/automation_helpers) section to learn more about the available Azure helpers. - -The most common helper would be the `vendor.azure.put` helper that uploads a file using various authentication methods. - -## Authentication Methods - -The Azure integration supports multiple authentication methods: - -### 1. Using Connection String -```js -vendor.azure.put( - env.AZURE_CONTAINER_NAME, - tarFile, - "", // accountName (not needed with connection string) - "", // accountKey (not needed with connection string) - env.AZURE_CONNECTION_STRING -); -``` - -### 2. Using Account Name and Key -```js -vendor.azure.put( - env.AZURE_CONTAINER_NAME, - tarFile, - env.AZURE_ACCOUNT_NAME, - env.AZURE_ACCOUNT_KEY -); -``` - -### 3. Using Default Azure Credentials (Managed Identity) -```js -vendor.azure.put( - env.AZURE_CONTAINER_NAME, - tarFile, - env.AZURE_ACCOUNT_NAME -); -``` - -## Configuration Examples - -### Connection String Method -In either `config.yaml` or `values.yaml`, the environment variables should look like this: -```shell -scripting: - env: - AZURE_CONTAINER_NAME: - AZURE_CONNECTION_STRING: "DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net" -``` - -### Account Name and Key Method -```shell -scripting: - env: - AZURE_CONTAINER_NAME: - AZURE_ACCOUNT_NAME: - AZURE_ACCOUNT_KEY: -``` - -### Managed Identity Method -```shell -scripting: - env: - AZURE_CONTAINER_NAME: - AZURE_ACCOUNT_NAME: -``` - -## Function Signature - -The `vendor.azure.put` function accepts the following parameters: - -- `containerName` (string, required): The name of the Azure Blob Storage container -- `path` (string, required): The local file path to upload -- `accountName` (string, optional): The Azure storage account name -- `accountKey` (string, optional): The Azure storage account key -- `connectionString` (string, optional): The Azure storage connection string - -## Return Value - -The function returns the blob URL of the uploaded file: -``` -https://.blob.core.windows.net//_/ -``` - -## Notes - -- Files are automatically prefixed with the node name and IP address for organization -- The function automatically detects the content type of the uploaded file -- When using managed identity, ensure your cluster has the appropriate Azure identity configuration -- The container must exist before uploading files to it \ No newline at end of file diff --git a/src/pages/en/integrations_credentials.md b/src/pages/en/integrations_credentials.md deleted file mode 100644 index 788dc18..0000000 --- a/src/pages/en/integrations_credentials.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Customer-provided Credentials -description: Credentials required for integrations. -layout: ../../layouts/MainLayout.astro ---- - -**kubeshark** keeps data secure by working on-prem and exporting data only to customer-provided systems using customer-provided credentials. - -Some integrations requires credentials and require obtaining its credentials prior to using it. - -## Influx DB - -[InfluxDB](https://en.wikipedia.org/wiki/InfluxDB) is a highly popular [time series database](https://en.wikipedia.org/wiki/Time_series_database) that is used to store metrics. InfluXDB can easily be configured to be used as a data source in applications such as [Grafana](https://grafana.com/). If you don't have access to an InfluxDB instance, you can follow the instructions in the [InfluxDB and Grafana](/en/integrations_influxdb#practical-example) section to get one. - -The following credentials are necessary: -- InfluxDB URL -- InfluxDB token - -> Learn more about the InfluxDB integration and the required credentials [here](/en/integrations_influxdb). - -## Elasticsearch - -Elasticsearch is a popular schema free JSON document repository. - -To access [Elastic Cloud](https://www.elastic.co/cloud), the following credentials are necessary: - -- Elasticsearch Cloud ID -- ELasticsearch API Key - -> For a self-hosted Elasticsearch instance, different authentication strategies are possible. See [`vendor.elastic`](/en/automation_helpers#vendorelasticurl-string-index-string-data-object-username-string-password-string-cloudid-string-apikey-string-servicetoken-string-certificatefingerprint-string) helper for more info. - -## AWS S3 - -AWS S3 can be used to export forensics in the form of PCAP files (or any other files) to an immutable datastore. - -The following credentials are necessary to use AWS S3: - -- AWS Access Key ID -- AWS Secret Access Key - -Additional required AWS S3 properties: -- AWS Region -- AWS S3 Bucket - -> Learn more about the AWS S3 integration and the required credentials [here](/en/integrations_aws_s3). - -## Slack - -Slack is useful to send alerts, whether in real-time or once a certain task is completed. - -The following credentials are necessary: - -- Incoming Webhook URL address; or -- OAuth Token & Channel ID - -> Learn more about the Slack integration and the required credentials [here](/en/integrations_slack). - -## Keeping Credentials Secure - -As these credentials are confidential, we highly recommend to keep them in the [Kubeshark](https://kubeshark.com) configuration file as environment variables. - -Here's an example of a suggested configuration file clause: - -```yaml -configpath: /home/xxx/.kubeshark/config.yaml -headless: false -license: FT7YKAYBAE.... -scripting: - env: - AWS_ACCESS_KEY_ID: AKIA... - AWS_REGION: us-east-2 - AWS_SECRET_ACCESS_KEY: YZv.. - INFLUXDB_TOKEN: edO4... - INFLUXDB_URL: http://10.0.0.99:8086 - S3_BUCKET: my-bucket-name - SLACK_AUTH_TOKEN: xoxb-43... - SLACK_CHANNEL_ID: C0.. - WEBHOOK_URL: https://webhook.site/08c9a... -``` diff --git a/src/pages/en/integrations_elastic.md b/src/pages/en/integrations_elastic.md deleted file mode 100644 index 4490130..0000000 --- a/src/pages/en/integrations_elastic.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: Send Logs to Elasticsearch -description: Elasticsearch integration enables persistent storage and search in an Elasticsearch instance. -layout: ../../layouts/MainLayout.astro ---- - -> This integration is part of the [Pro edition](https://kubeshark.com/pricing). - -This integration enables you to send schema-free JSON documents directly to any [Elasticsearch](https://en.wikipedia.org/wiki/Elasticsearch) local or cloud-hosted instance by using the [`vendor.elastic`](/en/automation_helpers#vendorelasticurl-string-index-string-data-object-username-string-password-string-cloudid-string-apikey-string-servicetoken-string-certificatefingerprint-string) helper. - - - -## Sending Select Traffic Logs to Elasticsearch - -A ready-to-run script that enables you to send select traffic logs to Elasticsearch can be found [here](https://github.com/kubeshark/scripts/tree/master/telemetry#send-select-traffic-logs-to-elasticsearch). - -```js -var kflQuery = "gql and (src.name == 'my-pod-name' or dst.name == 'my-pod-name')"; -var ACTIVE = true; // change to false to disable this script - -// Use environment variables (recommended) or change these variables locally -var elaIdx = env.ELASTIC_INDEX; -var elaCloId = env.ELASTIC_CLOUD_ID; -var elaApiKey = env.ELASTIC_API_KEY; - -function onItemCaptured(data) { - if (!ACTIVE) return; - try{ - if (kflQuery.match(kflQuery, data)){ - vendor.elastic( - "", // URL is ignored for Elastic Cloud - elaIdx, - data, // Payload - "", // Username is ignored for Elastic Cloud - "", // Password is ignored for Elastic Cloud - elaCloId, - elaApiKey - ); - } - } - catch(error){ - console.error("Elastic Traffic Logs", error); - } -} -``` - -Change the KFL query and the Elasticsearch Authentication variables at the top of the script to fit your use-case and put the script in the script folder. - -## TL;DR - -The script uses the [`onItemCaptured`](/en/automation_hooks#onitemcaptureddata-object) hook, the [`kfl.match` helper](https://docs.kubeshark.com/en/automation_helpers#kflmatchquery-string-data-object-boolean) in conjunction with a [KFL](https://docs.kubeshark.com/en/filtering) query to identify the select traffic logs. - -For example, the following KFL query can be used to filter pod specific ingress/egress GraphQL traffic: -```js -gql and (src.name == "my-pod-name" or dst.name == "my-pod-name") -``` - -The `vendor.elastic` helper is used to send the traffic logs to an Elasticsearch cloud instance. - -## Prerequisites - -Using the Elasticsearch integration requires the Pro edition and is currently supported only by the CLI. If you haven't done so: -1. Install the CLI, by following [these instructions](/en/install#cli). -2. Sign up to the Pro edition by running the `pro` command: -```shell -kubeshark pro -``` -> More information about upgrading to the Pro edition can be found in the [Upgrading & Downgrading](/en/pro_upgrade) section. - -Ensure your configuration file includes the following configuration at a minimum: -```shell -license: FT7YKAYBAE****************AA= -scripting: - env: - ELASTIC_CLOUD_ID: - ELASTIC_API_KEY: - ELASTIC_INDEX: - source: /path/to/your/script/folder/ - watchScripts: true -``` - - - - diff --git a/src/pages/en/integrations_gcs.md b/src/pages/en/integrations_gcs.md deleted file mode 100644 index c58d209..0000000 --- a/src/pages/en/integrations_gcs.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: Upload Files to Google Cloud Storage -description: Kubeshark enables you to generate files (e.g. PCAPs) and upload them to an immutable datastore (e.g. GCS). -layout: ../../layouts/MainLayout.astro ---- - -The GCS integration provides an immutable datastore option in case you want to export files outside of the Kubernetes cluster. - -You can read the [helper](/en/automation_helpers) section to learn more about the available GCS helpers. - -The most common helper would be the [`vendor.gcs.put`](/en/automation_helpers#vendorgcsputbucket-string-path-string-sakeyobj-json-string) helper that uploads a file using a service account key JSON content. - -## Use Specific Auth Credentials -```js -vendor.gcs.put( - env.GCS_BUCKET, - tarFile, - JSON.parse(env.GCS_SA_KEY_JSON) -); -``` - -In either `config.yaml` or `values.yaml`, the environment variable should look something like this: -```shell -scripting: - env: - GCS_BUCKET: - GCS_SA_KEY_JSON : '{ - "type": "service_account", - "project_id": , - "private_key_id": "14a....81", - "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0.....HBJsfVHn\nRvUJH6Yxdzv3rtDAYZxgNB0=\n-----END PRIVATE KEY-----\n", - "client_email": "k..om", - "client_id": "104..72", - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://oauth2.googleapis.com/token", - "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", - "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/k...nt.com", - "universe_domain": "googleapis.com" -}' -``` \ No newline at end of file diff --git a/src/pages/en/integrations_influxdb.md b/src/pages/en/integrations_influxdb.md deleted file mode 100644 index 1c9253e..0000000 --- a/src/pages/en/integrations_influxdb.md +++ /dev/null @@ -1,136 +0,0 @@ ---- -title: Send Metrics to InfluxDB -description: Kubeshark enables you to send metrics directly to any InfluxDB local instance or cloud-hosted solution using the InfluxDB helper. You can then use InfluxDB's capabilities as a time-series database for reporting and further aggregation (e.g. to Grafana). -layout: ../../layouts/MainLayout.astro ---- - -[Kubeshark](https://kubeshark.com) enables you to send metrics directly to any [InfluxDB](https://www.influxdata.com/) local instance or cloud-hosted solution using the InfluxDB helper. You can then use InfluxDB's capabilities as a time-series database for reporting and further aggregation (e.g. to Grafana). - -## Sending Identity-aware Latency Information - -The following example utilizes the [`onItemCaptured`](/en/automation_hooks#onitemcaptureddata-object) hook and calls [`vendor.influxdb`](/en/automation_helpers#vendorinfluxdburl-string-token-string-organization-string-bucket-string-measurement-string-data-object-tags-object) to send latency and status-code data to an InfluxDB instance on every API call. In addition to the latency and status metrics, each message includes the service, path and namespace as tags to enable easy filtering in InfluxDB. - -```js -function onItemCaptured(data) { - if (data.protocol.name !== "http") return; // ignore non-HTTP traffic - - vendor.influxdb( - env.INFLUXDB_URL, - env.INFLUXDB_TOKEN, - "my-org-name", // Organization - "my-bucket-name", // Bucket - "PerformanceKPIs" , // Measurement - { - latency: data.elapsedTime - status: data.response.status - }, // Key-Value Metrics - { - service: data.dst.name, - path: data.request.path, - namespace: data.namespace - } // Key-Value Tags - ); -} -``` -The example assumes the key properties that are required for authentication are stored in the [Kubeshark](https://kubeshark.com)'s configuration file as [environment variables](/en/config#scripts). - -> See [`vendor.influxdb`](/en/automation_helpers#vendorinfluxdburl-string-token-string-organization-string-bucket-string-measurement-string-data-object-tags-object) for more info. - -Read the [onItemCaptured](/en/automation_hooks#onitemcaptureddata-object) hook section to learn more about data that becomes available when using this hook. - -## Practical Example - -Installing a local instance of InfluxDB is pretty straight forward and shouldn't take more than a few minutes. - -Follow the [InfluxDB's documentation](https://docs.influxdata.com/influxdb/v2.6/install/) to install a local instance or go to [InfluxData Website](https://www.influxdata.com/), the company behind InfluxDB to sign up and use a cloud-hosted version. - -### Install a local Instance - -As an example, you can use the following command to install a local instance of InfluxDB on Mac OS and then start it: - -```shell -brew update -brew install influxdb -influxd -``` -### Retrieving the Required Properties - -To send a message to your InfluxDB instance you need the following properties: -- InfluxDB URL -- API Token -- Organization -- A bucket -- Measurement -- Metrics: Key-value set -- Tags: Key-value set - -Follow the [InfluxDB's documentation](https://docs.influxdata.com/influxdb/v2.6/get-started/setup/) to create the following: -- API Token -- Organization -- A bucket - -The **InfluxDB URL** is simply the instance's URL that can be copied from the browser once you log in to your instance. - -The other three properties (e.g. Measurement, Metrics, Tags) can be defined on the run. - -While you can created numerous metrics, queries and graphs, some properties are unlikely to change and therefore it is recommended to keep them in the [Kubeshark](https://kubeshark.com) configuration file under the environment variable section. - -### API Call Latency Query and a Graph - -This query presents a latency graph for each API call including the service name, the path and namespace of each data point. Having these tags is useful to filter the APIs based on service, path and namespace properties. - -```shell -from(bucket: "Metrics") - |> range(start: v.timeRangeStart, stop: v.timeRangeStop) - |> filter(fn: (r) => r["_measurement"] == "PerformanceKPIs") - |> filter(fn: (r) => r["_field"] == "latency") - |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) - |> yield(name: "mean") -``` -This would be the graph resulting from this query: - -![InfluxDB Latency Graph](/influx-latency.png) - -## API Call Status Code Query and a Graph - -The following query, which is almost similar to the previous one, uses the data sent using the `onItemCaptured` hook (see a couple of paragraph above) to present the status code of each API call with the path and service name information included for every data point. - -```shell -from(bucket: "Metrics") - |> range(start: v.timeRangeStart, stop: v.timeRangeStop) - |> filter(fn: (r) => r["_measurement"] == "PerformanceKPIs") - |> filter(fn: (r) => r["_field"] == "status") -``` -![InfluxDB Status Codes](/influxdb-status-codes2.png) - -## Connecting to Grafana - -Metrics that are transmitted to InfluxDB can be aggregated to be used in Grafana, when selecting your InfluxDB instance as a source. - -### Adding InfluxDB as a Data Source in Grafana - -Follow one of these resources to add your InfluxDB instance to Grafana as a data source: - -- [https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-influxdb/](https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-influxdb/) -- [https://www.influxdata.com/blog/getting-started-influxdb-grafana/](https://www.influxdata.com/blog/getting-started-influxdb-grafana/) - -![Grafana InfluxDB Data Source](/grafana-influxdb-data-source.png) - -Go ahead, copy and paste the query from InfluxDB to Grafana amd continue manipulating the data in Grafana. Below is a snapshot from Grafana after connecting to your InfluxDB instance. - -![Grafana InfluxDB Query](/grafana-influxdb-export.png) - -Defining the above chart and form selectors in Grafana is done with this query: - -```shell -from(bucket: "Metrics") - |> range(start: v.timeRangeStart, stop: v.timeRangeStop) - |> filter(fn: (r) => r["_measurement"] == "PerformanceKPIs") - |> filter(fn: (r) => r["_field"] == "latency") - |> filter(fn: (r) => contains(value: r["namespace"], set: ${Namespace:json})) - |> filter(fn: (r) => contains(value: r["service"], set: ${Service:json})) - |> filter(fn: (r) => contains(value: r["path"], set: ${Path:json})) - |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) - |> yield(name: "mean") -``` - diff --git a/src/pages/en/integrations_lens.md b/src/pages/en/integrations_lens.md deleted file mode 100644 index e117785..0000000 --- a/src/pages/en/integrations_lens.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: Kubeshark-Lens Extension -description: -layout: ../../layouts/MainLayout.astro ---- - -[Kubernetes Lens](https://k8slens.dev/), sometimes referred to as the [Kubernetes](https://kubernetes.io/) IDE, allows users to connect and manage multiple Kubernetes clusters on Mac, Windows, and Linux platforms. It is an intuitive graphical interface that allows users to deploy and manage clusters directly from the console. - -[Kubeshark](https://kubeshark.com)'s Lens extension enable Kubernetes Lens users to view real-time traffic of clusters, namespaces, nodes, pods or services, by simply clicking a new menu item added by the extension. - -## Installing - -1. Open Lens and navigate to the Extensions page (or press Command + Shift + E on macOS). -2. Enter the following URL into the Install Extension box: `@kubeshark/lens` -3. Click the Install button. - -## Configuration - -In the preference page, you can enter the [Kubeshark](https://kubeshark.com) URL. The default URL is `http://localhost:8899/`. - -![Enter Kubeshark URL](/lens_preferences.png) - -## How to View - -Once the extension is loaded, a new menu item will appear for each specific Kubernetes component. This menu item indicates that you can view real-time traffic for the corresponding component. By clicking on the menu item, [Kubeshark](https://kubeshark.com) will open and present the traffic associated with the selected Kubernetes component. - -For example, if you want to view traffic in a particular namespace, follow these steps: - -1. Find the namespace in KubernetesLens. -2. Locate and click on the menu item that indicates viewing traffic for that namespace. - -![Right click to view traffic](/lens_menu_item.png) - -Once you press the menu item, [Kubeshark](https://kubeshark.com)'s dashboard will open automatically configured to filter traffic related only to the selected Kubernetes component. - -## Limitations - -[Kubeshark](https://kubeshark.com) will only display traffic under the following circumstances: -- The component is active and either receiving or generating traffic. -- [Kubeshark](https://kubeshark.com) was able to successfully capture and dissect the traffic (there could be various reasons why this may not happen). - - - - - diff --git a/src/pages/en/integrations_slack.md b/src/pages/en/integrations_slack.md deleted file mode 100644 index a3ef88b..0000000 --- a/src/pages/en/integrations_slack.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -title: Slack Alerts -description: Kubeshark can send Slack messages to any channel with the purpose of alerting against anomalies in the Kubernetes network traffic. -layout: ../../layouts/MainLayout.astro ---- - -Real-time Slack alerts can be triggered when suspicious network behaviors occur. Messages can include event information and forensics (e.g. network traces - PCAPs). - - - -The following example uses an L7 hook and the Slack helper to trigger areal-time alert to a Slack channel whenever the HTTP response status code is `500`. The alert includes a brief description and a network trace. You can invite additional team members to the Slack channel to help identify the culprit. - -```js -// Report To a Slack Channel If HTTP Response Status Code is 500 Example - -function onItemCaptured(data) { - // Check if it's an HTTP request and the response status is 500 - if (data.protocol.name === "http" && data.response.status === 500) { - var files = {}; - - // Get the path of the PCAP file that this stream belongs to - var pcapPath = pcap.path(data.stream); - files[data.stream + ".pcap"] = pcapPath; - - // Dump the `data` argument into a temporary JSON file - var dataPath = file.temp("data", "", "json"); - file.write(dataPath, JSON.stringify(data, null, 2)); - files["data.json"] = dataPath; - - // Send a detailed Slack message with 2 attached files - vendor.slackBot( - SLACK_AUTH_TOKEN, - SLACK_CHANNEL_ID, - "Server-side Error in Kubernetes Cluster", // Pretext - "An HTTP request resulted with " + data.response.status + " status code:", // Text - "#ff0000", // Color - { - "Service": data.dst.name, - "Namespace": data.namespace, - "Node": data.node.name, - "HTTP method": data.request.method, - "HTTP path": data.request.path - }, - files - ); - - // Delete the temporary file - file.delete(dataPath); - } -} - -``` - -[Kubeshark](https://kubeshark.com) supports two message sending methods: - -1. Using an [incoming webhook](https://api.slack.com/messaging/webhooks). See [`vendor.slack`](/en/automation_helpers#vendorslackwebhookurl-string-pretext-string-text-string-color-string) helper. -2. Using an auth token and a channel ID. See [`vendor.slackBot`](/en/automation_helpers#vendorslackbottoken-string-channelid-string-pretext-string-text-string-color-string-fields-object-files-object) helper. - -## Creating a Slack App - -Slack provides an easy way to create a Slack App that enables maximum flexibility and customization. This step is required for both methods. - -Creating a Slack App shouldn't take more than 5 minutes. - -Start [here](https://api.slack.com/apps) and press the **Create New App** button to create a Slack app (Bot): - -![Create Slack App](/slack-create-app.png) - -To make it easy we provided short manifest that is sufficient for [Kubeshark](https://kubeshark.com) to send messages to a Slack channel. - -Choose to create an app from a manifest: - -![From a Manifest](/slack-manifest.png) - -Select your workspace: - -![Select your workspace](/slack-workspace.png) - -Copy the manifest from below and paste to the Slack Manifest window. Be sure to choose the YAML tab: - -```yaml -display_information: - name: [Kubeshark](https://kubeshark.com) - description: The API Traffic Analyzer for Kubernetes - background_color: "#1c50d4" - long_description: Think Wireshark re-invented for Kubernetes, [Kubeshark](https://kubeshark.com) provides deep visibility and real-time monitoring of all traffic going in, out and across containers, pods, namespaces, nodes, and clusters, elevating your ability to debug, troubleshoot, and protect your Kubernetes clusters. -features: - bot_user: - display_name: [Kubeshark](https://kubeshark.com) - always_online: true -oauth_config: - scopes: - bot: - - chat:write - - chat:write.public - - files:write - - incoming-webhook - - remote_files:share -``` - -Install the App to your workspace: - -![Install the App to your workspace](/slack-install-app.png) - -### Using an Incoming Webhooks - -If you'd like to use the **Incoming Webhooks** method, hop over to the **Incoming Webhooks** section and retrieve the **Incoming Webhook URL**. - -![Webhook URL](/slack-webhook.png) - -That's it, you can now use the Slack helper [`vendor.slack`](/en/automation_helpers#vendorslackwebhookurl-string-pretext-string-text-string-color-string) like this: - -```js -vendor.slack( - SLACK_WEBHOOK, - "Server-side Error", - JSON.stringify(data), - "#ff0000" -); -``` -As the Webhook is a confidential piece of information, we highly recommend to keep it in the [Kubeshark](https://kubeshark.com) configuration file. - -### Using an Auth Token and a Channel ID - -An alternative way to send messages requires obtaining an OAuth Token and a Channel ID which are also very easy to get. - -#### Obtaining the Auth Token - -To obtain the OAuth Token, go to the new Slack App page and hop over to the `OAuth & Permissions` section and copy the OAuth Token. - -![Slack OAuth Token](/slack-oauth.png) - -We suggest keeping the OAuth token as an [environment variables](/en/config#scripts) in the [Kubeshark](https://kubeshark.com) configuration file. - -#### Obtaining the Channel ID - -When you have a certain channel you'd like to send message to, you can obtain its Channel ID by pressing the down arrow that is adjacent to the channel name. - -![Slack Channel Down Arrow](/slack-channel-down-arrow.png) - -Now, copy the channel ID at the bottom of the about section. - -![Slack Channel About](/slack-channel-about.png) - -#### Adding Kubeshark to the Channel - -Last required action is to add [Kubeshark](https://kubeshark.com) to the channel. This can be achieved by calling the Bot using `@[Kubeshark](https://kubeshark.com)` - -![Slack Channel Add](/slack-adding-to-channel.png) diff --git a/src/pages/en/integrations_webhook.md b/src/pages/en/integrations_webhook.md deleted file mode 100644 index 90285b7..0000000 --- a/src/pages/en/integrations_webhook.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: Use a Webhook -description: A webhook integration enables uploading anything to anywhere as long as webhooks are supported. -layout: ../../layouts/MainLayout.astro ---- - -The helper [`vendor.webhook`](/en/automation_helpers#vendorwebhookmethod-string-url-string-body-string) enables uploading anything to anywhere as long as webhooks are supported. It does an HTTP request to the webhook (the HTTP endpoint) that’s defined by HTTP method and URL in the url argument with the HTTP body as the string in the body argument. - -The following example calls a webhook for each health check: - -```js -function onItemCaptured(data) { - if (data.request.path === "/health") - response = vendor.webhook( - "POST", - "https://webhook.site/a42ca96d-4984-45dc-8f72-a601448399dc", - JSON.stringify(data), - { - "content-type": "application/json" - } - ); -} -``` - -> See [`vendor.webhook`](/en/automation_helpers#vendorwebhookmethod-string-url-string-body-string) for more info. diff --git a/src/pages/en/irsa.md b/src/pages/en/irsa.md deleted file mode 100644 index 6a3f74a..0000000 --- a/src/pages/en/irsa.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: IRSA -description: Kubeshark enables you to generate files (e.g. PCAPs) and upload them to an immutable datastore (e.g. AWS S3). -layout: ../../layouts/MainLayout.astro ---- - -## EKS and IRSA - -***** **IRSA does not work when deploying [Kubeshark](https://kubeshark.com) to the `default` namespace** ***** - -[IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) is a method for not using specific credentials but rather use a role associated with a service account. -Prerequisite to using this method is completing with the list of steps described in [this article](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). Specifically having: -1. A role -2. A policy -3. A trust relationship - -### Annotating the Kubeshark Service Account -The IRSA method works by providing an annotation to the service account that is associated with the pod that is performing the AWS operations. -**In our case:** (FYI only, nothing to do here) -- Service Account: kubeshark-service-account -- Pod: [Kubeshark](https://kubeshark.com) Worker - -To use IRSA, you'd need to: -1. Provide annotation of the IAM role -2. Use shared configuration - -To provide the annotation, append the following line to either the CLI's `tap` or the Helm command: - -```shell ---set-json 'tap.annotations={"eks.amazonaws.com/role-arn":"arn:aws:iam::7456....3350:role/s3-role"}' -``` -You can also populate the config element in `~/.kubeshark/config.yaml` when using the CLI. - -### Helm Command - -```shell -helm install kubeshark kubeshark/kubeshark -n kubeshark --create-namespace \ ---create-namespace \ ---set license= \ ---set-json 'scripting.env={"AWS_ACCESS_KEY_ID":"", "AWS_SECRET_ACCESS_KEY": "", "AWS_REGION":"us-east-2", "S3_BUCKET":"demo-kubeshark-b"}' \ ---set-json 'tap.annotations={"eks.amazonaws.com/role-arn":"arn:aws:iam::74.....50:role/s3-role"}' \ ---set tap.ingress.enabled=true \ ---set tap.ingress.host=demo.kubeshark.io \ ---set "tap.ingress.auth.approveddomains={kubeshark.com}" \ ---set tap.release.namespace=kubeshark \ ---set tap.resources.worker.limits.memory=2Gi -``` - -* The above includes optional value that can come handy. diff --git a/src/pages/en/long_term_retention.md b/src/pages/en/long_term_retention.md deleted file mode 100644 index 7bc443f..0000000 --- a/src/pages/en/long_term_retention.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: Long-Term Retention -description: -layout: ../../layouts/MainLayout.astro ---- -Recordings' content can be automatically uploaded to an immutable data store for long-term retention. Data may be stored in either Amazon S3 or Google Cloud Storage (GCS). Uploads can be triggered automatically at the end of a recording or on-demand at the user's discretion. - -To enable either automatic or on-demand uploads, simply add the appropriate credentials. Once the proper credentials are in place, the recording's content will be uploaded: -1. When the recording ends -2. Once an hour -3. On-demand at the user's discretion - -## Triggering an Upload On-demand -Users can utilize the Jobs dashboard to initiate an on-demand upload of the `tar` files. Simply press the `Run All Jobs` button to compress and upload all accumulated content. - -![Run All Jobs](/s3-jobs.jpg) - -## Automatic Upload -By default, content is uploaded hourly and at the end of the recording window, as long as the proper crednetials are in place. - -## Bucket Content - -Within the bucket, you can expect a folder per node, where each folder contains a `tar` file for each upload, following this pattern: - -`/_kubeshark_.gz.tar` - -For example: -![image](/s3-nodes.jpg) -![image](/s3-gzs.jpg) - -## Content of the Tar File -Each tar file includes all JSON files and PCAP files that belong to the traffic recording. - -## Use Specific Auth Credentials - -- **S3_BUCKET**: The name of the S3 bucket -- **AWS_REGION**: The region where the bucket resides -- **AWS_ACCESS_KEY_ID**: AWS Access Key -- **AWS_SECRET_ACCESS_KEY**: AWS Secret Access Key - -> Learn more about the required AWS credentials [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). - -### Configuration Example - -In either `config.yaml` or `values.yaml`, the environment variable should look something like this: -```shell -scripting: - env: - S3_BUCKET: - AWS_REGION: - AWS_ACCESS_KEY_ID: AK..4Z - AWS_SECRET_ACCESS_KEY: K..r -``` - -## AWS S3 IRSA or Kube2IAM - -In scenarios using shared authentication (e.g., IRSA, kube2iam), the S3 helper will utilize the default authentication method and will not require specific AWS Auth credentials. In these cases, the required credentials include: - -- **S3_BUCKET**: The name of the S3 bucket -- **AWS_REGION**: The region where the bucket resides - -> Read more about [IRSA](/en/irsa) in this article. - -## GCS - -Uploading to GCS requires the following credentials: -- **GCS_BUCKET**: GCS Bucket -- **GCS_SA_KEY_JSON**: GCS Key JSON - -### Configuration Example - -In either `config.yaml` or `values.yaml`, the environment variable should look something like this: -```shell -scripting: - env: - GCS_BUCKET: - GCS_SA_KEY_JSON : '{ - "type": "service_account", - "project_id": , - "private_key_id": "14a....81", - "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0.....HBJsfVHn\nRvUJH6Yxdzv3rtDAYZxgNB0=\n-----END PRIVATE KEY-----\n", - "client_email": "k..om", - "client_id": "104..72", - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://oauth2.googleapis.com/token", - "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", - "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/k...nt.com", - "universe_domain": "googleapis.com" -}' -``` diff --git a/src/pages/en/mcp/autonomous_development.md b/src/pages/en/mcp/autonomous_development.md deleted file mode 100644 index 3e91f24..0000000 --- a/src/pages/en/mcp/autonomous_development.md +++ /dev/null @@ -1,245 +0,0 @@ ---- -title: Autonomous Development & Testing -description: Bridge the last mile to production with network-level feedback that helps AI coding tools identify and fix issues before release. -layout: ../../../layouts/MainLayout.astro -mascot: Cute ---- - -**Kubeshark doesn't write code—it bridges the last mile to production.** - -AI coding assistants (Claude Code, Cursor, Copilot) can write and deploy code, but they lack visibility into how that code actually behaves in a Kubernetes environment. Kubeshark closes this gap by providing real-time network feedback, enabling AI tools to identify issues and fix them *before* releasing to production. - -Instead of "deploy and pray," you get "deploy, verify, and fix"—all in one autonomous loop. - ---- - -## The Problem: The Last Mile Gap - -AI coding tools can write sophisticated code, but they operate blind when it comes to production behavior: - -- **No visibility** — AI generates code but can't see how it behaves in a real cluster -- **Delayed feedback** — Issues surface in production, long after the code was written -- **Limited context** — Unit tests pass, but integration failures hide in network interactions -- **Manual verification** — Developers must manually check logs and traces to verify behavior - -The result: code that "works" in isolation but fails in production. AI tools keep repeating the same integration mistakes because they never see the actual network behavior. - ---- - -## How Kubeshark Bridges the Gap - -Kubeshark provides the missing feedback loop. When connected to an AI coding assistant via MCP, it enables the AI to: - -1. **See actual behavior** — Every API call, payload, header, and response -2. **Identify issues** — Malformed requests, missing headers, unexpected calls -3. **Correlate cause and effect** — Link code changes to network behavior -4. **Iterate until correct** — Fix issues and re-verify automatically - -``` -+-----------------------------------------------------------------+ -| CLOSED-LOOP DEVELOPMENT | -| | -| +-----------+ +-----------+ +-----------+ | -| | AI Writes |---->| Deploy |---->| Test | | -| | Code | | to K8s | | | | -| +-----------+ +-----------+ +-----+-----+ | -| ^ | | -| | v | -| +-----+-----+ +-----------+ | -| | AI Fixes |<----------------------| Kubeshark | | -| | Issues | Network Feedback | MCP | | -| +-----------+ +-----------+ | -| | -| Kubeshark provides the production insight AI tools lack | -+-----------------------------------------------------------------+ -``` - -**Example prompts:** - -> *"Deploy my changes to the local cluster and verify the new /api/orders endpoint works correctly."* - -> *"Run the integration tests and use Kubeshark to verify the API calls are correct—check payloads, headers, and downstream calls."* - -> *"Test my authentication middleware. Verify it properly rejects invalid tokens and check what headers it adds to valid requests."* - -> *"I changed the retry logic. Deploy and verify that failed requests are retried exactly 3 times with exponential backoff."* - ---- - -## What Network Feedback Catches - -Kubeshark provides feedback that logs, metrics, and test assertions miss: - -| Traditional Testing | Network-Level Verification | -|---------------------|---------------------------| -| "Test passed (200 OK)" | "Response was 200, but payload had wrong field types" | -| "No errors in logs" | "3 unexpected retries occurred, masking a connection issue" | -| "Latency within SLA" | "Latency is fine, but you're making 5 redundant DB calls" | -| "Auth test passed" | "Auth works, but token is sent to downstream services unmasked" | - -**Specific issues the AI catches:** -- Malformed payloads that "work" but violate API specs -- Extra API calls that shouldn't happen (N+1 queries) -- Missing or incorrect headers (auth, correlation IDs, content-type) -- Serialization bugs (wrong date formats, encoding issues) -- Downstream services called with wrong parameters -- Retry/timeout logic that doesn't behave as intended -- Connection pooling issues - ---- - -## The Autonomous Loop in Action - -``` -+---------------------------------------------------------------------------------+ -| Terminal - claude | -+---------------------------------------------------------------------------------+ -| | -| > I've updated the order-service to add caching. Deploy it and verify | -| it works correctly. | -| | -| * Building and deploying order-service | -| - Running unit tests... 23 passed | -| - Building container image... | -| - Deploying to local cluster... | -| - Waiting for rollout... Ready | -| | -| * Running integration tests with network verification | -| - Executing test suite... | -| - Creating Kubeshark snapshot... | -| - Analyzing 847 API calls... | -| - Verifying network behavior... | -| | -| ! Issues detected: | -| | -| 1. Cache miss on every request | -| Expected: First request hits DB, subsequent requests use cache | -| Actual: All 12 requests to /api/orders/{id} called the database | -| Evidence: 12 calls to postgres-service for same order ID | -| | -| 2. Missing Cache-Control header | -| Expected: Response includes Cache-Control: max-age=300 | -| Actual: Header not present in any response | -| | -| Root cause: The cache key includes a timestamp, making every lookup unique. | -| | -| Suggested fix: Remove timestamp from cache key in order-service/cache.go:47 | -| | -| > Yes, apply the fix | -| | -| * Applying fix | -| - Updating order-service/cache.go... | -| - Rebuilding and redeploying... | -| - Re-running verification... | -| | -| * Network verification results | -| - First request: DB call OK | -| - Subsequent requests: Cache hit (no DB call) OK | -| - Cache-Control header present OK | -| - Response times: 45ms (cold) -> 2ms (cached) OK | -| | -| OK All verifications passed. Your caching implementation is working correctly. | -| | -| > _ | -| | -+---------------------------------------------------------------------------------+ -``` - ---- - -## Another Example: Verifying Retry Logic - -``` -+---------------------------------------------------------------------------------+ -| Terminal - claude | -+---------------------------------------------------------------------------------+ -| | -| > I implemented retry logic for the payment service. It should retry | -| 3 times with exponential backoff (1s, 2s, 4s). Deploy and verify. | -| | -| * Deploying and testing retry logic | -| - Deploying payment-client... | -| - Configuring payment-service to fail first 2 requests... | -| - Triggering test payment... | -| - Analyzing retry behavior... | -| | -| ! Retry logic has issues: | -| | -| Expected: 3 retries with 1s, 2s, 4s delays | -| Actual: 3 retries with 1s, 1s, 1s delays | -| | -| Network evidence: | -| Request 1: 14:23:15.000 -> 503 Service Unavailable | -| Request 2: 14:23:16.012 -> 503 Service Unavailable (delay: 1.01s) | -| Request 3: 14:23:17.018 -> 503 Service Unavailable (delay: 1.00s) | -| Request 4: 14:23:18.025 -> 200 OK (delay: 1.00s) | -| | -| The backoff multiplier isn't being applied. | -| | -| Found in payment-client/retry.go:23: | -| delay := baseDelay // Should be: delay := baseDelay * (1 << attempt) | -| | -| > Fix it | -| | -| * Fixing and re-verifying | -| - Applying fix to retry.go... | -| - Redeploying... | -| - Re-running test... | -| | -| OK Retry logic verified: | -| Request 1: 14:25:00.000 -> 503 (initial) | -| Request 2: 14:25:01.005 -> 503 (delay: 1.00s) OK | -| Request 3: 14:25:03.011 -> 503 (delay: 2.00s) OK | -| Request 4: 14:25:07.018 -> 200 (delay: 4.00s) OK | -| | -| > _ | -| | -+---------------------------------------------------------------------------------+ -``` - ---- - -## Why This Matters - -Kubeshark transforms AI coding tools from "code generators" into "production-ready code generators": - -| Without Kubeshark | With Kubeshark | -|------------------|----------------| -| AI writes code, hopes it works | AI writes code, verifies it works | -| Issues found in production | Issues found before commit | -| Manual debugging after release | Automatic detection and fix | -| "Works in tests" ≠ works in prod | Verify actual production behavior | - -**Key benefits:** - -- **Shift-left production visibility** — Catch integration issues before they reach staging -- **Autonomous verification** — AI handles the test-analyze-fix cycle without manual intervention -- **Evidence-based fixes** — AI sees the exact network behavior causing issues -- **Confidence to release** — Code is verified against real Kubernetes behavior, not mocks - ---- - -## The Complete Picture - -**What AI coding tools do:** -- Write and modify code -- Deploy to Kubernetes clusters -- Run tests and trigger API calls -- Apply fixes based on feedback - -**What Kubeshark provides (via MCP):** -- Real-time visibility into every API call -- Full request/response payloads and headers -- Timing data and latency analysis -- Evidence of actual behavior vs expected behavior -- Detection of issues AI tools can't see otherwise - -**Together:** The AI writes code, Kubeshark shows what that code actually does, and the AI fixes issues based on real network evidence—all in one continuous loop. - ---- - -## What's Next - -- [Conversational Debugging](/en/mcp/troubleshooting) — Debug integration issues -- [AI Use Cases](/en/mcp_use_cases) — Common scenarios for AI-powered analysis -- [MCP in Action](/en/mcp_in_action) — See a complete investigation example diff --git a/src/pages/en/mcp/troubleshooting.md b/src/pages/en/mcp/troubleshooting.md deleted file mode 100644 index 0b2e58e..0000000 --- a/src/pages/en/mcp/troubleshooting.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -title: Conversational Debugging -description: Use AI to debug integration issues, verify API behavior, and troubleshoot service communication through natural conversation. -layout: ../../../layouts/MainLayout.astro -mascot: Cute ---- - -Day-to-day debugging gets faster when you can ask questions instead of writing queries. - ---- - -## The Problem - -Developers spend significant time: -- Figuring out why their service isn't receiving expected requests -- Understanding what data another service is actually sending -- Debugging integration issues between services -- Verifying that their changes work correctly in the cluster -- Comparing what's documented vs. what's actually happening - ---- - -## How MCP Helps - -Ask debugging questions naturally: - -**Example prompts:** - -> *"Is my user-service receiving requests from the API gateway? Show me the last 10 requests."* - -> *"What exactly is the notification-service sending to the email-provider? Show me the request body."* - -> *"The frontend says it's sending the right headers, but the backend disagrees. Show me what's actually in the HTTP request."* - -> *"Find all 4xx errors returned by my service in the last 30 minutes. What requests caused them?"* - -> *"My service calls the inventory API but gets empty responses. Show me the actual request and response."* - -> *"Why is my service getting 'connection refused' errors to the database?"* - ---- - -## What the AI Can Do - -- **Show actual request/response payloads** -- **Verify headers, query parameters, and body content** -- **Find errors and their corresponding requests** -- **Compare what's sent vs. what's received** -- **Filter traffic to/from specific pods or deployments** -- **Identify mismatches** between expected and actual behavior -- **Track retry attempts and their outcomes** -- **Show connection-level issues** (TCP resets, timeouts) - ---- - -## Example Debugging Session - -``` -+---------------------------------------------------------------------------------+ -| Terminal - claude | -+---------------------------------------------------------------------------------+ -| | -| > My webhook endpoint isn't receiving events from the payment provider. | -| What's going on? | -| | -| * Investigating webhook traffic | -| - Searching for traffic to webhook endpoints... | -| - Checking for payment provider traffic... | -| - Analyzing connection attempts... | -| | -| Found the issue: | -| | -| Payment provider IS sending webhooks, but they're failing: | -| | -| Last 5 webhook attempts (from 203.0.113.50 - Stripe IP): | -| +----------------------------------------------------------------+ | -| | Time | Endpoint | Status | Reason | | -| +------------+-----------------------+--------+-----------------+ | -| | 14:23:15 | /webhooks/payment | 415 | Unsupported | | -| | 14:18:22 | /webhooks/payment | 415 | Media Type | | -| | 14:13:30 | /webhooks/payment | 415 | | | -| | 14:08:45 | /webhooks/payment | 415 | | | -| | 14:03:52 | /webhooks/payment | 415 | | | -| +----------------------------------------------------------------+ | -| | -| Root cause: Content-Type mismatch | -| | -| Payment provider sends: | -| Content-Type: application/json; charset=utf-8 | -| | -| Your endpoint expects: | -| Content-Type: application/json (exact match) | -| | -| The "; charset=utf-8" suffix is causing your framework to reject it. | -| | -| > Show me the full request that failed | -| | -| Request at 14:23:15 UTC: | -| | -| POST /webhooks/payment HTTP/1.1 | -| Host: api.yourcompany.com | -| Content-Type: application/json; charset=utf-8 | -| Stripe-Signature: t=1642262595,v1=abc123... | -| Content-Length: 847 | -| | -| { | -| "id": "evt_1234567890", | -| "type": "payment_intent.succeeded", | -| "data": { | -| "object": { | -| "id": "pi_abcdef", | -| "amount": 2000, | -| "currency": "usd" | -| ... | -| } | -| } | -| } | -| | -| Response: 415 Unsupported Media Type | -| Body: {"error": "Content-Type must be application/json"} | -| | -| > _ | -| | -+---------------------------------------------------------------------------------+ -``` - ---- - -## Common Debugging Scenarios - -| Issue | What to Ask | -|-------|-------------| -| **Not receiving requests** | "Is my service receiving any traffic? Show me recent requests." | -| **Wrong response** | "What exactly is service X returning for endpoint Y?" | -| **Header issues** | "Show me all headers being sent to my endpoint" | -| **Payload mismatch** | "Compare what's sent vs. what's received" | -| **Connection errors** | "Why am I getting connection refused/reset errors?" | -| **Auth failures** | "What auth headers are being sent? Are they valid?" | -| **Timeout issues** | "Which requests are timing out and how long do they take?" | - ---- - -## What's Next - -- [AI Use Cases](/en/mcp_use_cases) — Common scenarios for AI-powered analysis -- [Autonomous Development](/en/mcp/autonomous_development) — Verify code in local cluster -- [MCP in Action](/en/mcp_in_action) — See a complete investigation example diff --git a/src/pages/en/metrics.md b/src/pages/en/metrics.md deleted file mode 100644 index 5a93f4a..0000000 --- a/src/pages/en/metrics.md +++ /dev/null @@ -1,191 +0,0 @@ ---- -title: Prometheus Metrics & Grafana Dashboard -description: Learn how to integrate Kubeshark metrics with Grafana for enhanced monitoring. -layout: ../../layouts/MainLayout.astro -mascot: Cute ---- -## Custom Metrics from Network Information - -You can create custom metrics using information gathered from network traffic and export them to Prometheus. Hooks are used to monitor the traffic and can be utilized to create custom metrics. A helper is available to export metrics to Prometheus, typically as part of a scheduled job, ensuring minimal impact on CPU and memory consumption. - -**Examples of custom metrics:** -1. Count DNS requests per pod. -2. Count status codes per API call. -3. Count occurrences of a specific pattern (e.g., when a response field is empty). - -## Creating a Custom Metric (Best Practice) - -It is recommended to create custom metrics using one of the available hooks. - -**Example:** -```javascript -var dnsCounts = {}; - -function onItemCaptured(data) { - if (data.protocol && data.protocol.name === "dns" && data.src && data.src.name) { - var podName = data.src.name; - if (!dnsCounts[podName]) { - dnsCounts[podName] = 0; - } - dnsCounts[podName]++; - } -} -``` - -## Prometheus Metrics Helper - -[Kubeshark](https://kubeshark.com) can export both pre-existing and custom Prometheus metrics. Almost any network-based event can be turned into a custom metric. See below for a list of built-in metrics. A [script](/en/automation_scripting) can also be used to calculate and export custom metrics using the `prometheus.metric` helper. - -**Example:** -```javascript -prometheus.metric( - _metric_name_, // Metric name - "Metric description", // Metric description - 1, // Metric type: 1 - Counter, 2 - Gauge, 3 - Untyped (float) - 64, // Value - { // Labels - s_metric: "dnscounts", - s_pod: podName - } -); -``` - -## Exporting Custom Metrics - -It is recommended to export custom metrics as part of a scheduled job to ensure efficient resource utilization. - -**Example:** -```javascript -jobs.schedule("export-metrics", "*/10 * * * * *", function () { - for (var podName in dnsCounts) { - prometheus.metric( - "dnscounts_" + podName, - "DNS request count per pod", - 1, - dnsCounts[podName], - { s_metric: "dnscounts", s_pod: podName } - ); - } -}); -``` -## Visualizing in Grafana - -Once the script is executed, you can use the following PromQL query to display the custom metric on the Grafana dashboard: - -```javascript -rate({s_metric="dnscounts"}[$__rate_interval]) -``` - -![Custom Metric in Grafana](/custom-metric-prom.png) - -## Configuration - -By default, [Kubeshark](https://kubeshark.com) uses port `49100` to expose metrics through the service `kubeshark-worker-metrics`. - -If you are using the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) community Helm chart, you can configure additional scraping for [Kubeshark](https://kubeshark.com) using the following configuration: - -```yaml -prometheus: - enabled: true - prometheusSpec: - additionalScrapeConfigs: | - - job_name: 'kubeshark-worker-metrics' - kubernetes_sd_configs: - - role: endpoints - relabel_configs: - - source_labels: [__meta_kubernetes_pod_name] - target_label: pod - - source_labels: [__meta_kubernetes_pod_node_name] - target_label: node - - source_labels: [__meta_kubernetes_endpoint_port_name] - action: keep - regex: ^metrics$ - - source_labels: [__address__, __meta_kubernetes_endpoint_port_number] - action: replace - regex: ([^:]+)(?::\d+)? - replacement: $1:49100 - target_label: __address__ - - action: labelmap - regex: __meta_kubernetes_service_label_(.+) -``` - -## Existing Metrics - -Existing metrics provide essential insights for monitoring [Kubeshark](https://kubeshark.com). - -> For the most up-to-date details, visit the [metrics section](https://github.com/kubeshark/kubeshark/blob/master/helm-chart/metrics.md) in the repository. - -### Available Metrics - -| Name | Type | Description | -|-------------------------------------------|---------|-----------------------------------------------| -| kubeshark_received_packets_total | Counter | Total number of packets received | -| kubeshark_dropped_packets_total | Counter | Total number of packets dropped | -| kubeshark_processed_bytes_total | Counter | Total number of bytes processed | -| kubeshark_tcp_packets_total | Counter | Total number of TCP packets | -| kubeshark_dns_packets_total | Counter | Total number of DNS packets | -| kubeshark_icmp_packets_total | Counter | Total number of ICMP packets | -| kubeshark_reassembled_tcp_payloads_total | Counter | Total number of reassembled TCP payloads | -| kubeshark_matched_pairs_total | Counter | Total number of matched pairs | -| kubeshark_dropped_tcp_streams_total | Counter | Total number of dropped TCP streams | -| kubeshark_live_tcp_streams | Gauge | Number of live TCP streams | - -## Ready-to-use Dashboard - -You can import a ready-to-use dashboard from [Grafana's Dashboards Portal](https://grafana.com/grafana/dashboards/21332-kubeshark-dashboard-v3-10/). - -## TL;DR - -### Metric - -A time-series data stream identified by a name and a set of key-value pairs (called **labels**). Supported metric types: -- **Counter**: A cumulative value that only increases (e.g., number of requests). -- **Gauge**: A value that can increase or decrease (e.g., memory usage). -- **Untyped**: A metric that does not strictly conform to the semantics of other metric types and is essentially a floating-point value (float64). - -**Example of a metric:** -```plaintext -http_requests_total{method="POST", handler="/api"} -``` - -- `http_requests_total` is the metric name. -- `{method="POST", handler="/api"}` are the labels. - -### Install Prometheus Community Version - -```bash -helm upgrade -i prometheus prometheus-community/kube-prometheus-stack \ ---namespace prometheus --create-namespace \ --f kube_prometheus_stack.yaml - -kubectl port-forward -n prometheus svc/prometheus-grafana 8080:80 -``` - -**Example `kube_prometheus_stack.yaml` file:** -```yaml -grafana: - additionalDataSources: [] -prometheus: - prometheusSpec: - scrapeInterval: 10s - evaluationInterval: 30s - additionalScrapeConfigs: | - - job_name: 'kubeshark-worker-metrics' - kubernetes_sd_configs: - - role: endpoints - relabel_configs: - - source_labels: [__meta_kubernetes_pod_name] - target_label: pod - - source_labels: [__meta_kubernetes_pod_node_name] - target_label: node - - source_labels: [__meta_kubernetes_endpoint_port_name] - action: keep - regex: ^metrics$ - - source_labels: [__address__, __meta_kubernetes_endpoint_port_number] - action: replace - regex: ([^:]+)(?::\d+)? - replacement: $1:49100 - target_label: __address__ - - action: labelmap - regex: __meta_kubernetes_service_label_(.+) -``` \ No newline at end of file diff --git a/src/pages/en/network_sniffing.md b/src/pages/en/network_sniffing.md deleted file mode 100644 index 23a28e4..0000000 --- a/src/pages/en/network_sniffing.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Network Sniffing -description: Kubeshark can sniff both encrypted and unencrypted traffic in your cluster using various methods and APIs built into the Linux kernel. -layout: ../../layouts/MainLayout.astro ---- - -[Kubeshark](https://kubeshark.com) can sniff both encrypted and unencrypted traffic in your cluster using various methods and APIs built into [Linux kernel](https://www.kernel.org/). - -## Direct Packet Capture - -[Kubeshark](https://kubeshark.com)'s [Worker](/en/anatomy_of_kubeshark#worker) works at the Kubernetes Node level and uses direct packet capture to sniff the [TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol) and [UDP](https://en.wikipedia.org/wiki/User_Datagram_Protocol) traffic in your cluster using one of [libpcap](https://www.tcpdump.org/) or [AF_PACKET](https://man7.org/linux/man-pages/man7/packet.7.html. - -The **Worker** continuously captures TCP and UDP packets and saves locally in a local [PCAP](https://datatracker.ietf.org/doc/id/draft-gharris-opsawg-pcap-00.html) storage that is limited in size. - -Packets are dissected on-demand either by an active [Dashboard](/en/ui) connection or when [scripting](/en/automation_scripting) is used. The stored PCAP files have a very short expiration date, from seconds to a few minutes, depending on the storage limitation governed by the `tap.storageLimit` value. - -> For longer retention of traffic and offline analysis, please read the [Traffic Recording & Offline Analysis](/en/cloud_forensics) section. - -**Workers** dissect only packets that match one of the supported protocols (e.g. [HTTP](https://datatracker.ietf.org/doc/html/rfc2616), [AMQP](https://www.rabbitmq.com/amqp-0-9-1-reference.html), [Apache Kafka](https://kafka.apache.org/protocol), [Redis](https://redis.io/topics/protocol), [gRPC](https://grpc.github.io/grpc/core/md_doc__p_r_o_t_o_c_o_l-_h_t_t_p2.html), [GraphQL](https://graphql.org/learn/serving-over-http/) and [DNS](https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml)). Packets of other protocol will not be dissected and will be discarded, unless recorded by a script. \ No newline at end of file diff --git a/src/pages/en/observability.md b/src/pages/en/observability.md deleted file mode 100644 index 147f02c..0000000 --- a/src/pages/en/observability.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Observability & Telemetry -description: Kubeshark as an observability and telemetry tool. -layout: ../../layouts/MainLayout.astro ---- - -Harness the information that is carried across K8s network traffic, to understand your cluster and measure its performance. - -![Grafana](/grafana.png) - -## Workload Map - -Understand your cluster and service dependencies with **Kubeshark**'s Workload Map. - -> Read more in the [Workload Map](/en/v2/service_map) section. - -## Metrics & Logs - -**Kubeshark** enables you to send metrics and logs to your favorite telemetry or logs provider and enjoy dashboards and alerts. - -> Read more in the [InfluxDB & Grafana](/en/integrations_influxdb) and [Elasticsearch](/en/integrations_elastic) sections. diff --git a/src/pages/en/offline_analysis.md b/src/pages/en/offline_analysis.md deleted file mode 100644 index bb74b19..0000000 --- a/src/pages/en/offline_analysis.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -title: Recorded Traffic Offline Analysis -description: -layout: ../../layouts/MainLayout.astro ---- - -Each recording's content is stored in a dedicated folder. Access this content at any time, provided the folder hasn't been erased (intentionally or unintentionally). - -To view the traffic, use the `record` KFL helper. This helper, when used with the job name as part of a KFL statement, applies the KFL statement exclusively to the recording content. - -![Recording Helper](/record_helper.png) - -## Analyzing Multiple Recordings and Real-Time Traffic - -Assuming you have multiple recording jobs running, both currently and in the past, you can analyze the traffic of both jobs in conjunction with real-time traffic simultaneously. -For example, if you have recordings named `recording_1` and `recording_2`, you can use the dashboard to analyze traffic across both recordings as well as real-time traffic: - -![Multiple Recording and TLS Traffic](/recording_tls.png) - -This example filters TLS traffic from both recordings and real-time TLS traffic. - - -## Long-Term Immutable Traffic Retention - -It is recommended to export recorded traffic to external data stores, such as Google Cloud Storage (GCS) or AWS S3, to store traffic for extended periods in immutable data stores. - -> Read more in the [AWS S3](/en/integrations_aws_s3) and [GCS](/en/integrations_gcs) sections. - -> Go next to [Recordings Management](/en/recordings_management) \ No newline at end of file diff --git a/src/pages/en/packet_capture.md b/src/pages/en/packet_capture.md deleted file mode 100644 index c27696f..0000000 --- a/src/pages/en/packet_capture.md +++ /dev/null @@ -1,63 +0,0 @@ ---- - -title: Packet Capture Options -description: -layout: ../../layouts/MainLayout.astro ---- - -[Kubeshark](https://kubeshark.com) supports numerous packet capture options catering to a wide range of kernel versions and available capabilities. Each option has its own pros and cons. Having a variety of options to choose from guarantees traffic capture is performant and robust. - -## eBPF - -**How to Use**: `--set packetCapture=ebpf` - -While eBPF is the most modern packet capture option and the one we recommend using, it has some limitations. - -**Pros**: Performant, robust information (e.g. superior pod-name/IP resolution), taps both the network interface, the kernel OS, and XDP. - -**Cons**: Supported from kernel version 5.0 and up. Requires elevated security capabilities. - -The eBPF option requires the following security capabilities: -- SYS_ADMIN -- SYS_PTRACE -- SYS_RESOURCE -- IPC_LOCK - -> The eBPF packet capture option is different from the [eBPF-based TLS packet capture](/en/packet_capture#tls-packet-capture). - -## AF_PACKET - -[Kubeshark](https://kubeshark.com) uses AF_PACKET as the default packet capture option. -To set explicitly, use: `--set packetCapture=af_packet`. - -AF_PACKET operates at the network interface level. It is a mature and reliable mechanism that is available by default in most operating systems and supported in most kernel versions. AF_PACKET is a good enough option in most cases. - -**Pros**: Widely supported by most OSs and kernel versions, doesn't require elevated security capabilities. - -**Cons**: Taps only the network interface and not the OS, and no support for XDP. Not as performant as eBPF or PF-RING. Doesn't provide pod-name/IP resolution, instead has to rely on Kubernetes event-based pod-name/IP resolution. - -The AF_PACKET option requires the following security capabilities: -- NET_RAW -- NET_ADMIN - -## LIBPCAP - -**How to Use**: `--set packetCapture=libpcap` - -In the rare occasions where AF_PACKET isn’t compatible with your OS, the LIBPCAP option can be explicitly set. -LIBPCAP is an older, yet fully functional option for packet capture. - -## PF-RING, AF_XDP - -[Kubeshark](https://kubeshark.com) provides support for additional packet capture methods, however, at this point, using one of these is safe for specific use-cases. - -## Which Method Was Used - -How to learn which packet capture option was used to capture the specific traffic: -The request-response pair metadata includes information related to the packet capture method. - -![Packet Capture Method](/pc_method.jpg) - -## TLS Packet Capture - -[Kubeshark](https://kubeshark.com) uses eBPF to intercept TLS traffic. This packet capture option resembles, yet is different from the eBPF traffic capture option mentioned above. This option is enabled by default. Not using this option will disable TLS packet capture. It also uses the same security capabilities. It has wider OS and kernel version support. diff --git a/src/pages/en/pcap.md b/src/pages/en/pcap.md deleted file mode 100644 index aa50cce..0000000 --- a/src/pages/en/pcap.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: PCAPs, L4 Streams and API metadata -description: With its distributed PCA-based architecture, optimized for performance and scale, Kubeshark makes PCAP or It Didn't Happen possible. -layout: ../../layouts/MainLayout.astro -mascot: ---- -**Optimized for Performance and Scale** - -[Kubeshark](https://kubeshark.com) captures all L4 (UDP and TCP) streams and stores each L4 stream in a separate PCAP file in the root file system of each node in the cluster. - -In addition following successful dissection, all of the API information (e.g. headers, path payload) is stored in a dedicated JSON file for each request/response pair. - -## PCAP - Network Traces - -PCAP, short for packet capture, is an API and a [file format](https://datatracker.ietf.org/doc/id/draft-gharris-opsawg-pcap-00.html#name-introduction) that is commonly used by network analysis tools such as [Wireshark](https://wireshark.org), [Fiddler](https://www.telerik.com/fiddler) and [TCPdump](https://www.tcpdump.org/). - -PCAP provides all packet information from the Ethernet header all the way to the application payload, providing you with the full visibility of the application and network interaction, pre- and post-event. - -PCAP file format is suitable to contains network traces that include all communication information relevant to the trace. - -## API metadata - -Complete API metadata including every element that is visible in the Dashboard and more is stored in JSON file. - -Here's some of the information available in the API metadata JSON file. The file is available to view or download by pressing the API index link: -!![API JSON link](/api_json.png) -```json -{ - "dst": {..}, - "elapsedTime": 0, - "entryFile": "000000017720_pcap-0_entry.json", - "error": null, - "failed": false, - "id": "10.0.41.65:30001/000000017720.pcap-0", - "index": 0, - "node": {..}, - "outgoing": false, - "passed": false, - "protocol": {..}, - "record": "", - "request": {..}, - "requestSize": 111, - "response": {..}, - "responseSize": 164, - "src": {..}, - "startTime": "2023-12-19T20:44:30.360984398Z", - "stream": "000000017720.pcap", - "timestamp": 1703018670360, - "tls": false, - "worker": "10.0.41.65:30001" -} -``` - -## Short-term Vs Long-term Data Retention - -[Kubeshark](https://kubeshark.com) provides real-time visibility into API traffic, enabling its users to view full API traffic details within a short-term retention window. This window is determined by the throughput and the storage allocated for it, all at the Kubernetes node level. Once outside of this window, details are no longer available to view and attempting to do so will result in an error. - -![Error Message](/storage_error.png) - -> For information on long-term traffic retention, read the [Traffic Recorder](/en/traffic_recorder) section. - -## Guardrails - -Because storage can fill up rapidly, we have established certain guardrails to prevent overuse of disk resources. - -### PCAP and JSON TTLs - -PCAPs have a TTL of 10s and JSONs have TTL of 5 minutes. That means that the metadata is available for viewing up to 5 minutes from the point of capture and PCAPs stay on disk only for 10 seconds, enough for a script to handle the PCAP file, before being deleted. - -### Storage Limit - -The storage limitation is regulated by the `tap.storageLimit` configuration value, with the default set to `10Gi`. This value represents a hard limit. If storage surpasses this limit, it will result in pod eviction. When the Worker pod is evicted, the storage is purged and the pod immediately restarts from scratch. - -To increase this limit, simply provide a different value (e.g., setting it to 1GB with `--set tap.storageLimit=1Gi`). - -When an L4 stream is dissected, a JSON file is generated with all relevant details. This file has a time to live of 5 minutes. - -The actual L4 stream (PCAP) has a time to live of 10 seconds, enough for a script to copy it to a different folder if retention is required. - - - - diff --git a/src/pages/en/pcapdump.md b/src/pages/en/pcapdump.md deleted file mode 100644 index d482ee5..0000000 --- a/src/pages/en/pcapdump.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -title: Take a cluster-wide PCAP snapshot -description: -layout: ../../layouts/MainLayout.astro ---- - -You can take a cluster-wide traffic snapshot based on the `pcapdump` configuration values. By default, [Kubeshark](https://kubeshark.com) allows you to create and download a snapshot of the last 60 minutes of cluster-wide traffic. This functionality is available through the recording dialog window. - -![Cluster-wide traffic snapshot](/snapshot.png) - -The snapshot includes all Layer 4 (L4) supported protocols—such as TCP, UDP, and SCTP—including [decrypted TLS](/en/encrypted_traffic) and mTLS traffic. This functionality is similar to the popular [tcpdump](https://www.tcpdump.org/). - -[Kubeshark](https://kubeshark.com) automatically and continuously captures and stores traffic while managing both time window and storage constraints. This allows you to export recorded traffic to a local folder at any time for retention or further analysis using [Wireshark](https://www.wireshark.org/). - ---- - -## Configuration Options - -See the [Capture Filters](/en/pod_targeting) section to learn more about how to set proper capture filters. - -Specific traffic snapshot (`pcapdump`) properties can be set in the `settings` section or in the Helm values. - -[Kubeshark](https://kubeshark.com) handles retention automatically by deleting old files outside the defined time window or beyond the allocated storage quota. - -For example, with a 24-hour time window and a 50 MB storage limit, [Kubeshark](https://kubeshark.com) retains only the files recorded in the last 24 hours and within the 50 MB cap. Older or excess files are removed automatically. - -![Traffic Snapshot Settings](/pcapdump_settings.png) - -Example `pcapdump` configuration: - -```yaml -pcapdump: - enabled: true # Enable or disable PCAP recording - maxTime: 1h # Time window for retaining PCAP files - maxSize: 50MB # Maximum storage size for PCAP files -``` - ---- - -## CLI Installation - -You can also take traffic snapshots using the [Kubeshark](https://kubeshark.com) CLI. - -First, ensure the CLI is installed. You can install it via Homebrew (`brew install kubeshark`) or other supported methods. - -> See [installation instructions](/en/install) for more options. - ---- - -## Taking a Snapshot - -Use the following CLI commands to export recorded traffic: - -```bash -kubeshark pcapdump --dest=/tmp # Export all PCAP files to /tmp -kubeshark pcapdump --dest=/tmp --time=5m # Export only the last 5 minutes to /tmp -kubeshark pcapdump # Export all PCAP files to the current directory -kubeshark pcapdump --time=5m # Export the last 5 minutes to the current directory -``` - -> The `kubeshark` CLI is available via [Homebrew](/en/install#homebrew) or can be [downloaded from GitHub](https://github.com/kubeshark/kubeshark/releases). - -Each command produces a single `.pcap` file saved to the specified directory (or the current one, if no destination is set). \ No newline at end of file diff --git a/src/pages/en/performance.md b/src/pages/en/performance.md deleted file mode 100644 index d0a6fc2..0000000 --- a/src/pages/en/performance.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -title: CPU & Memory Consumption -description: AF_PACKET, AF_XDP, PF-RING. -layout: ../../layouts/MainLayout.astro ---- - -## Topics -- [Topics](#topics) -- [Resource Consumption](#resource-consumption) -- [Kubeshark Operations](#kubeshark-operations) -- [Resource Limitations](#resource-limitations) - - [Container Memory and CPU Limitations](#container-memory-and-cpu-limitations) - - [Worker Storage Limitation](#worker-storage-limitation) - - [OOMKilled and Evictions](#oomkilled-and-evictions) -- [Predictable Consumption](#predictable-consumption) - - [Capture Filters (Pod Targeting)](#capture-filters-pod-targeting) - - [Traffic Sampling](#traffic-sampling) -- [The Browser](#the-browser) - -## Resource Consumption - -[Kubeshark](https://kubeshark.com)'s resource consumption largely depends on the cluster workload and the amount of dissection required. Most resource-consuming operations are performed by the Worker at the node level. - -## Kubeshark Operations - -[Kubeshark](https://kubeshark.com) captures and stores all traffic in memory. It then filters traffic based on pod targeting rules, which include pod regex and a list of namespaces. Traffic filtered out by these rules is discarded. Traffic filtered in is dissected. Among all [Kubeshark](https://kubeshark.com) operations—traffic capturing, storing, filtering, and dissection—dissection is the most resource-intensive and is performed on-demand when a client requests it (e.g., the dashboard, a recording, a running script). - -## Resource Limitations - -While resource consumption can increase based on the amount of traffic targeted for dissection, it can be limited by setting configuration values. - -### Container Memory and CPU Limitations - -Container resources are limited by default. However, allocations can be adjusted in the configuration: - -```yaml -tap: - resources: - hub: - limits: - cpu: 750m - memory: 1Gi - requests: - cpu: 50m - memory: 50Mi - sniffer: - limits: - cpu: 750m - memory: 1Gi - requests: - cpu: 50m - memory: 50Mi - tracer: - limits: - cpu: 750m - memory: 1Gi - requests: - cpu: 50m - memory: 50Mi -``` - - -### Worker Storage Limitation - -Traffic is recorded and stored by the Worker at the Kubernetes node level. [Kubeshark](https://kubeshark.com) generates a PCAP file per L4 stream and a JSON file per API message. Files are deleted based on a TTL: -- PCAP - 10s TTL -- JSON - 5m TTL - -If storage exceeds its limit, the pod is evicted. The storage limit is controlled by setting the `tap.storageLimit` configuration value. To increase this limit, provide a different value (e.g., setting it to 5GB with `--set tap.storageLimit=5Gi`). - -### OOMKilled and Evictions - -Whenever a container surpasses its memory limit, it will get OOMKilled. If Worker storage surpasses its limitation, the pod will get evicted. - -## Predictable Consumption - -While limitations ensure [Kubeshark](https://kubeshark.com) does not consume resources above set limits, it is insufficient to ensure proper operation if the available resources aren't adequate for the amount of traffic [Kubeshark](https://kubeshark.com) must process. - -To consume fewer resources and not surpass limitations, [Kubeshark](https://kubeshark.com) offers two methods to control the amount of processed traffic: - -### Capture Filters (Pod Targeting) - -[Kubeshark](https://kubeshark.com) allows targeting specific pods using pod regex and a list of namespaces. This ensures only traffic related to targeted pods is processed, and the rest is discarded. -Utilizing Pod Targeting can significantly optimize resource consumption: - -![Pod Targeting Changes](/pod_targeting_grafana.png) - -> Read more in the [Capture Filter](/en/pod_targeting) section - -### Traffic Sampling - -`TrafficSampleRate` is a number representing a percentage between 0 and 100. This number causes [Kubeshark](https://kubeshark.com) to randomly select L4 streams, not exceeding the set percentage. - -For example, this configuration will cause [Kubeshark](https://kubeshark.com) to process only 20% of traffic, discarding the rest: - - -```yaml -tap: - trafficSampleRate: 20 -``` - -## The Browser - -Be aware that your browser can consume a significant amount of CPU when displaying a substantial amount of traffic. \ No newline at end of file diff --git a/src/pages/en/performance_benchmark.md b/src/pages/en/performance_benchmark.md deleted file mode 100644 index a563235..0000000 --- a/src/pages/en/performance_benchmark.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: TBD -description: TBD. -layout: ../../layouts/MainLayout.astro ---- - -## Running Kubeshark at a steady state - -Running [Kubeshark](https://kubeshark.com) at a steady state, without using the Dashboard should require very little resources in terms of CPU and memory. In this state, [Kubeshark](https://kubeshark.com) captures all traffic and save it to the Worker nodes local storage. - -## Using the Dashboard - -Depending on the amount of traffic and the various queries perform in the dashboard, the dashboard can consume significant efforts. - -In any event, Workers will not consume more resources than the provided limits. - -If the traffic workload is high, you can assign dedicated resources that will be used only when you conduct your investigation and will offload . - - -The following elements impact performance and resource consumption: -- Using the dashboard -- The KFL - -## Running Kubeshark without the dashboard - -## Using the Dashboard - -## Using Scripts - -| KPI | Description | -| -------- | ------- | -| ALLOC | Bytes of allocated heap objects including all reachable objects, as well as unreachable objects that the garbage collector has not yet freed. Specifically, ALLOC increases as heap objects are allocated and decreases as the heap is swept and unreachable objects are freed. Sweeping occurs incrementally between GC cycles, so these two processes occur simultaneously, and as a result ALLOC tends to change smoothly. | -| RSS | Resident set size. The portion of memory occupied by a process that is held in main memory (RAM).| -| CPU | 100 = 1 CPU | -| PROCESSED BYTES | Amount of bytes processed over period of time| -| TOTAL PACKETS | Amount of packets processed over period of time | -| REASSEMBLED | Amount of messages that were successfully reassembled over period of time | -| PACKETS RECEIVED | Amount of packets received over period of time | -| PACKETS DROPPEDs | Amount of packets dropped over period of time | - - - - -(1) Medium load test, using two workers of size c5.xlarge, over a short period of time, using AF_PACKET with no dashboard - -| # | Period | Test Type | Dashboard | Library | ALLOC | CPU | PROCESSED BYTES | TOTAL PACKETS | REASSEMBLED | PACKETS RECEIVED | PACKETS DROPPED | -| -------- | ------- | ------- | ------- | -------- | ------- | ------- | ------- | -------- | ------- | ------- | ------- | -| (1) | 46m25 | Load Test | No | AF_PACKET | 91.1MB | 13% | 1.8GB | 2.6MB | 549K | 2.6M |0 | - -![](/cloud_watch_1.png) - - -Scenarios 1-2 did not consider whether the dashboard was open or not. -(1) Small load testing over a short period of time, using AF_PACKET and KFL to focus only on two pods: no reassembly -(2) Small load testing over a short period of time, using AF_PACKET and KFL to focus only on two pods: with reassembly -(3) Stress testing over a long period of time, using AF_PACKET and KFL to focus only on two pods - -| Scenario | Period | Test Type | Library | ALLOC | RSS | CPU | PROCESSED BYTES | TOTAL PACKETS | REASSEMBLED | PACKETS RECEIVED | PACKETS DROPPED | -| -------- | ------- | ------- | ------- | -------- | ------- | ------- | ------- | -------- | ------- | ------- | ------- | -| (1) | 48m50 | Small Load | AF_PACKET | 41.23MB | 102.8MB | 8 | 2.7M | 2887 | 0 | 3170 | 0 | -| (2) | 1h1m45 | Small Load | AF_PACKET | 76.19MB | 162.2MB | 11.6 | 130MB | 384K | 48K | 263K | 0 | -| (3) | 13h43m15 | Stress | AF_PACKET | 620.3MB | 832.5MB | 60 | 4.6GB | 2.5M | 1M | 1.2M | 358 | - - diff --git a/src/pages/en/permissions.md b/src/pages/en/permissions.md deleted file mode 100644 index 1c32e8b..0000000 --- a/src/pages/en/permissions.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: Kubeshark's Security Context & RBAC -description: This document outlines the necessary Kubernetes permissions for the efficient operation of Kubeshark, a network monitoring tool. -layout: ../../layouts/MainLayout.astro ---- - -## The Worker DaemonSet - -[Kubeshark](https://kubeshark.com)'s Worker DaemonSet is a critical component designed to monitor network traffic within the Kubernetes cluster. To function effectively, it requires specific capabilities that go beyond the standard set. These capabilities are essential for enabling network sniffing and detailed traffic analysis. The security context for the Worker DaemonSet is defined as follows: - -#### Capabilities - -Below is the list of capabilities assigned to their respective features. If you disable certain features, their corresponding capabilities are not requested: -```yaml - capabilities: - networkCapture: - - NET_RAW - - NET_ADMIN - serviceMeshCapture: - - SYS_ADMIN - - SYS_PTRACE - - DAC_OVERRIDE - - CHECKPOINT_RESTORE - kernelModule: - - SYS_MODULE - ebpfCapture: - - SYS_ADMIN - - SYS_PTRACE - - SYS_RESOURCE - - CHECKPOINT_RESTORE -``` -This configuration can be changed via the `values.yaml`. - -## Service Account - -[Kubeshark](https://kubeshark.com) utilizes a dedicated Service Account named `kubeshark-service-account` for all its components. This account is specifically configured to provide the necessary access permissions for [Kubeshark](https://kubeshark.com)'s operations within the Kubernetes environment, ensuring secure and efficient performance. - -## Cluster Role - -The Cluster Role in [Kubeshark](https://kubeshark.com) is designed to grant broad permissions across the entire Kubernetes cluster. This role is crucial for [Kubeshark](https://kubeshark.com) to access and monitor various Kubernetes resources at a cluster-wide level. The Cluster Role Binding, detailed below, outlines these permissions: - -```yaml -rules: - - apiGroups: - - "" - - extensions - - apps - resources: - - pods - - services - - endpoints - - persistentvolumeclaims - verbs: - - list - - get - - watch -``` - -## Namespace Specific Role - -Within the specific namespace where [Kubeshark](https://kubeshark.com) is deployed, a Role Binding is used to grant targeted permissions for namespace-level resources. This ensures [Kubeshark](https://kubeshark.com)'s access to essential configurations and secrets within its operational namespace: - -```yaml -rules: - - apiGroups: - - "" # Core API group - - v1 # Version 1 of the core API group - resourceNames: - - kubeshark-secret # Specific secret for [Kubeshark](https://kubeshark.com) - - kubeshark-config-map # Specific config map for [Kubeshark](https://kubeshark.com) - resources: - - secrets # Access to secrets resource - - configmaps # Access to configmaps resource - verbs: - - get # Permission to get resource details - - watch # Permission to watch for changes in resources - - update # Permission to update resources -``` - -These permissions are integral for [Kubeshark](https://kubeshark.com)'s self-configuration and adaptive operation within the Kubernetes environment. \ No newline at end of file diff --git a/src/pages/en/pf_ring.md b/src/pages/en/pf_ring.md deleted file mode 100644 index e62fac1..0000000 --- a/src/pages/en/pf_ring.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: PF_RING -description: PF-RING is a traffic processing library, which is considered more performant than AF_PACKET and therefore more suitable for high-speed networks. -layout: ../../layouts/MainLayout.astro ---- - -[PF-RING](https://www.ntop.org/products/packet-capture/pf_ring/) is a high-performance traffic processing library, offering superior performance compared to [AF_PACKET](https://man7.org/linux/man-pages/man7/packet.7.html). It is particularly well-suited for high-speed network environments. - -The integration of PF_RING with [Kubeshark](https://kubeshark.com) is depicted in the following diagram: - -![Worker's Architecture](/worker-architecture.png) - -Due to PF_RING's requirement for specific support based on the Node's kernel version, and the vast variety of kernel versions, [Kubeshark](https://kubeshark.com)'s support for PF_RING cannot be universally guaranteed. - -## Adding Support for Your Kernel Version - -We consolidate all PF_RING module files into a single image named `kubeshark/pf-ring-module:all`. To check if your kernel version is supported, visit [this link](https://github.com/kubeshark/pf-ring-compiler/tree/main/modules/ko). - -Should your kernel version lack support, you can undertake the following steps to incorporate a new PF_RING module into the `kubeshark/pf-ring-module:all` image: - -1. Download the latest release of `pf-ring-compiler` for your platform from [here](https://github.com/kubeshark/pf-ring-compiler/releases). -2. Execute `pfring-compiler` for your target platform (for instance, EKS, which uses Amazon Linux 2): - -``` -pfring-compiler compile --target al2 -{"level":"info","msg":"creating compile job default/al2-pf-ring-compiler","time":"2024-03-21T14:17:52+02:00"} -{"level":"info","msg":"compile job default/al2-pf-ring-compiler created","time":"2024-03-21T14:17:52+02:00"} -{"level":"info","msg":"waiting for compile job to start","time":"2024-03-21T14:17:52+02:00"} -{"level":"info","msg":"compile job started","time":"2024-03-21T14:17:54+02:00"} -{"level":"info","msg":"waiting for compile pod to start","time":"2024-03-21T14:17:54+02:00"} -{"level":"info","msg":"compile pod started","time":"2024-03-21T14:18:12+02:00"} -{"level":"info","msg":"waiting for compile job to complete pf-ring module compilation","time":"2024-03-21T14:18:12+02:00"} -{"level":"info","msg":"pf-ring module compilation completed","time":"2024-03-21T14:19:41+02:00"} -{"level":"info","msg":"copying kernel module to local fs","time":"2024-03-21T14:19:41+02:00"} -{"level":"info","msg":"kernel module copied to pf-ring-5.10.210-201.852.amzn2.x86_64.ko","time":"2024-03-21T14:19:43+02:00"} -{"level":"info","msg":"cleaning up compile job","time":"2024-03-21T14:19:43+02:00"} -``` - -You will obtain a module named `pf-ring-5.10.210-201.852.amzn2.x86_64.ko` in the current folder, formatted as `pf-ring-.ko`. - -3. Clone the pf-ring-compiler repository: - -``` -git clone https://github.com/kubeshark/pf-ring-compiler -cd pf-ring-compiler -git checkout -b "add-module-" -``` - -4. Transfer the kernel module from step 2 into the `modules/ko` folder. -5. Commit your modifications and submit a pull request (PR): - -``` -git add modules/ko/pf-ring-5.10.210-201.852.amzn2.x86_64.ko -git commit -S -m "Add PF_RING module for kernel 5.10.210-201.852.amzn2.x86_64" -git push origin add-module- -``` - -After creating the PR, wait for it to be reviewed and merged. Following the merge, a GitHub Action pipeline will construct a new `kubeshark/pf-ring-module:all` image, enabling [Kubeshark](https://kubeshark.com) to support PF_RING for your kernel version. \ No newline at end of file diff --git a/src/pages/en/pod_to_pod_connections.md b/src/pages/en/pod_to_pod_connections.md deleted file mode 100644 index 588e6e5..0000000 --- a/src/pages/en/pod_to_pod_connections.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -title: Pod-to-pod Connection Analysis -description: Pod-to-pod connection analysis detects and analyzes connections between pods and external services by processing all TCP packets. The `tcp` dissector is disabled by default due to its impact on performance and must be manually enabled. Use with caution, especially in small or targeted clusters. Future updates will improve usability, reduce resource use, and add UDP analysis. -layout: ../../layouts/MainLayout.astro ---- - -This functionality is particularly useful for answering the following questions: - -1. Which pod is attempting to connect to a specific external IP or service? -2. Why is a certain pod-to-service connection failing? -3. Why am I not seeing certain traffic? - -## Enabling and Filtering - -To analyze pod-to-pod connections, follow these steps: -1. Ensure the TCP dissector is enabled by following the instructions in the [protocols section](/en/protocols#dynamically-configuring-available-dissectors). -2. Enter the `tcp` KFL alias in the display filter box. - -The above instructions will cause [Kubeshark](https://kubeshark.com) to analyze and show only TCP packets. - -Pod-to-pod connection analysis enables you to detect every connection between pods and external services. It displays all connections and allows the user to search for specific ones. Regardless of the protocol or encryption, as long as it runs over TCP, it will appear in the [Kubeshark](https://kubeshark.com) Dashboard. - -For example, the following image illustrates a namespace connectivity map, showing the connection between namespaces in the cluster and connections to external services. In this case, two external services, `grafana.com` and `gorest.co.in`, are clearly marked with a red rectangle. - -![Connectivity map](/connectivity.png) - -Building a connectivity map and viewing the raw packet content of pod-to-pod communication becomes possible when enabling the `tcp` dissector, which starts processing all TCP packets. In addition to the connectivity map, you can view and filter all TCP packets. - -![TCP log](/tcp_log.png) - -## Performance Impact - -> Performance impact is expected to improve significantly in the near future. - -Using this dissector can cause elevated CPU, memory, and storage consumption levels. We currently recommend using this functionality with caution. For example, we do not recommend using this functionality in busy clusters or targeting all pods in the cluster. Instead, we suggest using this functionality in conjunction with [pod targeting](/en/pod_targeting). - -In the following example, we see all the connectivity related to a single pod when setting the proper capture filter: - -![Showing only one pod](/one_pod.png) - -The following image shows that enabling this functionality increases CPU levels. However, using it in conjunction with a [capture filter](/en/pod_targeting) significantly reduces CPU consumption: - -![One pod performance](/one_pod_perf.png) - -## Useful Display Filter Queries - -- Show all cluster-to-external traffic: `dst.namespace==""` -- Show all traffic between two namespaces (e.g., ns1<=>ns2): `(src.namespace=="ns1" and dst.namespace=="ns2") or (src.namespace=="ns2" and dst.namespace=="ns1")` -- Show traffic going to an external domain: `dst.name=="www.domain.com"` - -> You can experience this functionality in our [live demo portal](https://kubeshark.kubeshark.com/) by entering `tcp` in the display filter. - -## What to Expect - -Pod-to-pod connection analysis by inspecting all TCP packets is a new feature that we plan to improve in subsequent releases. Our goals include: - -1. Making it easier to enable/disable this functionality. -2. Reducing resource consumption to normal levels, allowing this functionality to work in busy production clusters. -3. Adding significant insights derived from such analysis. -4. Adding UDP to the analysis. - -> The `dns` and `icmp` dissectors can also help in this analysis, usually indicating an intent to make a connection or an issue in the network. \ No newline at end of file diff --git a/src/pages/en/pro.md b/src/pages/en/pro.md deleted file mode 100644 index 5e5c73b..0000000 --- a/src/pages/en/pro.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: Community, Pro & Enterprise -description: Exploring the Differences Between Pro, Enterprise, and Community Editions -layout: ../../layouts/MainLayout.astro ---- - -## Community (Free) - -The Community edition is offered at no cost and supports clusters of up to 4 nodes and up to 80 pods. Requires cloud login. - -> To get started with the Community edition, [install **Kubeshark**](/en/install). - -## Pro - -Volume-based pricing for any cluster size. Download a license key from the [License Portal](https://console.kubeshark.com) for cluster-wide, multi-user access — no cloud login required. - -Pro licenses require an active internet connection (telemetry must succeed). - -See [Plans](/en/plans) for pricing tiers. - -## Enterprise - -With support for clusters of any size with dedicated support and SSO. -Our enterprise pricing starts at $20 per node per month. Operates in air-gapped environments with no internet required. - -### Obtaining an Enterprise POC License - -To obtain an Enterprise POC license, you'll first need an account in the [License Portal](https://console.kubeshark.com/), which is used to manage licenses. By default, a complimentary Pro license will be provisioned upon sign-up. - -Once your account is created, contact us using the [contact-us](https://kubeshark.com/contact-us) form, and provide the email address used to create the License Portal account. We'll convert your complimentary Pro license to an Enterprise license and notify you once it's ready. - -To retrieve your license, log in to the [License Portal](https://console.kubeshark.com/) and download your license key. diff --git a/src/pages/en/pro_upgrade.md b/src/pages/en/pro_upgrade.md deleted file mode 100644 index c9fb81c..0000000 --- a/src/pages/en/pro_upgrade.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: Upgrading & Downgrading -description: Upgrading & Downgrading -layout: ../../layouts/MainLayout.astro ---- - -## Upgrading to Pro - -Subscribe to a Pro plan at the [License Portal](https://console.kubeshark.com/?tab=pro), then download your license key from the portal. - -Apply the license key using one of the following methods: - -**With Helm:** - -```shell -helm install kubeshark kubeshark/kubeshark \ - --set license= -``` - -**With the CLI:** - -```shell -kubeshark tap --set license= -``` - -**Via configuration file** (~/.kubeshark/config.yaml): - -```yaml -license: -``` - -Once the license key is set, all users in the cluster can access Kubeshark without individual authentication. - -> **Note:** Pro licenses require an active internet connection. Telemetry must succeed for the license to remain valid. - -## Downgrading - -Using the Pro edition requires having a valid license key in the Kubeshark configuration file, that usually resides at ~/.kubeshark/config.yaml. - -To downgrade, simply erase the license key and Kubeshark will use the community version. - -No need to save the license key. You can always download it again from the [License Portal](https://console.kubeshark.com). - - - - diff --git a/src/pages/en/quick_start.md b/src/pages/en/quick_start.md deleted file mode 100644 index 8ae297d..0000000 --- a/src/pages/en/quick_start.md +++ /dev/null @@ -1,124 +0,0 @@ ---- -title: Quick Start -description: Get Kubeshark running in your Kubernetes cluster in under 60 seconds. -layout: ../../layouts/MainLayout.astro -mascot: Cute ---- - -Get [Kubeshark](https://kubeshark.com) running in your Kubernetes cluster in under 60 seconds. - ---- - -## Prerequisites - -Before you begin, ensure you have: - -- A running Kubernetes cluster (v1.16+) -- [kubectl](https://kubernetes.io/docs/tasks/tools/) configured to access your cluster -- [Helm](https://helm.sh/docs/intro/install/) v3.0+ installed - ---- - -## Installation - -### Step 1: Add the Helm Repository - -```bash -helm repo add kubeshark https://helm.kubeshark.com -helm repo update -``` - -### Step 2: Install Kubeshark - -```bash -helm install kubeshark kubeshark/kubeshark -``` - -
- -For production deployments, see the [full installation guide](/en/install) for configuration options. - -
- -### Step 3: Access the Dashboard - -Forward the dashboard port to your local machine: - -```bash -kubectl port-forward svc/kubeshark-front 8899:80 -``` - -Open [http://localhost:8899](http://localhost:8899) in your browser. - ---- - -## Verify Installation - -Check that all [Kubeshark](https://kubeshark.com) pods are running: - -```bash -kubectl get pods -l app.kubernetes.io/name=kubeshark -``` - -You should see output similar to: - -``` -NAME READY STATUS RESTARTS AGE -kubeshark-front-xxxxx 1/1 Running 0 1m -kubeshark-hub-xxxxx 1/1 Running 0 1m -kubeshark-worker-xxxxx 1/1 Running 0 1m -``` - ---- - -## First Steps in the Dashboard - -Once the dashboard loads: - -1. **View Live Traffic**: API calls appear in real-time in the traffic stream -2. **Filter Traffic**: Use the search bar to filter by protocol, pod, or content -3. **Inspect Details**: Click any request to see headers, payload, and response -4. **View Service Map**: Click the "Service Map" tab to visualize service dependencies - ---- - -## Common Commands - -| Command | Description | -|---------|-------------| -| `helm repo add kubeshark https://helm.kubeshark.com` | Add the Helm repository | -| `helm install kubeshark kubeshark/kubeshark` | Install [Kubeshark](https://kubeshark.com) | -| `helm upgrade kubeshark kubeshark/kubeshark` | Upgrade to latest version | -| `helm uninstall kubeshark` | Remove [Kubeshark](https://kubeshark.com) | -| `kubectl port-forward svc/kubeshark-front 8899:80` | Access dashboard | - ---- - -## What's Next? - -- [Dashboard Overview](/en/ui) - Learn the [Kubeshark](https://kubeshark.com) interface -- [Capture Filters](/en/pod_targeting) - Target specific pods and namespaces -- [TLS Decryption](/en/encrypted_traffic) - View encrypted traffic -- [Protocol Support](/en/protocols) - Configure protocol dissectors - ---- - -## Troubleshooting - -
- -If pods are not starting, check for common issues: - -
- -**Pods stuck in Pending state:** -```bash -kubectl describe pod -l app.kubernetes.io/name=kubeshark -``` - -**Check Kubeshark logs:** -```bash -kubectl logs -l app.kubernetes.io/name=kubeshark-hub -``` - -For more help, see the [Troubleshooting Guide](/en/troubleshooting). diff --git a/src/pages/en/raw_traffic_capture.md b/src/pages/en/raw_traffic_capture.md deleted file mode 100644 index 020e7dc..0000000 --- a/src/pages/en/raw_traffic_capture.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: PCAP Dumper (Capturing Raw Traffic) -description: The PCAP Dumper dumps the raw traffic into a named pipe directly from the node at a stage before any processing is done. This capability is particularly useful for debugging traffic when it isn't visible in the dashboard and some debugging is required. -layout: ../../layouts/MainLayout.astro -mascot: Hello ---- - -> The PCAP Dumper is a debug tool that aids in troubleshooting situations when traffic doesn't appear in the dashboard. - -The PCAP Dumper is akin to running `tcpdump` on the node. It dumps the raw traffic into a named pipe directly from the node at a stage before any processing is done. This capability is particularly useful for debugging traffic when it isn't visible in the dashboard and some debugging is required. - -Traffic capture occurs after a BPF expression is applied, if such an expression exists. This expression can be set if one of the [traffic targeting rules was applied](/en/pod_targeting). - -Raw traffic is piped on-demand and can be viewed in tools such as [Wireshark](https://www.wireshark.org/) or [Tshark](https://www.wireshark.org/docs/man-pages/tshark.html). Seeing the raw traffic can help identify any reasons why the traffic isn't visible in the dashboard. - -## How to View Raw Traffic - -Each node has a named pipe named `ksdump.pcap` located in the `data/` folder in the `sniffer` container of the Worker DaemonSet. - -To view the raw traffic of a certain node, you can use the following command: - -```shell -kubectl exec -c sniffer -- cat data//ksdump.pcap -``` - -Or - -```shell -kubectl exec -c sniffer -- tshark -i data//ksdump.pcap -w traffic.pcap -``` \ No newline at end of file diff --git a/src/pages/en/recordings_management.md b/src/pages/en/recordings_management.md deleted file mode 100644 index 065491b..0000000 --- a/src/pages/en/recordings_management.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: Recordings Management -description: Recordings Management -layout: ../../layouts/MainLayout.astro -mascot: Hello ---- - -Manage existing recordings effortlessly with the traffic recorder user interface, even if they are scheduled to start in the future. - -Available operations for managing recordings include: -- Listing all recordings -- Searching for specific recordings -- Deleting recordings -- Viewing recordings - -![Recordings Management](/recordings_management.png) diff --git a/src/pages/en/redaction.md b/src/pages/en/redaction.md deleted file mode 100644 index a549d99..0000000 --- a/src/pages/en/redaction.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: Sensitive Data Redaction -description: -layout: ../../layouts/MainLayout.astro -mascot: Cute ---- - -**Sensitive data can be redacted to maintain privacy and security.** - -## Enabling Sensitive Data Redaction - -Redaction works by running a script continuously. The script examines all traffic going from the **Workers to the browser**—and only that traffic. - -For example, if a display filter is used, only the traffic that passes the filter will be examined. Additionally, if no traffic is flowing from the Workers to the browser, the script remains idle. - -The script analyzes structured payloads and redacts what is perceived to be sensitive information. - -The redaction script can be activated as-is by selecting and enabling it from the script templates. - -![Activate the redaction script](/redact.png) - -The redaction process is performed **at the source**, at the individual **Worker level**, ensuring that sensitive information never travels over the network. - -## What is Redacted - -You can examine the script to see what data is redacted and from where. Essentially, the script processes all **headers, cookies, and JSON request and response payloads**. Values of keys considered sensitive are replaced with the string `[redacted]`. - -![Redacted sensitive information](/redact2.png) - -For illustration purposes, see this **code snippet** copied from the redaction script. You can inspect the entire script in the template section as demonstrated above. - -```javascript -var sensitiveFields = [ - // Authentication and Authorization - "password", "pwd", "pass", "passwd", "secret", - "token", "access_token", "refresh_token", "id_token", "session_token", - "auth_token", "authentication_token", "api_token", "jwt", "oauth_token", - "bearer_token", "client_token", "secret_token", "security_token", - "apikey", "api_key", "api_secret", "client_id", "client_secret", - "auth_code", "authorization_code", "login_token", "session_id", - - // Personally Identifiable Information (PII) - "email", "username", "user_id", "user", "phone", "phone_number", - "address", "street", "city", "state", "zip", "zipcode", "country", - "firstname", "lastname", "fullname", "name", "dob", "date_of_birth", - "ssn", "social_security_number", "nid", "national_id", - "passport_number", "driver_license", "dl_number", - - // Payment and Financial Information - "credit_card", "cc", "ccn", "card_number", "cvv", "cvc", - "expiry_date", "expiration_date", "iban", "swift", "routing_number", - "bic", "bank_account", "account_number", "balance", - - // Security and Encryption - "private_key", "public_key", "ssh_key", "ssl_key", "encryption_key", - "key", "api_secret_key", "secret_key", "cert", "certificate", - "fingerprint", "security_question", "security_answer", "pin", - "otp", "one_time_password", "recovery_key", "secure_code", - - // Healthcare Information - "health_id", "medical_record", "insurance_number", "policy_number", - "diagnosis", "treatment", "prescription", "patient_id", - - // Cookies and Tokens - "cookie", "set-cookie", "csrf_token", "xsrf_token", "csrf", "xsrf", - "tracking_cookie", "session_cookie", "authentication_cookie", - - // Miscellaneous - "id", "user_key", "user_secret", "authenticator", - "security_code", "access_code", "key_id", "app_key", "app_secret", - "device_id", "mac_address", "serial_number", "imei", "hostname", - "ip_address", "geolocation", "location", "coordinates" -]; - -var sensitiveHeaderFields = [ - // Authorization and Tokens - "authorization", "proxy-authorization", "x-api-key", "x-auth-token", - "x-access-token", "x-session-id", "x-session-token", "cookie", - "x-csrf-token", "x-xsrf-token", "csrf-token", "xsrf-token", - "x-client-id", "x-client-secret", "x-authentication-token", - "x-refresh-token", "x-bearer-token", "x-auth-code", - - // Custom Header Variations - "x-user-id", "x-username", "x-email", "x-password", - "x-token", "x-id-token", "x-oauth-token", "x-api-secret", - "x-authorization", "x-credentials", "x-signature", - "x-hmac-signature", "x-request-signature", "x-webhook-signature", - "x-hmac-key", "x-encryption-key", "x-secret-key", - "x-client-key", "x-public-key", "x-private-key", - - // Session and Cookies - "set-cookie", "x-cookie", "x-cookie-id", "x-tracking-cookie", - "x-session-cookie", "x-auth-cookie" -]; -``` -## Customization - -The script can be customized to meet specific user needs and hosted locally for better control and flexibility. diff --git a/src/pages/en/scope.md b/src/pages/en/scope.md deleted file mode 100644 index 98bfb44..0000000 --- a/src/pages/en/scope.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: Scope of Traffic Capture -description: Kubeshark enables you to describe the scope of traffic capture with support for namespaces and PODs -layout: ../../layouts/MainLayout.astro -mascot: ---- - -While capturing all traffic is possible, it is a storage and CPU intensive operation. [Kubeshark](https://kubeshark.com) enables you to describe the scope of traffic capture with support for namespaces and PODs. - -### Pods selection - -#### Specific Pod: - -```shell -kubeshark tap catalogue-b87b45784-sxc8q -``` - -#### Set of Pods Using a Regex: - -You can use a regular expression to indicate several pod names as well as dynamically changing names. - -In the example below using the regex `(catalo*|front-end*)` will catch the following three Pods: -* catalogue-868cc5ffd6-p9njn -* catalogue-db-669d5dbf48-8hnrl -* front-end-6db57bf84f-7kss9 - -```shell -kubeshark tap "(catalo*|front-end*)" -``` - -![PODS](/pods.png) - -### Namespaces - -By default, [Kubeshark](https://kubeshark.com) is deployed into the `default` namespace. -To specify a different namespace: - -``` -kubeshark tap -n sock-shop -``` - -### Specify All Namespaces - -The default deployment strategy of [Kubeshark](https://kubeshark.com) waits for the new Pods -to be created. To simply deploy to all existing namespaces run: - -``` -kubeshark tap -``` \ No newline at end of file diff --git a/src/pages/en/service_map.md b/src/pages/en/service_map.md deleted file mode 100644 index 08887b2..0000000 --- a/src/pages/en/service_map.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: Service Map in the Context of Kubernetes -description: Kubeshark offers a Service Dependency Graph that visualizes the relationships within your Kubernetes cluster. -layout: ../../layouts/MainLayout.astro ---- - -**Kubeshark** provides a comprehensive understanding of the complex network of interactions inside your cluster. You can toggle between pod and namespace perspectives and monitor latency, bandwidth, and throughput for pods, nodes, and namespaces. - -The **Service Map** displays the following data in a visual graph: -- Multiple identity layers such as namespaces, pod names, service names for internal workloads, and DNS identities for external workloads. -- Key Performance Indicators (KPIs) like latency, throughput, and bandwidth. -- Service connectivity details including protocol, direction, and density. - -![Service Dependency Graph](/new-service-map.png) - -To access the **Service Map**, click its button located in the upper right corner. - -## Focus on Specific Parts of the Cluster - -Dense clusters can yield cluttered **Service Maps**. With **Kubeshark**, you can zero in on particular sections of your cluster by pairing it with a [KFL2 filter](/en/v2/kfl2), narrowing down your analysis to specific segments of your cluster's traffic. - -## View Real-Time Changes - -By default, the **Service Map** refreshes in real-time, displaying changes inferred directly from incoming traffic. \ No newline at end of file diff --git a/src/pages/en/short_retention.md b/src/pages/en/short_retention.md deleted file mode 100644 index 23e1779..0000000 --- a/src/pages/en/short_retention.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: Short-term Traffic Retention -description: What to expect and how to control the short-term traffic retention -layout: ../../layouts/MainLayout.astro ---- - -[Kubeshark](https://kubeshark.com) provides real-time visibility into API traffic, enabling its users to view full API traffic details within a short-term retention window. This window is determined by the throughput and the storage allocated for it, all at the Kubernetes node level. Once outside of this window, details are no longer available to view and attempting to do so will result in an error. - -![Error Message](/storage_error.png) - -> For information on long-term traffic retention, read the [Traffic Recorder](/en/traffic_recorder) section. - -## Storage Limit & Guardrails - -Because storage can fill up rapidly, we have established certain guardrails to prevent overuse of disk resources. - -The storage limitation is regulated by the `tap.storageLimit` configuration value, with the default set to `10Gi`. This value represents a hard limit. If storage surpasses this limit, it will result in pod eviction. When the Worker pod is evicted, the storage is purged and the pod immediately restarts from scratch. - -To increase this limit, simply provide a different value (e.g., setting it to 1GB with `--set tap.storageLimit=1Gi`). - -When an L4 stream is dissected, a JSON file is generated with all relevant details. This file has a time to live of 5 minutes. - -The actual L4 stream (PCAP) has a time to live of 10 seconds, enough for a script to copy it to a different folder if retention is required. \ No newline at end of file diff --git a/src/pages/en/simple_ingress.md b/src/pages/en/simple_ingress.md deleted file mode 100644 index 3001237..0000000 --- a/src/pages/en/simple_ingress.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: Ingress -description: This article describes how to self host Kubeshark using Ingress. -layout: ../../layouts/MainLayout.astro ---- - -[Kubeshark](https://kubeshark.com) provides an option for self-hosting as a secure, authenticated web service that allows team members to access [Kubeshark](https://kubeshark.com) using a web browser with their corporate identities from remote. - -## Benefits - -Deploying Ingress is superior to using `port-forward` or `kubernetes proxy`. - -### Security - -When self-hosted as a web service, developers and security engineers can access [Kubeshark](https://kubeshark.com) remotely using a browser, authenticated with their corporate ID, without requiring RBAC / kubectl permissions. - -### Performance & Stability - -Compared to [port-forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) or Kubernetes proxy, [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) is much lighter and more stable. - -## Ingress Configuration - -Your cluster needs to have an ingress controller such as [Nginx](https://www.nginx.com/products/nginx-ingress-controller/) deployed. If you don't already have one in your cluster, you can install it by following one of the examples at the end of this article. - -### Ingress Config Values - -```shell -tap: - ingress: - enabled: false - classname: kubeshark-ingress-class - controller: k8s.io/ingress-nginx - host: ks.svc.cluster.local - tls: [] - certmanager: letsencrypt-prod -``` - -- To enable ingress set `tap.ingress.enabled` to `true`. -- Use the `tap.ingress.classname` and `tap.ingress.controller` when necessary for a more granular control. -- **host:** the IP/LB of the Ingress. You made a note earlier for that. - - -## Install Ingress - -```shell -helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx -helm install kubeshark-ingress ingress-nginx/ingress-nginx --namespace kubeshark-ingress --create-namespace -``` -Once installed, you'll need the IP or DNS of the Ingress Controller. You can retrieve it by running `kubectl get services `. It should look something like this: - -```shell -kubectl --namespace kubeshark-ingress get services -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -kubeshark-ingress-ingress-nginx-controller LoadBalancer 10.100.80.51 af5dde9619d1a44dfb01109571266776-16181046.us-east-2.elb.amazonaws.com 80:30550/TCP,443:30859/TCP 74m -kubeshark-ingress-ingress-nginx-controller-admission ClusterIP 10.100.159.17 443/TCP 74m -``` - -Copy the Ingress' external IP and use it in the `tap.ingress.host` field. \ No newline at end of file diff --git a/src/pages/en/support.md b/src/pages/en/support.md deleted file mode 100644 index 772d47c..0000000 --- a/src/pages/en/support.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: Getting Support -description: Explore the various ways to receive support. -layout: ../../layouts/MainLayout.astro ---- - -When you need support, be assured that we are continuously monitoring our [Slack channel](https://join.slack.com/t/kubeshark/shared_invite/zt-3jdcdgxdv-1qNkhBh9c6CFoE7bSPkpBQ). Feel free to [open a GitHub issue](https://github.com/kubeshark/kubeshark/issues), contact us through our [online form](https://kubeshark.com/support), or send an email to support@kubeshark.com. We will do our best to respond promptly. diff --git a/src/pages/en/traffic_investigation.md b/src/pages/en/traffic_investigation.md deleted file mode 100644 index c29150c..0000000 --- a/src/pages/en/traffic_investigation.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Deep Network Observability -description: Kubeshark as a real-time traffic investigation and debugging. -layout: ../../layouts/MainLayout.astro ---- - -**Kubeshark** offers real-time, cluster-wide, identity-aware, protocol-level visibility into API traffic, empowering its users to see in their own eyes what's happening in all (hidden) corners of their K8s clusters. Observe all traffic, including payloads, entering, exiting, and traversing containers, pods, namespaces, nodes, and clusters, with support for REST, GraphQL, gRPC, Redis, Kafka, RabbitMQ (AMQP), DNS, TLS, mTLS, ICMP and TCP (to diagnose TCP errors). - -Whether you are resolving issues with your infrastructure, analyzing potential threats, or delving into a security incident, **Kubeshark** can be highly valuable in pinpointing the responsible factors. - -## Protocol-level Visibility - -**Kubeshark** captures, dissects and monitors all traffic and payloads going in, out and across containers, pods, nodes and clusters. You can view the dissected protocol messages in the dashboard all the way to the payload level. - -![Protocol-level visibility](/ui-full.png) - -## Kubeshark Filter Language (KFL2) - -As K8s network is massive, [KFL2 filtering](/en/v2/kfl2) enables you to find the \`needle in the haystack\`. - -Here are a few examples: - -#### Filtering traffic that uses a specific token (or tokens in general) - -```python -request.headers["Authorization"] == r"Token.*" -``` - -![Detecting Tokens](/kfl-token.png) - -#### Focusing on a Certain Node - -When you'd like to analyze the traffic at a specific node or set of nodes. - -![KFL Node](/kfl-node.png) - -## Workload Map - -**Kubeshark** offers an instant [Workload Map](/en/v2/service_map) that updates in real-time, and can be used to focus your analysis on specific parts of the cluster. - -In conjunction with its filtering language ([KFL2](/en/v2/kfl2)), **Kubeshark** enables you to focus on specific parts of your cluster and reduce the scope of analysis to only a subset of your cluster's traffic. - -For example, the following query will analyze the ingress traffic of two pods and the egress traffic of a third pod: - -![Query a Subset of Traffic](/query-subset.png) - -The resulting query will show the following Workload Map: - -![Service Map Subset](/service-map-subset.png) - diff --git a/src/pages/en/traffic_recorder.md b/src/pages/en/traffic_recorder.md deleted file mode 100644 index 1a83ccd..0000000 --- a/src/pages/en/traffic_recorder.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: Traffic Recorder -description: -layout: ../../layouts/MainLayout.astro ---- - -## Getting Started Quickly - -To start a recording job quickly and seamlessly, follow these steps: - -1. Click the **recording button** located next to the KFL statement box. - ![Traffic Recorder Button](/record_button.png) -2. Ensure the `Start Time` field is empty. If it's not, click the `X` to clear it. This will start the recording immediately. -3. Click **CREATE**. -4. Wait a few minutes, then return to the recording dialog to view the captured traffic. - -> The default [Kubeshark](https://kubeshark.com) configuration is not optimized for the Traffic Recorder, especially in busy clusters. For best results, review the [configuration section](/en/traffic_recorder#troubleshooting--configuration-tuning). - -The **Traffic Recorder** supports multiple concurrent recording jobs. Each job captures traffic independently, based on a specific [KFL](/en/filtering) statement and schedule. Recordings can be filtered and analyzed using [Kubeshark](https://kubeshark.com)’s advanced query language. - ---- - -### Recording Job Properties - -Configure recording job settings in the recording dialog window using the following parameters: - -| Property | Example Values | Description | -|------------------|----------------------------------------|-----------------------------------------------------------------------------| -| KFL Statement | `http or dns` | The KFL pattern used to filter and record traffic. | -| Name | `my_recording_no_15` | The name of the job and the corresponding folder. Used to retrieve data. | -| Start Time | `08:45 UTC` or leave empty | Scheduled start time in UTC. Leave empty to start immediately. | -| Daily Iterations | `1` (once), `0` (forever), `n` (days) | Number of days the job should run. Use `1` for once or `0` to run forever. | -| Duration | `60`, `1440` | Duration in minutes to record traffic after the job starts. | -| Expiration | `1440`, `4320` | Time in minutes before the recording folder is deleted to free storage. | - -![Traffic Recorder Dialog](/recording_dialog.png) - ---- - -## Behavior-Based Recording - -KFL statements can filter traffic by Kubernetes attributes (e.g., pods, namespaces) and behavior. For example: - -```yaml -(response.status == 500) and -(request.headers["User-Agent"] == "kube-probe/1.27+") and -(src.name == "kube-prometheus-stack-prometheus-node-exporter") -``` - -This captures L4 streams for API traffic that meets specific criteria, such as being initiated by a known service and returning HTTP status code `500`. - -> Next: [Recorded Traffic Offline Analysis](/en/offline_analysis) - ---- - -## Troubleshooting & Configuration Tuning - -By default, [Kubeshark](https://kubeshark.com) is optimized for real-time traffic viewing in small clusters. However, the **Traffic Recorder** requires tuning for performance in busy or large-scale environments. Incorrect configuration can lead to Worker pod eviction and loss of PCAP data. - ---- - -### Signs of Misconfiguration - -#### Eviction - -The `tap.storageLimit` default is `10Gi`. When exceeded, the Worker pod is evicted, its storage is purged, and it is restarted. - -Large volumes of PCAP or JSON data can exceed this limit quickly, especially in busy clusters. Once evicted, all recorded data is lost, and the recording will appear empty. - ---- - -### Suggested Configuration Values - -Tuning depends on your available resources and traffic volume. Below is a sample configuration—feel free to increase values as needed for your environment. - -```yaml ---set tap.storageLimit=5Gi --set tap.misc.pcapTTL=30s -``` - -or as YAML: - -```yaml -tap: - misc: - pcapTTL: 30s - storageLimit: 5Gi -``` - -> There's no need to adjust JSON TTL settings for the Traffic Recorder to function effectively. \ No newline at end of file diff --git a/src/pages/en/traffic_troubleshooting.md b/src/pages/en/traffic_troubleshooting.md deleted file mode 100644 index 54f3245..0000000 --- a/src/pages/en/traffic_troubleshooting.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: Traffic Troubleshooting -description: Troubleshoot traffic across all nodes in your Kubernetes cluster -layout: ../../layouts/MainLayout.astro ---- - -To help network engineers troubleshoot traffic, [Kubeshark](https://kubeshark.com) added two interesting tools that can help learn more about specific traffic. -Traffic is huge and it's impossible to track all traffic at all times. -Kubesahrk provides tools to segment the traffic and learn as much as possible and as a result address questions one can have about said traffic. - -## AF_PACKET - -When using the AF_PACKET library, Kubesahrk by default taps to the `any` interface which is the sum of all interfaces. - -In a Linux system, the any interface is a special network interface that is used to capture traffic from all available network interfaces. Unlike standard network interfaces such as eth0 or wlan0, the any interface is not associated with a specific hardware device. Instead, it is a virtual interface that allows tools like packet sniffers to capture network traffic across all interfaces simultaneously. - - - -## eBPF - -Traffic Targeting -Explicitely seeting a BPF expression will override any logic imposed by pod targeting rules -BPF Pipes - -Listeinign to a single interface -Tapping a virtual interface -no problems with any. diff --git a/src/pages/en/v2/API_dissection.md b/src/pages/en/v2/API_dissection.md deleted file mode 100644 index 1d373e0..0000000 --- a/src/pages/en/v2/API_dissection.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: L7 API Dissection -description: Reconstructing the API Context by Correlating Information from All Layers - -layout: ../../../layouts/MainLayout.astro -mascot: ---- - -## API Dissection - -API dissection goes beyond basic traffic inspection. It reconstructs complete API calls from network data by identifying requests and responses, protocol metadata, and payloads. - -This process requires buffering both ingress and egress traffic between two peers, matching requests to responses, detecting the underlying protocol, and parsing payloads according to the protocol specification. - -Once API calls are successfully dissected, each call is enriched with workload and application identities. These identities are derived by correlating Kubernetes events from the Kubernetes API server with operating system context collected from distributed nodes via eBPF. The result links each API call to its originating pod, service, namespace, labels, and, when available, the specific process that generated the traffic. - -Unlike raw packet data, which lacks higher-level context, API dissection provides structured, semantically meaningful information. - -![API Dissection](/api_dissect.png) \ No newline at end of file diff --git a/src/pages/en/v2/tcp_udp_connections.md b/src/pages/en/v2/tcp_udp_connections.md deleted file mode 100644 index cf65445..0000000 --- a/src/pages/en/v2/tcp_udp_connections.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: TCP/UDP Connections -description: View and analyze L4 TCP and UDP connections across your Kubernetes cluster. -layout: ../../../layouts/MainLayout.astro ---- - -**TCP/UDP Connections** provide visibility into L4 network connections across your Kubernetes cluster. Each L7 API message is connected to an L4 connection, and each L4 connection can include one or multiple L7 messages. - -## What's in an L4 Connection - -An L4 (TCP/UDP) connection holds all data from the beginning of the connection to its end, including: - -- **Network metadata** - Source/destination IPs, ports, protocol -- **Kubernetes context** - Labels, workloads, pods, services, namespaces, nodes -- **Operating system info** - Process ID and other eBPF-captured data -- **PCAP data** - Raw packets for the entire connection - -## L4 to L7 Mapping - -When a connection is dissected, L7 objects (API calls) are created and mapped to the L4 connection. This enables: - -- Correlating API calls to their underlying network connection -- Viewing all L7 messages within a single L4 connection -- Accessing raw PCAP data for any API call - diff --git a/src/pages/en/verify.md b/src/pages/en/verify.md deleted file mode 100644 index 5bcd473..0000000 --- a/src/pages/en/verify.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -title: Installation Checklist -description: -layout: ../../layouts/MainLayout.astro -mascot: Bookworm ---- -[Kubeshark](https://kubeshark.com)'s default configuration, as provided in the [values.yaml](https://github.com/kubeshark/kubeshark/blob/master/helm-chart/values.yaml) file, is suited for small dev/test clusters. - -We recommend starting with a smaller, quieter cluster before moving to larger deployments. - -It’s helpful to go through this checklist to ensure everything works correctly in the backend. - -It's a good practice to initially allocate more memory than expected, while optimizing various configuration settings. - -### Sufficient Memory for the Hub - -Check if the Hub is being OOMKilled. On larger clusters, we recommend [increasing the memory allocation](/en/performance#container-memory-and-cpu-limitations). - -### Sufficient Memory, CPU, and Storage Resources for Worker DaemonSets - -The workers consume resources based on the amount of traffic they process. Check the resource allocations and [increase them as needed](/en/performance#container-memory-and-cpu-limitations). Alternatively, [disable redundant protocol dissectors](/en/protocols#configuring-available-dissectors) or [apply capture filters](/en/pod_targeting). - -### Protocol Dissectors - -TCP and DNS processing can consume significant CPU, memory, and storage. We recommend [enabling them on demand](/en/protocols#configuring-available-dissectors), especially TCP, which is more resource-intensive than DNS. Disabling unnecessary protocol dissectors can help reduce resource consumption. - -### Capture Filters - -Applying [capture filters](/en/pod_targeting) will limit the amount of traffic [Kubeshark](https://kubeshark.com) processes, reducing the CPU, memory, and storage usage by the Worker DaemonSets. - -Many of the above values can be adjusted dynamically. Therefore, we suggest limiting [Kubeshark](https://kubeshark.com)'s processing when it's not in use and scaling resources as needed on demand. - -### Ensure You’re Capturing the Right Traffic - -Especially when using [Kubeshark](https://kubeshark.com) for the first time, make sure you’re capturing the traffic you expect. - -### Troubleshooting - -If you follow the above guidance and [Kubeshark](https://kubeshark.com) is still not functioning as expected, consult our [troubleshooting page](/en/troubleshooting). - - -## Suggested Configuration Values - -```yaml -tap: - disableTlsLog: true - capture: - dissection: - enabled: true - resources: - hub: - limits: - memory: 2750Mi - sniffer: - limits: - cpu: 1750m - memory: 2750Mi - tracer: - limits: - memory: 750Mi - storageLimit: 10Gi - ingress: - enabled: true - className: nginx - host: - annotations: - cert-manager.io/cluster-issuer: letsencrypt - kubernetes.io/tls-acme: "true" -``` \ No newline at end of file