Skip to content

Commit 8e78454

Browse files
stumeliusdatamie-simo
stumelius
and
datamie-simo
authored
Add 'dbt-cloud audit-log get' command (#58)
* Add 'dbt-cloud audit-log get' command * Add unit test * Update README * Add integration test Co-authored-by: Simo Tumelius <simo@datamie.fi>
1 parent dfa344d commit 8e78454

10 files changed

Lines changed: 278 additions & 7 deletions

File tree

‎.circleci/config.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,14 @@ jobs:
155155
account_count=$(cat accounts.json | jq '.data | length')
156156
[[ $account_count > 0 ]] && exit 0 || exit 1
157157
158+
- run:
159+
name: Test 'dbt-cloud audit-log get'
160+
command: |
161+
dbt-cloud audit-log get > audit_logs.json
162+
cat audit_logs.json | jq '.data[] | {id: .id}'
163+
log_count=$(cat audit_logs.json | jq '.data | length')
164+
[[ $log_count > 0 ]] && exit 0 || exit 1
165+
158166
- run:
159167
name: Test 'dbt-cloud metadata query'
160168
command: |

‎README.md‎

Lines changed: 72 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ Group | API endpoint | Command | Description |
3939
| --- | --- | --- | --- |
4040
| Accounts | [https://cloud.getdbt.com/api/v2/accounts/](https://docs.getdbt.com/dbt-cloud/api-v2#operation/listAccounts) | [dbt-cloud account list](#dbt-cloud-account-list) | Retrieves all available accounts |
4141
| Accounts | [https://cloud.getdbt.com/api/v2/accounts/{accountId}/](https://docs.getdbt.com/dbt-cloud/api-v2#operation/getAccountById) | `dbt-cloud account get` | Not implemented yet |
42+
| Audit Logs | https://cloud.getdbt.com/api/v3/accounts/{accountId}/audit-logs/ | [dbt-cloud audit-log get](#dbt-cloud-audit-log-get) | Retrieves audit logs for the dbt Cloud account |
4243
| Projects | https://cloud.getdbt.com/api/v2/accounts/{accountId}/projects/ | [dbt-cloud project list](#dbt-cloud-project-list) | Returns a list of projects in the account |
4344
| Projects | [https://cloud.getdbt.com/api/v2/accounts/{accountId}/projects/{projectId}](https://docs.getdbt.com/dbt-cloud/api-v2#operation/getProjectById) | `dbt-cloud project get` | Not implemented yet |
4445
| Environments | https://cloud.getdbt.com/api/v3/accounts/{accountId}/projects/{projectId}/environments | [dbt-cloud environment list](#dbt-cloud-environment-list) | Retrieves environments for a given project |
@@ -59,6 +60,7 @@ Group | API endpoint | Command | Description |
5960
# Commands
6061

6162
* [dbt-cloud account list](#dbt-cloud-account-list)
63+
* [dbt-cloud audit-log get](#dbt-cloud-audit-log-get)
6264
* [dbt-cloud project list](#dbt-cloud-project-list)
6365
* [dbt-cloud environment list](#dbt-cloud-environment-list)
6466
* [dbt-cloud job run](#dbt-cloud-job-run)
@@ -143,6 +145,72 @@ This command retrieves all available dbt Cloud accounts. For more information on
143145
```
144146
</details>
145147

148+
## dbt-cloud audit-log get
149+
150+
❗ **This command is available for Enterprise accounts only.**
151+
152+
This command retrieves audit logs for the dbt Cloud account. For more information on the command, run `dbt-cloud audit-log get --help`. This command uses the API v3 which has no official documentation yet.
153+
154+
<details>
155+
<summary><b>Usage</b></summary>
156+
157+
```bash
158+
>> dbt-cloud audit-log get --logged-at-start 2022-05-01 --logged-at-end 2022-05-07 --limit 1
159+
{
160+
"status": {
161+
"code": 200,
162+
"is_success": true,
163+
"user_message": "Success!",
164+
"developer_message": ""
165+
},
166+
"data": [
167+
{
168+
"account_id": 123456,
169+
"service": "SERVICE_DBT_CLOUD",
170+
"source": "SOURCE_CLOUD_UI",
171+
"routing_key": "v1.events.auth.credentialsloginsucceeded",
172+
"actor_type": "ACTOR_USER",
173+
"actor_name": "REDACTED",
174+
"actor_id": 123454,
175+
"logged_at": "2022-05-05 06:51:10+00:00",
176+
"uuid": "8868c439-8928-4e8c-924b-77558d65db0b",
177+
"actor_ip": "REDACTED",
178+
"metadata": {
179+
"auth_credentials": {
180+
"user": {
181+
"id": "REDACTED",
182+
"email": "REDACTED"
183+
}
184+
}
185+
},
186+
"internal": false,
187+
"id": 1809583,
188+
"state": 1,
189+
"created_at": "2022-05-05 06:51:12.454677+00:00",
190+
"updated_at": "2022-05-05 06:51:12.454677+00:00"
191+
}
192+
],
193+
"extra": {
194+
"filters": {
195+
"account_id": 123456,
196+
"limit": 1,
197+
"offset": 0,
198+
"logged_at__range": [
199+
"2022-05-01 00:00:00Z",
200+
"2022-05-07 00:00:00Z"
201+
],
202+
"internal": false
203+
},
204+
"order_by": "-logged_at",
205+
"pagination": {
206+
"count": 1,
207+
"total_count": 4
208+
}
209+
}
210+
}
211+
```
212+
</details>
213+
146214

147215
## dbt-cloud project list
148216
This command returns a list of projects in the account. For more information on the API endpoint arguments and response, run `dbt-cloud project list --help` and check out the [dbt Cloud API docs](https://docs.getdbt.com/dbt-cloud/api-v2#operation/listProjects).
@@ -717,7 +785,7 @@ This command deletes a job in a dbt Cloud project. Note that this command uses a
717785

718786
## dbt-cloud job delete-all
719787

720-
**This command is a composition of one or more base commands.**
788+
💡 **This command is a composition of one or more base commands.**
721789

722790
This command fetches all jobs on the account, deletes them one-by-one after user confirmation via prompt and prints out the job delete responses. For more information on the command and its arguments, run `dbt-cloud job delete-all --help`.
723791

@@ -848,7 +916,7 @@ Job 54659 was deleted.
848916

849917
## dbt-cloud job export
850918

851-
**This command is a composition of one or more base commands.**
919+
💡 **This command is a composition of one or more base commands.**
852920

853921
This command exports a dbt Cloud job as JSON to a file and can be used in conjunction with [dbt-cloud job import](#dbt-cloud-job-import) to copy jobs between dbt Cloud projects.
854922

@@ -908,7 +976,7 @@ This command exports a dbt Cloud job as JSON to a file and can be used in conjun
908976

909977
## dbt-cloud job import
910978

911-
**This command is a composition of one or more base commands.**
979+
💡 **This command is a composition of one or more base commands.**
912980

913981
This command imports a dbt Cloud job from exported JSON. You can use JSON manipulation tools (e.g., [jq](https://stedolan.github.io/jq/)) to modify the job definition before importing it.
914982

@@ -1215,7 +1283,7 @@ This command cancels a dbt Cloud run. For more information on the API endpoint a
12151283

12161284
## dbt-cloud run cancel-all
12171285

1218-
**This command is a composition of one or more base commands.**
1286+
💡 **This command is a composition of one or more base commands.**
12191287

12201288
This command fetches all runs on the account, cancels them one-by-one after user confirmation via prompt and prints out the run cancellation responses. For more information on the command and its arguments, run `dbt-cloud run cancel-all --help`.
12211289

‎dbt_cloud/cli.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
DbtCloudProjectListCommand,
2020
DbtCloudEnvironmentListCommand,
2121
DbtCloudAccountListCommand,
22+
DbtCloudAuditLogGetCommand,
2223
)
2324
from dbt_cloud.demo import data_catalog
2425
from dbt_cloud.serde import json_to_dict, dict_to_json
@@ -71,6 +72,11 @@ def account():
7172
pass
7273

7374

75+
@dbt_cloud.group(help="Interact with dbt Cloud audit logs (Enterprise only).")
76+
def audit_log():
77+
pass
78+
79+
7480
@dbt_cloud.group(help="Interact with the dbt Cloud Metadata API.")
7581
def metadata():
7682
pass
@@ -332,6 +338,13 @@ def list(**kwargs):
332338
response = execute_and_print(command)
333339

334340

341+
@audit_log.command(help=DbtCloudAuditLogGetCommand.get_description())
342+
@DbtCloudAuditLogGetCommand.click_options
343+
def get(**kwargs):
344+
command = DbtCloudAuditLogGetCommand.from_click_options(**kwargs)
345+
response = execute_and_print(command)
346+
347+
335348
@metadata.command(help=DbtCloudMetadataQueryCommand.get_description())
336349
@click.option(
337350
"-f",

‎dbt_cloud/command/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,6 @@
1616
from .project import DbtCloudProjectListCommand
1717
from .environment import DbtCloudEnvironmentListCommand
1818
from .account import DbtCloudAccountListCommand
19+
from .audit_log import DbtCloudAuditLogGetCommand
1920
from .metadata import DbtCloudMetadataQueryCommand
2021
from .command import DbtCloudAccountCommand

‎dbt_cloud/command/audit_log/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
from .get import DbtCloudAuditLogGetCommand

‎dbt_cloud/command/audit_log/get.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
import requests
2+
from typing import Optional
3+
from pydantic import Field, PrivateAttr
4+
from dbt_cloud.command.command import DbtCloudAccountCommand
5+
6+
7+
class DbtCloudAuditLogGetCommand(DbtCloudAccountCommand):
8+
"""Retrieves audit logs for the dbt Cloud account."""
9+
10+
logged_at_start: Optional[str] = Field(
11+
description="Start date (YYYY-MM-DD) for the returned logs."
12+
)
13+
logged_at_end: Optional[str] = Field(
14+
description="End date (YYYY-MM-DD) for the returned logs."
15+
)
16+
offset: Optional[int] = Field(
17+
0,
18+
ge=0,
19+
description="Offset for the returned logs. Must be a positive integer.",
20+
)
21+
limit: Optional[int] = Field(
22+
100,
23+
ge=0,
24+
description="A limit on the number of logs to be returned. Must be a positive integer.",
25+
)
26+
_api_version: str = PrivateAttr("v3")
27+
28+
@property
29+
def api_url(self) -> str:
30+
return f"{super().api_url}/audit-logs/"
31+
32+
def execute(self) -> requests.Response:
33+
response = requests.get(
34+
url=self.api_url,
35+
headers=self.request_headers,
36+
params=self.get_payload(
37+
exclude=["api_token", "dbt_cloud_host", "account_id"]
38+
),
39+
)
40+
return response

‎dbt_cloud/command/run/list.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@ class DbtCloudRunListCommand(DbtCloudAccountCommand):
2020

2121
limit: Optional[int] = Field(
2222
100,
23-
gte=1,
24-
lte=100,
23+
ge=1,
24+
le=100,
2525
description="A limit on the number of objects to be returned, between 1 and 100.",
2626
)
2727
environment_id: Optional[str] = Field(description="Filter runs by environment ID.")

‎pytest.ini‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,5 @@ markers =
66
run: tests related to dbt Cloud runs
77
project: tests related to dbt Cloud projects
88
environment: tests related to dbt Cloud environments
9-
account: tests related to dbt Cloud accounts
9+
account: tests related to dbt Cloud accounts
10+
audit_log: tests related to dbt Cloud audit logs

‎tests/conftest.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
DbtCloudRunListCommand,
1616
DbtCloudEnvironmentListCommand,
1717
DbtCloudAccountListCommand,
18+
DbtCloudAuditLogGetCommand,
1819
)
1920

2021

@@ -154,6 +155,17 @@ def load_response(response_name):
154155
"get",
155156
marks=pytest.mark.account,
156157
),
158+
pytest.param(
159+
"audit_log_get",
160+
DbtCloudAuditLogGetCommand(
161+
api_token=API_TOKEN,
162+
logged_at_start="2022-05-01",
163+
logged_at_end="2022-05-07",
164+
),
165+
load_response("audit_log_get_response"),
166+
"get",
167+
marks=pytest.mark.audit_log,
168+
),
157169
]
158170

159171

‎tests/data/audit_log_get_response.json‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
{
2+
"status": {
3+
"code": 200,
4+
"is_success": true,
5+
"user_message": "Success!",
6+
"developer_message": ""
7+
},
8+
"data": [
9+
{
10+
"account_id": 123456,
11+
"service": "SERVICE_DBT_CLOUD",
12+
"source": "SOURCE_CLOUD_UI",
13+
"routing_key": "v1.events.auth.credentialsloginsucceeded",
14+
"actor_type": "ACTOR_USER",
15+
"actor_name": "REDACTED",
16+
"actor_id": 123454,
17+
"logged_at": "2022-05-05 06:51:10+00:00",
18+
"uuid": "8868c439-8928-4e8c-924b-77558d65db0b",
19+
"actor_ip": "REDACTED",
20+
"metadata": {
21+
"auth_credentials": {
22+
"user": {
23+
"id": "REDACTED",
24+
"email": "REDACTED"
25+
}
26+
}
27+
},
28+
"internal": false,
29+
"id": 1809583,
30+
"state": 1,
31+
"created_at": "2022-05-05 06:51:12.454677+00:00",
32+
"updated_at": "2022-05-05 06:51:12.454677+00:00"
33+
},
34+
{
35+
"account_id": 123456,
36+
"service": "SERVICE_DBT_CLOUD",
37+
"source": "SOURCE_CLOUD_UI",
38+
"routing_key": "v1.events.auth.credentialsloginsucceeded",
39+
"actor_type": "ACTOR_USER",
40+
"actor_name": "REDACTED",
41+
"actor_id": 123454,
42+
"logged_at": "2022-05-05 05:42:26+00:00",
43+
"uuid": "029625d0-8dcf-4fa7-9567-26f8f4648122",
44+
"actor_ip": "REDACTED",
45+
"metadata": {
46+
"auth_credentials": {
47+
"user": {
48+
"id": "REDACTED",
49+
"email": "REDACTED"
50+
}
51+
}
52+
},
53+
"internal": false,
54+
"id": 1809076,
55+
"state": 1,
56+
"created_at": "2022-05-05 05:42:30.635635+00:00",
57+
"updated_at": "2022-05-05 05:42:30.635635+00:00"
58+
},
59+
{
60+
"account_id": 123456,
61+
"service": "SERVICE_DBT_CLOUD",
62+
"source": "SOURCE_CLOUD_UI",
63+
"routing_key": "v1.events.auth.credentialsloginsucceeded",
64+
"actor_type": "ACTOR_USER",
65+
"actor_name": "REDACTED",
66+
"actor_id": 123454,
67+
"logged_at": "2022-05-04 16:13:33+00:00",
68+
"uuid": "ba0c717d-6ea7-4e10-b4ba-48e956163938",
69+
"actor_ip": "REDACTED",
70+
"metadata": {
71+
"auth_credentials": {
72+
"user": {
73+
"id": "REDACTED",
74+
"email": "REDACTED"
75+
}
76+
}
77+
},
78+
"internal": false,
79+
"id": 1802985,
80+
"state": 1,
81+
"created_at": "2022-05-04 16:13:34.295300+00:00",
82+
"updated_at": "2022-05-04 16:13:34.295300+00:00"
83+
},
84+
{
85+
"account_id": 123456,
86+
"service": "SERVICE_DBT_CLOUD",
87+
"source": "SOURCE_CLOUD_UI",
88+
"routing_key": "v1.events.auth.credentialsloginsucceeded",
89+
"actor_type": "ACTOR_USER",
90+
"actor_name": "REDACTED",
91+
"actor_id": 123454,
92+
"logged_at": "2022-05-02 06:33:48+00:00",
93+
"uuid": "fe0dfd93-8654-4568-8c13-ecf61eaf9cdb",
94+
"actor_ip": "REDACTED",
95+
"metadata": {
96+
"auth_credentials": {
97+
"user": {
98+
"id": "REDACTED",
99+
"email": "REDACTED"
100+
}
101+
}
102+
},
103+
"internal": false,
104+
"id": 1770400,
105+
"state": 1,
106+
"created_at": "2022-05-02 06:33:49.777343+00:00",
107+
"updated_at": "2022-05-02 06:33:49.777343+00:00"
108+
}
109+
],
110+
"extra": {
111+
"filters": {
112+
"account_id": 123456,
113+
"limit": 10,
114+
"offset": 0,
115+
"logged_at__range": [
116+
"2022-05-01 00:00:00Z",
117+
"2022-05-07 00:00:00Z"
118+
],
119+
"internal": false
120+
},
121+
"order_by": "-logged_at",
122+
"pagination": {
123+
"count": 4,
124+
"total_count": 4
125+
}
126+
}
127+
}

0 commit comments

Comments
 (0)