[Bug]: Experimental search driver for log analytics passes an incorrect timespan to the API

[JPvRiel]

Describe the bug

The new experimental AzureSearchDriver does not seem to pass the expected timespan parameter to the API.

A 'Timespan must be supplied for /search API' error is returned by the API.

When use browser dev tools, I see the Azure Monitor web app calls the API with an HTTP parameter, e.g. search?timespan=2025-02-14T00:00:00.000Z/2025-02-15T12:00:52.000Z.

In debug output for msticpy with httpx logging, I do not see the API called using a timespan param.

In the source code it appears that the timespan info is being converted and passed as a query body instead of being set as HTTP parameters. See:

msticpy/msticpy/data/drivers/azure_search_driver.py

Line 127 in a4b0b72

query_body = {

To Reproduce

Steps to reproduce the behavior:

1. Clone from this git repo in order to test PR Ianhelle/az monitor search driver 2025 02 05 #825 included in main.
2. Create an editable venv from the source and activate.
3. Configure msticpyconfig.yaml with a 'Sentinel' workspac.
4. Run a test script with the AzureSearchDriver on a table with the 'basic' plan.

E.g. editable venv from source

# Clone upstream main branch
if [ ! -d './src/msticpy' ]; then
    mkdir -p ./src/msticpy
    git clone --depth 1 --branch main https://github.com/microsoft/msticpy.git src/msticpy
fi
cd src/msticpy
git checkout main
git pull --depth 1
cd -

# Setup a venv for src main upstream
if [ ! -d './venvs/src/main' ]; then
    mkdir -p './venvs/src/main'
    python3.11 -m venv ./venv/src/main
fi
source ./venv/src/main/bin/activate

# Install msticpy from src main in editable mode
pip install --editable ./src/msticpy

# Activate venv
source ./venv/src/main/bin/activate

E.g. code to trigger API error

import datetime

# Set debug logging
import logging
logging.basicConfig(level=logging.DEBUG)

# Inherit log level
import msticpy
print(f'msticpy version: {msticpy.__version__}')

# Config
msticpy.init_notebook()

# Unsure why init_notebook does not get this driver module loaded. 
# Workaround for "AttributeError: module 'msticpy.data.drivers' has no attribute 'azure_search_driver'"
import msticpy.data.drivers.azure_search_driver  

drv_basic_search = msticpy.data.drivers.azure_search_driver.AzureSearchDriver(debug=True)
drv_basic_search.connect(workspace='BasicLogs')

# Prep a small time range to limit basic logs query costs
lookback_period = datetime.timedelta(hours=1)
ingest_grace_period = datetime.timedelta(minutes=15)
end = datetime.datetime.now(datetime.timezone.utc) - ingest_grace_period
start = end - lookback_period
time_span=dict(
    start=start.isoformat(timespec='seconds'),
    end=end.isoformat(timespec='seconds')
)
print(f'time_span: {time_span}')

# Test query
df, results = drv_basic_search.query_with_results('SyslogBasic_CL | take 1', time_span=time_span)
print(results)
print(df)

Expected behavior

The timespan is passed to the API and a dataframe is returned.

Screenshots and/or Traceback

...
INFO:msticpy.data.drivers.azure_monitor_driver:Time parameters set TimeSpan(start=2025-02-16 18:56:29+00:00, end=2025-02-16 19:56:29+00:00, period=0 days 01:00:00)
...
INFO:httpx:HTTP Request: POST https://api.loganalytics.io/v1/workspaces/<*** workspace_id ***>/search "HTTP/1.1 400 Bad Request"
...
ERROR:msticpy.data.drivers.azure_search_driver:Request failed: 400, {"error":{"message":"The request had some invalid properties","code":"BadArgumentError","correlationId":"7d404bef-fc2d-446d-b71d-2dab6248e3d7","innererror":{"code":"QueryValidationError","message":"Timespan must be supplied for /search API"}}}
relationId":"7d404bef-fc2d-446d-b71d-2dab6248e3d7","innererror":{"code":"QueryValidationError","message":"Timespan must be supplied for /search API"}}}
Traceback (most recent call last):
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/test_data_driver_azure_search.py", line 41, in <module>
    df, results = drv_basic_search.query_with_results('SyslogBasic_CL | take 1', time_span=time_span)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/src/msticpy/msticpy/data/drivers/azure_search_driver.py", line 137, in query_with_results
    results = self._query_search_endpoint(search_url, query_body, timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/src/msticpy/msticpy/data/drivers/azure_search_driver.py", line 170, in _query_search_endpoint
    raise MsticpyKqlConnectionError(
msticpy.common.exceptions.MsticpyKqlConnectionError: ("we've hit an error while running", 'Error 400 from /search endpoint: {"error":{"message":"The request had some invalid properties","code":"BadArgumentError","correlationId":"7d404bef-fc2d-446d-b71d-2dab6248e3d7","innererror":{"code":"QueryValidationError","message":"Timespan must be supplied for /search API"}}}', ('Connecting to Microsoft Sentinel', 'https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#connecting-to-an-azure-sentinel-workspace'))

Environment (please complete the following information):

• Python Version: 3.11
• OS: Ubuntu
• Python environment: editable venv from cloned msticpy source with main branch checked out
• MSTICPy Version: 2.16 (pre-release, main branch, commit a4b0b72)

Additional context

PR #825 was merged to main and closed #819.

See https://github.com/MicrosoftDocs/azure-monitor-docs/blob/main/articles/azure-monitor/logs/basic-logs-query.md#api

https://api.loganalytics.io/v1/workspaces/%7BworkspaceId%7D/search?timespan=P1D

Also see https://learn.microsoft.com/en-us/rest/api/loganalytics/query/get?view=rest-loganalytics-2022-10-27-preview&tabs=HTTP#uri-parameters

The timespan over which to query data. This is an ISO8601 time period value. This timespan is applied in addition to any that are specified in the query expression.

[ianhelle]

I'll see if I can get to this, but I don't have a lot of time to spend on the project.
Very happy if you want to fix the problem and submit a PR though.