Skip to content

README.md

Latest commit

 

History

History
134 lines (101 loc) · 6.44 KB

README.md

File metadata and controls

134 lines (101 loc) · 6.44 KB

ScanIQ Azure Function App

This Azure Function App contains two main functions for the ScanIQ service:

  1. GetScanIQSearchTerms: A timer-triggered function that runs on a schedule to fetch search terms from the database and initiate a data scraping batch process.
  2. ProcessBlob: A blob-triggered function that activates when a new file is uploaded to the designated Azure Storage container, processing the data within that file.

The application is built to be run locally for development and deployed to Azure for production.

Local Development and Testing

Follow these steps to run and test the Function App on your local machine.

Prerequisites

  1. Python 3.9+: Ensure you have a compatible version of Python installed.
  2. Node.js and npm: Required to install the Azure Functions Core Tools.
  3. Azure Functions Core Tools: A command-line tool that lets you develop, test, and publish Azure Functions from your local computer. Install it globally using npm: 
    npm install -g azure-functions-core-tools@4

Setup and Running

  1. Navigate to the App Directory: Open your terminal and change the directory to this folder.

    cd ScanIQFunctionApp

  2. Install Python Dependencies: Create a virtual environment and install the required packages from requirements.txt.

    python -m venv .venv
source .venv/bin/activate  # On Windows use `.venv\Scripts\activate`
pip install -r requirements.txt

  3. Configure Local Settings: The local.settings.json file holds all the necessary connection strings and secrets for the application to run. Ensure all values are correctly filled in. This file should not be committed to source control.

  4. Start the Local Functions Host: Run the following command to start the local runtime. It will load your functions and display the local endpoints.

    func start

Testing and Debugging

  • Timer Trigger (GetScanIQSearchTerms): The function will run automatically when the application starts and then again according to the SCHEDULE defined in local.settings.json. You can also manually invoke it for testing via the endpoint URL that the func start command outputs.
  • Timer Trigger (GetPropertyPulseIQSearchTerms): This function runs according to the PPI_SCHEDULE defined in local.settings.json.
  • Blob Trigger (ProcessBlob): To test this, upload a file to the Azure Storage container and path specified in your local.settings.json (AZURE_STORAGE_CONTAINER). The function will be triggered automatically.

How to Test After Editing Code

When you edit and save any of the Python (.py) files in the project, the Azure Functions Core Tools host will automatically detect the changes and reload the function workers. You do not need to restart the func start command manually. Simply save your file, watch the terminal for the reload confirmation, and you can test the new version of your function immediately.

Deployment to Azure

These instructions will guide you through deploying the Function App to a Consumption Plan (Serverless), which is eligible for the Azure Functions free grant.

Prerequisites

  1. Azure Account: You need an active Azure subscription. You can create one for free.
  2. Azure CLI: You need the Azure command-line interface installed. You can install it here.

Deployment Steps

  1. Login to Azure: Open your terminal and run:

    az login

  2. Create a Resource Group: A resource group is a container that holds related resources for an Azure solution. Replace <ResourceGroupName> and <Location> (e.g., eastus).

    az group create --name <ResourceGroupName> --location <Location>

  3. Create a Storage Account: This storage account will be used by the Function App for its internal operations (AzureWebJobsStorage). Replace <StorageAccountName> with a globally unique name.

    az storage account create --name <StorageAccountName> --location <Location> --resource-group <ResourceGroupName> --sku Standard_LRS

  4. Create the Function App: This command creates the Function App on a serverless Consumption plan. Replace <FunctionAppName> with a globally unique name.

    az functionapp create \
    --name <FunctionAppName> \
    --storage-account <StorageAccountName> \
    --resource-group <ResourceGroupName> \
    --consumption-plan-location <Location> \
    --runtime python \
    --python-version 3.9 \
    --functions-version 4

  5. Configure Application Settings: Your app needs the same settings that are in local.settings.json. Set them in the Azure environment using the following command for each key-value pair. This is a critical step.

    Get the connection string for your storage account:

    STORAGE_CONNECTION_STRING=$(az storage account show-connection-string --name <StorageAccountName> --resource-group <ResourceGroupName> --query "connectionString" --output tsv)

    Set the required settings:

    az functionapp config appsettings set --name <FunctionAppName> --resource-group <ResourceGroupName> --settings "AzureWebJobsStorage=$STORAGE_CONNECTION_STRING"
az functionapp config appsettings set --name <FunctionAppName> --resource-group <ResourceGroupName> --settings "AZURE_STORAGE_CONNECTION_STRING=$STORAGE_CONNECTION_STRING"
az functionapp config appsettings set --name <FunctionAppName> --resource-group <ResourceGroupName> --settings "AZURE_STORAGE_CONTAINER=scaniq"
# Add all other secrets from your .env file here
az functionapp config appsettings set --name <FunctionAppName> --resource-group <ResourceGroupName> --settings "DB_USER=<YourDbUser>"
az functionapp config appsettings set --name <FunctionAppName> --resource-group <ResourceGroupName> --settings "DB_PASSWORD=<YourDbPassword>"
# ... and so on for all other required secrets.

  6. Deploy the Code: Finally, from within the ScanIQFunctionApp directory, publish your project to Azure.

    # Make sure you are in the ScanIQFunctionApp directory
func azure functionapp publish uncapscrapefunction

Your Function App is now live on Azure. ```

Your Function App is now live on Azure.