added new task to dag that gets remaining api calls to github api#30
added new task to dag that gets remaining api calls to github api#30LuisJG8 merged 1 commit intoMyProjectsfrom
Conversation
Summary of ChangesHello @LuisJG8, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
/gemini review |
LuisJG8
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
The pull request introduces a new task to an Airflow DAG to check the GitHub API rate limit before initiating a Celery task. This enhancement improves the DAG's resilience by preventing unnecessary API calls when the rate limit is low. However, there are several areas for improvement, including correcting type hints, using Airflow's logging system instead of print statements, and addressing unused imports and variables for better code hygiene. Additionally, a magic number used for the rate limit threshold should be replaced with a named constant for clarity.
| @task | ||
| def run_the_queue(): | ||
| app.send_task("worker.get_data_from_queue", args=[100, 500]) | ||
| def run_the_queue(rate_limit: str): |
There was a problem hiding this comment.
The type hint for rate_limit is str, but the check_rate_limit task returns a dictionary. This type hint is incorrect and could lead to confusion or runtime errors if strict type checking were enforced. Please update the type hint to dict to accurately reflect the expected input.
| def run_the_queue(rate_limit: str): | |
| def run_the_queue(rate_limit: dict): |
| from pendulum import datetime | ||
| from celery import Celery | ||
| import os | ||
| from github import Auth, Github, GithubException |
There was a problem hiding this comment.
The GithubException import is not used in this file. It's good practice to remove unused imports to keep the code clean and reduce unnecessary dependencies.
| from github import Auth, Github, GithubException | |
| from github import Auth, Github |
| api_token, api_token_two = os.getenv("GITHUB_API_TOKEN"), os.getenv("GITHUB_API_TOKEN_SECOND_ACCOUNT") | ||
| auth, auth_two = Auth.Token(api_token), Auth.Token(api_token_two) | ||
| gh, gh_two = Github(auth=auth), Github(auth=auth_two) |
There was a problem hiding this comment.
The variables api_token_two, auth_two, and gh_two are defined but not used within this DAG file. Unused variables can lead to confusion and unnecessary resource allocation. Consider removing them if they are not intended for use here, or add a comment explaining their future purpose if they are placeholders.
| api_token, api_token_two = os.getenv("GITHUB_API_TOKEN"), os.getenv("GITHUB_API_TOKEN_SECOND_ACCOUNT") | |
| auth, auth_two = Auth.Token(api_token), Auth.Token(api_token_two) | |
| gh, gh_two = Github(auth=auth), Github(auth=auth_two) | |
| api_token = os.getenv("GITHUB_API_TOKEN") | |
| auth = Auth.Token(api_token) | |
| gh = Github(auth=auth) |
| @task | ||
| def check_rate_limit(): | ||
| rate_limit = gh.rate_limiting | ||
| print(f"Rate limit: {rate_limit[0]} remaining / {rate_limit[1]} total") |
There was a problem hiding this comment.
Using print statements directly within Airflow tasks is not ideal for production environments. Airflow provides a robust logging system that should be utilized for better observability and integration with monitoring tools. Please replace print with logging.info() or logging.debug() after importing the logging module.
| def run_the_queue(): | ||
| app.send_task("worker.get_data_from_queue", args=[100, 500]) | ||
| def run_the_queue(rate_limit: str): | ||
| print(f'rate limit: {rate_limit["total"]}, remaining {rate_limit["remaining"]}') |
There was a problem hiding this comment.
Similar to the check_rate_limit task, using print statements directly within Airflow tasks is not ideal for production environments. Please replace print with logging.info() or logging.debug() for better logging practices.
| def run_the_queue(rate_limit: str): | ||
| print(f'rate limit: {rate_limit["total"]}, remaining {rate_limit["remaining"]}') | ||
|
|
||
| if rate_limit["remaining"] > 4900: |
There was a problem hiding this comment.
The number 4900 is a magic number. It's better to define such thresholds as named constants at the top of the file for clarity, easier modification, and improved maintainability. For example, MIN_RATE_LIMIT_THRESHOLD = 4900.
There was a problem hiding this comment.
Code Review
This pull request adds a new task to an Airflow DAG to check the GitHub API rate limit before processing data, which is a good proactive measure. However, the implementation has some issues that should be addressed. The GitHub client is initialized in the global scope of the DAG file, which is an Airflow anti-pattern. This should be moved into the task that uses it. There's also an incorrect type hint and a magic number that should be refactored for better code quality and maintainability. I've provided specific suggestions to resolve these points.
| api_token, api_token_two = os.getenv("GITHUB_API_TOKEN"), os.getenv("GITHUB_API_TOKEN_SECOND_ACCOUNT") | ||
| auth, auth_two = Auth.Token(api_token), Auth.Token(api_token_two) | ||
| gh, gh_two = Github(auth=auth), Github(auth=auth_two) |
There was a problem hiding this comment.
Initializing the GitHub client at the top level of the DAG file is an Airflow anti-pattern. This code is executed on every DAG parsing cycle, which is inefficient and can lead to performance issues. This logic should be moved into the check_rate_limit task. Also, the variables suffixed with _two are unused and should be removed. Please see my other comment on the check_rate_limit task for a suggested implementation. These lines should be removed.
| def check_rate_limit(): | ||
| rate_limit = gh.rate_limiting |
There was a problem hiding this comment.
As mentioned in my other comment, the GitHub client should be initialized within the task to follow Airflow best practices. This avoids running authentication logic during DAG parsing. For better security, it's also recommended to use Airflow Connections to manage secrets like API tokens.
def check_rate_limit():
api_token = os.getenv("GITHUB_API_TOKEN")
auth = Auth.Token(api_token)
gh = Github(auth=auth)
rate_limit = gh.rate_limiting| @task | ||
| def run_the_queue(): | ||
| app.send_task("worker.get_data_from_queue", args=[100, 500]) | ||
| def run_the_queue(rate_limit: str): |
There was a problem hiding this comment.
The type hint for rate_limit is str, but the check_rate_limit task returns a dictionary. This is misleading and could cause issues with static analysis tools or future maintenance. It should be dict.
def run_the_queue(rate_limit: dict):| def run_the_queue(rate_limit: str): | ||
| print(f'rate limit: {rate_limit["total"]}, remaining {rate_limit["remaining"]}') | ||
|
|
||
| if rate_limit["remaining"] > 4900: |
There was a problem hiding this comment.
The number 4900 is a magic number, which makes the code harder to understand and maintain. It should be defined as a constant with a descriptive name at the module level, for example: MIN_API_CALLS_THRESHOLD = 4900.
if rate_limit["remaining"] > MIN_API_CALLS_THRESHOLD:1 participant
No description provided.