Skip to content

EricMymj/auto-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

34 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Hot100.ai Auto-Submission Tools

A comprehensive toolkit for automating submissions to Hot100.ai, including browser automation, API discovery, and a browser extension.

πŸš€ Features

  • Playwright Automation Script: Full browser automation for submitting projects
  • Browser Extension: Semi-automated helper for quick form filling
  • API Monitor: Discovers and documents API endpoints
  • Session Management: Saves and restores authentication sessions
  • Postman Collection Generator: Auto-generates API documentation

πŸ“ Project Structure

auto-crawler/
β”œβ”€β”€ hot100ai_submission_research.md  # Research findings
β”œβ”€β”€ hot100ai_automation.js          # Main automation script
β”œβ”€β”€ api_monitor.js                  # API discovery tool
β”œβ”€β”€ package.json                    # Node.js dependencies
β”œβ”€β”€ .env.example                    # Environment variables template
β”œβ”€β”€ browser-extension/              # Chrome extension
β”‚   β”œβ”€β”€ manifest.json
β”‚   β”œβ”€β”€ content.js
β”‚   β”œβ”€β”€ background.js
β”‚   β”œβ”€β”€ popup.html
β”‚   β”œβ”€β”€ popup.js
β”‚   └── styles.css
└── README.md                       # This file

πŸ”§ Installation

Prerequisites

  • Node.js v14 or higher
  • Google Chrome browser
  • A Google account for authentication

Setup

  1. Clone the repository:
cd /Users/zenglei/Workspace/auto-crawler
  1. Install dependencies:
npm install
  1. Set up environment variables:
cp .env.example .env
# Edit .env with your Google credentials

🎯 Usage

1. Playwright Automation

Run the main automation script:

npm run submit
# or
node hot100ai_automation.js

Features:

  • Automatic Google OAuth login
  • Session persistence
  • Form auto-filling
  • Thumbnail upload
  • API request capture

2. API Discovery

Monitor and discover API endpoints:

npm run monitor
# or
node api_monitor.js

This will:

  • Navigate through Hot100.ai pages
  • Capture all API requests
  • Generate endpoint documentation
  • Create a Postman collection

3. Browser Extension

Installation:

  1. Open Chrome and go to chrome://extensions/
  2. Enable "Developer mode"
  3. Click "Load unpacked"
  4. Select the browser-extension folder

Usage:

  • Click the extension icon when on hot100.ai
  • Fill in project details
  • Click "Fill Form on Page" to auto-fill
  • Use "Capture API" to monitor requests

πŸ” Key Findings

Authentication

  • Method: Google OAuth 2.0
  • Backend: Supabase
  • Token Storage: localStorage (sb-wscvwmwxpdzszdcxnaja-auth-token)

API Endpoints

GET  /api/submissions?filter=hot100
GET  /api/chart/stats
GET  /api/stats/total-vibes
POST /api/submissions (requires auth)

Submission Requirements

  • Google account authentication
  • Project title
  • Description
  • Project URL
  • Thumbnail image (PNG/JPG)
  • Tags/categories

⚠️ Important Considerations

Legal & Ethical

  • Review Hot100.ai's Terms of Service before automation
  • Respect rate limits to avoid being blocked
  • Consider reaching out for official API access
  • Use automation responsibly

Security

  • Never hardcode credentials
  • Use environment variables for sensitive data
  • Keep authentication tokens secure
  • Don't commit .env files to version control

Technical Limitations

  • Google OAuth may require 2FA handling
  • CAPTCHAs may block automation
  • Site structure may change (BETA 2.0)
  • Manual review may be required for submissions

πŸ› οΈ Troubleshooting

Common Issues

  1. Authentication fails

    • Check Google account credentials
    • Ensure 2FA is handled if enabled
    • Verify no security blocks on account
  2. Form fields not found

    • Site structure may have changed
    • Update selectors in automation script
    • Use API monitor to rediscover endpoints
  3. Rate limiting

    • Add delays between submissions
    • Reduce automation frequency
    • Consider manual review process

πŸ“Š API Monitoring Output

The API monitor generates:

  • hot100ai-api-discovery-[timestamp].json - Full API call log
  • hot100ai-endpoints-[timestamp].json - Simplified endpoint list
  • hot100ai-postman-[timestamp].json - Postman collection

πŸ”„ Next Steps

  1. Contact Hot100.ai for official API access
  2. Implement rate limiting in automation scripts
  3. Add error handling for edge cases
  4. Create test suite for validation
  5. Build dashboard for submission tracking

πŸ“ License

MIT License - Use at your own risk and responsibility

🀝 Contributing

Feel free to submit issues and enhancement requests!

⚑ Quick Start Example

const submitter = new Hot100AISubmitter();
await submitter.initialize();
await submitter.login(email, password);

await submitter.submitProject({
  title: 'My AI Project',
  description: 'An innovative AI tool',
  url: 'https://myproject.com',
  tags: ['ai', 'automation']
});

await submitter.close();

Note: This toolkit is for educational purposes. Always respect the target website's terms of service and implement responsible automation practices.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors