Skip to content

Remove crawl API endpoint (#733) #1065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dduugg wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Remove crawl API endpoint (#733) #1065

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions app/appEndpoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ const app = express();
// and this says express can use it
app.set('trust proxy', true);

// allows ex. res.render('crawl.html');
// allows ex. res.render('email.njk');
nunjucks.express(app);

// modify every request
Expand Down Expand Up @@ -38,7 +38,6 @@ app.use(express.urlencoded({extended:false}), express.json());
// each of these is a javascript file
// containing a get ( or post ) export.
const endpoints = [
"crawl",
"delete_event",
"events",
"ical",
Expand Down
10 changes: 2 additions & 8 deletions app/config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ const config = {
appPath,
api: {
header: 'Api-Version',
version: "3.60.0",
version: "3.61.0",
},
db: getDatabaseConfig(dbType, isTesting),
// maybe bad, but some code likes to know:
Expand All @@ -40,7 +40,7 @@ const config = {
name: "SHIFT to Bikes",
listen,
path: "/",
// used for crawl url, shareable links, and the manage url sent in email
// used for shareable links, and the manage url sent in email
url(...parts) {
const base = `${siteHost}${config.site.path}`;
return base + parts.join("/");
Expand Down Expand Up @@ -115,12 +115,6 @@ const config = {
'image/png' : '.png',
},
},
crawl: {
image: 'https://www.shift2bikes.org/images/shiftLogo_plain.gif',
title: 'Shift/Pedalpalooza Calendar',
description: `Find fun bike events and make new friends!` +
`Shift helps groups and individuals to promote their "bike fun" events.`,
},
cal: {
pedalp: {
name: 'Pedalpalooza Bike Calendar',
Expand Down
60 changes: 0 additions & 60 deletions app/endpoints/crawl.js
View file
Open in desktop

This file was deleted.

45 changes: 0 additions & 45 deletions app/test/crawl_test.js
View file
Open in desktop

This file was deleted.

26 changes: 0 additions & 26 deletions app/views/crawl.html
View file
Open in desktop

This file was deleted.

53 changes: 1 addition & 52 deletions docs/CALENDAR_API.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ Base URL:

Most responses are in JSON format, except for:
* event export returns vCalendar format
* event crawl returns HTML

## Viewing events

Expand Down Expand Up @@ -395,57 +394,6 @@ Errors:
* date range too large (100 days maximum)


### Crawling an event

Endpoint:
* GET `crawl`

Example request:
* `/crawl.php?id=1234`

URL parameters:
* `id`: `caldaily` event ID

Unknown parameters are ignored.

This endpoint is used by web crawlers such as search engines.

Success:
* status code: `200`
* returns a simple HTML rendering of ride data
* if `id` parameter is not present, a short, general message about Shift


Example response:

<html>
<head>
<title>Shift to Pedalpalooza Ride</title>
<meta property="og:title" content="Shift to Pedalpalooza Ride">
<meta property="og:url" content="https://www.shift2bikes.org/calendar/event-9300">
<meta property="og:image" content="https://www.shift2bikes.org/eventimages/6245.jpg">
<meta property="og:type" content="article">
<meta property="og:description" content="Have you ever wondered how Pedalpalooza happens every year...and did you know we have a team of programmers who work on the shift calendar and website. There is a lot of rewarding volunteer work that goes on behind the scenes and we are recruiting for new folks who are interested in helping out next year and beyond. Come on this ride and we will talk a little bit about the history of shift and try to find you a place to help out in the future. We will end at a family friendly watering hole. First round of drinks is on shift. We will be done by 8 so you can check out other rides.">
<meta property="og:site_name" content="SHIFT to Bikes">
<meta name="description" content="Have you ever wondered how Pedalpalooza happens every year...and did you know we have a team of programmers who work on the shift calendar and website. There is a lot of rewarding volunteer work that goes on behind the scenes and we are recruiting for new folks who are interested in helping out next year and beyond. Come on this ride and we will talk a little bit about the history of shift and try to find you a place to help out in the future. We will end at a family friendly watering hole. First round of drinks is on shift. We will be done by 8 so you can check out other rides.">
<meta name="keywords" content="bikes,fun,friends,Portland,exercise,community,social,events,outdoors">
</head>
<body>
<h2>Mon, Jun 5th, 6:00 PM - Shift to Pedalpalooza Ride</h2>
<p>Have you ever wondered how Pedalpalooza happens every year...and did you know we have a team of programmers who work on the shift calendar and website. There is a lot of rewarding volunteer work that goes on behind the scenes and we are recruiting for new folks who are interested in helping out next year and beyond. Come on this ride and we will talk a little bit about the history of shift and try to find you a place to help out in the future. We will end at a family friendly watering hole. First round of drinks is on shift. We will be done by 8 so you can check out other rides.</p>
<p>877 SW park</p>
<img src="https://www.shift2bikes.org/eventimages/6245.jpg">
</body>
</html>

Errors:
* status code: `404`
* body of response is empty
* possible errors
* `id` not found
* `id` of a hidden (unpublished) event


## Managing events

### Retrieving all event data
Expand Down Expand Up @@ -787,3 +735,4 @@ As with v1, there were probably revisions to v2 during this time, but changelog
* 3.59.9: (2025-12-22) Updated dependencies: nodemailer
* 3.59.10: (2026-01-15) Changed dependency management to only allow patch updates; updated dependencies: MySQL. Also removed unused example data.
* 3.60.0: (2026-02-19) ICS export now supports either single occurrence (`event_id`; new default) or the series (`series_id`; previous default). The existing `id` parameter aliases to `series_id` for backwards compatibility, but clients are encouraged to specify the ID type explicitly. Also updated Node to v24.x (latest LTS).
* 3.61.0: (2026-05-30) Removed the `crawl` endpoint, which is no longer used by any known clients or crawlers.