Skip to content

Firefox support #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tbodt wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Firefox support #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Memrise Course Dump
This **Google Chrome** extension downloads word lists from [**Memrise**](https://memrise.com/) courses as ".csv" spreadsheets along with all associated <ins>images</ins>, <ins>audio</ins>, and <ins>video</ins> files. It also supports [batch download](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#batch-download) of Memrise courses. The format of the downloaded data is suitable for subsequent import into [**Anki**](https://apps.ankiweb.net/).
This **Chrome** and **Firefox** extension downloads word lists from [**Memrise**](https://memrise.com/) courses as ".csv" spreadsheets along with all associated <ins>images</ins>, <ins>audio</ins>, and <ins>video</ins> files. It also supports [batch download](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#batch-download) of Memrise courses. The format of the downloaded data is suitable for subsequent import into [**Anki**](https://apps.ankiweb.net/).

The extension *does not* download personal study data (although it is planned to be added in the future). It also *does not* download the words you have marked as "ignored" on Memrise. You might want to unignore them before downloading a course or make a separate clean Memrise account specifically for downloading purposes.

Expand All @@ -10,7 +10,7 @@ At the top of this page click `Code` and then `Download ZIP` (Note, that the `Co
<img src="https://user-images.githubusercontent.com/93875472/212447995-ec0370a5-af67-4a7b-96ec-b7eb2dd4e803.png">
</picture></p>

## Installation
## Installation (Chrome)
1. [Download](https://github.com/Eltaurus-Lt/CourseDump2022/archive/refs/heads/main.zip) the ***CourseDump2022-main.zip*** archive and extract ***CourseDump2022-main*** folder from it. At this step, you can move the extension folder to any place in your filesystem.
2. In *Google Chrome* click the `Extensions` button <picture><source media="(prefers-color-scheme: dark)" srcset="https://github.com/user-attachments/assets/6cca563b-8149-421b-a217-0664c3b872f2"><img src="https://github.com/user-attachments/assets/89838937-f887-4aa7-bff9-9f5293fa04cb" alt="Chrome extension icon"></picture> and then <picture><source media="(prefers-color-scheme: dark)" srcset="https://github.com/user-attachments/assets/bff14d50-3a4c-4b89-be0a-a739beeb111c"><img src="https://github.com/user-attachments/assets/d07bfc2b-e281-4e79-bf9f-5bc5f9c50611" alt="Manage extensions"></picture><br>
<sub>(alternatively go to the Main menu <picture><source media="(prefers-color-scheme: dark)" srcset="https://github.com/user-attachments/assets/f36fcb90-886f-4445-98e9-7fb5d81646e1"><img src="https://github.com/user-attachments/assets/eb8d2e95-82d3-46ae-ad96-35e6d3db159b" alt="`Menu`"></picture> in the top right corner and click `Extensions` → `Manage Extensions`)</sub>
Expand All @@ -20,6 +20,15 @@ At the top of this page click `Code` and then `Download ZIP` (Note, that the `Co
5. (_optional_) Click the `Extensions` button from step 2 again and pin the extension to the toolbar by clicking the pin button<p>
<picture><source media="(prefers-color-scheme: dark)" srcset="https://github.com/user-attachments/assets/bc69c932-19a1-4ada-9c9d-e8dfd32ee000"><img src="https://github.com/user-attachments/assets/cc502262-e429-44c2-a5c2-fb277cec67c4"></picture></p>

## Installation (Firefox)
1. [Download](https://github.com/Eltaurus-Lt/CourseDump2022/archive/refs/heads/main.zip) the ***CourseDump2022-main.zip*** archive and extract ***CourseDump2022-main*** folder from it. At this step, you can move the extension folder to any place in your filesystem.
2. Install the [`web-ext`](https://extensionworkshop.com/documentation/develop/getting-started-with-web-ext/#installation-section) command line tool.
3. In the extension directory, run `web-ext build`. This outputs `web-ext-artifacts/memrise_course_dump-9.1.zip`.
4. Go to `about:debugging` in your browser.
5. Switch to "This Firefox" tab
6. Click "Load Temporary Add-on..." and select the .zip file from step 3
7. (_optional_) Click the extensions button in the toolbar and pin the extension to the toolbar using the gear menu

## 💡 Downloading a Memrise Course

>---
Expand Down Expand Up @@ -61,7 +70,7 @@ After a download is complete, you should see the progress bar turning green:
<img src="https://github.com/user-attachments/assets/0b6cb1fd-9114-4a9c-b356-da0dec19e07e">
</picture></p>

The downloaded files should appear in your Chrome downloads directory, in a subfolder with the name comprised of the id, name, and author of that course:
The downloaded files should appear in your downloads directory, in a subfolder with the name comprised of the id, name, and author of that course:

<p><picture>
<source media="(prefers-color-scheme: dark)" srcset="https://github.com/user-attachments/assets/9921a71a-d513-4112-882f-8434a302cf44">
Expand Down Expand Up @@ -265,7 +274,7 @@ Note, that decks can be nested inside each other (via drag-and-drop) to group co
5. **Learnable IDs**: Appends an additional column to the course spreadsheet containing a unique ID for each item. Can be used to manage duplicates inside Anki (if imported into the sorting field), or to cross-reference against other Memrise data downloaded separately, such as [archived mems](https://github.com/Eltaurus-Lt/MemDump)
6. **Video files**: Allows excluding video files from a download: when turned off overwrites the `Download media` setting for video files while leaving images and audio unaffected (has no effect if the `Download media` toggle is turned off)
7. **Skip media download**: Allows skipping media files during the file download phase. In contrast to the `Download media` setting, does not remove the respective columns from the spreadsheet when turned off. It can be helpful if a course spreadsheet needs to be recompiled with different settings without downloading the whole media folder again
8. **Course metadata**: Enables downloading three metadata files in addition to the basic spreadsheet and media: an `info.md` file containing the text description of a course, the course's thumbnail image, and the course author's avatar. When turned off, the ".csv" spreadsheet and respective media folder (when applicable) will be placed directly into the Chrome download folder, instead of being bundled together with meta files in a separate course folder
8. **Course metadata**: Enables downloading three metadata files in addition to the basic spreadsheet and media: an `info.md` file containing the text description of a course, the course's thumbnail image, and the course author's avatar. When turned off, the ".csv" spreadsheet and respective media folder (when applicable) will be placed directly into the download folder, instead of being bundled together with meta files in a separate course folder

## Discussion
If you encounter errors, have further questions regarding the extension, or need any help with using the downloaded materials in Anki, please leave a comment in this thread: [An alternative to Memrise2Anki](https://forums.ankiweb.net/t/an-alternative-to-memrise2anki-support-thread/30084)
110 changes: 58 additions & 52 deletions background.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ function ciddsParse(cidd_strings) {
}
}

chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => {
chrome.runtime.onMessage.addListener((arg, sender, sendResponse) => {
//messages from menu
if (arg.type === "coursedump_checkOngoing") {
sendResponse({ "ongoing-status": !!ongoingTab });
Expand Down Expand Up @@ -177,34 +177,40 @@ chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => {
let urls = [];
if (arg.type === "coursedump_downloadFiles") {
let terminated = false;
if (arg.file_queue) urls = arg.file_queue;
const todo = urls.length;
if (arg.maxThreads) maxConnections = arg.maxThreads;
console.log(`max threads set to : ${maxConnections}`);
let done = 0;
let pids = Array(maxConnections).fill().map((_, i) => i + 1);
const results = await Promise.allSettled(pids.map(async pid => {
while (ongoingTab && urls.length) {
// const [url, filename] = ["some url", "some filename.ext"]; //emu
if (arg.file_queue) urls = arg.file_queue;
const todo = urls.length;
if (arg.maxThreads) maxConnections = arg.maxThreads;
console.log(`max threads set to : ${maxConnections}`);
let done = 0;
let pids = Array(maxConnections).fill().map((_, i) => i + 1);
Promise.allSettled(pids.map(async pid => {
while (ongoingTab && urls.length) {
// const [url, filename] = ["some url", "some filename.ext"]; //emu
// const emu = urls.shift();//emu
const [url, filename] = urls.shift();
await sleep(200);
let did;
try {
did = await downloadFile({url, filename, conflictAction: "overwrite" });
let [url, filename] = urls.shift();
if (url instanceof Blob) {
url = URL.createObjectURL(url);
}
await sleep(200);
let did;
try {
did = await downloadFile({url, filename, conflictAction: "overwrite" });
//did = await sleep(Math.floor(Math.random() * 600 + 300)); //emulate download
} catch (err) {
console.error(filename, err);
} catch (err) {
console.error(filename, err);
chrome.tabs.sendMessage(tabId, {
type: "coursedump_error",
error: err.message,
url, filename
}).catch(err => {});
}
if (did !== undefined) {
await chrome.downloads.erase({ id: did });
}
done++;
} finally {
if (new URL(url).protocol === "blob:")
URL.revokeObjectURL(url);
}
if (did !== undefined) {
await chrome.downloads.erase({ id: did });
}
done++;
chrome.tabs.sendMessage(tabId, {
type: "coursedump_progressMedia_upd",
done, todo
Expand All @@ -214,37 +220,37 @@ chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => {
console.log('Downloading tab appears to be closed. terminating file download.');
terminated = true;
ongoingTab = null;
}
})
}
}));

for (let i = 0; i < results.length; i++) {
const r = results[i];
if (r.status === "rejected") {
console.error(`pid ${i + 1}: ${r.reason}`);
}
}

if (ongoingTab) {
chrome.tabs.sendMessage(tabId, {
type: "coursedump_mediaFinished",
status: "done"
}).catch(err => {});
ongoingTab = null;
} else {
if (!terminated) {
console.log('Download stopped by user during file downloading phase');
} })
}
})).then((results) => {
for (let i = 0; i < results.length; i++) {
const r = results[i];
if (r.status === "rejected") {
console.error(`pid ${i + 1}: ${r.reason}`);
}
}

if (ongoingTab) {
chrome.tabs.sendMessage(tabId, {
type: "coursedump_mediaFinished",
status: "done"
}).catch(err => {});
ongoingTab = null;
} else {
console.log('Downloading tab was closed during file downloading phase');
menuAlert("Downloading tab was closed. Download terminated",
"no open menus left, download termination alert was not sent");
if (!terminated) {
console.log('Download stopped by user during file downloading phase');
} else {
console.log('Downloading tab was closed during file downloading phase');
menuAlert("Downloading tab was closed. Download terminated",
"no open menus left, download termination alert was not sent");
}
chrome.tabs.sendMessage(tabId, {
type: "coursedump_mediaFinished",
status: "stopped"
}).catch(err => {});
}
chrome.tabs.sendMessage(tabId, {
type: "coursedump_mediaFinished",
status: "stopped"
}).catch(err => {});
}
updAllMenus();
updAllMenus();
});
return true;
}
});
25 changes: 11 additions & 14 deletions coursescan.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,21 +66,19 @@ async function fetchMeta(cidd) {
}

function meta2txt(meta) {
let text = 'data:md/plain;charset=utf-8,' + encodeURIComponent(
let text =
`# **${meta['proper name'] || meta['url name']}**\n` +
`### by _${meta['author']}_\n` +
`### (${meta['number of items']} learnable items)\n` +
`\n` +
meta['description']
);
meta['description'];
if (!ANKI_HEADERS) {
text = text + encodeURIComponent(
text = text +
`\n\n` +
`## Course Fields\n` +
`| ${meta['course fields']} |`
);
`| ${meta['course fields']} |`;
}
return text
return new Blob([text], {type: 'text/markdown'});
}

//THE MAIN FUNCTION FOR SCANNING ALL LEVELS OF A COURSE
Expand Down Expand Up @@ -434,7 +432,6 @@ async function scanCourse(cidd, threadN) {
"#columns:" + course_fields.join(",") + "\n" +
csv_data;
}
const csv_encoded = 'data:text/csv;charset=utf-8,%EF%BB%BF' + encodeURIComponent(csv_data);

//names for directory and spreadsheet
let course_filename, course_folder;
Expand All @@ -451,7 +448,7 @@ async function scanCourse(cidd, threadN) {
};

//add all files to global queue
file_queue.unshift([csv_encoded, `${course_folder}${course_filename}_(${meta['number of items'].toString()}).csv`]);
file_queue.unshift([new Blob([csv_data], {type: 'text/csv'}), `${course_folder}${course_filename}_(${meta['number of items'].toString()}).csv`]);
if (settings["course_metadata"]) {
file_queue.unshift([meta2txt(meta), `${course_folder}info.md`]);
file_queue.unshift([meta['ava'], `${course_folder}${meta['author']}.${meta['ava'].split(".").slice(-1)}`]);
Expand Down Expand Up @@ -580,11 +577,11 @@ async function batchDownload() {


//global variables
if (typeof cidds === 'undefined') {var cidds = []} //should be defined as an argument passed from menu.js through background.js
if (typeof batch_done === 'undefined') {var batch_done = 0} //global progress counter
if (typeof file_queue === 'undefined') {var file_queue = []} //global list of files
if (typeof cidds === 'undefined') {cidds = []} //should be defined as an argument passed from menu.js through background.js
if (typeof batch_done === 'undefined') {batch_done = 0} //global progress counter
if (typeof file_queue === 'undefined') {file_queue = []} //global list of files
if (typeof threads !== 'undefined') {
alert('Script is already executing on this page. Reload to retry'); //should be impossible to trigger if the menu state is correct
} else {
batchDownload();
}
batchDownload().catch(console.error);
}
3 changes: 2 additions & 1 deletion manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
"default_popup": "menu.html"
},
"background": {
"service_worker": "background.js"
"service_worker": "background.js",
"scripts": ["background.js"]
},
"content_security_policy": {},
"web_accessible_resources": [{
Expand Down