feat(fetcher): implement SearchBasedFetcher for ScienceDirect using scopus search API

[Siva-Sai22]

Closes #14421

Implemented web search capability for ScienceDirect by integrating the Scopus Search API, since the ScienceDirect Search API requires institutional access that most users don't have. Added a new ScopusQueryTransformer to convert JabRef search queries to Scopus query syntax, and wrote the Scopus class to implement PagedSearchBasedParserFetcher interface, enabling paginated search results with proper JSON response parsing, this allows users to search academic literature via the "Web search" feature in JabRef, which was previously not available for ScienceDirect.
Modified the ScienceDirect to use the api key of scopus.

Steps to test

• Go to settings -> preferences -> websearch.
• Add the Scopus API key and test it.
• Then search using the Scopus (Example query: author = Einstein)

Mandatory checks

• I own the copyright of the code submitted and I license it under the MIT license
• I manually tested my changes in running JabRef (always required)
• I added JUnit tests for changes (if applicable)
• I added screenshots in the PR description (if change is visible to the user)
• I described the change in CHANGELOG.md in a way that is understandable for the average user (if change is visible to the user)
• I checked the user documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request updating file(s) in https://github.com/JabRef/user-documentation/tree/main/en.

[github-actions]

Hey @Siva-Sai22! 👋

Thank you for contributing to JabRef!

We have automated checks in place, based on which you will soon get feedback if any of them are failing.

After all automated checks pass, a maintainer will also review your contribution. Once that happens, you can go through their comments in the "Files changed" tab and act on them, or reply to the conversation if you have further inputs.

Please re-check our AI Usage Policy to ensure that your pull request is in line with it. It also contains links to our contribution guide in case of any other doubts related to our contribution workflow.

[koppor]

According to https://dev.elsevier.com/technical_documentation.html Scopus is not ScienceDirect.

I see two options:

1. To avoid confusion, ScienceDirect should be renamed to Elsevier. But then, we have a "big ball of mud", because the methods in that class are not coupled, but only semnatically correlated
2. Add new functionality to Scopus - this should make maintainence easier. This fetcher should be added to the list of fetchers in the "Web Search" prefernces

Please do option 2 to have the fetcher class small.

• API key should renamed from "Scopus" to "Esevier", since it covers both.

---

Future work: Split up API key configuration and fetcher enablement.

[koppor]

There must be a method to convert an inputstream to a JSON object directly. But maybe, not mit dem JSONObject.

[Siedlerchr]

** JSONObject response = JsonReader.toJsonObject(inputStream);
**

[koppor]

Another view - from

[Siva-Sai22]

• API key should renamed from "Scopus" to "Esevier", since it covers both.

@koppor
But the ScienceDirect is accessing the api key using the FETCHER_NAME (ScienceDirect). If we rename it to Esevier we have to change the fetcher name also
importerPreferences.getApiKey(getName())

I think it would be better to take a scopus api key also.

[koppor]

• API key should renamed from "Scopus" to "Esevier", since it covers both.

@koppor But the ScienceDirect is accessing the api key using the FETCHER_NAME (ScienceDirect). If we rename it to Esevier we have to change the fetcher name also importerPreferences.getApiKey(getName())

I think it would be better to take a scopus api key also.

I searched for the code you gave

It seems that EACH FETCHER uses that thing.

Since the new fetcher is a SearchBasedFetcher displayed in the preferences, the old fetcher should use the key of the other fetcher. With that, both fetchers will share the same key and it can be configured.

Update That means, my idea with "Elsevier" was wrong and a concrete fetcher name is used. -- Please double check with the preferences code.

[Siva-Sai22]

@koppor I have checked the preferences code
The api key is stored in a keyring and retrieved using the same when called using the concrete fetcher name
So I think we can't have the same api key name for the two different fetchers.

I have changes in which we allow the user to input the api key for scopus (same as the scienceDirect one) and can display the search results.

If you are okay with this approach I will update the PR.

[github-actions]

Your pull request conflicts with the target branch.

Please merge with your code. For a step-by-step guide to resolve merge conflicts, see https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/resolving-a-merge-conflict-using-the-command-line.

[koppor]

@Siva-Sai22 It is difficult for me to follow. Please provide a link (you can search for files on GitHub by typing T). Provide a deep link to the line number. - It is also OK to provide a class name and line number. Answers might then even take longer, since we mostly do JabRef during the day in short breaks without an IDE opened.

[koppor]

I have changes in which we allow the user to input the api key for scopus (same as the scienceDirect one) and can display the search results.

No, this is not OK, because it seems that only one fetcher is displayed and not the other one.

---

To move on, you extract your new SearchBasedFetcher out and revert the original class to their own state. Then we maybe see easier how the key handling works.

[Siva-Sai22]

@koppor

• This is how api key is accesed in ScienceDirect (or any other fetcher)

  jabref/jablib/src/main/java/org/jabref/logic/importer/fetcher/ScienceDirect.java

  Line 140 in 75b867f

  .header("X-ELS-APIKey", importerPreferences.getApiKey(getName()).orElse(""))

• This is how the api keys are set (using the fetcher names)

  jabref/jablib/src/main/java/org/jabref/logic/preferences/JabRefCliPreferences.java

  Line 2315 in 75b867f

  private Set<FetcherApiKey> getFetcherKeys() {

So the keys are fetched using the fetcher name.
The method you discussed above:

API key should renamed from "Scopus" to "Esevier", since it covers both.

This cannot be done because the api key is fetched using fetcher name.

As suggested by you the scienceDirect should be kept as it is (and not make it a search based fetcher) and instead make a new searchbasedfetcher known as Scopus which uses the Esevier's api for searching.

The scienceDirect and this scopus one will use the same api key. But inorder to use these the user has to put in the same api key twice (one for using scienceDirect which is fulltextFetcher (not modified) and one for the new Scopus which is searchBasedFetcher)

These are the changes : main...Siva-Sai22:jabref:test

[koppor]

@Siva-Sai22 I seee that the class ScienceDirect itself calls the api key. One can replace getName() by Elsevier.

[Siva-Sai22]

@koppor updated the pr with required changes. Please let me know if any other changes are required.

[koppor]

I accept that we missed the alphabetical ordering here.

[koppor]

[ant:checkstyle] [ERROR] /home/runner/work/jabref/jabref/jablib/src/test/java/org/jabref/logic/importer/fetcher/ScopusTest.java:27:15: Unused import - org.junit.jupiter.api.Assertions.assertFalse. [UnusedImports]