Create example scrapers with example results

[klartext]

It would be nice to have real-world example-json files
together with directory/file-collection, which are created by
running a scraper with a certain scraperJSON-json file.

That would be helpful to implement a scraper that follows the scraperJSON scheme/policy.

A *.zip or *.tgz file for results (or json-file and results) would make sense as examples, IMHO.

[petermr]

Chris (copied) has written a tutorial on this which we released last week
and this should provide what you want. Chris, can you point to this and see
if it's what is wanted? thx

On Sat, Aug 8, 2015 at 3:57 PM, klartext notifications@github.com wrote:

It would be nice to have real-world example-json files
together with directory/file-collection, which are created by
running a scraper with a certain scraperJSON-json file.

That would be helpful to implement a scraper that follows the scraperJSON
scheme/policy.

A *.zip or *.tgz file for results (or json-file and results) would make
sense as examples, IMHO.

—
Reply to this email directly or view it on GitHub
#10.

Peter Murray-Rust
Reader in Molecular Informatics
Unilever Centre, Dep. Of Chemistry
University of Cambridge
CB2 1EW, UK
+44-1223-763069

[blahah]

@klartext

• there are a lot of examples in https://github.com/ContentMine/journal-scrapers/tree/master/scrapers
• this issue feature: collect multiple downloads in a subdirectory #7 covers having a directory of downloaded files

[blahah]

Ah, I now realise that you perhaps mean that the example scrapers should come with example results - is that the case? If so, that's an excellent idea and I will make it a priority.

[klartext]

Yes, I meant examples that has three things:

• json-file
• realworld example (what paper is downloaded and how - e.g. example data that is input via file/stdin/cli/gui to select those papers that want to be downloaded)
• realworld results in form of example directory-with-downloaded-content

Just fom the json-files it's not clear, how to interpret them.
Is ee a lot of selector-strings like "//meta[@name='citation_publisher']"
but how is that be used?
Is this an OPTION to select via cli/gui/... or what does this mean?
So there is some ambiguity in interpreteing the json files.
So a realworld example (e.g. element/selector foobar is used with cli-switc foobar ??? and searchkeywords are e.g. "horizontal._gene._transfer" or so, and this results in a dir with a pdf....)

[petermr]

@klartext

Or recent workshop is covered in:

https://github.com/ContentMine/workshop-resources/

You will find many of your questions addressed here. Suggest you work
though the getpapers ans Scraper tutorials and let us know if what you
want is not there.

On Sat, Aug 8, 2015 at 9:21 PM, klartext notifications@github.com wrote:

Yes, I meant examples that has three things:

• json-file
• realworld example (what paper is downloaded and how - e.g. example
  data that is input via file/stdin/cli/gui to select those papers that want
  to be downloaded)
• realworld results in form of example
  directory-with-downloaded-content

Just fom the json-files it's not clear, how to interpret them.
Is ee a lot of selector-strings like "//meta[@name
https://github.com/name='citation_publisher']"
but how is that be used?
Is this an OPTION to select via cli/gui/... or what does this mean?
So there is some ambiguity in interpreteing the json files.
So a realworld example (e.g. element/selector foobar is used with
cli-switc foobar ??? and searchkeywords are e.g. "horizontal._gene._transfer"
or so, and this results in a dir with a pdf....)

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

Peter Murray-Rust
Reader in Molecular Informatics
Unilever Centre, Dep. Of Chemistry
University of Cambridge
CB2 1EW, UK
+44-1223-763069

[klartext]

@petermr
OK, I read the "getpapers-tutorial.md" from the workshop-ressources.
This has explained a lot and also answered some questions.

But where does scraperJSON fit in?
That is not explained, and I could not find json-files for getpaper.
Is scraperJSON just a new idea for newer/planned scrapers?
Or is it used somewhere already? And if so: where, and how to interpret it?

[blahah]

@klartext yes, I think we need some overview documentation.

getpapers and quickscrape are both tools you can use to get scientific papers en masse for content mining.

getpapers allows you to search for papers on EuropePubMed, ArXiv or IEEE. You get metadata of all the hits to your query. You can optionally also try to download PDF, XML and/or supplementary data for the hits, but not all papers are downloadable this way.

quickscrape is a web scraping tool. You give it a URL and a scraper definition (in scraperJSON format) and it will scrape the URL using the scraper definition to guide it. We have a collection of scraperJSON definitions for major publishers and journals over at journal-scrapers. quickscrape is useful when you want to download things that (a) were in your getpapers results but getpapers couldn't get the PDF/XML/supp or (b) are not contained in the source databases that getpapers uses.

So, getpapers can data, very fast, from a subset of the literature. Quickscrape can get the same data, it takes more work but you can theoretically use it on any page on the internet.

[petermr]

@klartext as Richard says you need to read about quickscrape. The
workshop tutorial is at
https://github.com/ContentMine/workshop-resources/tree/master/software-tutorials/quickscrape
which should give a reasonable introduction to quickscrape and the format
of scrapers.

On Sun, Aug 9, 2015 at 2:31 AM, klartext notifications@github.com wrote:

@petermr https://github.com/petermr
OK, I read the "getpapers-tutorial.md" from the workshop-ressources.
This has explained a lot and also answered some questions.

But where does scraperJSON fit in?
That is not explained, and I could not find json-files for getpaper.
Is scraperJSON just a new idea for newer/planned scrapers?
Or is it used somewhere already? And if so: where, and how to interpret it?

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

Peter Murray-Rust
Reader in Molecular Informatics
Unilever Centre, Dep. Of Chemistry
University of Cambridge
CB2 1EW, UK
+44-1223-763069

[klartext]

@petermr: OK, I read the quickscrape-tuorial.
It explains how it works.
But how to create / interpret the jsonSCRAPER-files? (Should print that question in a loop?)
The link to "create your own definitions" gives me 404-error.
The link to ctree also gives me 404-error.

[blahah]

@klartext how about this: https://github.com/ContentMine/ebi_workshop_20141006/tree/master/sessions/6_scrapers

It's the session on scrapers that I wrote for a workshop last year. It includes guides on creating selectors, as well as basic and advanced scraperJSON.

We need to update some of these resources as there are now more features available, but that should get you started.

[petermr]

@klartext,
Thanks for this. Your engagement helps to drive our documentation and also
shows up places where we have a need to develop software.

"But how to create / interpret the jsonSCRAPER-files? "
Probably trivial point: we expect people to use a text editor, probably
starting with a generic scraper template/example. Not sure whether it's
worth developing a specific tool.

If you aren't familiar with XPath see https://en.wikipedia.org/wiki/XPath.
(We use version 1.0). There are also many online tutorials and some are
interactive.

On Sun, Aug 9, 2015 at 10:15 PM, klartext notifications@github.com wrote:

@petermr https://github.com/petermr: OK, I read the quickscrape-tuorial.
It explains how it works.
But how to create / interpret the jsonSCRAPER-files? (Should print that
question in a loop?)
The link to "create your own definitions" gives me 404-error.
The link to ctree also gives me 404-error.

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

Peter Murray-Rust
Reader in Molecular Informatics
Unilever Centre, Dep. Of Chemistry
University of Cambridge
CB2 1EW, UK
+44-1223-763069

[klartext]

@blahah thanks, that text did help a lot. I was not knowing XPATH-stuff in detail, so the selector-syntax looked strange to me. After reading that, I saw, what the definition is about.
So, the selector selects tags/data from html-pages, and what is selected, is written in XPATH-syntax.
Some other things could be explained in the scraperJSON-docs: for example
(from example in scraperJSON-doc) "attribute": "content".
I guess, this means the part between opening and closing tag, like "ThisStuff" in "ThisStuff",
would mean "content" here...