author first pass word frequency

[kyook]

Karen Yook kyook@wormbase.org

Jul 18 (1 day ago)

to Yuling, Textpresso, wormbase
Hi Yuling,

Would it be possible for you to run a word frequency analysis on this
list of papers? These are all papers for which an author responded
and we would like to know see if there are any keywords that stand
out.

Thanks,
Karen

[kyook]

Yuling Li

Jul 18 (1 day ago)

to kyook, wormbase, Textpresso
Hi Karen,

Counting word-frequency is easy but most of chances those common words "is",
"We", "do", "a" etc... will stand out, as we can imagine. In order to get
some words that "really" make those papers stand out, we will need a
negative set... i.e. these papers stand out of "what" crowd? we will need
rest of the crowd to prove those words. Do we want to use ALL Other WBPapers
as negatives in this matter? Or we have a collection of papers that authors
don't respond to yet? But if you only want a traditional word-frequency
list, then it's easy to program, just count and rank.

Thanks,
Yuling

[kyook]

Karen Yook kyook@wormbase.org

Jul 18 (1 day ago)

to Yuling, wormbase, Textpresso
do you have a stop word list that you can use?

[kyook]

Yuling Li

Jul 18 (1 day ago)

to kyook, wormbase, Textpresso
I have a stop word list that Textpresso uses, which contains 192 common
words, I can use it as a filter when counting frequency. So let's just go
with the traditional frequency counting with this filter.

[kyook]

Karen Yook kyook@wormbase.org

Jul 18 (1 day ago)

to Yuling, wormbase, Textpresso
That would be enough for this run.

Thanks,
Karen

[kyook]

Yuling Li

1:29 AM (13 hours ago)

to kyook, wormbase, Textpresso
Hi Karen,

The attached spreadsheet has two tabs(ranks):

1. A word is counted only one time if it appears in the paper.
2. A word is counted whatever times it appears in the paper.

It's already filtered with Textpresso stop words list and also

1. pure numbers were filtered
2. one-letter words were filtered
   Frequency_count.xlsx Frequency_count.xlsx
   6918K View Download

[kyook]

At the textpresso meeting today we discussed rerunning the frequency analysis on the afp papers by sections, with title being one of them (thanks Kimberly and Ranjana for the suggestion).

[liyuling]

The re-run of all sections of a paper has been done:

http://131.215.52.209/karen/frequency.tar.gz

contains files with names "section"_one and "section"_multi. which means a word is counted only one time in the paper or counted multiple times in the paper.

unfortunately these files are sorted by the value(alphabetically), not numerically. You may want to import them into spreadsheet and let the spreadsheet sort by second column. Let me know if you have difficulties doing that, I can do that after I come back from NIF.

Thanks.

[kyook]

From http://131.215.52.209/karen/frequency.tar.gz of all afp papers
Created a worksheet for each "section"_multi in an excel spreadsheet
First sheet contains Title_multi word frequencies
Deleted all words with less than 10 hits
SUBJECTIVELY highlighted, bolded, and italicized words that are reminiscent of a major biological process being represented in the afp set.

The group settled on papers containing 'stress' in the title.
There are 45 papers in the afp set that contain this word. Curators should take a look at the set, compare it to their own backlog, for the data types they are responsible for and get a sense of how process-biased curation works for them and what would make it better.

We can discuss our individual experiences in this thread and perhaps decide on new steps that need to be taken to move this curation style forward for all of us.

WBPaper00031692
WBPaper00031694
WBPaper00031842
WBPaper00031873
WBPaper00032236
WBPaper00032241
WBPaper00033114
WBPaper00032321
WBPaper00034757
WBPaper00035114
WBPaper00036083
WBPaper00036413
WBPaper00036090
WBPaper00036135
WBPaper00035965
WBPaper00037147
WBPaper00037595
WBPaper00037886
WBPaper00038233
WBPaper00039783
WBPaper00039990
WBPaper00038093
WBPaper00040006
WBPaper00039878
WBPaper00039835
WBPaper00040166
WBPaper00039788
WBPaper00040384
WBPaper00038464
WBPaper00040697
WBPaper00040849
WBPaper00041075
WBPaper00040133
WBPaper00040902
WBPaper00041277
WBPaper00041295
WBPaper00041568
WBPaper00041528
WBPaper00041150
WBPaper00041610
WBPaper00041663
WBPaper00041866
WBPaper00042148
WBPaper00042067
WBPaper00042178

[draciti]

Below the results for Expression pattern (otherexpr)

here is what I have done: Took the list from afp stress, compared with backlog:
Only one paper overlapping backlog and afp stress list -WBPaper00031694. I checked it but was negative for expression (reserpine study influencing lifespan).

I then did a more detailed analysis. Out of 45:
9 were already curated
15 were already validated as negative
21 were never checked as they were never flagged by any method. I randomly selected half of those to see if there was any false negative for expression but were all true negatives. I did not continue further.

One way to ensure completeness for gene expression coverage for a specific process would be the following. Get a list of the genes involved from the paper set and make sure that each gene has a described expression pattern. I could then use textpresso to retrieve genes that are not covered.