Nenuzhny Andrey

.gitignore

@@ -10,7 +10,7 @@ __pycache__/
 .Python
 build/
 develop-eggs/
-dist/
+#dist/  Need for the final's second iteration task
 downloads/
 eggs/
 .eggs/
@@ -35,7 +35,7 @@ MANIFEST
 pip-log.txt
 pip-delete-this-directory.txt
 
-# Unit test / coverage reports
+# Unit tests / coverage reports
 htmlcov/
 .tox/
 .coverage
@@ -102,3 +102,27 @@ venv.bak/
 
 # mypy
 .mypy_cache/
+
+# MAC filesystem's files
+.DS_Store
+
+# pycharm files
+.idea/
+
+# tests data files
+cover/
+tests/data/help.txt
+tests/data/yahoo.txt
+
+# news storage
+rss_reader/storage
+*sqlite3.db
+
+# pdf & html files
+*.pdf
+*.html
+
+# Unused files
+rss_reader/utils/dejavu_font/DejaVuSansCondensed.cw127.pkl
+rss_reader/utils/dejavu_font/DejaVuSansCondensed.pkl
+rss_reader/utils/no_imagepng.png

LICENSE.txt

@@ -0,0 +1,19 @@
+Copyright (c) 2019 The Python Packaging Authority (PyPA)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

MANIFEST.in

@@ -0,0 +1,10 @@
+include requirements.txt
+include *.sh
+include *.txt
+include config.cfg
+include mypy.ini
+recursive-include dist *.gz
+recursive-include dist *.whl
+recursive-include rss_reader *.ttf
+recursive-include tests *.py
+recursive-include tests *.xml

README.md

@@ -0,0 +1,112 @@
+# Rss reader hometask for EpamTrainee
+Python RSS-reader.
+
+Url for cloning:
+`https://github.com/Nenu1985/PythonHomework.git`
+
+Version 4
+```shell
+usage: rss_reader [-h] [--verbose] [--limit LIMIT] [--json] [-v]
+                  [--width WIDTH] [--date DATE] [--to_pdf TO_PDF]
+                  [--to_html TO_HTML]
+                  url
+
+Rss reader. Just enter rss url from your favorite site and app will print
+latest news.
+
+positional arguments:
+  url                url of rss
+
+optional arguments:
+  -h, --help         show this help message and exit
+  --verbose          Outputs verbose status messages
+  --limit LIMIT      Limit news topics if this parameter provided
+  --json             Print result as JSON in stdout
+  -v, --version      Print version info
+  --width WIDTH      Define a screen width to display news
+  --date DATE        Date of stored news you want to see. Format: %Y%m%d
+  --to_pdf TO_PDF    Convert and store news you are looking for to pdf
+  --to_html TO_HTML  Convert and store news you are looking for to html
+
+
+```
+
+## Code description
+* Code uses `argparse` module.
+* Codebase covered with unit tests with at 90% coverage.
+* Any mistakes are printed in human-readable error explanation.
+Exception tracebacks in stdout are prohibited.
+* Docstrings are mandatory for all methods, classes, functions and modules.
+* Code corresponds to `pep8` (used `pycodestyle` utility for self-check).
+* Feedparser module is used for rss parsing;
+* There are several bots for parsing news: default bor for unimplemented RSS urls and
+    custom bots (yahoo, tut) with detailed approach to parsing.
+
+## Code self-checking
+Use ./pycodestyle.sh to check the code corresponding to `pep8`
+(pycodestyle package must be installed)
+
+## Testing
+Tests are available at `https://github.com/Nenu1985/PythonHomework`
+Launching:
+```
+./make_tests.sh
+```
+- to pass test with coverage
+(nose and coverage packages must be installed)
+
+## Version 2: Distribution
+Utility wrapes into distribution package with setuptools.
+This package exports CLI utility named rss-reader.
+
+To generate distribution package (setuptool and wheel must be installed).
+Launch:
+
+``` python3 setup.py sdist bdist_wheel```
+
+In the ./dist repo you'll find a .tar and .whl files.
+
+Wheel package for the second iteration task 
+(maybe is discarded but it works) on the Google Drive:
+```https://drive.google.com/file/d/1RbMYxvpEXTx77Dk61xPkwSChD_jTf0jf/view?usp=sharin```
+
+Actual packages you may find in the './dist' repo if you don't want to generate it manually.
+
+Installing: 
+
+```python3 -m pip install ./dist/rss_reader-4.0-py3-none-any.whl```
+
+OR
+```
+python3 -m pip install -r requirements.txt
+pip install ./dist/rss-reader-4.0.tar.gz
+```
+## Version 3: News cashing
+News cashing implemented by using Sqlite3 DB. DB consists of 4 related tables: feed, news_item, links, imgs.
+The implementation is in the rss_reader/utils/sqlite.py file. It contains RssDB class. Builtin sqlite3 lib is
+used.
+Base RssParser class imports RssDB class and uses for storing and loading data. RssParser's method print_news() 
+is decorated with call_save_news_after_method() (rss_parser/utils/decorators) that calls appropriate function 
+for storing news data (_store_news()).
+
+## Version 4: Converters
+Utility implements news converting to pdf and html formats. See according files: rss_reader/utils/pdf.py and 
+rss_reader/utils/html_writer.py files.
+Pdf converter uses pyFPDF package. To correct print cyrillic symbols djvu fonts are imported. Html2Pdf method
+doesn't use because of unsupported utf-8 encoding. That's why I had to parse htmls and generate pdf object 
+manually.
+Html converter uses lxml.html library to parse and generate html content. 
+
+
+
+## Docker deployment
+docker run -it python /bin/bash
+git clone https://github.com/Nenu1985/PythonHomework.git 
+cd PythonHomework
+pip install -r requirements.txt
+pip install .
+rss-reader --help
+
+
+
+

config.cfg

@@ -0,0 +1,13 @@
+[metadata]
+# This includes the license file(s) in the wheel.
+# https://wheel.readthedocs.io/en/stable/user_guide.html#including-license-files-in-the-generated-wheel-file
+license_files = LICENSE.txt
+
+[bdist_wheel]
+# This flag says to generate wheels that support both Python 2 and Python
+# 3. If your code will not run unchanged on both Python 2 and 3, you will
+# need to generate separate wheels for each Python version that you
+# support. Removing this line (or setting universal to 0) will prevent
+# bdist_wheel from trying to make a universal wheel. For more see:
+# https://packaging.python.org/guides/distributing-packages-using-setuptools/#wheels
+universal=1

make_tests.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+# Run this script to launch the tests
+nosetests --with-coverage --cover-erase --cover-package=rss_reader --cover-html --traverse-namespace

mypy.ini

@@ -0,0 +1,38 @@
+# Global options:
+[mypy]
+# Logistics of what code to check and how to handle the data.
+scripts_are_modules = False
+show_traceback = True
+
+[mypy-bs4]
+ignore_missing_imports = True
+
+[mypy-feedparser]
+ignore_missing_imports = True
+
+[mypy-fpdf]
+ignore_missing_imports = True
+
+[mypy-lxml]
+ignore_missing_imports = True
+
+[mypy-lxml.html]
+ignore_missing_imports = True
+
+[mypy-rss_reader]
+ignore_missing_imports = True
+
+[mypy-setuptools]
+ignore_missing_imports = True
+
+[mypy-terminaltables]
+ignore_missing_imports = True
+
+[mypy-urllib]
+ignore_missing_imports = True
+
+[mypy-utils]
+ignore_missing_imports = True
+
+[mypy-utils.RssInterface]
+ignore_missing_imports = True

pycodestyle.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+YELLOW='\033[0;33m'
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+NC='\033[0m' # No Color
+
+echo -e "${YELLOW}Starting static code analys ${NC}"
+
+# utils/
+python3 -m pycodestyle rss_reader/utils/ --max-line-length=120
+echo -e "${YELLOW}utils/ ${GREEN}PASSED${NC}"
+
+# rss.py
+python3 -m pycodestyle rss_reader/rss.py --max-line-length=120
+echo -e "${YELLOW}rss.py/ ${GREEN}PASSED${NC}"
+
+# bots/
+python3 -m pycodestyle rss_reader/bots/ --max-line-length=120
+echo -e "${YELLOW}bots/ ${GREEN}PASSED${NC}"

requirements.txt

@@ -0,0 +1,9 @@
+attrs==19.3.0
+bs4==0.0.1
+coverage==4.5.4
+feedparser==5.2.1
+fpdf==1.7.2
+nose==1.3.7
+python-dateutil==2.8.1
+terminaltables==3.1.0
+lxml==4.4.1

rss_reader/__init__.py

@@ -0,0 +1 @@
+from . import bots, utils

rss_reader/__main__.py

@@ -0,0 +1,2 @@
+from rss_reader import rss
+rss.main()

rss_reader/bots/default.py

@@ -0,0 +1,7 @@
+"""Default (not specified) rss parser bot"""
+from ..utils.rss_interface import BaseRssBot
+
+
+class Bot(BaseRssBot):
+    """Default (not specified) rss parser bot"""
+    pass

rss_reader/bots/tut.py

@@ -0,0 +1,94 @@
+"""Tut.by specified rss parser bot"""
+import attr
+import bs4
+import feedparser
+import typing
+
+from rss_reader.utils.rss_interface import BaseRssBot
+from ..utils.data_structures import NewsItem, News
+
+
+@attr.s(frozen=True)
+class TutNewItem(NewsItem):
+    """Extended NewsItem class to store tags and authors"""
+    tags: typing.List[str] = attr.ib()
+    authors: typing.List[str] = attr.ib()
+
+
+class Bot(BaseRssBot):
+    """Tut.by specified rss parser bot"""
+
+    def _feed_to_news(self, feed: feedparser.FeedParserDict) -> News:
+        """
+        Returns str containing formatted news from internal attr self.feed
+
+        :return: str with news
+        """
+        news_items = []
+
+        for i, item in enumerate(feed.get('items')[:self.limit]):
+
+            news_items.append(TutNewItem(
+                title=item.get('title', ''),
+                link=item.get('link', ''),
+                published=item.get('published', ''),
+                imgs=[img.get('url', '') for img in item.get('media_content', '')],
+                links=[link.get('href', '') for link in item.get('links', '')],
+                html=item.get('summary', ''),
+                authors=[author.get('name', '') for author in item.get('authors', '')],
+                tags=[tag.get('term', '') for tag in item.get('tags', '')],
+            ))
+
+        news = News(
+            feed=feed.get('feed', '').get('title', ''),
+            link=feed.get('feed', '').get('link', ''),
+            items=news_items,
+        )
+        self.logger.info(f'Feedparser object is converted into news_item obj with TUT news')
+
+        return news
+
+    def _parse_news_item(self, news_item: TutNewItem) -> str:
+        """
+        Forms a human readable string from news_item and adds it to the news_item dict
+        :param news_item: news_item content
+        :return: human readable news content
+        """
+        self.logger.info(f'_parse_news_item_tut.by  Extending {news_item.title}')
+
+        out_str = ''
+        out_str += f"\nTitle: {news_item.title}\n" \
+                   f"Date: {news_item.published}\n" \
+                   f"Link: {news_item.link}\n"
+        if type(news_item) == TutNewItem:
+            out_str += f"Authors: {', '.join(news_item.authors)}\n"
+            out_str += f"Tags: {', '.join(news_item.tags)}\n"
+
+        html = bs4.BeautifulSoup(news_item.html, "html.parser")
+
+        links = news_item.links
+        imgs = news_item.imgs
+
+        for tag in html.descendants:
+            if tag.name == 'a':
+                pass
+            elif tag.name == 'img':
+                src = tag.attrs.get('src')
+                # src = src.replace('thumbnails/', '')
+                if src in imgs:
+                    img_idx = imgs.index(src) + len(links) + 1
+                else:
+                    imgs.append(src)
+                    img_idx = len(imgs) + len(links)
+
+                out_str += f'\n[image {img_idx}:  {tag.attrs.get("alt")}][{img_idx}]'
+            elif tag.name == 'p':
+                out_str += '\n' + tag.text
+            elif tag.name == 'br':
+                out_str += '\n'
+        out_str += f'\n{html.getText()}\n'
+        out_str += 'Links:\n'
+        out_str += '\n'.join([f'[{i + 1}]: {link} (link)' for i, link in enumerate(links)]) + '\n'
+        out_str += '\n'.join([f'[{i + len(links) + 1}]: {link} (image)' for i, link in enumerate(imgs)]) + '\n'
+
+        return out_str