Skip to content

ShoppingData Amazon Scrape #1080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Erdem-cevik wants to merge 201 commits into
base: develop
Choose a base branch
from
Open

ShoppingData Amazon Scrape #1080

Erdem-cevik wants to merge 201 commits into from

Conversation

@Erdem-cevik
Copy link
Contributor

@Erdem-cevik Erdem-cevik commented Nov 9, 2023 •
edited by sdmuki
Loading

Release Notes

JIRA Link

Summary:

Three new packages developed for Shopping Data:

  1. ai-scraper
  2. nlp
  3. shopping-data

Changes:

  1. PurchaseRepository
  2. Module Registration
  3. And new types in objects package
  4. Browser Extension
  5. Extension-onboarding

Potential Failures:

  • LLM is still producing non-deterministic output

Testing Notes:

  • How has this been tested?

Pre-Flight Checks

  • Has QA approved this change for dev?
  • [y ] Are all unit tests passing?
  • Have you added this description to this week's release notes?

@adhocmaster
initialized package
fc84f48
@adhocmaster
fixed paths
5c038c2
@adhocmaster
URLUtils to be tested
54b9e18
@adhocmaster
asyncified
9cf0ccb
@adhocmaster
fixed compilation errors. brands cannot be used in enums
b81bf5d
@adhocmaster
why are tests not compiled?
c11938e
@adhocmaster
working on the classifier
fff3841
@adhocmaster
fix types
17f6457
@adhocmaster
fixed typing
088221c
@adhocmaster
url keywords are hard
84a1ebe
@adhocmaster
match started
ebe3092
@adhocmaster
prototype for detection is done
6f5fd39
@adhocmaster
refactored
20bb043
@adhocmaster
most methods in URLUtils are synchronized calls. Sets instead of List…
049550b 
…s for faster search
@adhocmaster
Update KeywordRepository.ts
d8a875c
@adhocmaster
Merge branch 'develop' into feat/ENGT-1699
65ca28f
@adhocmaster
updated yarn lock
d7d476e
@adhocmaster
interfaces
086ba04
@adhocmaster
next builder patterns for prompts
44566cf
@adhocmaster
Update README.md
b7f93f8
@adhocmaster
builder
20bfcab
@adhocmaster
Create ProductUtils.ts
27f953c
@adhocmaster
more primitives and first prompt builder
b7c93bd
@adhocmaster
working on the director
5edda16
@adhocmaster
rest of chalie's comments
f05f178
@adhocmaster
Merge branch 'develop' into feat/ENGT-1699
d42c829
@adhocmaster
Merge branch 'feat/ENGT-1699' into feat/ENGT-1717
193d4a9
@adhocmaster
refactoring, supporting mechanism for multiple provider structures
9e3abcc
@adhocmaster
now unit tests
1bafd1f
@adhocmaster
started tests
4beb8d2
Erdem-cevik and others added 20 commits December 20, 2023 01:35
@Erdem-cevik
extension navigate issue fixed
da8029e
@Erdem-cevik
pieChart (arc) issue done
b672d19
@Erdem-cevik
Merge branch 'develop' into feat/ENGT-1752
c4ddfe3
@adhocmaster
chatgpt model update
da76b5e
@adhocmaster
Update ChatGPTRepository.ts
01a8f8c
@adhocmaster
back to 4k
ae1f7e9
@adhocmaster
Update LLMPurchaseHistoryUtilsChatGPT.ts
6130ec6
@Erdem-cevik
blank issue at product table fixed
fcd9754
@Erdem-cevik
Merge branch 'develop' into feat/ENGT-1752
0b6b90c
@Erdem-cevik
Merge branch 'develop' into feat/ENGT-1752
5d7b81e
@adhocmaster
trimmed halucinated purchases
daaa862
@adhocmaster
wordwrap fix. logs added for debug only
b619ea2
@adhocmaster
Update LLMScraperService.ts
6c15896
@adhocmaster
Update WebpageClassifier.ts
0b39dd0
@Erdem-cevik
console logs
7bef470
@adhocmaster
Update LLMPurchaseValidator.ts
406e93f
@adhocmaster
working on validator for malformed JSON
2983f9b
@adhocmaster
tests done for malformed LLMResponse
04b0938
@adhocmaster
moved MockLogUtils to commons package. some tests are failing in the …
35d6faf 
…ai-scraper
@adhocmaster
added sanitization for product meta
4a17b5d
@sdmuki sdmuki requested a review from SnickerChar January 25, 2024 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sdmuki sdmuki Awaiting requested review from sdmuki

@okanzdmr okanzdmr Awaiting requested review from okanzdmr

@SnickerChar SnickerChar Awaiting requested review from SnickerChar

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Erdem-cevik @SnickerChar @okanzdmr @sdmuki @adhocmaster