Upgrade to Mongo 2.0 Syntax #148

yuvalkarmi · 2017-07-16T13:01:52Z

I've gone ahead and made the necessary changes in across the board for this gem to work with MongoDB versions 3.4+, which have not been previously supported.
I've also added support for sharded clusters, and the ability to use SSL when connecting.
Please see the full list of updates in the changelog.
One note -- I did not test the sync ability or connection to sqlite3 databases.
The setup I did test this with was MySQL + MongoDB 3.4
You're more than welcome to edit this as needed.
Thanks for making this gem, and I hope my contribution helps! :)

houndci-bot · 2017-07-16T13:02:05Z

spec/mongify/database/no_sql_connection_spec.rb


    it "should return a db" do
-      @mongodb_connection.db.should be_a Mongify::Database::NoSqlConnection::DB
+      @mongodb_connection.db.should be_a Mongify::Database::NoSqlConnection::Database


Line is too long. [85/80]

houndci-bot · 2017-07-16T13:02:06Z

spec/mongify/database/no_sql_connection_spec.rb

      it "should drop the index" do
-        @collection.should_receive(:drop_index).with('pre_mongified_id_1')
-        @collection.stub(:update)
+        @indexes.should_receive(:drop_one).with('pre_mongified_id_1')


Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.

houndci-bot · 2017-07-16T13:02:06Z

spec/mongify/database/no_sql_connection_spec.rb

      it "should call update with unset" do
-        @collection.should_receive(:update).with({},{'$unset' => {'pre_mongified_id' => 1}}, {:multi=>true})
-        @collection.stub(:drop_index)
+        @collection.should_receive(:update_many).with({},{'$unset' => {'pre_mongified_id' => 1}})


Space missing after comma.
Space inside { missing.
Redundant curly braces around a hash parameter.
Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
Line is too long. [97/80]
Space inside } missing.

houndci-bot · 2017-07-16T13:02:06Z

spec/mongify/database/no_sql_connection_spec.rb

+        @indexes.stub(:drop_one)
+        @collection.stub(:indexes).and_return(@indexes)
+        @collection.stub(:update_many)
+        @indexes.stub(:get).and_return('pre_mongified_id_1' => 'something')


Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.

houndci-bot · 2017-07-16T13:02:06Z

spec/mongify/database/no_sql_connection_spec.rb

    it "should create index for pre_mongified_id" do
-      @collection.should_receive(:create_index).with([["pre_mongified_id", Mongo::ASCENDING]]).and_return(true)
+      @collection.indexes
+      @collection.indexes.should_receive(:create_many).with([ { key: { pre_mongified_id: 1 } } ]).and_return(true)


Space inside square brackets detected.
Line is too long. [114/80]

houndci-bot · 2017-07-16T13:02:06Z

lib/mongify/translation/process.rb

            id = row["_id"]
            attributes = fetch_reference_ids(t, row)
-            no_sql_connection.update(t.name, id, {"$set" => attributes}) unless attributes.blank?
+            no_sql_connection.connection[t.name].update_one( { "_id" => id } , { "$set"  => attributes}) unless attributes.blank?


Space inside parentheses detected.
Space found before comma.
Redundant curly braces around a hash parameter.
Line is too long. [129/80]
Unnecessary spacing detected.
Operator => should be surrounded by a single space.
Space inside } missing.

houndci-bot · 2017-07-16T13:02:06Z

lib/mongify/database/no_sql_connection.rb

      # speedy lookup for collections via the pre_mongified_id
      def create_pre_mongified_id_index(collection_name)
-        db[collection_name].create_index([['pre_mongified_id', Mongo::ASCENDING]])
+        db[collection_name].indexes.create_many([ { key: { pre_mongified_id: 1 } } ])


Space inside square brackets detected.
Line is too long. [85/80]

houndci-bot · 2017-07-16T13:02:07Z

lib/mongify/database/no_sql_connection.rb

      # @raise MongoDBError if index isn't found
      def drop_mongified_index(collection_name)
-        db[collection_name].drop_index('pre_mongified_id_1') if db[collection_name].index_information.keys.include?("pre_mongified_id_1")
+        db[collection_name].indexes.drop_one('pre_mongified_id_1') unless db[collection_name].indexes.get('pre_mongified_id_1').nil?


Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
Line is too long. [132/80]

houndci-bot · 2017-07-16T13:02:07Z

lib/mongify/database/no_sql_connection.rb

      def remove_pre_mongified_ids(collection_name)
        drop_mongified_index(collection_name)
-        db[collection_name].update({}, { '$unset' => { 'pre_mongified_id' => 1} }, :multi => true)
+        db[collection_name].update_many({}, { '$unset' => { 'pre_mongified_id' => 1} })


Redundant curly braces around a hash parameter.
Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
Line is too long. [87/80]
Space inside } missing.

houndci-bot · 2017-07-16T13:02:07Z

lib/mongify/database/no_sql_connection.rb

      # Returns a row of a item from a given collection with a given pre_mongified_id
      def get_id_using_pre_mongified_id(colleciton_name, pre_mongified_id)
-        db[colleciton_name].find_one('pre_mongified_id' => pre_mongified_id).try(:[], '_id')
+        db[colleciton_name].find('pre_mongified_id' => pre_mongified_id).try(:[], '_id')


Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
Line is too long. [88/80]

houndci-bot · 2017-07-16T13:06:41Z

spec/mongify/database/no_sql_connection_spec.rb

    context "get_id_using_pre_mongified_id" do
      it "should return new id" do
-        @collection.should_receive(:find_one).with({"pre_mongified_id"=>1}).and_return({'_id' => '123'})
+        @collection.should_receive(:find).with({"pre_mongified_id"=>1}).and_return([{'_id' => '123'}])


Space inside { missing.
Redundant curly braces around a hash parameter.
Surrounding space missing for operator =>.
Space inside } missing.
Line is too long. [102/80]
Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.

houndci-bot · 2017-07-16T13:06:41Z

lib/mongify/database/no_sql_connection.rb

      # Returns a row of a item from a given collection with a given pre_mongified_id
      def get_id_using_pre_mongified_id(colleciton_name, pre_mongified_id)
-        db[colleciton_name].find_one('pre_mongified_id' => pre_mongified_id).try(:[], '_id')
+        db[colleciton_name].find('pre_mongified_id' => pre_mongified_id).try(:first).try(:[], '_id')


Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
Line is too long. [100/80]

anlek · 2017-07-17T00:59:31Z

Hey @yuvalkarmi,
Thank you for taking this on! It looks excellent. However, there are tests which are failing.
https://dsc.cloud/Anlek/Screen-Shot-2017-07-16-17-58-29

IsamAljawarneh · 2017-07-30T10:50:37Z

Hey @anlek ,
Mongify is an intersting tool, i tested it with mlab.com node and it works good.
however, with Atlas MongoDB's cluster , it does not seem to work.
i was searching for the problem for three days now, and the reason is clear now.
MongoDB 3.4 is deployed in Atlas which is not supported by Mongify currently. also, in Atlas SSL is enabled and here is an example of the minimum Ruby connection config. required:
............................................................................................................................................
client = Mongo::Client.new('mongodb://kay:myRealPassword@mycluster0-shard-00-00-wpeiv.mongodb.net:27017,mycluster0-shard-00-01-wpeiv.mongodb.net:27017,mycluster0-shard-00-02-wpeiv.mongodb.net:27017/admin?ssl=true&replicaSet=Mycluster0-shard-0&authSource=admin')
...................................................................................................................................................

i hope that you upgrade Mongify as soon as possible to work with MongoDB 3.4, also, i suggest you provide a support for SSL configuration in database.config. and what about replicaSet and authSource in the previous connection string?

Thank you and Best Regards,

IsamAljawarneh · 2017-07-30T11:12:01Z

hi @yuvalkarmi ,
how to upgrade from Mongify 1.3.1 to Mongify 1.4.0?

yuvalkarmi · 2017-07-31T00:18:51Z

hey @doltiana -- so as @anlek mentioned, there are still a few tweaks to make before this can be pulled into core (notably, some tests are failing). However, the core functionality is working, and you can try it (at your own risk). For now, you can simply use my fork to do it, until we get everything sorted out:

https://github.com/yuvalkarmi/mongify

There's an updated readme that outlines support for MongoDB 3.4 and SSL.

I'll take care of the outstanding issues as soon as I get a bit of free time, and then Andrew would be able to merge it into core.

Happy to answer any other questions you may have,
Yuval

IsamAljawarneh · 2017-08-04T19:43:50Z

hey @yuvalkarmi,
ok i will try the fork you made.
actually i don't normally work with Ruby. so, how to pull changes you made in my local system.
is it a command to install directly like this:
gem install mongify

or updating Gemfile , or what exactly?

Thank You.

anlek · 2017-08-04T21:10:06Z

@doltiana if you pull the code, you can follow the instructions under "Development" in the README.

IsamAljawarneh · 2017-08-05T10:51:14Z

Hey @anlek , can you provide me with specific instructions of how to pull the mongify gem from the fork by @yuvalkarmi , i tried different ways i found online but nothing seems to work.
actually i need to test mongify with real data over a cloud cluster, and that cluster supports MongoDB 3.4 and SSL is enabled among other properties.

like currently in my system i have installed mongify using: gem install mongify
then it works fine with a local instance of MongoDB, and with a remote MongoDB instance hosted in https://mlab.com/. however, it does not work with a MongoDB cluster hosted by MongoDB Atlas, since the latter supports only MongoDB 3.4 , and SSL is enabled.
now, having this in mind, what should i do exactly to update my current version of mongify so that it works as exactly as modified by @yuvalkarmi ?
Thanks

anlek · 2017-08-11T22:45:09Z

@doltiana you can git pull https://github.com/yuvalkarmi/mongify.git to get @yuvalkarmi code.

As for the remote MongoDB, I would recommend testing on a local copy as remote has network delays that add up fast!

robertangeles · 2017-10-29T22:22:25Z

Hi Gents. Hope all is well. I found out that mongoDB 3.6 is in the horizon and was wondering if the changes being done in mongify will also support this new release.

Thanks!

anlek · 2017-10-30T23:22:54Z

Unfortunately, at the moment, I'm unable to put time into Mongify, but I am open to reviewing people's code and merging it in.

Yuval Karmi added 7 commits June 25, 2017 13:56

Update all syntax to match Mongo 2.0 and add support for Rails 5

fa7b0ea

add support for ssl config option

2d2a68a

version bump, update change log, update readme

bcda14d

remove byebug require statement

50bb3f1

add support for auth_source and update readme

a461263

decrease log level, update drop database syntax for Mongo 2

829729d

remove references to byebug

6ccb2d7

houndci-bot reviewed Jul 16, 2017

View reviewed changes

correct syntax for find

ae923cb

houndci-bot reviewed Jul 16, 2017

View reviewed changes

anlek mentioned this pull request Jul 22, 2017

Can i use mongoDB version 3.4 with mongify? #144

Open

aravinreizend mentioned this pull request Jan 29, 2018

Not able to process the large data (more than 10k records) when processing using batch_size from Oracle #161

Open

Upgrade to Mongo 2.0 Syntax #148

Are you sure you want to change the base?

Upgrade to Mongo 2.0 Syntax #148

Uh oh!

Conversation

yuvalkarmi commented Jul 16, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anlek commented Jul 17, 2017

Uh oh!

IsamAljawarneh commented Jul 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IsamAljawarneh commented Jul 30, 2017

Uh oh!

yuvalkarmi commented Jul 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IsamAljawarneh commented Aug 4, 2017

Uh oh!

anlek commented Aug 4, 2017

Uh oh!

IsamAljawarneh commented Aug 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anlek commented Aug 11, 2017

Uh oh!

robertangeles commented Oct 29, 2017

Uh oh!

anlek commented Oct 30, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

IsamAljawarneh commented Jul 30, 2017 •

edited

Loading

yuvalkarmi commented Jul 31, 2017 •

edited

Loading

IsamAljawarneh commented Aug 5, 2017 •

edited

Loading