size limitation

[pejmansaeghe]

Hi there,
I tried running the programme with 100k segments in-domain, and 100k segments mix-domain dedicating 2G RAM memory in the command line and I received out of memory exceptions. However, when I ran the same command on 50k in-domain and 100k mix-domain, there was no exceptions and code ran fine.
I was wondering if you have tested to find the optimum values for in and mix domain number of segments, and also if is there a way to increase this limitation to be able to deal with much larger corpora. For instance, millions of segments both in and mix domain.

Regards

[amirkamran]

Hi,
Actually, I was running experiments with 1.9m mix-domain and 100k in-domain sentences and I was using 20G memory for that. But thanks for pointing it out, I will try to run smaller experiments to report min memory requirements. Although, it is a memory intensive algorithm, but I will give it another shot to minimize the memory usage.

By the way, I just fix a bug, so if you are testing please pull the latest commit.

Thanks !

[pejmansaeghe]

Thanks for your reply, I would like to try this on 1.9m in-domain and 2m to 3m mix-domain. Memory usage might exponentially increase then?

Thanks

[amirkamran]

It will not increase exponentially, but definitely a linear relation with size of corpus. I am using bi directional translation models for both in-domain and mix-domain plus four language models and everything is loaded in memory.

[turchmo]

Hi
I'm running the code in a different scenario of the previous posts (large mix domain: 20 M parallel segments. and small in domain: 30k), I assigned 60G of RAM but the software is two days working at the same step:

2015-10-30 17:13:23.197 [pool-2-thread-415] INFO nl.uva.illc.dataselection.InvitationModel - Updating translation table ...

my feeling is that it is a problem of memory, any comments on this?

If the use of memory is linear with the size of the corpus, following the numbers you suggested in the previous posts, it means that I need to allocate 200G, is it correct?

Thanks a lot!

[amirkamran]

Hi Turchmo,

Memory should not be the issue because if it will reach 60g+ JVM will crash with an out of memory error. Basically, it is based on IBM Model 1, so the memory requirements and processing time is directly related to the size of source and target vocabulary.

Can you send me an email with your corpus details, specially the vocabulary sizes of both source and target and also the machine specs on which you are running the application.

I am able to find out some possibilities to speed up the code. If you give me a day to test it, I will update the code. I am currently running with a corpus of 52M and within 60gb memory I am able to complete an EM iteration in 1 hour.