hash generated on aws does not correspond with windows

[kianwei96]

Hash value (72a7) generated on AWS for spike train object. Also included branch details in the screenshot.

Trying to load the file in windows, it expects a different hash value (9d1b).

windows branch details.

Same issue found for other objects (eg. Unity, Umaze, RPLParallel)

[grero]

Hm, that's unfortunate. It seems that we cannot use the hash then, unless it stems from the dirs variable, since they would be different on a windows system?

[kianwei96]

i added a few lines in the hash function, and tried it on both aws and windows. they both show an empty args, which is to be expected, but somehow the hash value in the file name is different?

[kianwei96]

i dont think the issue is regarding the dirs variable, seems like the pickle.dumps is giving different outputs:

b'\x80\x03}q\x00.'
vs
b'\x80\x04}\x94.'

even though self.args are they same

[grero]

Hm, OK. I think this means that we'll have to load each file to check the arguments instead of relying on the hash. Or we could try and create our own hash, like what we do for get_shortname for levels.

[kianwei96]

i'm not sure what's the issue here though. i was able to work with objects generated on aws, on my windows machine for the past month or so. the issue just cropped in the past few days. from what i can see, the hashing function hasn't been modified recently right?

[grero]

That's right, we haven't made any changes to the function. Can you check if the pickle version is the same on both systems?

[kianwei96]

print(pickle.format_version) gives 4.0 for both systems

[kianwei96]

hash function seems to behave differently for different python versions. makes sense because i just reinstalled anaconda today, and had a version upgrade to 3.8:

perhaps we can consider switching to this library (hashlib) instead.
https://stackoverflow.com/questions/39989534/hash-returning-different-values-on-different-oss
https://docs.python.org/3.5/library/hashlib.html?highlight=hashlib

[grero]

I'm starting to think we'll have to create our own hashes. The reason is that, mostly, these hashes are not used for what we are using them for. I don't think we have any guarantees that the hash values they generate won't change with future versions of the library. I'll think a bit this a bit and see if i can come up with something that works.

[kianwei96]

I think it should be save to use hashlib with md5 hashing. It’s a standard hash function, which probably has to be platform/version independent. The hashlib library is also packaged in the default python installation I think, so support should be quite reliable

[grero]

Another alternative is to keep a database in each directory that maps hashes to dictionaries of arguments. Something like _index.json, where the keys are the hashes and the values are the arguments that hash represents. When creating an object, the database can be consulted and if a file already exists, it will be loaded. I realise that this adds quite a bit of complexity, so it might not be worth it.

[kianwei96]

But yes, I can see how a custom implementation would be more attractive - we can maybe even encode it in such a way that we can see which arguments differ based off the file name alone

[grero]

Could be, let's do a check first, though, before we commit to it. Let's find out if hashes computed using hashlib are stable across python versions, and more importantly, across OSs.

[grero]

I was experimenting with hashlib initially, but it wasn't clear how you would actually go about hashing data, which is why I opted for the pickle solution. We could have another look at it, though.

[grero]

We don't actually need a hash in the traditional sense, just a consistent abbreviation of the arguments used to compute the objects.