Unload don't create files while concurrency scaling enabled

[maxim-lisovsky-gismart]

When we enabled concurrency scaling in redshift locopy.redshift.unload_and_copy stopped creating files on S3 during critical load and logged the corresponding message:

No files generated from unload

Tried both parquet and csv formats

[fdosani]

Could you provide more details? Like a minimal example of the code and stack trace etc. it's hard to debug just based on what you've provided. TIA!

[fdosani]

Also do any of these limitations apply to you setup? https://docs.aws.amazon.com/redshift/latest/dg/concurrency-scaling.html

[maxim-lisovsky-gismart]

I do something like this:

query = "select * from foo.bar"
with connection.get_redshift(secret_id, database=database, host=host) as redshift_locopy:
    redshift_locopy.unload_and_copy(
        query=query,
        s3_bucket=bucket,
        s3_folder=bucket_dir,
        export_path=False,
        raw_unload_path=data_dir,
        delete_s3_after=False,
        parallel_off=False,
        unload_options=["PARQUET", "PARALLEL ON", "CLEANPATH"],
    )

Haven't found any of these restrictions relevant for me

[fdosani]

I'm not sure this is a locopy issue though. Sounds like concurrency scaling is changing the behaviour of redshift which means the data it is querying isn't available to download? I think one of those limitations is applying to your setup. It works without it enabled correct?

[maxim-lisovsky-gismart]

Yes, it works without concurrency. I also have an idea that it can be caused by this:

SELECT path FROM stl_unload_log WHERE query = pg_last_query_id() ORDER BY path;

Probably last query id is not so reliable when it comes to concurrency

[fdosani]

It say this in the docs:

It doesn't support queries that access system tables, PostgreSQL catalog tables, or no-backup tables.

That would make sense.

[maxim-lisovsky-gismart]

But it's locopy's code, unload_and_copy method

[fdosani]

But it's locopy's code, unload_and_copy method

If you don't use locopy and have concurrency scaling enabled does your query work. Keep in mind I don't have your setup or your actual code, so I'm going off very little here to understand what is actually happening.

I'd suggest removing locopy from the testing and see if you can manually query/download the data, this would help me in tracking down where the issue is if it is in locopy.