Skip to content

Commit f404ebb

Browse files
committed
Feed ttls into the DB in batches [fix]
This fixes issues with the comadn crashign on to many files! :-)
1 parent 0c977f2 commit f404ebb

1 file changed

Lines changed: 8 additions & 4 deletions

File tree

run/fill-db

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -181,15 +181,19 @@ rm -Rf "$db_tmp_dir"
181181
mkdir -p "$db_tmp_dir"
182182
echo "Loading data into the DB ..."
183183
# --loader=parallel \
184-
time find \
184+
ttl_list_file="/tmp/ttls_$RANDOM.csv"
185+
find \
185186
"$ontology_ttls_root" \
186187
"$data_ttls_root" \
187188
-name "*.ttl" \
188-
-print0 \
189-
| xargs -0 "$jena_db_data_injector" \
189+
> "$ttl_list_file"
190+
time while mapfile -t -n 10000 batch && ((${#batch[@]}))
191+
do
192+
"$jena_db_data_injector" \
190193
--loc "$db_tmp_dir" \
191194
--syntax turtle \
192-
2>&1 | awk -v ORS='' \
195+
"${batch[@]}"
196+
done < "$ttl_list_file" 2>&1 | awk -v ORS='' \
193197
'
194198
# This AWK script only serves to reformat the output
195199
# of the loading procedure into a less spammy

0 commit comments

Comments
 (0)