Skip to content

Commit 51e4521

Browse files
committed
updated links, new TOC & minor changes
1 parent 4f76275 commit 51e4521

File tree

1 file changed

+68
-40
lines changed

1 file changed

+68
-40
lines changed

README.md

Lines changed: 68 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,40 @@
11
# Keboola Python Component library
22

3+
- [Keboola Python Component library](#keboola-python-component-library)
4+
- [Introduction](#introduction)
5+
- [Links](#links)
6+
- [Quick start](#quick-start)
7+
- [Installation](#installation)
8+
- [Core structure \& functionality](#core-structure--functionality)
9+
- [CommonInterface](#commoninterface)
10+
- [Initialization](#initialization)
11+
- [Loading configuration parameters:](#loading-configuration-parameters)
12+
- [Processing input tables - Manifest vs I/O mapping](#processing-input-tables---manifest-vs-io-mapping)
13+
- [Manifest \& input folder content](#manifest--input-folder-content)
14+
- [Using I/O mapping](#using-io-mapping)
15+
- [I/O table manifests and processing results](#io-table-manifests-and-processing-results)
16+
- [Get input table by name](#get-input-table-by-name)
17+
- [Working with Input/Output Mapping](#working-with-inputoutput-mapping)
18+
- [Accessing Input Tables from Mapping](#accessing-input-tables-from-mapping)
19+
- [Creating Output Tables based on Output Mapping](#creating-output-tables-based-on-output-mapping)
20+
- [Combining Input and Output Mapping](#combining-input-and-output-mapping)
21+
- [Processing input files](#processing-input-files)
22+
- [Grouping Files by Tags](#grouping-files-by-tags)
23+
- [Creating Output Files](#creating-output-files)
24+
- [Processing state files](#processing-state-files)
25+
- [Logging](#logging)
26+
- [ComponentBase](#componentbase)
27+
- [Table Schemas in ComponentBase](#table-schemas-in-componentbase)
28+
- [JSON Table Schema example file](#json-table-schema-example-file)
29+
- [Out table definition from schema example](#out-table-definition-from-schema-example)
30+
- [Sync Actions](#sync-actions)
31+
- [Creating Sync Actions](#creating-sync-actions)
32+
- [Returning Data from Sync Actions](#returning-data-from-sync-actions)
33+
- [Validation Message Action](#validation-message-action)
34+
- [No output](#no-output)
35+
- [License](#license)
36+
37+
338
## Introduction
439

540
![Build & Test](https://github.com/keboola/python-component/workflows/Build%20&%20Test/badge.svg?branch=main)
@@ -21,25 +56,18 @@ to simplify the I/O handling.
2156

2257
### Links
2358

24-
- API Documentation: [API docs](https://keboola.github.io/python-component/interface.html)
25-
- Source code: [https://github.com/keboola/python-component](https://github.com/keboola/python-component)
26-
- PYPI project
27-
code: [https://test.pypi.org/project/keboola.component-kds/](https://test.pypi.org/project/keboola.component-kds/)
28-
-
29-
30-
Documentation: [https://developers.keboola.com/extend/component/python-component-library](https://developers.keboola.com/extend/component/)
31-
32-
- Python Component Cookiecutter template
33-
project: [https://bitbucket.org/kds_consulting_team/cookiecutter-python-component](https://bitbucket.org/kds_consulting_team/cookiecutter-python-component)
59+
- [PyPI](https://pypi.org/project/keboola.component/) & [TestPyPI](https://test.pypi.org/project/keboola.component/)
60+
- [Keboola Components for developers](https://developers.keboola.com/extend/component/)
61+
- [Python Component Cookiecutter template project](https://github.com/keboola/cookiecutter-python-component)
3462

3563
# Quick start
3664

3765
## Installation
3866

39-
The package may be installed via PIP:
67+
The package may be installed via uv 💜:
4068

4169
```
42-
pip install keboola.component
70+
uv add keboola.component
4371
```
4472

4573
## Core structure & functionality
@@ -261,7 +289,7 @@ with open(out_table.full_path, 'w', newline='') as f:
261289
"status": "completed",
262290
"value": "123.45"
263291
})
264-
292+
265293
# Write manifest
266294
ci.write_manifest(out_table)
267295
```
@@ -285,15 +313,15 @@ out_table = ci.create_out_table_definition(
285313

286314
# Add columns using different data type methods
287315
# Method 1: Using BaseType helper
288-
out_table.add_column("id",
316+
out_table.add_column("id",
289317
ColumnDefinition(
290318
primary_key=True,
291319
data_types=BaseType.integer()
292320
)
293321
)
294322

295323
# Method 2: Using SupportedDataTypes enum
296-
out_table.add_column("created_at",
324+
out_table.add_column("created_at",
297325
ColumnDefinition(
298326
data_types=BaseType(dtype=SupportedDataTypes.TIMESTAMP)
299327
)
@@ -303,15 +331,15 @@ out_table.add_column("created_at",
303331
out_table.add_column("status", ColumnDefinition())
304332

305333
# Method 4: Using BaseType with parameters
306-
out_table.add_column("price",
334+
out_table.add_column("price",
307335
ColumnDefinition(
308336
data_types=BaseType.numeric(length="10,2"),
309337
description="Product price with 2 decimal places"
310338
)
311339
)
312340

313341
# Method 5: Backend-specific data types
314-
out_table.add_column("metadata",
342+
out_table.add_column("metadata",
315343
ColumnDefinition(
316344
data_types={
317345
"snowflake": DataType(dtype="VARIANT"),
@@ -323,7 +351,7 @@ out_table.add_column("metadata",
323351
)
324352

325353
# Update existing column (example of column modification)
326-
out_table.update_column("price",
354+
out_table.update_column("price",
327355
ColumnDefinition(
328356
data_types={
329357
"snowflake": DataType(dtype="NUMBER", length="15,4"),
@@ -345,7 +373,7 @@ with open(out_table.full_path, 'w', newline='') as f:
345373
"price": "99.9999",
346374
"metadata": '{"category": "electronics", "brand": "TechCorp"}'
347375
})
348-
376+
349377
# Write manifest
350378
ci.write_manifest(out_table)
351379
```
@@ -409,16 +437,16 @@ input_tables = ci.configuration.tables_input_mapping
409437
for table in input_tables:
410438
# Get the destination (filename in the /data/in/tables directory)
411439
table_name = table.destination
412-
440+
413441
# Load table definition from manifest
414442
table_def = ci.get_input_table_definition_by_name(table_name)
415-
443+
416444
# Print information about the table
417445
print(f"Processing table: {table_name}")
418446
print(f" - Source: {table.source}")
419447
print(f" - Full path: {table_def.full_path}")
420448
print(f" - Columns: {table_def.column_names}")
421-
449+
422450
# Read data from the CSV file
423451
with open(table_def.full_path, 'r') as input_file:
424452
csv_reader = csv.DictReader(input_file)
@@ -444,20 +472,20 @@ for i, table_mapping in enumerate(output_tables):
444472
# Get source (filename that should be created) and destination (where it will be stored in KBC)
445473
source = table_mapping.source
446474
destination = table_mapping.destination
447-
475+
448476
# Create output table definition
449477
out_table = ci.create_out_table_definition(
450478
name=source,
451479
destination=destination,
452480
incremental=table_mapping.incremental
453481
)
454-
482+
455483
# Add some sample data (in a real component, this would be your processed data)
456484
with open(out_table.full_path, 'w', newline='') as out_file:
457485
writer = csv.DictWriter(out_file, fieldnames=['id', 'data'])
458486
writer.writeheader()
459487
writer.writerow({'id': f'{i+1}', 'data': f'Data for {destination}'})
460-
488+
461489
# Write manifest file
462490
ci.write_manifest(out_table)
463491
```
@@ -481,27 +509,27 @@ output_tables = ci.configuration.tables_output_mapping
481509
for i, out_mapping in enumerate(output_tables):
482510
# Find corresponding input table if possible (matching by index for simplicity)
483511
in_mapping = input_tables[i] if i < len(input_tables) else None
484-
512+
485513
# Create output table
486514
out_table = ci.create_out_table_definition(
487515
name=out_mapping.source,
488516
destination=out_mapping.destination,
489517
incremental=out_mapping.incremental
490518
)
491-
519+
492520
# If we have an input table, transform its data
493521
if in_mapping:
494522
in_table = ci.get_input_table_definition_by_name(in_mapping.destination)
495-
523+
496524
# Read input and write to output with transformation
497525
with open(in_table.full_path, 'r') as in_file, open(out_table.full_path, 'w', newline='') as out_file:
498526
reader = csv.DictReader(in_file)
499-
527+
500528
# Create writer with same field names
501529
fieldnames = reader.fieldnames
502530
writer = csv.DictWriter(out_file, fieldnames=fieldnames)
503531
writer.writeheader()
504-
532+
505533
# Transform each row and write to output
506534
for row in reader:
507535
# Simple transformation example - uppercase all values
@@ -513,7 +541,7 @@ for i, out_mapping in enumerate(output_tables):
513541
writer = csv.DictWriter(out_file, fieldnames=['id', 'data'])
514542
writer.writeheader()
515543
writer.writerow({'id': f'{i+1}', 'data': f'Sample data for {out_mapping.destination}'})
516-
544+
517545
# Write manifest
518546
ci.write_manifest(out_table)
519547
```
@@ -543,14 +571,14 @@ for file in input_files:
543571
print(f"Processing file: {file.name}")
544572
print(f" - Full path: {file.full_path}")
545573
print(f" - Tags: {file.tags}")
546-
574+
547575
# Example: Process image files
548576
if 'images' in file.tags:
549577
# Process image using appropriate library
550578
print(f" - Processing image: {file.name}")
551579
# image = Image.open(file.full_path)
552580
# ... process image ...
553-
581+
554582
# Example: Process document files
555583
if 'documents' in file.tags:
556584
print(f" - Processing document: {file.name}")
@@ -912,19 +940,19 @@ class Component(ComponentBase):
912940
def run(self):
913941
# Main component logic
914942
pass
915-
943+
916944
@sync_action('testConnection')
917945
def test_connection(self):
918946
"""
919947
Tests database connection credentials
920948
"""
921949
params = self.configuration.parameters
922950
connection = params.get('connection', {})
923-
951+
924952
# Validate connection parameters
925953
if not connection.get('host') or not connection.get('username'):
926954
raise UserException("Connection failed: Missing host or username")
927-
955+
928956
# If no exception is raised, the connection test is considered successful
929957
# The framework automatically returns {"status": "success"}
930958
```
@@ -949,7 +977,7 @@ class Component(ComponentBase):
949977
{"id": "orders", "name": "Order History"},
950978
{"id": "products", "name": "Product Catalog"}
951979
]
952-
980+
953981
# Return as list of SelectElement objects for UI dropdown
954982
return [
955983
SelectElement(value=table["id"], label=table["name"])
@@ -972,23 +1000,23 @@ class Component(ComponentBase):
9721000
Validates the component configuration
9731001
"""
9741002
params = self.configuration.parameters
975-
1003+
9761004
# Check configuration parameters
9771005
if params.get('extraction_type') == 'incremental' and not params.get('incremental_key'):
9781006
# Return warning message that will be displayed in UI
9791007
return ValidationResult(
9801008
"Incremental extraction requires specifying an incremental key column.",
9811009
MessageType.WARNING
9821010
)
983-
1011+
9841012
# Check for potential issues
9851013
if params.get('row_limit') and int(params.get('row_limit')) > 1000000:
9861014
# Return info message
9871015
return ValidationResult(
9881016
"Large row limit may cause performance issues.",
9891017
MessageType.INFO
9901018
)
991-
1019+
9921020
# Success with no message
9931021
return None
9941022
```

0 commit comments

Comments
 (0)