22 questions
0
votes
0
answers
99
views
meltano handling of encoding
I am currently pulling my hair out trying to get meltano to correctly write my data to a target.
Step 1:
The data is extracted correctly using tap-mssql (database has collation ...
- 71
2
votes
0
answers
149
views
Programatically configuring Meltano
I am considering using Meltano as the EL tool in an application that allows users to configure their data sources so they are replicated to a data lake. For that, it would be necessary to configure ...
1
vote
1
answer
275
views
Debugging and troubleshooting singer-tap development for a SQL source using the Meltano SDK
Does anyone have experience developing SQL singer-taps using the Meltano SDK?
Problem
My biggest problem is troubleshooting the exceptions I am experiencing when testing my tap.
Context
I am ...
- 21
0
votes
0
answers
474
views
Poetry configuration error when adding Meltano tap-csv extractor
I'm attempting to add the tap-csv extractor to my Meltano project in a virtual environment but run into the following error when running meltano add extractor tap-csv:
RuntimeError
The Poetry ...
- 1
2
votes
1
answer
517
views
Meltano with MWAA
I have an AWS MWAA environment to run an ETL Pipeline, and I was planning to consider using Meltano in it.
I seem to find no example(s) or documentation that is specific to MWAA. So, I was wondering ...
- 23
0
votes
1
answer
641
views
Meltano ETL not deleting rows in destination PostgreSQL database after deleting from source PostgreSQL database
I am using Meltano for ETL to extract data from a source PostgreSQL database using tap-postgres (MeltanoLabs variant) and load it into a destination PostgreSQL database using target-postgres (...
- 31
0
votes
1
answer
487
views
Where files from tap are kept in Meltano
I have the following combination of tap/target in Meltano: tap-marketo and target-s3-parquet.
I want to extract data from tap-marketo from data A to date B in the past.
I saw that we can only define ...
- 93
0
votes
1
answer
501
views
Meltano: (tag and tagret: postgresql) temporary file size exceeds temp_file_limit
I'm getting the issue while running Meltano with tag and target "postgresql" as source database, without changing the temp_file_limit params by the tap "postgresql".
how could I ...
- 13
1
vote
1
answer
934
views
Meltano: tag-postgres to target-postgres: data type such as uuid is converted to varchar
I'm actually working on a Meltano project where I have to extract data from one postgres database and load to the final warehouse (also a postgres database) using "Key-based Incremental ...
- 13
1
vote
1
answer
1k
views
Running Meltano commands raises an error: __init__() got an unexpected keyword argument 'request_timeout'
Running Meltano commands raises an error:
__init__() got an unexpected keyword argument 'request_timeout'
This is happening on a simple:
$ meltano state clear
# OR
$ meltano elt
- 11
2
votes
1
answer
1k
views
How to debug a Singer Tap using VS Code
In creating new Singer Taps using the Meltano SDK, it's not always clear how to setup testing in VS Code.
What's the best way to get the VS Code test features working?
- 2,640
6
votes
2
answers
2k
views
How to override a stream's primary key properties or incremental replication key in Meltano?
There are times when we want to override a stream's key properties (primary key) or it's incremental replication key. What's the best way to do this in Meltano?
- 2,640
2
votes
1
answer
461
views
Unable to connect to AWS redshift via Meltano
hope you all are doing great.
Can any one support me with an issue regarding Meltano? I want to perform EL (Extract Load) where I extract data from shopify and load it to AWS Redshift. To extract data ...
1
vote
1
answer
678
views
With Singer taps and targets, why are columns created in the target even if they are deselected?
When using singer taps and targets, we sometimes see the target creating columns for columns that are explicitly filtered out (deselected) in the tap.
Why does this this happen and how can we resolve ...
- 2,640
4
votes
1
answer
672
views
How to kill a running EL pipeline in Meltano
When running an Extract/Load pipeline with Meltano, what's the best way (or ways) to kill a running job?
Generally these would be run via Airflow but would be nice to have a process that worked also ...
- 2,640
