Tap and target performance

The following sections provide different strategies to improve the performance of your tap or target.

Use BATCH mode

See the BATCH messages page for more information.

Use a different message writer or reader

Starting from version 0.45.0, the Meltano Singer SDK supports different message writers and readers.

For example, to use a message writer or reader based on the msgspec serialization library, first specify the msgspec extra for the singer-sdk dependency:

uv

$ uv add singer-sdk --extra msgspec

Poetry

$ poetry add singer-sdk --extras msgspec

To update your tap:

from singer_sdk.contrib.msgspec import MsgSpecWriter


class MyTap(Tap):
    message_writer_class = MsgSpecWriter

To update your target:

from singer_sdk.contrib.msgspec import MsgSpecReader


class MyTarget(Target):
    message_reader_class = MsgSpecReader

Measuring performance

We’ve had success using viztracer to create flame graphs for SDK-based packages and find if there are any serious performance bottlenecks.

You can start doing the same in your package. Start by installing viztracer.

uv

$ uv add --group dev viztracer

Poetry

$ poetry add --group dev viztracer

Then simply run your package’s CLI as normal, preceded by the viztracer command

uv

$ uv run viztracer my-tap
$ uv run viztracer -- my-target --config=config.json --input=messages.json

Poetry

$ poetry run viztracer my-tap
$ poetry run viztracer -- my-target --config=config.json --input=messages.json

That command will produce a result.json file which you can explore with the vizviewer tool.

uv

$ uv run vizviewer result.json

Poetry

$ poetry run vizviewer result.json

The output should look like this

Note: Chrome seems to work best for running the vizviewer app.