Skip to content

Allow using pagination#98

Merged
claudiubelu merged 7 commits intocloudbase:masterfrom
petrutlucian94:pagination
May 6, 2026
Merged

Allow using pagination#98
claudiubelu merged 7 commits intocloudbase:masterfrom
petrutlucian94:pagination

Conversation

@petrutlucian94
Copy link
Copy Markdown
Member

At the moment, listing endpoint instances is the only Coriolis API that supports pagination. For this reason, Coriolis clients often retrieve much more database records than needed, leading to poor performance.

We're now adding pagination to other Coriolis APIs:

  • transfers
  • transfer executions
  • deployments

New optional parameters:

  • sort_key - sort key, repeatable. created_at and id are used by default.
  • sort_dir - sort direction, repeatable. asc or desc (default).
  • marker - the last seen ID, omitted from the results.
  • limit - the maximum number of records to retrieve.

Example:

GET http://server:7667/v1/transfers?marker=a7061715-e56c-470c-a6ac-80bb02f1f198&limit=2&sort_key=id&sort_dir=asc

At the moment, listing endpoint instances is the only Coriolis
API that supports pagination. For this reason, Coriolis clients
often retrieve much more database records than needed, leading to
poor performance.

For this reason, we're now adding pagination to other Coriolis
APIs, starting with the transfer executions. A transfer can have
a large amount of executions, especially in case of cron jobs.

New optional parameters:
* limit - the maximum number of entries to retrieve
* marker - the last seen id, will not be retrieved again

The pagination will be performed on the db API side, leveraging the
"utils.paginate_query" helper from oslo_db.
Python 3.9 does not support modern type hints. It's no longer
supported by coriolis-core, as such we'll drop it from the
python-coriolisclient pipeline.
@petrutlucian94
Copy link
Copy Markdown
Member Author

This implies that we'll have two sort parameters, one for the db side sort and pagination and another one for the standard client side sorting:

List transfer executions

positional arguments:
  transfer      The transfer's id

options:
  -h, --help            show this help message and exit
  --marker MARKER
                        The id of the last retrieved execution.
  --limit LIMIT
                        Maximum number of executions to retrieve.
  --sort SORT   Comma-separated list of sort keys and directions in the form of <key>[:<asc|desc>]. The direction defaults to descending if not specified.

output formatters:
  output formatter options

  -f {csv,json,table,value,yaml}, --format {csv,json,table,value,yaml}
                        the output format, defaults to table
  -c COLUMN, --column COLUMN
                        specify the column(s) to include, can be repeated to show multiple columns
  --sort-column SORT_COLUMN
                        specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated
  --sort-ascending      sort the column(s) in ascending order
  --sort-descending     sort the column(s) in descending order

@petrutlucian94 petrutlucian94 requested a review from claudiubelu May 5, 2026 18:18
Comment thread coriolisclient/cli/deployments.py Outdated
Comment thread coriolisclient/cli/deployments.py Outdated
Comment thread coriolisclient/cli/transfers.py Outdated
query={
"some_filter": "some_value",
"some_other_filter": "some_other_value"
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: query should be a list, not a dict, as seen in the test below, and in the calls in the tests above. It has to be a list because the same key can be specified multiple times.

Copy link
Copy Markdown
Member Author

@petrutlucian94 petrutlucian94 May 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

urlparse.urlencode accepts both, which is why _list does the same:

def _list(self, url, response_key=None, obj_class=None, json=None,
              values_key='values', query: dict | list | None = None):

test_list_with_dict_query covers dict query params, test_list_with_tuple_list_query handles lists of tuples.

@claudiubelu claudiubelu merged commit 6f8f1e1 into cloudbase:master May 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants