Category: C#

Rest API Paging Strategy

There is a lot written about paging strategies for REST API’s. This is my simple and quick take on the subject and what I generally implement when rolling my own APIs.

Any API that returns a collection should have some form of return result limiting. This is to avoid killing your web servers, database servers, networks and avoiding a super easy distributed denial of service (DDoS) attack.

I don’t cover other optimisations like filtering and sorting, although these do have a major impact too.

There are two major API standards/frameworks that I would follow when implementing my own paging strategy:

Adding to that, there are two primary pagination styles:

  • Offset
    • Most common. Set the number of records in a page and then the record offset to use.
    • If there are more results, the server also returns some metadata that contains information about the current page, such as the total number of pages and the link for the next page.
  • Keyset / Cursor based
    • Less common. Harder to program. Uses a specific cursor, which is a unique identifier for a specific item in the list, then returns this record and the next specified number of results.
    • If there are more results, the server also returns a cursor for the next page.
Standard / FrameworkPrimary Paging MethodQuery Properties
ODataOffset– skip (the offset from the beginning of the results)
– top (the number of records it wants to retrieve in each page)
GraphQLKeyset / Cursor based– first (the number of records to return)
– after (specify the cursor of the record to show after)
Table showing high-level paging methods

My General Principals

  1. Use Offset paging unless, you have large data and performance challenges.
  2. Use ‘skip’ and ‘top’ for the parameters, as this matches OData which you may want to implement later anyway.
  3. Always have a default page size (‘top’) and use it if ‘top’ is not specified.
  4. Always return a total query set count (integer) by default. e.g., $count defaults to true, but can be turned off, if necessary, by passing false.
  5. If more records exist, then always return a ‘nextLink’ URL
  6. Put the results array into the ‘value’ field to match that of the OData.

Response Structure and Paging Metadata

  • nextLink
  • totalCount

Example OData Return Results

{
    "@odata.context": "http://localhost:5000/odata/$metadata#Books",
    "@odata.count": 100,
    "@odata.nextLink": "http://localhost:5000/odata/Books?$count=true&$top=2&$skip=1",
    "value": [
        {
            "Id": 1,
            "ISBN": "978-0-321-87758-1",
            "Title": "Essential C#5.0",
            "Author": "Mark Michaelis"
        },
        {
            "Id": 2,
            "ISBN": "063-6-920-02371-5",
            "Title": "Enterprise Games",
            "Author": "Michael Hugos",
        }
    ]
}

Example of hand rolled API Return Results

{
    "count": 100,
    "nextLink": "http://localhost:5000/Books?$count=true&$top=2&$skip=1",
    "value": [
        {
            "Id": 1,
            "ISBN": "978-0-321-87758-1",
            "Title": "Essential C#5.0",
            "Author": "Mark Michaelis"
        },
        {
            "Id": 2,
            "ISBN": "063-6-920-02371-5",
            "Title": "Enterprise Games",
            "Author": "Michael Hugos",
        }
    ]
}

Data Changes Complications

There are a few complexities that may be of a concern if you are requiring an exact immutable record set. In most cases, this is not critical, but could be in a financial or audit type scenario. That is the complete recordset will need to be locked and remain unaltered during all the paging activities.

An example would be where a field in the recordset must add up exactly to a total at a specific point in time.

As most dataset are dynamic, new records can be added and removed at any point and that means you can’t guarantee to not return the same record in different pages or that the records on a given page/offset will remain the same, and that the total count does not change.

There are several solutions to this problem, and I will only suggest the easiest here and it’s not perfect either.
The dataset will require a created timestamp field, as well as a soft delete ‘isDeleted’ or ‘deletedAt’ type field. Then you can return all the records (including the soft deleted ones) prior to a specified creation time (i.e. the time of the first request in the sequence).

This would ensure consistency in the records returned, but not necessarily the values in those records.

References

Pagination | GraphQL

Pagination – OData | Microsoft Learn

PageSize, Top and MaxTop – OData | Microsoft Learn

Add NextPageLink and $count for collection property – OData | Microsoft Learn

IActionResult Return Types and StatusCodes quick reference

I’m always forgetting which return types are available directly from ASP.Net controllers, so have created a quick listing here.

For general information on HTTP Status codes see
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

ASP.Net Core 2.2

All Status Codes:

https://docs.microsoft.com/en-gb/dotnet/api/microsoft.aspnetcore.http.statuscodes?view=aspnetcore-2.2

ControllerBase IActionResult Return Types

  • OK (200)
  • BadRequest (400)
  • Forbid (403)
  • LocalRedirect (302)
  • LocalRedirectPermanent (301)
  • LocalRedirectPermanentPreserve (308)
  • LocalRedirectPermanentPreserveMethod (307)
  • NoContent (204)
  • NotFound (404)
  • RedirectToAction (302)
  • RedirectToActionPermanent (301)
  • RedirectToActionPermanentPreserve (308)
  • RedirectToActionPermanentPreserveMethod (307)
  • RedirectToPage (302)
  • RedirectToPagePermanent (301)
  • RedirectToPagePermanentPreserve (308)
  • RedirectToPagePermanentPreserveMethod (307)
  • RedirectToPage (302)
  • RedirectToPagePermanent (301)
  • RedirectToPagePermanentPreserve (308)
  • RedirectToPagePermanentPreserveMethod (307)
  • StatusCode (set own status code)
    • Use 409 – Conflict, for updates that fail due to conflicts such as already exists etc..
  • Unauthorized (401)
  • UnprocessableEntity (422)
  • ValidationProblem (400)

ApiController MVC Compatibility Shim

https://docs.microsoft.com/en-us/dotnet/api/system.web.http.apicontroller?view=aspnetcore-2.2

  • OK (200)
  • BadRequest (400)
  • Conflict (409)
  • Created (201)
  • CreatedAtRoute (201)
  • InternalServerError (500)
  • NotFound (404)
  • Redirect (302)
  • RedirectToRoute (302)
  • StatusCode (set own status code)
    • Use 409 – Conflict, for updates that fail due to conflicts such as already exists etc..
  • Unauthorized (401)
  • UnprocessableEntity (422)
  • ValidationProblem (400)

Azure Active Directory Graph API Wrapper to help make it a bit easier!

I have recently been trying to program against the Azure Active Directory (AAD) using the Microsoft.Azure.ActiveDirectory.GraphClient library. Unfortunately this library literally has no useful comments to assist understanding  or clarify parameters etc.. Let alone how best to use or implement objects and methods or what and why exception may occur.

Equally the MSDN documentation seems to be lacking in any examples and really has minimal comments (although I see it’s getting a bit better…I think).

To this end I have created a ‘wrapper / handler’ to simplify all sorts of AAD interactions called AADGraphHandler. It effectively will help manage the creation of the ActiveDirectoryClient and a bunch of it’s operations. You can find this on GitHub at https://github.com/nrogoff/AADGraphHandler

You can get access to the ActiveDirectoryClient directly, and so any methods not covered yet. (e.g. Adding and removing roles to a user. Just not needed it yet!)

Continue reading “Azure Active Directory Graph API Wrapper to help make it a bit easier!”

Unit Test Class ReSharper Template and Snippets for nUnit and MS Test

Quick helper boiler plates for setting up your Unit Test.

I have included ReSharper Template here that you can import directly Resharper Templates. Also thanks you to Alessandro Aeberli for making the these Visual Studio Snippets

After importing the ReSharper Templates then you can use ‘uTestBoiler’ or ‘uTestnUnitBoiler’ shortcuts to add the code below into any test class.

If you want to make your own snippets you can use the Visual Studios Snippet Manager by following the instructions here https://msdn.microsoft.com/en-us/library/ms165394.aspx

Continue reading “Unit Test Class ReSharper Template and Snippets for nUnit and MS Test”

Bulk update EXIF ‘Shot Taken Date’ on JPG photos using ‘ImageProcessing Console’.

If, like me, you have digital photo’s stretching back to the early days of digital cameras and even scans of film and slides, then you may have the same issue I had attaching the right date to the file. I have over 35,000 digital images, so developed a little command line tool to help called ‘Image Processing Console‘ (https://github.com/nrogoff/ImageProcessingConsole).

Continue reading “Bulk update EXIF ‘Shot Taken Date’ on JPG photos using ‘ImageProcessing Console’.”

Entity Framework – Re-querying or fetching latest foreign key relationship records

In entity framework, you can often include foreign key or relationship collections in your LINQ request by using the ‘Include’ statement.

For example you may want to get a collection of Customers and their related Invoices. This would look something like:

var customersWithInvoices = DataContext.Customers.Include("Invoices")

However there may be times where this is not possible or you want to ensure that you have the latest related collection or you only want to fetch that collection when you need it.

To do this you need when the related records are a collection and re-query the navigation property for your parent entity (here ‘customer’) we use the Collection() method as follows:

try
{
DataContext.Entry(customer)
.Collection<Invoice>("Invoices")
.Query()
.ToList();
}
catch(Exception e)
{
Console.WriteLine(e);
}

Where ‘Invoices’ is the name of your navigation property. Now you can access the latest related objects.

e.g.

var countOfInvoices = customer.Invoices.Count();

To do this you need for a one-to-one related single entity then use the Reference() method instead as follows (Say we wanted the Country dial code of our customers Country):

DataContext.Entry(customer).Reference<Country>("CountryOfResidence").Query().FirstOrDefault();

Here the navigation property is called ‘CountryOfResidence”.

We can now navigate to the Country entity and get the dial code.

e.g.

var dialCode = customer.CountryOfResidence.DialCode;